# Resource Allocation and Offloading Strategy for UAV-Assisted LEO Satellite Edge Computing
## 4 Algorithm Design
### Task and Computing Resource Allocation Strategy  
Due to the presence of correlations in several restrictions, to reduce the problem's difficulty, several deterministic factors have to be taken into account when offloading satellite edge tasks. These include the satellite edge server's state and the environment of the communication. Therefore, the optimizing problem is converted to an offloading scheme based on DRL-making methods and solving it by Markov decision. The DRL architecture consists of interactions through the agent to solve the above problem by training the best policy to maximise the cumulative reward [37]. DDPG-Based Task Offloading and Resource Allocation Algorithm  
Through iterative trials, reinforcement learning optimizes the action selection in multiple situations based on a given reward function. The intelligent body perceives the state and performs actions to change it. During each iteration, the competent body observes the state as input and selects the action to be completed. The execution of the action produces a reward, and the intellectual body judges the quality of the action by observing the prize. The selection of activities by intelligence tends to increase the long-term total compensation and maximize the reward function. The DDPG algorithm is one of the most popular methods for dealing with problems in RL, described as \((S,a,r_{t},\gamma)\), where S is the state, a denotes the action, \(r_{t}\) is the immediacy reward of the time slot, and \(\gamma\in(0,1)\). The anticipated long-term discounting compensation has the following definition:  
\[R_{t}=\sum_{t=1}^{T}\gamma^{t}r(s_{t},a_{t}) \tag{35}\]  
Here, at time \(t\), the state and action are \(s_{t}\) and \(a_{t}\), respectively; \(r(s_{t},a_{t})\) is the straight reward. Taking into account that we have to deal with continuous actions, we decide to follow a determined policy and write the value function, as follows  
\[Q^{\mu}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma Q^{\mu}(s_{t+1}.a_{t+1}) \tag{36}\]  
In an effort to achieve maximum expected discounted benefits over the long term, at each slots, we use the time-series difference method learned from the previous period's experience to update the action function. The DDPG algorithm makes the approximation \(Q(s,a|\theta)\). The actor uses the strategy function \(\mu(s|\theta)\) to decide on an action, the critic uses the value function to judge the policy functional, and the network of values and strategic network are renewed in accordance with the critic's output. The loss function has the form  
\[L\Big{(}\theta^{Q}\Big{)}=\Big{(}Q\Big{(}\theta^{Q}\Big{)}-y_{t}\Big{)}^{2} \tag{37}\]  
\[y_{t}=r(s_{t},a_{t})+\gamma Q\Big{(}s_{t+1},\theta^{Q}\Big{)} \tag{38}\]  
Refine the critic network by minimising \(L(\theta^{Q})\), and \(Q(\theta^{Q})\) is the maximum future payoff that will be earned by proceeding with the policy \(\mu(s_{t+1}|\theta^{Q})\) to the state following the implementation of action \(a_{t}\). The actor network must change the action parameters in the general direction where the maximum Q is more probable. Instead of updating all the parameters, the fixed goal Q-network can stabilise the learning by keeping a part of the parameters updated. Let \(\theta^{\mu}\) and \(\theta^{Q}\) be the parameters before the update, respectively, and \(\theta^{\mu^{\prime}}\) and \(\theta^{Q^{\prime}}\) after the update. As changes in task offloading are time-continuous and the offloading decision taken in the previous time slot has an impact on the current observation, we use LSTM to capture the correlation between the previous observation and the current observation and more potential information by learning a series of past experiences. The algorithm overcomes the inability of the Deep Deterministic Policy Gradient (DDPG) to handle partial observability and history-dependent decisions by adding a recursive mechanism. The DDPG-LSTM is therefore proposed, and the algorithm architecture is shown in Figure 2.  
The organisation of the DDPG-LSTM is mainly built on the actor-critic model.