# A Service-Caching Strategy Assisted by Double DQN in LEO Satellite Networks
## 3 Implementation  
### Introduction to Deep Reinforcement Learning  
Traditional reinforcement learning (RL) is a process in which the \"agent\" entity continuously, through trial and error, evaluates rewards and improves solutions in the unknown environment. Some classic methods include Q-learning, SARSA, etc. During the learning process, the \"agent\" selects corresponding actions based on the current state, obtains corresponding rewards, and transitions to the next state. It is worth noting that each action may affect the subsequent state. Through continuous iteration, the RL method aims to determine the optimal action in each state to obtain the largest reward in the target problem. Generally, reinforcement learning can be modeled as a Markov model: \(MDP=\langle S,A,P,R,\gamma\rangle\), where \(S\) represents the set of states, \(A\) represents the set of actions, \(P\) represents the probability of  
Figure 2: Cache structure. selecting different actions in each state, and \(R\) represents rewards. The parameter \(\gamma\) is a discount factor used to discount future rewards into current rewards. However, traditional RL methods can only handle limited and discontinuous states, which have certain limitations. With the combination of deep learning, the agent can learn to observe any state feature in the environment and perform corresponding decision training. The DQN algorithm is one of the classic DRL methods that has been improved on the basis of the Q-learning algorithm, and its process is shown in Figure 3.  
As shown in Figure 3, the DQN algorithm adopts an experience replay mechanism. For any state, \(s\), after taking an action, \(a\), it will enter another state, \(s^{{}^{\prime}}\), and receive the corresponding reward, \(r\). The quadruple \(\left\langle s,a,r,s^{{}^{\prime}}\right\rangle\) will form a sample and be stored in the replay buffer. After reaching the specified number, a batch of samples will be randomly selected for training. The DQN algorithm uses two Q-networks; one is the evaluation network, also known as the original network, which is responsible for controlling the \"agent\" and collecting experience, with weight parameters of \(\omega\), and the other is the target network, with weight parameters of \(\omega^{-}\), which is used to calculate the time difference (TD) target, as shown in the following equation. \[y=r+\gamma\cdot\arg\max_{a}Q(s^{{}^{\prime}},a;\omega^{-}) \tag{9}\]  
The DQN algorithm uses the gradient descent method to update the weight parameters; the update formula is as follows:  
\[\omega_{t+1}=\omega_{t}+\alpha\bigg{[}r+\gamma\cdot\arg\max_{a}Q(s^{{}^{\prime} },a,\omega^{-})-Q(s,a,\omega^{-})\bigg{]}\
abla Q(s,a,\omega^{-}) \tag{10}\]  
During the update process, we can adapt the approach by fixing the weight parameters of the original network, updating the weight parameters of the target network, and regularly copying the target network weight parameters to the original network in order to enhance the fitting stability. However, the DQN algorithm involves the weight parameters of the target network in its own update process, which leads to the accumulation of positive errors and the overestimation of the Q value. In this regard, the Double DQN algorithm has been improved on the basis of the DQN algorithm by using the origin Q-network to obtain the best action, which, to some extent, solves the problem of overestimation. In the Double DQN algorithm, the calculation formula for the TD target is as follows:  
\[y=r+\gamma\cdot\arg\max_{a}Q(s^{{}^{\prime}},a;\omega) \tag{11}\]  
Figure 3: Process of DQN algorithm. where \(s\) is the size of the transmitted content, and \(R_{s,\mu},R_{s,s^{\prime}},R_{c,\mu}\) are the transmission rates between the fog satellite and users, the fog satellite and its adjacent satellite, and cloud service centers and users, respectively. We set different reward functions based on the transmission delay of cached content in different situations, which are shown in the following equation:  
\[r(t)=\left\{