# Caching-Aware Intelligent Handover Strategy for LEO Satellite Networks
## 3 Caching-Aware Intelligent Handover Strategy
### Intelligent Handover Strategy with Multiple Attributes
So, \(\Omega_{i,p}^{K}\) is used to express the \(k\)th candidate satellite label of user \(i\). \(\Theta_{p}\) refers to the information of satellite channels, which can be expressed as \(\Theta_{p}\)= \([\Theta_{p}^{*},~{}\Theta_{p}^{1},~{}\Theta_{p}^{2},~{}\ldots,~{}\Theta_{p}^{K }]\). \(\Theta_{p}^{*}\) is defined as \([\Theta_{1,p}^{*},~{}\Theta_{2,p^{\prime}}^{*},~{}\ldots,~{}\Theta_{i,p}^{*}]\), and \(\Theta_{i,p}^{*}\) is the number of idle channels of the serving satellite of user \(i\). \(\Theta_{i,p}^{*}\in\{1,~{}2,~{}3,~{}\ldots,~{}I\}\), \(I\) is the total channel number of a single satellite. \(\Theta_{p}^{K}\) is defined as \([\Theta_{1,p^{\prime}}^{K},~{}\Theta_{2,p^{\prime}}^{K},~{}\ldots,~{}\Theta_{ i,p}^{K}]\), and \(\Theta_{i,p}^{K}\) is the number of idle channels for user \(i\)'s Kth candidate satellite. \(C_{p}=[C_{p}^{*},~{}C_{p}^{1},~{}C_{p}^{2},~{}\ldots,~{}C_{p}^{K}]\) and \(\mathrm{T}_{p}=[\mathrm{T}_{p}^{*},~{}\mathrm{T}_{p}^{1},~{}\mathrm{T}_{p}^{2},~{}\ldots,~{}\mathrm{T}_{p}^{K}]\) is the remaining caching capacity and service time. **Action space (A)**: For each time slot, the agent will decide whether the handover be activated or not and select the optimal target satellite for every user. The action of time slot \(p\) can be expressed as  
\[a_{p}=[\Lambda_{1,p},\Lambda_{1,p},\ldots,\Lambda_{i,p}], \tag{15}\]  
where \(\Lambda_{i,p}\in\{0,~{}1,~{}2,~{}3,~{}\ldots,~{}K\}\). The value '0' means that the handover for user \(i\) will not be activated. Other values mean that handover will be activated, and the corresponding number is the label of the candidate satellites. \(K\) is the maximum number of candidate satellites. **Transition probability (P)**: Since the system state in this paper is continuous and the state is affected by the handover decision, the transition probability from \(s_{p}\) to \(s_{p+1}\) with the action \(a_{p}\) is difficult to be obtain. Hence, a model-free DRL framework based on a deep Q-learning network (DQN) is adopted. **Reward function (R)**: We divide the reward function \(R(s_{p},a_{p})\) into two parts: gain function \(g_{p}\) and cost function \(l_{p}\). \[R(s_{p},a_{p})=\sum_{i=1}^{N}g_{i,p}-l_{p}, \tag{16}\]  
\[g_{i,p}= \begin{cases}w_{1}\Theta_{i,p}^{*}+w_{2}C_{i,p}^{*}+w_{3}\mathrm{T}_{i,p}^{*},\text{if a}_{\mathrm{p}}=0,\text{handover is not activated}\\ w_{1}\Theta_{i,p}^{a_{p}}+w_{2}\mathrm{C}_{i,p}^{a_{p}}+w_{3}\mathrm{T}_{i,p}^{a _{p}},\text{if a}_{\mathrm{p}}\
eq 0,\text{handover is activated}\end{cases} , \tag{17}\]  
\[l_{p}=w_{4}a_{p}+w_{5}\beta_{p}+w_{6}\delta_{p}. \tag{18}\]  
\(g_{i,p}\) is the gain of remaining communication resources. Three attributes, which are the number of idle channels, remaining caching capacity, and remaining service time, are normalized. \(w\) is the weight of each attribute, and \(w_{1}+w_{2}+w_{3}=1\). \(l_{p}\) is the cost function. \(\alpha_{p}\) is the number of successful handover times in time slots \(p\), and \(\beta_{p}\) is the number of failed handover times. The failed handover defined here refers to the handover failure caused by several reasons. For example, the candidate satellite does not have idle channels, or the remaining service time is reduced to zero, or the serving satellite does not have enough caching capacity. \(\delta_{p}\) is the number of dropped calls caused by insufficient resources (remaining service time, channel, etc.). According to the definitions of state, action, and the reward, we can calculate the target Q value:  
\[y_{p}=R(s_{p},a_{p})+\gamma\underset{a^{\prime}}{\max}\hat{Q}(s_{p},a^{\prime}; \theta^{-}). \tag{19}\]  
Then, the mean square error loss function is calculated as  
\[L(\theta)=E[(R(s_{p},a_{p})+\gamma\underset{a^{\prime}}{\max}\hat{Q}(s_{p},a^{ \prime};\theta^{-})-Q(s_{p},a;\theta))^{2}], \tag{20}\]  
where \(\theta\) is the main network parameters, and \(\theta^{-}\) is the target network parameters. After calculating the loss function, the gradient descent strategy is used to update the main network parameters.