## 2 LEO Satellite Networks Model
### Two-Hops State Aware Updating
\begin{array}{ll}r_{d}&N_{next}=N_{d}\\ -r_{c}&N_{next}\text{ }failed/\text{\emph{congested}}\\ -dif(N_{s},N_{d})&Other\end{array} \right. \tag{15}\]

where \(RAAN\) represents right ascension of ascending node. \(\omega\) represents the mean anomaly. \(\alpha\) and \(\beta\) are the weights of inter-plane ISLs and intra-plane ISLs. We define \(r_{d}\) as a high reward for success and \(-r_{c}\) as a punishment for mistake. ### Routing Algorithm

The DDQN model observes the information of the satellite networks topology through the training process. However, the training process of DDQN requires a large amount of overhead, and it takes a long time. Due to limited resources and processing capacity on the satellite, we simulate the flows of the satellite networks and complete the DDQN training process on the ground. The off-line training process enables the DDQN model to cope with all the link states that may be encountered. Then the trained DDQN models are stored on the satellite and no longer updated during the satellite routing process. Figure 3: Virtual Node (VN)-based satellite networks topology. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Parameters** & \(S\) & \(A\) & \(P\) & \(R\) \\ \hline
**Interpretation** & \(\text{State}([N_{s},N_{d},LST])\) & Action(\(N_{next}\) ) & Possibility(\(P_{next}\)) & Reward (\(r\)) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Agent model of reinforcement learning. #### 3.2.1 Double-DQN Offline Training Process

Double-DQN uses Deep Neural Networks (DNN) instead of the look-up table to represent all the states and actions. The inputs of the DNN are the current states, and the outputs are the Q-values of all the possible actions. We propose to use the DDQN which is composed of an online DNN with weight \(\theta^{\text{online}}\) and a target DNN with weights \(\theta^{\text{target}}\). DNN needs to be trained to achieve convergence state. The online DNN updates its weights \(\theta^{\text{online}}\) at each iteration. The target DNN resets its weights \(\theta^{\text{target}}\) to \(\theta^{\text{online}}\) in every \(N^{\text{target}}\) iterations and keeps weights \(\theta^{\text{target}}\) fixed at other iterations. The loss function at the current iteration is shown in Equation (16):

\[L^{DDQN}=[Y^{DDQN}-Q(s,a,\theta^{\text{online}})^{2}] \tag{16}\]

where the target value \(Y^{DDQN}\) is defined as

\[Y^{DDQN}=r+\gamma Q(s^{\prime},\text{argmax}Q_{i}(s^{\prime},a^{\prime};\theta ^{\text{online}});\theta^{\text{target}}) \tag{17}\]

To minimize the loss function, it needs to update the weights \(\theta\) by using the experience \(<s,a,r,s^{\prime}>\) to train the DNN. The DDQN can execute action \(a\) by the \(\varepsilon\)-greedy policy to balance its exploration and exploitation. Algorithm 2 shows the DDQN train algorithm which uses the DDQN to find the optional routing policy. ```
Input:\(A\);\(N^{\text{target}}\);\(N_{b}\);\(M\) Output: DDQN routing model Initialize:\(\theta^{\text{online}}\);\(\theta^{\text{target}}\) for episode \(i=\{1,\cdots,N\}\)do for iteration \(t=\{1,\cdots,T\}\)do  Execute action \(a\) according to \(\varepsilon\)-greedy policy  Receive reward \(r_{t}\)  Store experience \(<s,a,r_{t},s^{\prime}>\) in \(M\) if an episode terminates at iteration \(j+1\)then  Set \(Y^{DDQN}_{j}=r_{d}\) else  Determine \(a=\underset{a^{\prime}\in A}{\text{argmax}}Q(s^{\prime},a^{\prime};\theta^{ \text{online}})\)  Set \(Y^{DDQN}_{j}=r_{j}+\gamma Q(s^{\prime},a;\theta^{\text{target}})\) endif  Perform a gradient descent step on \(L^{DDQN}\) to update \(\theta^{\text{online}}\)  Reset \(\theta^{\text{target}}=\theta^{\text{online}}\) in every \(N^{\text{target}}\) iterations endfor endfor
```

**Algorithm 2** Double-DQN train algorithm

Accordingly, based on the experience \(e\), the online DNN computes the optional value \(Q(s^{\prime},a;\theta^{\text{target}})\). Then, the loss function \(L^{DDQN}\) and the target value \(Y^{DDQN}\) are calculated according to Equations (14) and (15) respectively.