## 2 LEO Satellite Networks Model
### Two-Hops State Aware Updating  
Taking a given satellite as the center, two-hops states consist of the link states of all the ISLs within two-hops. It is shown in Figure 2.  
In DRL-THSA, each satellite keeps both link state table (LST) and neighbors' link state tables (NLST). The link states are stored as the style in Table 2.  
To monitor the link connectedness, we adopt the HELLO packet strategy proposed in the open shortest path first routing scheme [22]. The satellite sends HELLO packets to its neighbors with the period \(t_{h}\) The connectedness is defined to be off if the current satellite does not receive the acknowledgment (ACK) message from the direction \(n\) within \(t_{d}\). It may not change states until it receives the HELLO packets periodically. When the change happens, the current satellite updates its LST and broadcasts the connectedness change messages to all the other neighbors. To monitor the link state, satellites check the buffer queue of all the directions with the period \(t_{c}\). If the current link state is different from the previous one, the current satellite will update its LST and send the link state change messages to its neighbor satellites. When satellite receives the message from neighbors, it updates NLST according to the information contained in the message. The process of dynamic two-hops state aware updating is shown in Algorithm 1. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Node** & **Direction** & **Connectedness** & **Link State** & **Timestamp** \\ \hline \(N\) & \(n\) & On/Off & Free/Busy/Congested & \(t\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Link state table. Figure 2: Inter-satellite links within two hops. ```
ConnectednessUpdating: while true do  broadcast\(HELLO\) packets  wait\(t_{h}\) endwhile if receive ACK within \(t_{d}\)then if(\(t\) is up to date)then \(LST(N,n)\).connectedness\(\leftarrow\) On else  drop the message endif else \(LST(N,n)\).connectedness\(\leftarrow\) Off endif
```  
Link StateUpdating: while true do  calculate \(T_{1}\) and \(T_{2}\)  evaluate link state \(LST(N,n)\).state\(\leftarrow\)state  wait \(t_{c}\) endwhile ```
NeighborLink StateUpdating: Receive link change message if(\(t\) is up to date)then  continue else  drop the message endif if(\(NLST(N,n)\).state is not equal to state)then \(NLST(N,n)\).state\(\leftarrow\)state else  drop the message endif
```

**Algorithm 1** Dynamic Two-Hops State Aware Updating

## 3 Routing Strategy Based on Deep Reinforcement Learning

### Deep Reinforcement Learning Model Setting

The satellite networks topology was converted into a 2D plan, shown in Figure 3. Reinforcement learning observes the information obtained in the satellite networks topology, which functions as the environment. In deep reinforcement learning, an agent is modeled as a four-tuple consisting of \(\{S,A,P,R\}\), where \(S\) is a set of states, \(A\) is a set of actions. \(P\) is the state transition probability that represents the probability of a switch from one state to another state, and \(R\) is a reward function that represents a reward \(r\) received from the operating environment. Combining the deep reinforcement learning with satellite routing strategy, we construct the model \(\{S,A,P,R\}\) as Table 3.

where \([N_{s},N_{d},LST]\) represents routing source node, destination node, and current satellite's LST. \(N_{next}\) represents the decision of next-hop satellite node. \(P\) is set to be \(P_{next}\) calculated by Equation (13):

\[P_{next}=\left(\sum_{i=1,2,\ldots,m}s_{i}\right)^{-1} \tag{13}\]

where \(m\) represents the number of neighbors for current satellite node, and \(s_{i}\) represents the number of link states of neighbor satellite \(i\). When the packet is routed from \(N_{s}\) to \(N_{d}\), the reward \(r\) is calculated by Equations (14) and (15):

\[dif(N_{s},N_{d})=\alpha\cdot\left(RAAN_{s}-RAAN_{d}\right)^{2}+\beta\cdot \min\left[\left|\omega_{s}-\omega_{d}\right|^{2},\left(2\pi-\left|\omega_{s}- \omega_{d}\right|\right)^{2}\right] \tag{14}\]

\[r=\left\{