## 2 LEO Satellite Networks Model
### Two-Hops State Aware Updating
The value of \(L^{DDQN}\) is used to update weights \(\theta^{\text{online}}\). To make sure the stability of the learning, DDQN uses the experience reply memory \(M\) to store experience \(e\), and a mini-batch of \(N_{b}\) experiences are taken at each iteration to train the DNNs. Because the network topology environment changes with the destination node. For the whole LEO satellite networks, the number of DDQN is equal to the number of satellites. #### 3.2.2 Double-DQN On-Board Running Process

The trained DDQN models are attached to the satellite. During the DDQN on-board running process, each satellite calculates the optimal next hop by inputting the two-hops link states to the corresponding DDQN model, and then the next hop node repeats the process until the packets arrive at the destination. In addition, DRL-THSA considers a total of four cases in LEO satellite networks including link failure, link recovery, link state change, and endless-loop route. The routing strategy can competently cope with the dynamic changes of the satellite networks by effectively handling these cases, observing the two-hops state information and training the DDQN. Algorithm 3 shows the workflow of DRL-THSA routing strategy. ```
1:Update the threshold \(T_{1}\) and \(T_{2}\) at whole directions of current satellite node \(N_{c}\)
2:Evaluate link states and update LST
3:if link state changes then  Broadcast LST to neighbors endif
4:Updating neighbor link state (NLST) by receiving link states information from neighbors
5:Load DDQN according to destination node \(N_{d}\)
6:Input\([N_{c},N_{d}, LST]\) to DDQN
7:Output the next-hop satellite \(N_{next}\)
8:Input\([N_{next},N_{d},NLST]\) to DDQN
9:Output the two-hops satellite \(N_{two}\)
10:if\(N_{two}\) is equal to \(N_{c}\)then  Suppose \(LST(N_{n})\)._connected_ness to be Off temporarily  Go to step 6; else  continue endif
11:transmit the packet to satellite node
```

**Algorithm 3** Workflow of DRL-THSA

* Link \((N,n)\) failure: Current satellite has not received ACK message from the direction \(n\) within \(t_{d}\). It is defined as link \((N,n)\) failure. Then it should update \(LST(N,n)\)._state_. When the routing task is coming, it is required to input the latest link state to DDQN. * Link \((N,n)\) recovery: When current satellite received ACK message from node \(n\) periodically, it is defined as link \((N,n)\) recovery. Then it needs to update \(LST(N,n)\)._state_. When the routing task is coming, it is required to input the latest link state to DDQN. * Link \((N,n)\) state change: The link states are divided into three cases according to the thresholds \(T_{1}\) and \(T_{2}\). When the link state changes from low load state to high load state, such as from FS to BS and from BS to CS, the current satellite will update \(LST(N,n)\)._state_, then send the link state change message and transmission rate reducing message. In contrast, if the link state changes from high load state to a low one, satellite updates \(LST(N,n)\)._state_ and sends the link state change message only. * Endless-loop route: This will cause a serious packet drop problem. Combining two-hops state aware with the observation of DDQN is the key to avoid endless-loop route. According to the calculation by DDQN, if the \(N_{two}\) is equal to \(N_{c}\), it means endless-loop occurs. We suppose the link connectedness of the current next-hop satellite \(N_{next}\) to be off during this routing process. Repeat the routing strategy until the \(N_{two}\) is not equal to \(N_{c}\). Then the \(N_{next}\) is chosen to transmit the packet. Traditional calculation strategies such as Dijkstra require too much computation resources because of global routing table updating. In fact, there may be only one link state change. DRL-THSA makesfull use of the two-hops link state information which is partially updated. It significantly reduces the updating overhead when only a few link states change. However, it is not applicable to the networks where the number of disconnected links is destructive.