## 2 LEO Satellite Networks Model
### Two-Hops State Aware Updating
The higher the value of the traffic distribution index is, the better the traffic is distributed over the entire constellation. Figures 10 and 11 show the traffic distribution index performance of these three routing strategies. It can be seen that the DRL-THSA outperforms the ELB, the TLR and the ELMDR. In order to verify that DRL-THSA can alleviate congestion and reduce queueing delays, the average queue occupancy of each satellite is shown in Figure 4.0 The simulations are performed for cases where the individual data transmission rate is fixed to 3.5 Mbps and the number of flows is fixed to 300. It can be seen that the DRL-THSA obtains the lowest average queue occupancy, which means that congestion is alleviated throughout the network. There are two reasons for DRL-THSA traffic with a more uniform traffic distribution than ELB, TLR and ELMDR. The first reason is that the DRL-THSA filters the shore-term light traffic load. The second reason is that the \(\varepsilon\)-greedy value of DDQN is set as 0.9. Therefore, DRL-THSA has a chance to explore the next-hop node autonomously, thus further diver

Figure 4.1 The traffic distribution index with different number of flows. Figure 4.2 The traffic distribution index with different sending rates. ## 5 Conclusions

In this paper, a Two-Hops State-Aware Routing Strategy Based on Deep Reinforcement Learning (DRL-THSA) for LEO satellite networks is presented. In DRL-THSA, we propose a mechanism to evaluate link states for adjusting the dynamic traffic in satellite networks and put forward a two-hops state-aware strategy to update the real-time link states. When the link states changes, it may broadcast the changes to its neighbors for updating the LST. To observe the information contained in LEO satellite networks topology, we train DDQN models through a simulated routing environment. According to the training efficiency and the convergence degree of the network, the \(\varepsilon\)-greedy value of DDQN is chosen as 0.9. Combined with the two-hops state-aware strategy, our models can figure out the optimal next hop for routing packets, and they can also handle situations including link failure, link recovery, link state change, and endless-loop route. Simulation results demonstrate that DRL-THSA performs well in terms of end-to-end delay, throughput, and packet drop rate and traffic distribution index. In future research, we will study the impact of deep learning network structure and parameter settings on routing strategy performance. **Author Contributions**: [PERSON] proposed the basic framework of the research scenario. In addition, [PERSON] was in charge of modeling the problem and proposed the routing strategy. [PERSON] did the simulations and wrote the paper. [PERSON] gave some suggestions on the mathematical model and formula derivation. **Funding**: This work was supported by the National Natural Science Foundation of China (NSFC) under the Grant No. 61801033. **Conflicts of Interest**: The authors declare no conflict of interest. ## References

* [PERSON] et al. (2017) [PERSON]; [PERSON]; [PERSON]; [PERSON] LEO Satellite Constellation for Internet of Things. _IEEE Access_**2017**, \(5\), 18391-18401. [CrossRef]
* [PERSON] et al. (2018) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] SDN-based end-to-end fragment-aware routing for elastic data flows in LEO satellite-terrestrial network. _IEEE Access_**2018**, \(7\), 396-410. [CrossRef]
* [PERSON] et al. (2017) [PERSON]; [PERSON]; [PERSON]; [PERSON] Network coding-based multi-path routing algorithm in two-layered satellite networks. _IET Commun._**2017**, _12_, 2-8. [CrossRef]
* [PERSON] et al. (2017) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] HGL: A hybrid global-local load balancing routing scheme for the Internet of Things through satellite networks. _Int. J. Distrib. Sens. Netw._**2017**, 13. [CrossRef]
* [PERSON] et al. (2007) [PERSON]; [PERSON]; [PERSON] An adaptive routing based on an improved ant colony optimization in LEO satellite networks.