# Fast-Convergence Reinforcement Learning for Routing in LEO Satellite Networks
## 5 Conclusions  
In this paper, we proposed a fast-convergence reinforcement learning algorithm to construct the routing issue in the LEO constellation. Aiming at addressing the characteristics of large dynamic satellite network status and unstable inter-satellite links, we designed a routing method named the fast-convergence reinforcement learning satellite routing (FRL-SR) for online decision-making. This method is always aware of the network link status and dynamically adjusts its routing strategy. The FRL-SR algorithm includes three parts: neighbor node discovery, offline training, and online training. By shortening the cycle time of agent obtaining network states, we accelerated the convergence speed of the algorithm, so that the routing decision was more suitable for the current network state. In addition, we also performed a complexity analysis, and the FRL-SR algorithm was superior to the Dijkstra algorithm in both time complexity and spatial complexity. The simulation results showed that the FRL-SR algorithm had a lower average delay and higher packet arriving ratio compared with the Dijkstra algorithm. In addition, the FRL-SR algorithm also had a good performance with respect to load balancing. It made full use of the resources of each node and reduced the probability of network congestion. Multi-agent cooperation is a promising method to solve the problem of large-scale satellite  
Figure 8: Measure the balance of node load in the network. network routing. In future work, we will continue to work on multi-agent reinforcement learning algorithms to better solve the problem of satellite network routing. Conceptualization, Z.D. ; Formal analysis, Z.D. ; Funding acquisition, H.L. ; Investigation, Z.D. ; Methodology, Z.D. ; Project administration, H.L. ; Resources, Z.Y. ; Software, Z.D. ; Supervision, N.W. ; Validation, F.T. ; Writing--original draft, Z.D. ; Writing--review & editing, Z.D. and F.T. All authors have read and agreed to the published version of the manuscript. * [PERSON] et al. (2017) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] State-of-the-art deep learning: Evolving machine intelligence toward tomorrow's intelligent network traffic control systems. _IEEE Commun. Surv. Tutor._**2017**, _19_, 2432-2455. [CrossRef]
* [PERSON] et al. (1995) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Topological design and routing for low-Earth orbit satellite networks. In Proceedings of the GLOBECOM'95, Singapore, 14-16 November 1995; Volume 1, pp. 529-535. * [PERSON] and [PERSON] (2018) [PERSON]; [PERSON] _Reinforcement Learning: An Introduction_; MIT Press: Cambridge, UK, 2018. * [PERSON] et al. (2021) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Tensor-based reinforcement learning for network routing. _IEEE J. Sel. Top. Signal Process._**2021**, _15_, 617-629. * [PERSON] et al. (2019) [PERSON]; [PERSON]; [PERSON]; [PERSON] A Multi-Agent Reinforcement Learning Routing Protocol for Underwater Optical Sensor Networks. In Proceedings of the ICC 2019--2019 IEEE International Conference on Communications (ICC), Shanghai, China, 20-24 May 2019; pp. 1-7. * [PERSON] et al. (2020) [PERSON]; [PERSON]; [PERSON]; [PERSON] Deep Reinforcement Learning-Based Beam Tracking for Low-Latency Services in Vehicular Networks. In Proceedings of the ICC 2020--2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 7-11 June 2020. * [PERSON] et al. (2017) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Multi-agent actor-critic for mixed cooperative-competitive environments. In Proceedings of the 31 st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4-9 December 2017; pp. 6379-6390. * [PERSON] et al. (2017) [PERSON]; [PERSON]; [PERSON] QGeo: Q-learning-based geographic ad hoc routing protocol for unmanned robotic networks. _IEEE Commun. Lett._**2017**, _21_, 2258-2261. [CrossRef]
* [PERSON] et al.