# Fast-Convergence Reinforcement Learning for Routing in LEO Satellite Networks
## 4 Simulation Results  
In this section, we present the simulation results of the FRL-SR algorithm. The experiment was mainly divided into two parts: First, we simulated different reward functions in reinforcement learning and found out which one could minimize the delay for our subsequent experiments. Then, we compared the performance differences between the FRL-SR algorithm proposed in this paper and the traditional Dijkstra algorithm, both running in the same network environment and with the same user data requirements. We compared these two algorithms in terms of average delay, packet arriving ratio, and network load balance, and explained the reasons for their performance differences. To make the experimental results more accurate, we repeated all the experiments three times and took the average of the results as the final result. In this paper, we chose the Dijkstra algorithm as the comparison algorithm because it is the most mature and widely used algorithm in satellite routing algorithm, and through experiments, we verified that the proposed algorithm can effectively improve the satellite routing strategy and better support communication services [27]. The parameters of the experiment are given in Table 3. The network had a total of 7 satellite orbits, each containing 7 satellites, for a total of 49 satellites [28]. Because interstellar links fail frequently and link recovery varies, to simulate a satellite network more realistically, we set the propagation delay to vary according to a sinusoidal curve. In the offline training phase, the algorithm ran for 30 episodes--the step for each episode was 200, which ensured each agent could learn the routing strategy in a limited number of training steps. In the online training phase, we observed the delivery of packets in the network every 1 s and recorded it. In order to ensure the stability of the network, we adjusted the learning rate of this stage to 0.2, and the corresponding learning rate of the offline training stage was 0.8. The performance of different rewards is shown in Figure 4. The reward1 function was designed to be related only to the length of the link between the two nodes, in which \(D_{ij}\) was the distance between node \(i\) and node \(j\). \[reward1= \begin{cases}980&\text{Next node is the destination}\\ -0.1*D_{ij}&\text{Otherwise}\end{cases}  \tag{7}\] \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Values** \\ \hline Number of satellites & 49 \\ Delay type & sinusoidal \\ Trials & 3 \\ Offline training network load & 3000 \\ Initial network load for online training & 3000 \\ Max queue length & 150 \\ Max transmit packages at one time & 10 \\ Number of episodes & 30 \\ Number of steps peer episode & 200 \\ discount factor & 0.9 \\ Learning rate for offline training & 0.8 \\ Learning rate for online training & 0.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Simulation parameters. The FRL-SR algorithm outperforms the Dijkstra algorithm in terms of average delay in the same network environment. This is because the FRL-SR algorithm adjusts its forwarding policy based on the current state of the network and then forwards the packets to the optimal port, while the Dijkstra algorithm makes forwarding decisions based on the previously collected global information. The link state of the satellite network changes rapidly, and the information collected by [PERSON] cannot accurately reflect the state of the satellite network at that time. As can be observed from the above graph, the average latency increases slowly with time because the inter-star link is prone to failure, resulting in packet loss. The algorithm in this paper does not have a data retransmission function, which means the delay of lost packets will keep increasing, resulting in a rising average delay. The relationship between cumulative number of packets and time is shown in Figure 7. We can see that, over time, the advantages of the FRL-SR algorithm over the Dijkstra algorithm gradually become apparent.