## 4 Simulation Experiment and Analysis
### Q-Learning Routing Algorithm Performance Evaluation  
In addition, take the 16-node inter-satellite network topology shown in the previous section as an example. In order to comprehensively evaluate the validity of the model proposed in this paper and the feasibility of the algorithm, this chapter takes the load of the service transmission path and the average transmission delay of the service as the main performance index. This chapter compares the proposed Q-routing routing algorithm with [PERSON]'s shortest path forwarding algorithm.  
Firstly, the proposed Q-learning based routing algorithm is explored with learning factor \(\alpha\) and discount factor \(\gamma\). According to the Formula (2g), it can be seen that the larger the learning factor \(\alpha\), the less the previous training results are retained, and the larger the discount factor \(\gamma\), the greater the effect of considering future rewards. Figure 11a,b, respectively describe the comparison of Q value fluctuation (Euclidean distance) when \(\alpha\) is set to 0.6 and 0.9, respectively, under different search steps with different discount factors.  
The simulation results in Figure 11 show that when the \(\alpha\) remains unchanged, the Q value fluctuation decays faster as the \(\gamma\) increases. When \(\alpha=0.6\) and \(\gamma\) is 0.3, it is about 30 steps that the Q value can be converged. When \(\gamma\) is 0.6 and 0.9, the convergence speed is accelerated, but there are still different degrees of oscillation. When the learning factor increases to 0.9, it can be seen that the convergence speed of the Q matrix is significantly accelerated, and the fluctuation is smaller, and the effect is better.  
However, speeding up the convergence rate of Q-routing routing needs further study. At the beginning of training, a larger learning factor can improve the convergence speed of Q matrix. However, as the training level increases, a larger learning factor causes the Q-matrix to move back and forth on both sides of the optimal point. Therefore, at the beginning of training, the learning factor can be set to a larger value, and with the increase of the number of iterations, the learning factor gradually decreases, that is, to achieve dynamic control of the learning factor. The learning factors are updated dynamically, and the update strategy is shown in Figure 12.  
Figure 10: Network Throughput Comparison Chart.  
Figure 11: The comparison between the degree of convergence of the Q matrix and the number of search steps under different values of \(\alpha\) and \(\gamma\). (**a**) \(\alpha=0.6\), under different \(\gamma\), the comparison chart of the convergence degree of Q matrix. (**b**) \(\alpha=0.9\), under different \(\gamma\), the comparison chart of the convergence degree of Q matrix.  
Figure 13 depicts the comparison of convergence before and after Q-routing algorithm updates the learning factor. The results show that after setting the dynamic learning factor, due to the initial \(\alpha\) value is larger, the learning rate is faster, and then decreases gradually. A lot of previous training results are retained, that is, using the previous experience, so that the convergence rate of Q-matrix is improved to some extent.  
Next, we compare the Q-routing algorithm with the Dijkstra algorithm and the Q-learning algorithm proposed in [22]. The Q-learning routing algorithm can be regarded as a combination of the Dijkstra algorithm and the Q-learning algorithm. It predicts the link load and switches to Q-learning algorithm after reaching 80%. The reward function is relatively simple. When reaching the target node, it obtains 30 immediate rewards. When reaching the neighbor node of the target node, it obtains 20 rewards. This routing algorithm converges faster. It can also alleviate link congestion to some extent.  
Figure 12: Learning Factors \(\alpha\) Dynamic Change Curve with Training Step.  
Figure 13: Q-matrix convergence before and after updating learning factors dynamically.