# DRL-Based Load-Balancing Routing Scheme for 6G Space-Air-Ground Integrated Networks
## 5 Simulation and Performance Analysis  
In this work, we employed the Win\(11\) 64-bit system, used the Python torch deep learning framework, and utilized the programming language Python3.9 to carry out the simulation experiments and analyze the results. The experiment was carried out under the constructed Iridium satellite model, and the Iridium satellite topology was generated by using the official satellite data files of the Iridium constellation and Python networkX. As shown in Figure 10, the generated Iridium constellation contained six orbits, and each orbit contained 12 satellites. At the same time, the delay and bandwidth were randomly generated for each inter-satellite link. In order to simulate the dynamic network environment in the topology below, 70 streams of data flow were generated in a loop at the same time to propagate in the topology, so that the agent could obtain dynamic queue utilization and link bandwidth. The DQN algorithm in this paper was configured with a neural network that consisted of an input layer, an output layer, and two hidden layers. The discount factor was set to 0.99, and the experience replay pool size was 4000. The DQN-LLRA algorithm trained the network with 10,000 randomly generated data streams and produced a routing decision model after convergence. The relevant simulation parameters are shown in Table 1. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Parameter** & **Symbol** & **Value** \\ \hline link delay & \(D_{ij}\) & random value (10–20 ms) \\ link bandwidth & \(B_{ij}\) & 10 Mbps \\ discount factor & \(\gamma\) & 0.99 \\ learning rate & \(\alpha\) & 0.005 \\  & \(\alpha_{i},\beta_{i},\gamma_{i},\omega_{i}\), & 0.45, 0.35, 0.1, 0.1 \\ reward parameter weights & \(i=1,2,3,4\) & 0.50, 0.30, 0.1, 0.1 \\  & \(i=1,2,3,4\) & 0.40, 0.40, 0.1, 0.1 \\  & \(0.50,0.30,0.1,0.1\) \\ reward adjustment factors & \(o,p,q\) & \(10,20,20\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameter settings for evaluation. Figure 10: Iridium satellite topology. Figure 11 shows the variation trend of the reward with the training data stream. The Dijkstra-QU algorithm was used to find the shortest path after transforming the queue utilization into the link weight. The algorithm aimed at the lowest queue utilization but did not consider the path hop count and path delay. We compared our proposed scheme with the routing scheme based on reinforcement learning in [45]. In this benchmark scheme, Q-learning was used as the solution method for the satellite-routing decision process. At the same time, the packets in the routing process were used as agents. Each packet made a routing decision for the next hop based on the surrounding environment of the current satellite and destination node. The following performance evaluation compared the traditional Dijkstra algorithm, the Dijkstra-QU algorithm, the Q-learning-based scheme, and the proposed DQN-LLRA algorithm. To simulate the overhead resulting from the agent-environment interaction in the machine learning algorithms, we decreased the bandwidth around the decision node in both the Q-Learning-based intelligent routing scheme and the proposed routing scheme. This promoted the fairness of the comparison with other benchmark algorithms. In this work, 100 pairs of the source and destination nodes were generated to evaluate the performance of the four different routing algorithms. For each algorithm, 100 routing paths were calculated, resulting in a total of 400 paths. A mathematical analysis was conducted to evaluate the delay performance of the algorithms. From Figure 12, it can be seen that in the test with 100 pairs of source-destination nodes, the Dijkstra algorithm had the lowest latency among the four algorithms. In contrast, the Dijkstra-QU algorithm with queue utilization as the optimization objective had the highest latency.