# DRL-Based Load-Balancing Routing Scheme for 6G Space-Air-Ground Integrated Networks
## 4 DQN-Based Load-Balancing Routing Algorithm
### DQN-LLRA Framework and Mechanisms
The reward value was calculated by the reward function when the current agent made a decision, and the state transition occurred. Our reward function is as follows in Equation (5). \[r_{i}=\left\{ \begin{array}{l}\alpha_{1}\frac{C_{i}-C_{j}}{p}+\beta_{1}(1-QU _{j})+\gamma_{1}\frac{D_{ij}}{q}+\omega_{1}\frac{B_{ij}}{\sigma},C_{i}\leq C_ {j},QU_{j}\leq 0.5\\ \alpha_{2}\frac{C_{i}-C_{j}}{p}+\beta_{2}(1-QU_{j})+\gamma_{2}\frac{D_{ij}}{q} +\omega_{2}\frac{B_{ij}}{\sigma},C_{i}\leq C_{j},QU_{j}>0.5\\ \alpha_{3}\frac{C_{i}-C_{j}}{p}+\beta_{3}(1-QU_{j})+\gamma_{3}\frac{D_{ij}}{q} +\omega_{3}\frac{B_{ij}}{\sigma},C_{i}>C_{j},QU_{j}\leq 0.5\\ \alpha_{4}\frac{C_{i}-C_{j}}{p}+\beta_{4}(1-QU_{j})+\gamma_{4}\frac{D_{ij}}{q} +\omega_{4}\frac{B_{ij}}{\sigma},C_{i}>C_{j},QU_{j}>0.5\end{array} \right., \tag{2}\]  
where \(D_{ij},B_{ij},C_{i},C_{j}\), and \(QU_{j}\), respectively, represent the delay and bandwidth of the inter-satellite link \(e_{(\gamma_{1},\mu_{2})}\), the shortest hops from node \(i\) to the destination node, the shortest hops from node \(j\) to the destination node, and the queue utilization rate of node \(j\). The symbols \(\alpha_{i},\beta_{i},\gamma_{i},\omega_{i},i=1,2,3,4\) denote the parameter weights where \(\alpha_{i}+\beta_{i}+\gamma_{i}+\omega_{i}=1\). \(o\), \(p\), and \(q\) are adjustment factors, which keep each state value at the same order of magnitude. Here, \(C_{i}\), \(C_{j}\), and \(QU_{i}\) were used for segmental processing of the reward function, and different parameter weights were set for each segment function, in order to increase the penalty value of a poor decision and accelerate the convergence speed of the training  
Figure 5: Architecture diagram of DQN-LLRA. and the accuracy of the decision. We used four parameter weights to adjust the final result of the reward function. We can adjust the ratio of the parameter weights according to different optimization goals. The ratio of the parameter weights directly affects the relative performance of the final training model. The main optimization objectives of this paper are the hop count and queue utilization of the path. Therefore, in the simulation part, we set the parameter weights of the first two items to be larger and the parameter weights of the latter two items to be smaller. In addition, we imposed the following constraints on the reward function as follows in Formula (6). \[r_{t}=\left\{ \begin{array}{l}1,done=True\\ r_{t},done=False\end{array} \right.. \tag{3}\]  
When the current node was the destination node, the decision was ended, the reward value was 1, and the cumulative reward was calculated as follows:  
\[R_{t}=\sum_{t=0}^{\infty}\gamma^{t}r_{t+1}, \tag{4}\]  
where, \(\gamma^{t}\) represents the decay value of the future reward at time \(t\), while \(r_{t+1}\) denotes the instantaneous reward at time \(t+1\). In summary, the training flowchart of the DQN-LLRA algorithm is as follows in Figure 6.  
Figure 6: The training flowchart of the DQN-LLRA algorithm. Initially, the experience replay pool, estimation network, and target network were initialized for training. The Iridium satellite topology constellation randomly generated source and destination node pairs, and the routing decision was made by the source node acting as the agent with an initialized state. In the decision-making process, the agent made decisions according to the following dynamic greedy strategy in Equation (8). \[\pi_{i}=\left\{ \begin{array}{l}\text{random action with probability }e^{-steps\cdot 0.4}\\ \arg\max\limits_{a_{t+1}}Q^{\prime}(s_{t+1},a_{t+1}\Big{|}\varrho^{Q^{\prime}} )\text{ with probability }1-e^{-steps\cdot 0.4}\end{array} \right., \tag{5}\]  
where steps is the number of training steps. With the progress of training, the decision tended to choose the next hop decision with the largest \(Q\) value. After each decision, the reward value was calculated according to the reward function in (5). After the agent made a decision, \((s_{t},a_{t},s_{t+1},done)\) was stored in the empirical replay pool for reserve.