# DRL-Based Load-Balancing Routing Scheme for 6G Space-Air-Ground Integrated Networks
## 4 DQN-Based Load-Balancing Routing Algorithm  
### DQN-LLRA Framework and Mechanisms  
Our paper presents a load-balancing routing algorithm for LEO satellites that leverages deep reinforcement learning. We modeled the satellite-routing decision-making process as a Markov decision process. The Markov decision process is a mathematical framework used to model decision-making processes, where only the immediate state affects the outcome of a current decision, rather than previous states. In our approach, each decision-making satellite was treated as an agent and interacted with the surrounding nodes to gather information about the current state in order to make informed routing decisions. The proposed algorithm used DQN to train the agents and make optimal routing decisions. At each routing node, the agent made a decision and selected the next route based on the information obtained from the surrounding nodes. The reward function was used to evaluate the quality of the decision made by the agent, and the agent moved on to the next routing node to continue making decisions. The DQN-LLRA algorithm architecture is shown in Figure 5. The proposed algorithm aims to optimize the routing decisions of LEO satellites by balancing the workload among the satellites effectively. By modeling the satellite-routing decision-making process as a Markov decision process and using deep reinforcement learning, the proposed algorithm can continuously learn and adapt to changing network conditions, leading to more efficient routing decisions. Figure 4: Schematic diagram of the inter-satellite links. The following section provides details on the DQN-LLRA algorithm, including the state space, action space, and the reward function settings. (1) The state space: The state of satellite node \(i\) at time \(t\) is represented as \(S_{t}^{i}\), where \(S_{t}^{i}=\left\{D_{ij},B_{ij},C_{i},C_{j},QU_{j},done\right\}\), with \(j=1,2,3,4\). \(D_{ij}\) denotes the delay of the inter-satellite link between node \(i\) and neighboring node \(j\), \(B_{ij}\) represents the bandwidth of the inter-satellite link between node \(i\) and neighboring node \(j\), \(C_{i}\) represents the shortest hop count from the current node \(i\) to the destination node, and \(C_{j}\) denotes the shortest hop count from neighbor node \(j\) to the destination node. The pair of numbers \(C_{i},C_{j}\) was introduced to enhance the algorithm's convergence rate and avoid undesired ping-pong routing. \(C_{i}\) and \(C_{j}\) are calculated by the shortest path algorithm, i.e., the Dijkstra algorithm. The agent leveraged the Dijkstra algorithm to calculate \(C_{i}\) and \(C_{j}\) when acquiring state information. When \(C_{i}-C_{j}>0\), the reward function gave positive feedback. When \(C_{i}-C_{j}\leq 0\), the reward function gave negative feedback. After multiple training rounds, the agent chose a node as close as possible to the destination node when making a decision. This avoided the occurrence of ping-pong routing and, at the same time, accelerated the agent to find the optimal path, thus speeding up the convergence speed of the algorithm. The symbol \(QU_{j}\) denotes the queue utilization of neighboring node \(j\), which indicates a load of node \(j\). The symbol \(Done\) denotes a two-valued Boolean number that indicates whether the current node is the destination node or not. \(Done\) is true if the current node is the target node and false otherwise. (2) The action: The action of the current satellite node \(i\) is expressed as \(a_{i}=\{\text{neighbor}_{j}\}\), \(j=1,2,3,4\), where \(\text{neighbor}_{j}\) represents that each decision agent selected a node \(j\) among the neighboring nodes according to the decision function. (3) The reward function: In order to meet the load balancing requirements in the LEO satellite scenario, our reward function introduced the queue utilization on the satellite node to represent the load situation of the node.