## 3 Algorithm Description
### Q-Table Initialization  
Due to the constrained computational resources and energy availability on satellites, as well as the potential for packet loss, network congestion, and significant propagation delays during the initial training phase, we employ ground-based pre-training to initialize the Q-table. This approach reduces the computational burden of updating the Q-tableduring the satellite's operational phase and ensures the reliability of the routing algorithm in the initial phase of satellite operation. The Q-table Initialization process is illustrated in Figure 4.  
Owing to the predictable orbital dynamics of satellites during the deployment phase, and as established in Section 2.1, each satellite node can establish connections with both inter-orbit neighbors (adjacent orbital planes) and intra-orbit neighbors (same orbital plane). For polar-orbit constellations such as Iridium, the system continuously maintains and updates visible satellite Ephemeris Data, enabling each node to form Ka-band connections with its four nearest neighbors according to the real-time orbital data. The Q-table Initialization process is conducted based on the initial topology of the satellite network, denoted as \(G_{t_{0}}\), and the output consists of the Q-tables distributed to each satellite node. To mitigate the effects of randomness and ensure the generalizability of the training outcomes, we will randomly generate the source and destination nodes of the data packets. In order to meet the actual satellite communication scenario, data packets will be generated upon the arrival of the previous packet, with randomly assigned source and destination nodes. Figure 4 illustrates the Q-table Initialization process. First, input the initialization of network environment and algorithm parameters. Second, each satellite node monitors its transmission queue \(q_{send}\). The queue determines the packet forwarding priority when it is not empty, while the node prioritizes transmitting the highest-priority packet first, thereby satisfying differentiated forwarding requirements for data of varying levels of importance. Third, the next-hop node \(m\) is selected according to the \(\varepsilon-greedy\) strategy. If the receiving queue \(q_{receive}\) of node m is not full, the packet is forwarded to \(m\), and the current node receives both the Q-table and reward value from node \(m\) and update state and Q-table  
Figure 4: Q-table Initialization flow chart. according to Equation (1). However, if the receiving queue is full or if node/link failure occurs (resulting in network congestion), the packet enters a waiting state until the next transmission opportunity. Finally, iterate through the above steps. To prevent convergence to local optima and enhance routing precision, we employ the \(\epsilon-greedy\) strategy. To enhance the convergence speed of the algorithm, a \(\epsilon\) decay factor \(\upmu\) is introduced. As the iterations progress, the probability of randomly selecting the next hop is reduced. As shown in Equation (5), the selection of the next hop will be impacted by \(\upmu\epsilon\). With probability \(\upmu\epsilon\), the next hop will not be an optimal selection. To expedite convergence, a decay factor \(\mu\), which progressively decreases \(\epsilon\) over time, thereby significantly enhances the convergence rate. This adaptive approach ensures a balance between exploration and exploitation, ultimately improving the efficiency of the routing algorithm. \[a_{t}= \begin{cases}\text{random action}&\text{probability}:\upmu\epsilon\\ \max(Q_{l+1})&\text{probability}:1-\upmu\epsilon\end{cases}  \tag{5}\]  
Furthermore, owing to the predictable nature of satellite networks, the Q-table Initialization scheme conducted via ground stations can be periodically reapplied. This approach mitigates potential inaccuracies in the online-trained Q-table that are caused by significant changes in the satellite network topology, thereby enhancing routing performance.