## 3 Algorithm Description
### Operational Phase  
Due to the dynamic nature of satellites, the topology and connectivity of the satellite network undergo continuous changes. To effectively address the dynamic characteristics of satellite links, a temporal network model is adopted during the operational phase of the satellites. This approach effectively handles the dynamic nature of the satellite network while reducing the computational burden on the satellites, enabling efficient data routing. The operational phase strategy is formulated as a problem of maximizing the Q-value. The reinforcement learning Q-value expression is Equation (6), which represents a simplified formulation of the final Q-value and combines the immediate reward with the cumulative rewards of all the subsequent actions following action \(a\). Here, \(R_{i+1}\) denotes the reward for the next action, while \(R_{n}\) indicates the terminal reward value upon reaching the destination node. Consequently, \(Q(s,a)\) is determined by the reward conditions of traversed nodes rather than all nodes in the network, where \(n\) specifies the index of the final destination node, \(\gamma\) is the discount factor, and \(\alpha\) is the learning rate. A higher Q-value indicates a more optimal forwarding path selection, and a higher total Q-value corresponds to the globally optimal path. Therefore, during the actual operation of the satellite network, it is necessary to select the action with the highest Q-value at each forwarding step. As shown in Equation (7), \(Q_{sum}\) represents the total Q-value, which serves as a critical metric for evaluating path optimality, \(\sum_{i}^{n}\) represents the sum of Q, through which the packet is forwarded; and \(Q_{i}(s_{i},a_{i})\) represents the Q-value of state \(s_{i}\) and action \(a_{i}\). \[Q_{i}(s,a)=(1-\alpha)R_{i}+\alpha\gamma(R_{i+1}+R_{i+2}+\cdots+R_{n}) \tag{6}\]  
\[Q_{sum}=\sum_{i}^{n}Q_{i}(s_{i},a_{i}) \tag{7}\]  
In traditional Q-routing algorithms, agents can only obtain the Q-tables and link-state information of neighboring nodes when forwarding data packets to them, subsequently updating their own Q-tables. This approach is passive and limited, leading to the incomplete acquisition of network state information. When no data packets are forwarded between nodes, the timely detection of link-state changes becomes impossible. Additionally, this method suffers from low learning efficiency, as only a single entry in the Q-table can be updated at a time. To address these issues, this study proposes a periodic broadcasting mechanism. Upon receiving a data packet, a node actively broadcasts its Q-table and link-state information, enabling receiving nodes to promptly update their own Q-tables and improving learning efficiency. When a satellite's neighbor nodes change, the affected node updates both its Q-table and link-state information, subsequently propagating these updates to new neighbors to maintain network-wide information consistency. The smaller the broadcasting period T, the more frequently agents can perceive network state changes. This improved mechanism ensures the rapid perception of network states while effectively controlling communication overhead through period adjustment, thereby enhancing the reliability of routing optimization in LEO satellite networks. The operational phase process is illustrated in Figure 5. A shorter broadcast interval enhances the agent's ability to detect network state changes more rapidly, thereby facilitating the implementation of globally optimal routing strategies. In online training, the routing process initiates by loading the ground-pre-trained routing policy model and ingesting user-specified data packets. At periodic intervals t, the system updates both the satellite link states and Q-tables. When packet forwarding is required, the following will occur: (1) The next-hop node is selected based on Q-table values.