## 3 Algorithm Description
### Model Setup  
This paper adopts an Iridium-like LEO satellite network model, with graphical \(G=(V,E)\) representation methodologies. In a graph, \(V\) is the set of satellite nodes and \(E\) is the set of ISL. The number of nodes in graph G is denoted as \(Num_{v}=\big{(}Num_{x},Num_{y}\big{)}\), where \(Num_{x}\) represents the number of orbital planes and \(Num_{y}\) represents the number of satellites deployed in each orbital plane. The index \(i\) of satellite node \(N_{i}\) can be calculated according to Equation (2). \[i=Num_{y}X_{i}+Y_{i} \tag{2}\]  
where \(X_{i}\in\{0,\dots,Num_{x}-1\}\) denotes the orbital index of the satellite node \(N_{i}\), and \(Y_{i}\in\{0,\dots,Num_{y}-1\}\) represents the sequence number of the satellite node \(N_{i}\) within its orbit plane. The relevant parameters are defined in Table 3.  
When a data packet arrives at satellite node \(N_{i}\), the node \(N_{i}\) will select the most appropriate action \(a_{t}\) from the action set \(A_{c}\) to forward the packet to node \(N_{j}\) on the basis of the current environment state \(se_{t}\). \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Node \(i\) State**} & \multicolumn{4}{c}{**Neighbor**} \\ \cline{2-5}  & \(N_{1}\) & \(N_{2}\) & \(N_{3}\) & \(\dots\) \\ \hline \((i,Des_{1})\) & \(Q_{i}((i,Des_{1}),N_{1})\) & \(Q_{i}((i,Des_{1}),N_{2})\) & \(Q_{i}((i,Des_{1}),N_{3})\) & \(\dots\) \\ \hline \((i,Des_{2})\) & \(Q_{i}((i,Des_{2}),N_{1})\) & \(Q_{i}((i,Des_{2}),N_{2})\) & \(Q_{i}((i,Des_{2}),N_{3})\) & \(\dots\) \\ \hline \((i,Des_{3})\) & \(Q_{i}((i,Des_{3}),N_{1})\) & \(Q_{i}((i,Des_{3}),N_{2})\) & \(Q_{i}((i,Des_{3}),N_{3})\) & \(\dots\) \\ \hline \(\dots\) & \(\dots\) & \(\dots\) & \(\dots\) & \(\dots\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The Q table of node \(i\). \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Definition** \\ \hline \(G\) & Satellite network graph \\ \(V\) & Set of satellite nodes \\ \(E\) & Set of inter-satellite links \\ \(Num_{v}\) & Number of satellites \\ \(N_{i}\) & Individual satellite nodes \\ \(D_{ij}\) & Delay between nodes \(N_{i}\) and \(N_{j}\) \\ \(S\) & Set of satellite states \\ \(A_{c}\) & Set of forwarding actions \\ \(Q\) & Q-value \\ \(q_{max}\) & Maximum satellite load \\ \(g_{j}\) & Load condition of node \(j\) \\ \(N_{b_{j}}\) & Set of neighbor nodes for \(N_{i}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Parameters and their definitions. The data forwarding process between satellites can be modeled as a finite-state Markov chain, where the terminal state corresponds to the successful delivery of the data packet to the destination node. The Markov decision process (MDP) can be defined by \((S,A,P,R)\). In the context of satellite network routing, S, A, and R can be defined as follows:  
* State: The global environmental state of the satellite network at time \(t\), denoted as \(se_{t}\in S\), is defined as \(se_{t}=\left\{N_{c},N_{a},q_{1}^{t},q_{2}^{t},\ldots,q_{Num_{v}}^{t}\right\}\), where \(N_{c}\) and \(N_{a}\) represent the current and destination nodes of the data packet, and \(q_{i}^{t}(1\leq i\leq Num_{v})\) indicates the queue length of \(i\) node at time \(t\). * Action: The action \(a_{t}\in Act=\left\{Q_{1},Q_{2},\ldots,Q_{p}\right\}\) corresponds to the forwarding decision for the data packet, where \(Q_{i}\) denotes the Q-value associated with forwarding to the \(i\) node. In LEO satellite networks, each satellite can establish connections with a maximum of four neighboring satellites [22], thus \(\max(p)=4\). * Reward: The reward function is influenced by two key factors--propagation delay \(D_{ij}\) and the load condition \(g_{j}\). A maximum load threshold \(q_{max}\) is defined and, according to Equation (4), \(g_{j}\) is related to the number of received packets \(q_{receive}\), sent packets \(q_{send}\), and occupied space \(q_{occupied}\), which represents the number of packets that the node has stored.