# Fast-Convergence Reinforcement Learning for Routing in LEO Satellite Networks
## 3 Proposed FRL-SR Algorithm
### Offline Training  
The algorithm proposed in this paper needs offline training before being applied to satellite nodes. The purpose of this process is to reduce the training time of the algorithm online and improve its initial performance. The network \(G_{t0}\) at \(t_{0}\) time is input as the initial state, and the output is that each satellite node receives a Q-table.  
Figure 2: Model framework of FRL–SR.  
To simulate the random nature of packet generation, the initial and destination nodes of packets are randomly selected from the set of satellite nodes. To reduce training time, if a packet is forwarded to its destination node, a new packet is generated, and its initial node and destination node are also randomly generated.  
To better explore the unknown space, the \(\epsilon\)-greedy strategy is used in the offline training phase. As shown in Equation (4), the agent randomly chooses an action with probability \(\epsilon\), and, with probability \(1-\epsilon\), it chooses the action with the maximum q-value. This strategy prevents local optimal solutions, but the convergence speed is low. To solve the above problem, the value of \(\epsilon\) gradually decreases as reinforcement learning progresses, which can speed up the convergence of the algorithm without affecting the learning effect.  
\[a_{t}=\begin{cases}\text{random action}&\text{w.p.}\epsilon\\ argmax_{a}Q_{t+1}&\text{w.p.}1-\epsilon\end{cases} \tag{4}\]  
Algorithm 1 is the process of offline training. First we need to initialize the training parameters: \(num_{step}\) is the total number of steps trained, \(\epsilon\) is the probability value of the greedy strategy, \(l_{r}\) is the learning rate of reinforcement learning, and \(\gamma\) is the discount factor of the Q value update function. At each step, the algorithm first cleans up the remaining packets in the network and then randomly generates a preset number of packets. For each satellite node, we first determine whether its sending queue \(q_{t}\) is empty, and the top packet pops up when it is not empty. Then, we select the next hop node m according to the \(\epsilon\)-greedy strategy. If the receive queue \(q_{r}\) of node m is not full, the packet is forwarded to m, and the current node will receive the Q-table and reward value of node m. After that, the Q-table of the current node should be updated according to Equation (2). Otherwise, the network is congested and packets will be inactive until the next round arrives.  
```
Input:\(G_{t0}=<N,E>;num_{step};\epsilon;l_{r};\gamma\) Output: Q-tables for each satellite node
1 initial env, Q-tables;
2 fork=1 tonum_stepdo
3 Clear all data packages;
4 Randomly generate \(x_{t}\) data packages;
5 forn=1 toNdo
6 ifq\({}_{t}\) in current node n \(\
eq\emptyset\)then
7 Pop a package p from \(q_{t}\);
8 Select an action m according to \(\epsilon\)-greedy;
9 ifq\({}_{r}\) of node m is not fullthen
10 Forward packets p to node m;
11 Node m send Q-table and reward to node n;
12 Update the state and Q-table of node n;
13 Update the state of node m;
14 ifm is the destination node of package pthen
15 reward = 20;
16 Randomly generate a package;
17
18 end
19
20 else
21 Wait a round;
22
23
24
25 end
26
27
28 end
29
30 end
310
32 end
33
34 end
35
36 end
37
38 end
39
40 end
410
42 end
43
44 end
45
46 end
47
48 end
49
50 end
510 end
520 end
53
54 end
55
56 end
57
58 end
59
60 end
611 end
62
63 end
64
65 end
666
```  
**Algorithm 1**FRL-SR offline training algorithm.