## 2 LEO Satellite Networks Model
### Two-Hops State Aware Updating
In addition, there is no need for DRL-THSA to recalculate when link states change owing to the fact that DDQN has known how to route them in the whole cases. ## 4 Simulation and Results

### Parameters Setup

To evaluate the proposed DRL-THSA, we use NS-3.29 (Network Simulator 3, Version 3.29) as the simulation tool to construct the simulations in an Iridium-like satellite network with 66 satellites distributed over six planes. Except for the satellites along the seam where cross-seam ISLs cannot be built, each satellite maintains two inter-plane ISLs and two intra-plane ISLs. Intra-plane ISLs keep connected all the time while inter-plane ISLs only work outside the polar area [23]. The capacity of ISLs is set to 25 Mbps. The average packet size is set to 1 KB and the queue length is set to 100 packets. We utilize 200 On-Off flows and the On-Off period of each flow follows a Pareto distribution with the shape of 1.5. The average burst and idle time are both set to 500 ms. Traffic load can be controlled by adjusting the data transmission rate of sources or the number of flows. The main system parameters are shown in Table 4. According to the experience, the greedy value of \(\varepsilon\) is set as 0.9. All the simulations are run for 60s equivalent to that in [9]. All scenarios are run 100 times, and the average values are considered as the final results. \begin{table}
\begin{tabular}{c c c} \hline \hline  & Altitude & 780 km \\  & Polar region boundary latitude & 70 \\  & Routing recomputation period & 600 ms \\
**General** & ISL bandwidth & 25 Mb/s \\  & Up/down bandwidth & 25 Mb/s \\  & Packet length & 1 kB \\  & Simulation time length & 60 s \\ \hline  & \(\alpha_{0}\) & 0.02 \\  & \(\alpha_{1}\) & 0.1 \\  & \(\alpha_{2}\) & 0.3 \\  & \(t_{c}\) queue check interval & 30 ms \\
**DRL-THSA** & ISL buffer queue size & 100 \\  & \(t_{s}\) the desired time for a satellite to & 200 ms \\  & reside in free state & 30 ms \\  & \(t_{h}\) the period of HELLO packet & 30 ms \\  & \(t_{d}\) the maximum acknowledgment & 30 ms \\  & time of HELLO packet & 30 ms \\  & \(\varepsilon\) the greedy value for DDQN & 0.9 \\ \hline  & ISL buffer queue size & 75 \\
**TLR** & Public waiting queue size & 100 \\  & Public waiting checking interval & 30 ms \\  & Maximum waiting time & 90 ms \\ \hline
**ELB** & ISL buffer queue size & 100 \\ \hline  & \(\alpha\) & 1 \\  & \(\rho\) & 0.75 \\
**ELMDR** & \(q_{1}\) & 0.2 \\  & \(c_{1}\) & 0.7 \\  & \(c_{2}\) & 0.3 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: System parameter. We evaluate our routing model, the ELB [9], the TLR [10] and the Extreme Learning Machine-based distributed routing (ELMDR) [12] under the same scenario for the comparison from three aspects: average end-to-end delay, packet drop rate, and system throughput. ### Results

#### 4.2.1 End-to-End Delay

The total delay of packets that arrived at their destination is recorded. To measure the performance of DRL-THSA under different traffic conditions, different individual data transmission rate and number of flows are used in the simulation. When the individual data transmission rate varies from 2.5 Mbps to 3.5 Mbps, the number of flows is fixed to 200. The purpose is to evaluate the system performance when the number of flows is unchanged and the transmission rate is changed. When the number of flows increases from 200 to 300, the individual data transmission rate is fixed to 3.5 Mbps. The purpose is to evaluate the system performance when the transmission rate is unchanged and the number of flows changes. The average end-to-end delay for different transmission rates is shown in Figure 4. The average end-to-end delay for different number of flows is shown in Figure 5.

It can be seen that the average end-to-end delay of DRL-THSA is smaller than that of ELB, TLR and ELMDR. The reason is that the DRL-THSA filters out the impact of short-term light traffic load while counting the average input rate and the output rate. The ELMDR needs to discover the routing path and pass back to the source node through the mobile agent.