## 4 Experiment  
### Simulation Model Establishment  
The experimental setup uses Windows 10 as the operating system, equipped with an AMD R5 4500 u CPU. Python 3.8 serves as the primary tool for reinforcement learning solutions. The parameters for reinforcement learning are set as follows: \(\partial=1.0,\ \gamma=0.8\) are commonly used numerical values in the Q-table update formula, and parameter \(\varepsilon=0.8\) is used for calculating actions during random selection in the algorithm iteration, indicating a higher weight on completion time. The reinforcement learning algorithm iterates 200 times.  
The data packet format used in this paper is shown in Figure 8. Type is used to differentiate data packets, representing AF, BF, and EF types of data packets. In brief, different values indicate different priorities. Packet Length indicates the length of the data packet. Source and Destination represent the source satellite node and the destination node of the data packet, respectively. Data denote the contents being transmitted in the packet. Check Code is used to verify if any errors occurred during the data packet transmission process.  
Figure 7: Reinforcement learning algorithm flowchart.  
The simulation of this paper adopts satellite constellations based on optical inter-satellite links using the Starlink mega-constellation as an example. The constellation consists of 1584 satellites, with an ideal inter-satellite distance of 60-300 km, considering the absence of reverse seams and pole issues, and the satellites are positioned at a height of 550 km above the ground. The inter-satellite optical link transmission speed is \(3\times 10^{8}\) km. The gate control scheduling conversion time within a satellite is 400 us. As shown in Figure 9, the 1584 satellite nodes are divided into 56 secondary subnetworks using the Louvain algorithm, from which 7 sets of subnetworks are extracted. For the sake of image simplification, this article only selects seven sets of satellites at mid-latitudes after division as examples. In addition to adjacent connections with neighboring subnetworks, each secondary subnetwork also establishes long-distance connections with visible subnetworks. Satellites in the 7 th subnetwork have established long-distance connections with different satellites in the 5 th, 9 th, 13 th, 19 th, 24 th, and 38 th subnetworks. Subsequently, rerouting and data scheduling will be based on the new topology structure.  
For constellation sizes of 4, 8, 16, 32, 66, 128, 512, and 1584 satellites, the computation time of the visibility model was recorded ten times each, and the average was calculated. The reason for selecting these constellation sizes is that they represent the number of satellites in satellite constellations that are suitable for different scenarios in practical applications. The experimental results are shown in Figure 10. It can be observed from the graph that when the constellation size is less than 100 satellites, the computation time of the visibility model is less than 100 ms. As the constellation size increases, the computation time of the visibility model also increases due to the need to traverse the visibility between all satellites. When the constellation size reaches 1584 satellites, the overall constellation computation time is 2.946 s, showing relatively low computational time. To simplify the  
Figure 8: Packet format.  
Figure 9: Partially partitioned optical inter-satellite subnetwork.  
complexity of this paper and reduce computation time, a pre-partitioned subnetwork is selected for the next step of routing scheduling and data transmission.