# Learning-Based Traffic Scheduling in Non-Stationary Multipath 5G Non-Terrestrial Networks
## 2 Materials and Methods
### Simulation Set-Up  
We simulated UAV satellite transmission with dual connectivity, in which one UAV could use two pieces of UEs to connect to two different satellites. Our goal was to train an AC learning agent to estimate the LOS model of the two satellites and the optimal policy for selecting appropriate links (transmission policy) while tracking the changes in the elevation angles of the satellites. As pointed out earlier, the LOS probability changes according to the elevation angle. Therefore, the learning agent must continuously track the variation of the LOS of the two satellites as a function of the elevation angles. To this end, using a satellite tracker ([[https://satellitemap.space](https://satellitemap.space)]([https://satellitemap.space](https://satellitemap.space)), accessed on 11 January 2023), pairs of Starlink satellites visible from Paris, France at a given time were selected. A satellite pair was selected that provided clear handoff events that forced the learning agent to retrain its model. This means that as the elevation angle of one satellite decreased and, consequently, the LOS probability decreased, the corresponding UAV interface connected to a new visible satellite with a higher elevation angle. Note, however, that the handoffs of the two interfaces did not happen simultaneously, since the two radio interfaces were independent of each other. Since the new connections had channel models different from the previous ones, the learning agent was forced to retrain to adapt to the new environment. The selected pairs of angles and the two handoff events are shown in Figure 3.  
Figure 4 shows the probability mass function of the satellite visibility at given elevation angles, evaluated using Equation (4), compared with the empirical values achieved from the real dataset collected by the satellite tracker during a window of 15 min (the maximum allowed). The ITU recommendations in [5] provide the link parameters required to compute the state transition probabilities at different elevation angles, propagation frequencies and environments. These parameters are reported in Table 1 for each state: the good state (G) and bad state (B). These include the mean (\(\mu\)), variance (\(\sigma\)) and minimum duration of each state in the given propagation environment. For the purpose of this work, we used the parameters for the dense urban scenario at 2.2 GHz and elevation angles of 45\({}^{\circ}\), 60\({}^{\circ}\) and 70\({}^{\circ}\), at which the satellites were visible from Paris, France, which was our reference scenario. By applying these parameters in Equations (6)-(8), we computed the corresponding state transition probabilities, as reported in Table 2. Other parameters used for the computation included the velocity of the mobile UE (\(\
u\) = 10 m/s) and the packet length (\(l\) = 1000 bits). Using the selected pairs of angles, shown in Table 4, and the computed transition probabilities, we constructed state transition matrices for each satellite and for each pair of elevation angles, obtaining a total of six transitions or _contexts_ (i.e., in each context or range of elevation angles, we transitioned to a different channel model). The duration of  
Figure 4: Probability that a satellite is visible from UE at given satellite elevation angles in Paris. Figure 3: Elevation angles of the pair of satellites connected to the UE in different contexts. the context was approximated to the minimum satellite visibility time. It is important to note that we did not draw any assumptions about the physical layer schemes or channel loss models because it was not mandatory for the training of the AC agent, according to the objective function that was designed. We then trained our AC agent using the channel traces of LOS and NLOS created above. The simulated AC networks consisted of 3 fully connected layers with 64 neurons on each layer.