# Learning-Based Traffic Scheduling in Non-Stationary Multipath 5G Non-Terrestrial Networks
## 2 Materials and Methods
### Problem Formulation  
LOS estimation on multiple links can be formulated as a Markov decision process (MDP). Specifically, it is modeled as a Partially Observable Markov Decision Process (POMDP) [27] because, at any observation time, the RL agent can only observe the link(s) it has selected. A POMDP is defined by the tuple \(\{\mathcal{S},\mathcal{A},P(s_{t+\Delta t}|s_{t},a_{t}),r_{t}\}\), where \(\mathcal{S}\) is the state space of the system and \(\mathcal{A}\) denotes the action space for achieving the optimal choice. \(P(s_{t+\Delta t}|s_{t},a_{t})\) is the probability of being in state \(s_{t+\Delta t}\in\mathcal{S}\) after a time interval \(\Delta t\) conditioned by the action \(a_{t}\in\mathcal{A}\) and the state \(s_{t}\in\mathcal{S}\), and \(r_{t}\) is the immediate reward for the action \(a_{t}\) that leads to the state transition from \(s_{t}\) to \(s_{t+\Delta t}\). In the following, we describe the POMDP for our problem, where we assume that the UE can select a subset of the \(N\) available satellite links to which it is connected.  
1. _State space_: We assumed the state of each selected channel to be a binary variable with values in \(\{\text{LOS},\text{NLOS}\}\) so that we could formally define the state of the link \(n=1\ldots N\) at time \(t\) as follows: \[s_{nt}=\left\{\begin{array}{ll}+1&\quad\text{if $s_{nt}=\text{LOS}$}\\ \\ -1&\quad\text{otherwise.}\end{array}\right.\] Each of the \(N\) links dynamically changes its state between LOS and NLOS according to its own transition matrix \(T_{n}\) as defined in [28]. In our use case, we assumed that the number of available links was \(N=2\) so that we could define the link state space as the set of vectors \(\mathcal{S}=\{\mathbf{s}_{t}\mid\mathbf{s}_{t}=[s_{1t},\ldots,s_{Nt}]\}\), which generates a state space \(\mathcal{S}=\{[\text{LOS},\text{LOS}],[\text{LOS},\text{NLOS}],[\text{NLOS },\text{NLOS}]\}\).
2. _Actions_: An action constitutes the choice of the appropriate transmission pattern (i.e., a subset of the \(N\) links). The action space is the set of vectors \(\mathcal{A}=\{\mathbf{a}_{t}\mid\mathbf{a}_{t}=[\rho_{1t},\ldots,\rho_{Nt}]\}\), where \(\rho_{nt}=1\) indicates that the \(n\)th link is selected and it is \(\rho_{nt}=0\) otherwise for \(n=1\ldots N\). In this case study, we assumed that we had a pair of radio interfaces (i.e., \(N=2\)), which led to having an action space \(\mathcal{A}=\{[0,1],[1,0],[1,1]\}\).
3. _Reward_: The immediate reward \(r_{t}\) is defined as a penalty to the agent and is proportional to the E2E loss rate above the threshold calculated over an episode, as in the following equation: \[r_{t}=\left\{\begin{array}{ll}\frac{-(\psi-\varphi)}{\eta}&\quad\text{if $\psi>=\varphi$}\\ \\ \frac{1}{\eta}&\quad\text{otherwise.}\end{array}\right.\] (9)  
\begin{table}
\begin{tabular}{c c c} \hline
**Elevation** & \(\boldsymbol{P(B\to G)}\) **(g)** & \(\boldsymbol{P(G\to B)}\) **(b)** \\ \hline
20\({}^{\circ}\) & 0.00014310 & 0.00047466 \\
30\({}^{\circ}\) & 0.00024460 & 0.00027570 \\
45\({}^{\circ}\) & 0.00020318 & 0.00007556 \\
60\({}^{\circ}\) & 0.00105161 & 0.00010797 \\
70\({}^{\circ}\) & 0.00052923 & 2.76683\times 10^{-6}\) \\ \hline \end{tabular}
\end{table}
Table 2: Satellite link state transition probabilities for the dense urban area in France at 2.2 GHz.  
where \(\psi\) is the E2E loss, \(\varphi\) is the E2E loss threshold and \(\eta\) is the number of transmission links used in the previous episode. The first term in the equation encourages the use of double transmissions when there are losses, while the second term conserves bandwidth in favorable link conditions.