# Learning-Based Traffic Scheduling in Non-Stationary Multipath 5G Non-Terrestrial Networks
## 2 Materials and Methods
### The Learning Process of the AC Agent  
At the beginning of each transmission window, the agent using the actor network will select the path (s) to use for transmission according to the observed channel state. It will then transmit the traffic over the selected paths. After several rounds of transmissions, it will receive feedback from the receiver on the loss rate over that window and the path states determined by the reception status. The reward is then computed and sent to the target-critic network to estimate the future state-action values. The critic network estimates the action-state value. The TD error is then computed as the difference between the critic and the target-critic estimates. Then, the actor and critic networks are updated using the TD error. Finally, after several transmission windows, the target-critic network is updated, copying the weights of the critic network. The full learning procedure is detailed in Algorithm 1.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Name** & **Value** \\ \hline Number of hidden layers & 3 \\ Number of neurons for hidden layer & 64 \\ Discount factor (\(\xi\)) & 0.96 \\ Learning rate for the actor (\(\beta\)) & 0.001 \\ Learning rate for the critic (\(\alpha\)) & 0.005 \\ Optimizer & ADAM \\ UAV velocity (\(v\)) & 10 m/s \\ Packet length (\(l\)) & 1000 bits \\ \hline \hline \end{tabular}
\end{table}
Table 3: Simulation parameters.  
```
1:Set to \(T\) the total length of the video to be transmitted, the number of iterations per episode to \(m\), and the learning rates for the actor and the critic networks to \(\beta\) and \(\alpha\), respectively. Set the initial state to \(\mathbf{s}_{0}\), the target-critic network update interval to \(n=km\) with \(k\) positive integer, and the counter for the target-critic update \(j=0\). Finally, set the parameters of the actor network, the critic network, and the target-critic network to \(\theta_{a}^{0}\), \(\theta_{c}^{0}\) and \(\theta_{lc}^{0}\), respectively.
2:while\(t\leq T\)do - Select the action \(\mathbf{a}_{t}\sim\pi_{\theta_{a}}(\mathbf{s}_{t})\) according to the available policy; - Set \(i=0\);
3:while\(i\leq m\)do - Transmit the video traffic in bits in each iteration by using the selected action \(\mathbf{a}_{t}\);
4:if\(i=m-1\)then - Receive feedback from the receiver on the loss rate and the future
5: states of the paths determined by the reception status in the last iteration; - Compute the reward \(r_{t}\) according to (9); - Compute the state-action value \(Q_{\theta_{a}}(\mathbf{s}_{t},\mathbf{a}_{t})\); - Calculate the future state-action values according to Equation (12); - Calculate the TD error according to Equation (13); - Update the critic parameters by minimizing: \(\delta_{t}^{2}\); - Update the actor policy according to Equation (16); - Update the agent's state observation: \(\mathbf{s}_{t}\);
6:endif - Set \(i=i+1\); - Set \(j=j+1\);
7:endwhile
8:if\(j=n\)then - Update the target-critic network according to Equation (18); - Set \(j=0\);
9:endif
10:endwhile
```  
**Algorithm 1** The training procedure for the AC-DRL agent for traffic scheduling  
For the purpose of comparison and performance evaluation of the RL method, an optimal traffic allocation policy was defined. This was considered optimal because we assumed that the channel state model was known in advance. Thus, the steady state probabilities were known accurately in each context and for all available paths.