# Learning-Based Traffic Scheduling in Non-Stationary Multipath 5G Non-Terrestrial Networks
## 2 Materials and Methods
### Actor-Critic RL Architecture  
In this part, we describe the architecture of the actor-critic algorithm. As shown in Figure 2, the architecture of our AC learning agent model consists of three networks: the actor \(\pi_{\theta_{a}}(\mathbf{s}_{t})\), the critic network \(Q_{\theta_{c}}(\mathbf{s}_{t},\mathbf{a}_{t})\) and the target critic network \(Q_{\theta_{tc}}(\mathbf{s}_{t},\mathbf{a}_{t})\), parameterized by \(\theta_{a}\), \(\theta_{c}\) and \(\theta_{tc}\) respectively:  
1. Actor: The actor explores a policy \(\pi\) that maps the agent's observation of the state to the action space \(\mathcal{A}\) : using the mapping policy which is the function of the state: \(\pi_{\theta_{a}}(\mathbf{s}_{t})\ :\ S\rightarrow\mathcal{A}\). To explore the optimal policy \(\pi^{*}\), the actor selects actions \(\mathbf{a}_{t}\) from the action space \(\mathcal{A}\) and optimizes its selection policy in order to maximize the long-term rewards. The selected action is given by \[\mathbf{a}_{t}\sim\pi_{\theta_{a}}(\mathbf{s}_{t})\] (10) The agent's optimization goal of the long-term rewards can be represented by \[\pi^{*}(\mathbf{a}_{t}|\mathbf{s}_{t})=\arg\max_{\mathbf{a}_{t}}E\Big{[}\sum _{t=0}^{\infty}\gamma^{t}r_{t}\Big{]}\] (11) where \(\gamma\) is the discounting factor.
2. Critic: The critic is used to estimate the state-action value \(Q_{\theta_{c}}(\mathbf{s}_{t},\mathbf{a}_{t})\), which gives the goodness of the action selected by the actor at time \(t\) and state \(\mathbf{s}_{t}\) and is used to optimize the agent's selection policy in the direction of maximizing future rewards.
3. Target-Critic: To overcome the instability problem of the critic due to frequent updates, we use a third network, the target-critic network, to perform the [PERSON]'s estimation of the future state-action values. When the action \(\mathbf{a}_{t}\) is taken at time \(t\) and executed by the agent by transmitting the traffic over the network, the received feedback from the environment is sent to the target critic to estimate the future state-action value. The feedback includes the instant reward \(r_{t}\) and the next state of the environment (transmission paths). The future state-action values are estimated as follows: \[Q_{\theta_{tc}}(\mathbf{s}_{t+\Delta t},\mathbf{a}_{t+\Delta t})=r_{t}+\gamma Q _{\theta_{c}}(\mathbf{s}_{t+\Delta t},\mathbf{a}_{t+\Delta t})\] (12)  
Figure 2: Architecture of the actor-critic learning agent.  
Updating the Networks  
The estimates of the critic and target-critic networks were used to compute the Time Difference (TD) error [29]:  
\[\delta_{t}=r_{t}+\gamma Q_{\theta_{t}}(\mathbf{s}_{t+\Delta t},\mathbf{a}_{t+ \Delta t})-Q_{\theta_{c}}(\mathbf{s}_{t},\mathbf{a}_{t}) \tag{13}\]  
Then, the critic was updated as given below:  
\[Q_{\theta_{c}}^{*}=\arg\min_{Q_{\theta_{c}}}(\delta)^{2}. \tag{14}\]  
The actor was updated by using the policy gradient. The policy gradient was computed using the TD error as follows:  
\[\
abla\theta_{a}J(\theta_{a})=E_{\pi_{\theta_{a}}}[\
abla_{\theta_{a}t}\text{ In}\pi_{\theta_{a}}(\mathbf{s}_{t},\mathbf{a}_{t})\delta_{t}] \tag{15}\]  
where \(\
abla\theta_{a}J(\theta_{a})\) is the policy gradient and \(J(\theta_{a})\) is the policy objective function.  
The actor was then updated using the gradient descent method as follows:  
\[\theta_{a}=\theta_{a}+\beta\
abla_{\theta_{a}}\text{In}\pi_{\theta_{a}}( \mathbf{s}_{t},\mathbf{a}_{t})\delta_{t} \tag{16}\]  
where \(\beta\) is the learning rate of the actor. Finally, we updated the target critic network by using a soft update method:  
\[\theta_{tc}=\alpha\,\theta_{tc}+(1-\alpha)\theta_{c}. \tag{17}\]  
We designed the actor and critic networks using the TensorFlow-2 and Keras libraries with an ADAM optimizer in a fully connected multilayer perceptron Neural Network (NN) with the parameters given in Table 3.