## 4 DRL Based TDIA Transmission Scheduling Algorithm (DRLSA-IA)
### The Describe of DRLSA-IA  
In this paper, we propose a deep reinforcement learning-based scheduling algorithm, called DRLSA-IA. An illustration of the DRLSA-IA is shown in Figure 7.  
Instead of obtaining transmission decisions by enumerating the decision space and state space, we use value function approximation to express the Q-value of each state-action pair, i.e., \(Q(S,a)\approx f(S,a,\omega)\). A vector containing the Q-values of all state-action pairs can be obtained when the state is input. We use the deep neural network to fit the value function. The Q-value of each action in each state can be calculated by the deep neural network. Figure 6: Time-domain interference alignment diagram. To avoid selecting overestimated Q-values that degrade performance, we use two deep neural networks to decouple the selection of the action from the calculation of target Q-values. The two deep neural networks are named as \(Q_{1}\) and \(Q_{2}\). Specifically, we use \(Q_{1}\) to find the action with the largest Q-value and use \(Q_{2}\) to estimate the target Q-value. \(Q_{1}\) and \(Q_{2}\) have the same network structure. For each node in the network, it is difficult to obtain the global network state. Thus, the class DQN does not perform well in this setting. In DRLSA-IA, we use an Long Short-Term Memory (LSTM) layer to replace the first fully connected layer of the deep neural network. Message transmission scheduling is a sequential decision problem. When making transmission decisions, it is not only based on the currently obtained information, but also needs to refer to previously stored prior information. LSTM maintains an internal state and aggregates observations over time. This gives the network the ability to estimate the true state using the history of the process. Considering the delayed reward caused by large propagation delays, we collect the _ktimes_ episodes at each iteration. All experiences in an episode are used to update. As shown in Equation (11), we create target values for all the episodes. By the LSTM layer, DRLSA-IA can aggregate observations from time slot \(t\) to time slot \(t+2G\) over time. \[ \begin{split}& Q_{target}\left(a^{j}(t)\right)=R(t)+\\ &\gamma\mathds{Q}_{2}\left(S^{j}(t+1),\arg\max_{a^{\prime}}Q_{1} \left(S^{j}(t+1),a^{\prime},\omega\right),\omega^{\prime}\right),\end{split}  \tag{11}\]  
where \(j\) is the \(j\)-sample experience. Each node in the network executes DRLSA-IA. The training process of DRLSA-IA is shown in Algorithm 1. Specifically, there are the following several steps: (1) The first step is when the node inputs the current observed state to the eval-net \(Q_{1}\) and obtains the estimation Q-values of all of the actions. (2) The second step occurs when the node determines to transmit a message to a destination node or wait according to the Q-values of all the actions. The node uses the \(\epsilon\)-greedy method to select the action. (3) The third step is when the node takes the action and obtains the reward. Then, the network transforms to the next state. (4) The next step is when the node feeds the new network state to the  
Figure 7: Illustration of the DRLSA-IA algorithm. eval-net \(Q_{1}\) and the target-net \(Q_{2}\), as well as obtains the Q-values of all the actions. (5) The target Q-value is computed according to Equation (11). After each iteration, the eval-net \(Q_{1}\) is trained with all experiences of _ktimes_ episode, as in Equation (12). After every \(c\) iteration, the parameters of the target-net \(Q_{2}\) are updated. We use the Mean Squared Error (MSE) loss function to update all parameters \(\omega\) of the eval-net \(Q_{1}\) through gradient backpropagation of the neural network, as shown in Equation (12). The node obtains all experiences of _ktimes_ episodes and uses these experiences to update the parameters of the model. The node inputs the current state \(S^{j}(t)\) of an experience to the eval-net \(Q_{1}\) and obtains the estimation Q-values of all of the actions. Then, target Q-values are calculated, as shown in Equation (11).