# User Pairing for Delay-Limited NOMA-Based Satellite Networks with Deep Reinforcement Learning
## 4 DRL for Delay-Constrained User Pairing
With this \(\varepsilon\)-greedy policy, \(Q\)-value function \(Q_{\pi}(s_{l},a_{l})\), which describes the expected \(R_{\pi}(l)\), can be given by  
\[Q_{\pi}(s_{l},a_{l})=E(R_{\pi}(l)|S=s_{l},A=a_{l}). \tag{25}\]  
This \(Q\)-value function is updated with  
\[Q_{\pi}(s_{l},a_{l})=Q_{\pi}(s_{l},a_{l})(1-\bar{\alpha})+\bar{ \alpha}\Bigg{(}R_{\pi}(l)+\hat{\gamma}\max_{a_{l+1}}Q_{\pi}(s_{l+1},a_{l+1}) \Bigg{)}, \tag{26}\]  
where \(\bar{\alpha}\) and \(\hat{\gamma}\) are the learning rate and discount factor, respectively. The best action can be written as \(a_{t}^{*}=\underset{a_{l}\in A}{max}Q_{\pi}(s_{l},a_{l})\). Following the environmental transition resultant from variations in users' link budgets and delay QoS limitations, the tuple \((s_{l},a_{l},R_{l},s_{l+1})\) at the \(l^{th}\) time slot is collected and stored in the experience pool, in which the old tuple gives space to the newest tuple (if the pool is full). Considering that the number of satellite users in a beam spot could be very large, the size and computation efficiency of \(Q\) values of (25) for all possible actions are large and low. In this context, deep neural networks parameterized by \(\theta^{{}^{\prime}}\) and \(\theta\), called target DQN and training DQN, respectively, are used in the neural network training phase to estimate the \(Q\)-value by function approximations. As shown in Figure 2, the target of the  
Figure 2: DQN-based NOMA user pairing model. DQN is to estimate the maximum \(Q\)-value for the next state, i.e., \(\max_{a_{t+1}}\!Q\Big{(}s_{t+1},a_{t+1};\theta^{{}^{\prime}}\Big{)}\). The training DQN network is deployed to make an action decision and estimate the \(Q\)-value for the current state, whose loss function can be written as  
\[L(\theta)=E\bigg{[}\Big{(}R_{\pi}(t)+\max_{a_{t+1}}\!Q\Big{(}s_{t+1},a_{t+1}; \theta^{{}^{\prime}}\Big{)}-Q(s_{t},a_{t};\theta)\Big{)}^{2}\bigg{]}. \tag{27}\]  
Using stochastic gradient descent to minimize the function in (27), the correct weights of \(\theta\) can be learned by the training DQN. The weights \(\theta^{{}^{\prime}}\) are frozen for several steps and then updated by setting \(\theta^{{}^{\prime}}=\theta\) for the goal of stabilizing the training. The specific steps for the training DQN to select one from many users to be User \(t\) is given in Algorithm 1. ```
Initialization: \(\bar{\alpha}\), \(\hat{\gamma}\), pool capacity \(D\), \(\theta\), \(\theta^{{}^{\prime}}\), state \(s_{0}\), l=k=1, parameter \(C\). while\(k\)\(\leq\)Kdo for\(l\)\(\leq\)\(T\)do  Observe state \(s_{l}\);  Choose action \(a_{l}\) under policy \(\pi\);  Perform action \(a_{l}\), observe reward \(R_{\pi}(l)\), and next state \(s_{l+1}\);  Store tuple \((s_{l},a_{l},R_{\pi}(l),s_{l+1})\) in pool; ifnumber of tuples in pool is larger than \(N_{p}\)then  Sample random mini-batch of tuples \((s_{l},a_{l},R_{\pi}(l),s_{l+1})\) from \(D\);  Update \(\theta\) by performing stochastic gradient descent on (27);  Set \(\theta^{{}^{\prime}}=\theta\) in every \(C\) steps. end if  end for k=k+1;  end while Output: In each time slot, a user who can achieve the biggest capacity with the power factor in (11) is selected to be User \(t\). ```  
**Algorithm 1**DQN Algorithm-based NOMA User Pairing in Satellite Networks.