# User Pairing for Delay-Limited NOMA-Based Satellite Networks with Deep Reinforcement Learning
## 4 DRL for Delay-Constrained User Pairing  
The deep \(Q\)-network (DQN) algorithm, which combines the advantages of \(Q\)-learning and deep neural networks, is one of the most representative value-based method in the DRL family, with which the expected returns of actions can be predicted based on a certain environmental observation; a framework of applying such an algorithm in user pairing for the considered system is provided in Figure 2. (The DQN method is the classical approach in the DRL family, whose complexity analysis is not provided in this paper--the interested reader can refer to [33].) Although DRL deployment causes more delay, it is believed that this delay can be significantly decreased with the improvement of chip processing speed). Since our objective in problem P1 is to choose an appropriate user to be User \(t\) at different time slots to maximize the power resource utilization, we thus define a tuple \(\bar{M}:=<S,A,R,\pi>\) to model this problem as a Markov decision process (MDP) for a stationary decision. Specifically, \(S\) means the state and observation space, \(A\) represents the set of actions, \(R\) means the designed reward, and \(\pi\) is the policy that makes the decision. Meanwhile, \(Q_{\pi}(s_{t},a_{t})\) is defined as the Q-value obtained with policy \(\pi\) when the environment is in state \(s_{l}\) while adopting action \(a_{l}\) at the \(l^{th}\) time slot. For the problem P1, key elements, such as the states, actions, and cost, in an MDP model are described in detail as follows:  
* State \(S\): At time slot \(l\), a tuple denoted by \(s_{l}=(P_{s_{l}},\Phi_{j},d_{j},g_{j},\theta_{j})\), \(s_{l}\in S\) is used to describe the system state, where \(P_{s},\Phi_{j},d_{j},g_{j},\theta_{j}\) are transmission power, antenna gains, location information, fading severity, and delay QoS exponent of User \(j\) (\(j=1,2,\cdots,c-1,c+1,\cdots,m\)), as analyzed in Sections 2 and 3, respectively. Since \(s_{l}\) varies in different time slots, the agent is required to adjust its action in each slot accordingly;
* Action \(A\): NOMA user pairing is important for NOMA-aided satellite networks with delay QoS constraints because it directly impacts the resource utilization efficiency. Thus, user selection should be designed based on current state; here, we set the action space as \([A=1,2,\cdots,c-1,c+1,\cdots,m]\), and then \(a_{l}=m\) means the \(m^{th}\) user is selected to be the User \(t\);* Reward design: Equation (11) must be satisfied to ensure that User \(c\)'s performance achieved with the NOMA scheme is not less than that achieved with the TDMA scheme. Based on this, our objective is to select a user to be User \(t\) who, with the remaining power resource, can achieve the largest effective capacity. Thus, if User \(j\) is selected at time slot \(l\), the reward is assigned as \[C_{j}^{N}\big{(}\theta_{j},l\big{)}=C_{j}^{N}\big{(}\theta_{j} \big{)},(j=1,2,\cdots,c-1,c+1,\cdots,m)\] (24)  
As can be seen from Figure 2, the DQN algorithm has two phases. In the data-generation phase, \(Q\)-learning with experience pool \(D\) is used to generate data for the next network-training phase. In this process, the agent chooses an action \(a_{l}\) according to its observation \(s_{l}\) under policy \(\pi\). To trade off between exploration and exploitation, \(\varepsilon\)-greedy exploration is used here, which means, for state \(s_{l}\), a random action with probability \(\varepsilon\) (\(0<\varepsilon<1\)) or the best action with probability \((1-\varepsilon)\) is chosen to be action \(a_{l}\).