# Deep Reinforcement Learning for the Coexistence of eMBB and URLLC Services in NOMA Systems

Joseph Doumit, Marie-Josépha Youssef, Charbel Abdel Nour, Catherine

Douillard

► To cite this version:

Joseph Doumit, Marie-Josépha Youssef, Charbel Abdel Nour, Catherine Douillard. Deep Reinforce-ment Learning for the Coexistence of eMBB and URLLC Services in NOMA Systems. ICC 2025: IEEE International Conference on Communications, Jun 2025, Montréal (Québec), Canada. pp.5265-5270, 10.1109/ICC52391.2025.11161603. hal-05045754

HAL Id: hal-05045754

https://imt-atlantique.hal.science/hal-05045754v1

Submitted on 24 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Deep Reinforcement Learning for the Coexistence of eMBB and URLLC Services in NOMA Systems

Joseph Doumit<sup>(1)</sup>, Marie-Josépha Youssef<sup>(2)</sup>, Charbel Abdel Nour<sup>(1)</sup>, Catherine Douillard<sup>(1)</sup>

<sup>(1)</sup>IMT Atlantique, Lab-STICC, UMR CNRS 6285, F-29238 Brest, France

<sup>(2)</sup>Capital Fund Management, Paris, France

Email: joseph.doumit, charbel.abdelnour, catherine.douillard@imt-atlantique.fr, mariejosephay@gmail.com

***Abstract***—This paper introduces a novel resource allocation framework for the coexistence of enhanced mobile broadband (eMBB) and ultra-reliable low latency communication (URLLC) services in a non-orthogonal multiple access (NOMA) system. The proposed solution utilizes a two-layer deep reinforcement learning (DRL) algorithm to optimize the dynamic allocation of subbands and power for both URLLC and eMBB users. In the top layer, a deep Q-network (DQN) is employed for discrete subband allocation while the bottom layer uses a multi-agent DRL approach to refine the continuous power allocation. This hierarchical approach ensures that low latency and reliability requirements of the URRLC services are met, without compromising the high throughput demands of eMBB. Extensive simulations validate the efficiency of the proposed algorithm, demonstrating improved system performance in terms of power and subband optimization for both user types.

***Index Terms***—Non-orthogonal multiple access, coexistence of eMBB and URLLC, deep reinforcement learning, power allocation, subband allocation.

## I. INTRODUCTION

The fifth-generation (5G) wireless communication system classifies service requirements into three main categories: enhanced mobile broadband (eMBB), ultra-reliable low-latency communication (URLLC), and massive machine-type communication (mMTC) [1]. eMBB users demand high data rates, URLLC services prioritize stringent reliability and low latency, while mMTC focuses on supporting a large number of simultaneous connections. In practice, eMBB users typically consume the majority of wireless resources, whereas URLLC users require resources sporadically but with critical timing constraints. A straightforward solution to handle URLLC traffic involves reserving dedicated resources; however, this approach leads to inefficient spectrum utilization, as reserved resources may remain unused for extended periods. Hence, efficient mechanisms for the dynamic coexistence of URLLC and eMBB on shared wireless resources have become a crucial area of investigation.

To address this challenge, the 3GPP has introduced superposition and puncturing [2], [3]. These techniques partition each transmission slot (1 ms) into several mini-slots (0.125 ms), allowing URLLC users to preempt resources that were initially allocated to eMBB users. During the mini-slots occupied by URLLC traffic, the transmission power of the involved eMBB users is set to zero [4]–[7]. This issue has been under study for several years. For example, the authors in [4] proposed a

slicing architecture to achieve orthogonal coexistence between URLLC and eMBB traffic. In contrast, the studies in [5], [6] explored non-orthogonal sharing of radio access network (RAN) resources among eMBB, mMTC, and URLLC devices in uplink communications. Furthermore, [7] modeled the coexistence challenge as a deep reinforcement learning (DRL) problem. Despite these efforts, achieving efficient coexistence between URLLC and eMBB users remains an open research challenge.

Non-orthogonal multiple access (NOMA) has emerged as a promising access technology due to its potential to enable massive connectivity, ensure user fairness, and enhance spectral efficiency [8], [9]. Several studies have applied NOMA to address the coexistence problem between URLLC and eMBB traffic. For instance, [10] proposed a NOMA-based multi-cell cloud radio access network (C-RAN) architecture to explore performance tradeoffs between URLLC and eMBB users. The authors in [11] examined uplink NOMA transmission in an analog C-RAN architecture. Additionally, in [12], the orthogonal and non-orthogonal slicing of radio resources for eMBB and URLLC was studied using a max-matching diversity (MMD) algorithm to allocate frequency subbands for eMBB users. Moreover, a communication-theoretic framework for 5G network slicing was introduced in [5], where radio resources are shared among various 5G services using both orthogonal and non-orthogonal strategies. However, this study did not consider the use of NOMA for multiple URLLC devices, which limits the number of connected URLLC devices per base station (BS).

Hence, there is a clear need for a scalable and efficient resource allocation strategy to handle the dynamic coexistence of eMBB and URLLC traffic in NOMA systems. Traditional resource allocation methods either over-provision resources for URLLC traffic or introduce complex coordination mechanisms that increase system overhead. To address this, we leverage deep reinforcement learning techniques, specifically a deep Q-network (DQN), to dynamically allocate resources. Our approach is conceived to optimize power and subband allocation for both traffic types, while meeting the latency requirements of URLLC and the throughput demands of eMBB. The main contributions of this paper are summarized as follows:

* We propose a novel two-layer DQN framework for grant-free resource allocation, where the top layer manages discrete subband allocation for both eMBB and URLLC
users, and the bottom layer optimizes continuous power allocation using a multi-agent DRL approach.

* We introduce a scalable method to optimize both power and subband allocation in NOMA systems, allowing multiple users to share the same subband while meeting the URLLC latency requirements, and maximizing the sum rates of eMBB users.

* We propose a new method to organize users for successive interference cancellation (SIC) decoding, which improves the efficiency of the resource allocation process in NOMA systems. This approach enhances the handling of interference, especially when multiple URLLC users share the same resources.

The remainder of the paper is organized as follows. Section II presents the system model and formulates the problem. In section III, we describe the proposed DQN-based resource allocation algorithm. Simulation results are presented in section IV and section V concludes the paper.

## II. SYSTEM MODEL

Consider the uplink of a communication system consisting of one single-antenna BS, S subbands, and K sensors uniformly deployed over the cell as shown in Fig. 1. The sensors are divided into two categories: the first contains $K_u$ URLLC sensors, each with a rate requirement $R_k^{\text{req}}$, while the second contains $K_e$ full-buffer sensors, i.e., continuously have data to transmit. Let $P_{\text{max}}$ denote the power budget of each sensor. The channel gain between sensor k and the BS on subband s at timeslot t is denoted by $h_{k,s}^{(t)}$. This gain accounts for both small-scale Rayleigh fading and large-scale fading, which includes path loss and log-normal shadowing. Furthermore, the channels are spatially correlated, meaning that sensors in close proximity or with similar propagation environments experience correlated fading effects. To enable multiple sensors to share the same frequency subband of bandwidth B, NOMA is employed. In this scenario, when multiple sensors transmit on the same subband, their received power levels at the BS must differ to ensure correct SIC decoding. To that end, the uplink NOMA power allocation model proposed in [9] is adopted. In this model, a set of L available received power levels $V = \{v_1, \dots, v_L\}$ ordered in decreasing magnitude is first determined [9]:

$$v_l = \Gamma_l N_0 B \prod_{l'=l+1}^{L} (\Gamma_{l'} + 1), \quad (1)$$

where $\Gamma_l$ is the target signal-to-interference plus noise ratio (SINR) for power level $l$ and $N_0$ is the noise power spectral density. When a sensor k selects the received power level $v_l$ on subband s, it can achieve the following data rate:

$$R_{k,s,l}^{(t)} = B \log_2 \left( 1 + \frac{v_l}{V_l + N_0 B} \right), \quad (2)$$

where $V_l$ represents the power level of interfering transmissions that are not canceled by SIC. This power is expressed as: $V_l = \sum_{l'=l+1}^{L} v_{l'}$.

<figure><img src="image_2.png" /><figcaption>Figure 1: System model</figcaption></figure>

To achieve the received power level $v_l$ at the BS, the needed transmit power value by sensor k over channel s at timeslot t is given by:

$$P_{k,s,l}^{(t)} = \frac{v_l}{\left(h_{k,s}^{(t)}\right)^2}. \quad (3)$$

Given a power budget $P_{\text{max}}$, sensor k can successfully transmit only on subbands and power levels where the required transmit power satisfies $P_{k,s,l}^{(t)} \le P_{\text{max}}$.

The objective of this work is to jointly optimize both subband and power allocations, aiming to maximize the achievable rates of the $K_e$ sensors, while ensuring that each active URLLC user $k_u \in \mathcal{A}(t)$ transmits its data before its latency limit expires, where $\mathcal{A}(t)$ denotes the set of active devices. We transform this constraint into a minimum rate requirement using the max-plus queuing theory [13]. The optimization problem can thus be formulated as:

$$(\tilde{P}) \max_{x,p} \sum_{k_e \in \mathcal{K}_e} \sum_{s \in S} x_{k_e,s}^{(t)} R_{k_e,s}^{(t)} \quad (4)$$

$$\text{such that} \quad \sum_{s \in S} x_{k_u,s}^{(t)} R_{k_u,s}^{(t)} \ge R_{k_u, \min}, \quad \forall k_u \in \mathcal{A}(t), \quad (4a)$$

$$\sum_{s \in S} x_{k_e,s}^{(t)} R_{k_e,s}^{(t)} \ge \sum_{s \in S} x_{k_e,s}^{(t)} R_{k_e, \min}, \quad \forall k_e \in \mathcal{K}_e, \quad (4b)$$

$$\sum_{s \in S} p_{k,s}^{(t)} \le P_k^{\text{max}}, \quad \forall k \in \mathcal{K}, \quad (4c)$$

$$\sum_{s \in S} x_{k,s}^{(t)} \le 1, \quad \forall k \in \mathcal{K}, \quad (4d)$$

$$x_{k,s}^{(t)} \in \{0, 1\}. \quad (4e)$$

Constraint (4a) represents the rate required for the URLLC users. Constraint (4b) corresponds to the rate requirements for each eMBB user, while constraint (4c) defines the power budget for all devices. Finally, constraint (4d) ensures that each user can select at most one subband s during timeslot t, for which $x_{k,s}^{(t)} = 1$.

($\tilde{P}$) is a mixed-integer optimization problem, making it inherently non-convex. Finding the optimal solution for ($\tilde{P}$) requires an exhaustive search over all possible subband and power assignments, which is computationally intractable. To solve it, we propose a solution based on the DRL framework to efficiently allocate the subbands, determine the decoding order, and assign the transmit power accordingly.
### III. DEEP REINFORCEMENT LEARNING-BASED SOLUTION

In this section, we introduce a distributed, grant-free resource allocation solution leveraging a DRL framework.

The resource allocation process for both eMBB and URLLC users is controlled by two decision variables: subband assignment and power allocation. While subband assignment involves discrete variables, power allocation represents a continuous optimization problem. To address these distinct problems, we employ separate processing blocks for subband assignment and power allocation. At the top layer, the subband assignment is managed through a DQN, which efficiently handles the discrete nature of the problem. In the bottom layer, power allocation is optimized through another DQN. This second DQN starts with discrete power levels and iteratively refines them, progressively approaching a continuous power allocation solution.

#### A. An Overview of Reinforcement Learning

A reinforcement learning agent learns its optimal strategy by observing the rewards resulting from trial-and-error interactions with its environment. At each timeslot $t$, the agent observes the current state of the environment $s^{(t)}$ and selects an action $a^{(t)}$ based on a policy $\pi$, where $\pi(s, a)$ defines the probability of choosing action $a$ when the environment is in state $s$. After taking action $a^{(t)}$, the environment transitions to a new state $s^{(t+1)}$, and the agent receives a reward $r^{(t+1)}$. This sequence of interactions forms an experience expressed as: $e^{(t+1)} = \{s^{(t)}, a^{(t)}, r^{(t+1)}, s^{(t+1)}\}$.

The Q-learning algorithm seeks to compute the optimal policy $\pi$, which maximizes the expected reward. However, as the network scale grows, the classical Q-learning algorithm becomes inefficient due to the large storage requirements of the Q-table and its slow convergence. To address these issues, a DQN with experience replay [14] can be employed.

In a DQN, a neural network approximates the Q-function, eliminating the need for a large lookup table. The DQN can be described as $q(s, a, \theta)$, where $\theta$ represents the relevant network parameters. Finding the optimal Q-values is thus equivalent to optimizing $\theta$. Similar to classical Q-learning, the DQN agent gathers experiences from its interactions with the environment, building a dataset $D$. Following the quasi-static target network approach [14], our approach uses two DQNs: a local DQN with parameters $\theta_{local}^{(t)}$ at timeslot $t$, and a target DQN with parameters $\theta_{target}^{(t)}$. Every $T_s$ timeslots, $\theta_{target}^{(t)}$ is updated to match $\theta_{local}^{(t)}$. Periodically, the agent selects a mini-batch of experiences of size $L_b$ from its memory $D^{(t)}$ and uses it to optimize the parameters $\theta_{local}^{(t)}$. The goal is to minimize the least-squares loss function, given by:

$$Loss(\theta_{local}^{(t)}) = \sum_{D^{(t)}} (y_{DQN}^{(t)}(r', s') - Q(s, a, \theta_{local}^{(t)}))^2, \quad (5)$$

where $y_{DQN}^{(t)}(r', s') = r' + \gamma \max_{a'} Q(s', a', \theta_{target}^{(t)})$.

#### B. Subband and Power Allocation for URLLC and eMBB Users

To address the grant-free subband and power allocation problem for URLLC and eMBB users, we propose a multi-

agent DRL framework. In this model, each user operates as an independent agent within a two-layer approach: At the top layer, users select their subbands, and at the bottom layer, optimize their transmit power. The subband and power allocation problems are modeled as Markov decision processes (MDPs) to enable the efficient learning of optimal policies.

##### 1) Subband Allocation as a Markov Process:

a) **Action space design:** The action space for each user is defined as $\mathcal{A}^C = \{0, 1, \dots, S\}$, where $S$ is the total number of subbands. At timeslot $t$, user $k$ selects an action $a_k^{C,(t)} \in \mathcal{A}^C$, with $a_k^{C,(t)} = 0$ indicating no transmission and $a_k^{C,(t)} = s$ indicating that user $k$ transmits on subband $s$.

b) **State space design:** The state encapsulates the relevant information for decision-making by user $k$ at time slot $t$. It is defined as: $s_k^{C,(t)} = \{a_k^{C,(t-1)}, h_{k,s}^{(t-1)}, \forall s \in S, p_k^{(t-1)}, \epsilon_k, \psi_k, r_k^{(t-1)}\}$, where:

* $a_k^{C,(t-1)}$: the subband selection in the previous time slot,

* $h_{k,s}^{(t-1)}$: channel gain over each subband $s$ in the previous time slot,

* $p_k^{(t-1)}$: transmit power in the previous time slot,

* $\epsilon_k$: a binary indicator showing whether the SIC constraint is satisfied,

* $\psi_k$: a binary indicator of interference constraint violation,

* $r_k^{(t-1)}$: the achieved reward from the previous time slot.

The state across all users forms the input to the learning model, guiding future subband and power decisions.

2) **Power Allocation as a Markov Process:** The power allocation problem involves continuous decision variables, but since DRL models are typically discrete, previous works [15]–[17] discretize the power levels into a fixed set between 0 and $P_{max}$. However, such discretization introduces a trade-off between granularity and system complexity. In this work, we propose a novel method to derive near-continuous power levels without significantly increasing system complexity.

a) **Action space design:** For power allocation, we define the action space of user $k$ as: $\mathcal{A}^P = \{l_k P_{max}, (l_k + \frac{u_k-l_k}{L-1}) P_{max}, \dots, u_k P_{max}\}$, where $l_k$ and $u_k$ are the lower and upper bounds of the feasible power range for user $k$, and $L$ is the number of considered power levels.

b) **State space design:** The state of user $k$ for power allocation at time slot $t$ is: $s_k^{P,(t)} = \{a_k^{P,(t)}, h_{k,a_k^{P,(t)}}^{(t)}, p_k^{(t-1)}, lb_k, ub_k, \epsilon_k, \psi_k, r_k^{(t-1)}\}$, where:

* $a_k^{P,(t)}$: the subband selected at time slot $t$,

* $h_{k,a_k^{P,(t)}}^{(t)}$: channel gain over the selected subband,

* $p_k^{(t-1)}$: power level in the previous time slot,

* $lb_k, ub_k$: lower and upper bounds of the power search space,

* $\epsilon_k$: SIC satisfaction indicator,

* $\psi_k$: interference violation indicator.

c) **Reward function:** The reward for user $k$ at time slot $t$ is defined as:

$$r_k^{(t)} = \begin{cases} R_k^{(t)} \times \epsilon_k, & \text{if } \psi_k = 1, \\ -1, & \text{otherwise.} \end{cases} \qquad (6)$$
This reward structure encourages users to maximize throughput while maintaining interference-management constraints.

3) Interference Constraints and Power Optimization: To ensure that users sharing the same subband meet their rate requirements, we impose an interference robustness threshold $I_{th,k,s}$ for each user $k$. This threshold ensures that interference on the selected subband remains below acceptable levels. The threshold is derived from the following rate requirement:

$$R_{k,\min} = B \log_2 \left( 1 + \frac{P_{\max}^k h_{k,a_k}^2 C_{k,(t)}}{I_{th,k,s} + N_0 B} \right), \quad (7)$$

which gives the corresponding interference threshold:

$$I_{th,k,s} = \frac{P_{\max}^k h_{k,a_k}^2 C_{k,(t)}}{2 \frac{R_{k,\min}}{B} - 1} - N_0 B.$$

Based on the calculated $I_{th,k,s}$ values, users sharing the same subband are decoded in descending order of their interference thresholds, ensuring that the user with the highest threshold is decoded first. This user is most robust to interference, while the user with the lowest threshold is decoded last, free of interference, assuming successful SIC. The selection of transmit power follows this ordering to meet rate and interference constraints efficiently.

a) Search space bound update: To achieve continuous power allocation, the search space bounds for user $k$ are updated iteratively based on the received reward. Specifically, if $r_k^{(t)} = -1$, indicating interference constraint violation, user $k$ reduces its power; if $r_k^{(t)} \ge 0$, it increases its power. The bounds are updated as follows:

$$\begin{aligned} u_k &= l_k + \frac{(L-2)(u_k - l_k)}{L-1} & \text{if } r_k^{(t)} &= -1, \\ l_k &= l_k + \frac{(u_k - l_k)}{L-1} & \text{if } r_k^{(t)} &\ge 0. \end{aligned} \quad (8)$$

The bounds are refined until $u_k - l_k \le \theta$, where $\theta$ is a small predefined threshold.

4) DRL-Based Grant-Free Resource Allocation for All Users: Inspired by the work in [18], the proposed resource allocation algorithm leverages DQNs, which are trained offline at the BS. Upon completion of the offline training, each user receives a copy of the trained DQN for efficient online decision-making. The training procedure of the DQNs is outlined in Algorithm 1.

The initialization phase begins by setting key training parameters, including the learning rate $\eta$, the discount factor $\gamma$, the initial $\epsilon$-greedy exploration probability $\epsilon_{start}$, the minimum exploration value $\epsilon_{end}$, and the decay rate $\epsilon_{decay}$. Additional parameters such as the mini-batch size $L_b$, the replay memory size $L_M$, the target DQN update frequency $T_s$, and the weights of the local and target DQNs, $\theta_{local}$ and $\theta_{target}^{(0)}$, are initialized such that $\theta_{target}^{(0)} = \theta_{local}^{(0)}$.

In the training phase, at each timeslot $t$, the agent first processes the network state and inputs it into the subband allocation DQN, which computes the Q-values for all available

subbands. The agent then selects the subband $a_k^{C,(t)}$ based on the adaptive $\epsilon$-greedy policy. Specifically, with a probability of $\epsilon^{(t)}$, a random action (subband) is chosen to encourage exploration, while with probability $1 - \epsilon^{(t)}$, the action corresponding to the maximum Q-value is selected. The exploration probability $\epsilon^{(t)}$ evolves over time as follows:

$$\epsilon^{(t)} = \max \left( \epsilon_{end}, \epsilon_{start} \times (\epsilon_{decay})^t \right), \quad (9)$$

which ensures high exploration during the early stages of training, gradually shifting to exploitation as the training progresses. This trade-off between exploration and exploitation is critical for achieving efficient convergence and preventing the agent from getting stuck in local optima.

Once the subband is selected, the agent proceeds to optimize the transmit power. The state $s_k^{P,(t)}$, which includes the current subband selection and relevant channel information, is input to the power-level DQN. Similar to subband selection, the power level $a_k^{P,(t)}$ is chosen using the adaptive $\epsilon$-greedy algorithm. The environment then transitions to a new state $s_k^{P,(t+1)}$ and returns a reward $r_k^{(t+1)}$, based on the achieved rate and interference conditions.

The agent subsequently updates its power-level action boundaries, as outlined in step 11 of Algorithm 1, to refine the search space. The transition tuple $(s_k^{P,(t)}, a_k^{P,(t)}, r_k^{(t+1)}, s_k^{P,(t+1)})$ is stored in the replay memory, enabling the agent to learn from past experiences. The DQN is trained using the experience replay technique [14], where a mini-batch of size $L_b$ is randomly sampled from the replay memory, and the local DQN is updated via the RMSprop algorithm [19]. This approach reduces the correlation between consecutive updates and stabilizes training.

Every $T_s$ timesteps, the target DQN weights are updated to match those of the local DQN, ensuring that the learning process maintains a balance between stability and adaptability (step 15 in Algorithm 1).

Once the transmit power is selected, the environment transitions to the new state $s_k^{C,(t+1)}$ corresponding to subband allocation. The transition $(s_k^{C,(t)}, a_k^{C,(t)}, r_k^{(t+1)}, s_k^{C,(t+1)})$ is similarly stored in the replay memory, and the DQN training for subband allocation is performed following the same mini-batch-based learning process. Additionally, every $T_s$ timesteps, the target DQN weights are updated to match the subband DQN weights (step 22 in Algorithm 1).

This iterative procedure ensures that both subband and power allocation decisions are continually improved as the agents interact with the environment. By employing decentralized DRL, each user independently optimizes its resource allocation while coordinating implicitly through the shared environment. This framework effectively captures the dynamic and decentralized nature of grant-free access in URLLC and eMBB coexistence scenarios.

## IV. SIMULATION RESULTS

Extensive simulations were conducted to validate the performance of the proposed DRL-based resource allocation technique. We consider an uplink communication system with
Algorithm 1 DRL-Based Resource Allocation Algorithm

Initialization:

1: Initialize the training parameters, the replay memory and the weights of the local and target DQNs.

Training phase:

2: for t = 0, 1, 2, . . . do do

3: For each user *k*, input state $s_k^{C,(t)}$ into the local DQN.

4: Select action $a_k^{C,(t)}$ according to the adaptive $\epsilon$-greedy algorithm.

5: $t_p = 0$

6: while ($u_k - l_k > \theta$ do

7: $t_p = t_p + 1$

8: Input state $s_k^{P,(t)}$ into the local DQN.

9: Select action $a_k^{P,(t)}$ according to the adaptive $\epsilon$-greedy algorithm.

10: The environment transitions into state $s_k^{P,(t+1)}$ and returns reward $r_k^{(t+1)}$.

11: Update $l_k$ and $u_k$.

12: Store transition $(s_k^{P,(t)}, a_k^{P,(t)}, r_k^{(t+1)}, s_k^{P,(t+1)})$ in the replay memory.

13: Sample a mini-batch of size $L_b$ from the replay memory and train local DQN.

14: if $t_p \% T_s = 0$ then

15: Update the weights of the target DQN.

16: end if

17: end while

18: The environment transitions into a new state $s_k^{C,(t+1)}$.

19: Store transition $(s_k^{C,(t)}, a_k^{C,(t)}, r_k^{(t+1)}, s_k^{C,(t+1)})$ in the replay memory.

20: Sample a random mini-batch of size $L_b$ of transitions from the replay memory and train the local DQN with the RMSprop algorithm.

21: if $t_p \% T_s = 0$ then

22: Update the weights of the target DQN.

23: end if

24: end for

one BS located at the cell center and $K = 10$ randomly distributed users. The user set consists of 4 URLLC users and 6 eMBB users. The minimum rate requirements are set to $R_{k_e,min} = 1$ Mbps for eMBB users and $R_{k_u,min} = 1.5$ Mbps for URLLC users. Both user types have a maximum transmit power constraint of $P_k^{max} = 1$ W. The system operates over $S = 4$ subbands, each with a bandwidth of $B = 360$ kHz.

The propagation environment is modeled using a path loss exponent of 3.76 with an additional shadowing component modeled by a log-normal distribution with zero mean and a standard deviation of 8 dB. The noise power spectral density is assumed to be $N_0 = 4 \times 10^{-21}$ W/Hz.

The hyperparameters for the DQN architecture used in our algorithm are detailed in Table I. Each DQN consists of an input layer followed by two fully connected (FC) layers with 100 neurons each. The FC layers are activated by ReLU functions. A dueling architecture is employed, splitting the Q-function into a state-value function and an action-dependent advantage function, which improves the learning efficiency by isolating the impact of states from actions. The final layer provides the Q-values for all actions.

<table><caption>Table I: DQN architecture hyperparameters</caption><thead><tr><td>Parameter</td><td>Value</td><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Learning rate η</td><td>10<sup>-4</sup></td><td>Discount factor γ</td><td>0.5</td></tr><tr><td>Initial exploration probability ϵ<sub>start</sub></td><td>1</td><td>Final exploration probability ϵ<sub>end</sub></td><td>10<sup>-4</sup></td></tr><tr><td>Decay rate ϵ<sub>decay</sub></td><td>1 − 5 × 10<sup>-4</sup></td><td>Mini-batch size L<sub>b</sub></td><td>256</td></tr><tr><td>Replay memory size L<sub>M</sub></td><td>10,000</td><td>Target DQN update interval T<sub>s</sub></td><td>10</td></tr></tbody></table>

A. DQN Convergence Analysis

We first analyze the convergence behavior of the DQNs through their loss functions. Figs. 2a and 2b show the mean squared loss for both the subband (top) and power DQNs. As the number of timeslots progresses, both DQNs demonstrate a gradual reduction in loss, indicating effective learning over time. However, the top DQN exhibits larger fluctuations compared to the power DQN. This phenomenon can be attributed to the interdependence between the two DQNs; the top DQN's actions (subband selections) are influenced by the output of the power DQN, which directly controls power allocation. Consequently, any noise or instability in the power DQN's learning process propagates to the top DQN, leading to increased variability and a slower convergence rate.

<figure><img src="image_3.png" /><figcaption>Figure 2: Training loss of the subband and power DQNs.</figcaption></figure>

B. Sum Rate Performance

Fig. 3 illustrates the evolution of the achieved sum rate over time. As the training progresses, the sum rate stabilizes, demonstrating the convergence of the proposed DRL-based resource allocation framework. The steady increase and eventual stabilization of the sum rate indicate that the algorithm has successfully learned the optimal subband and power allocation policy, achieving near-optimal performance. This convergence in the reward function underscores the effectiveness of the proposed method in adapting to the dynamic communication environment while fulfilling the rate requirements of both eMBB and URLLC users. To further validate the performance of the proposed algorithm, we compare it against the non-orthogonal slicing technique introduced in [6]. Fig. 4 shows the achieved sum rate for both URLLC and eMBB users. The proposed method consistently outperforms the benchmark, providing a higher overall sum rate. Notably, the proposed method
<figure><img src="image_4.png" /><figcaption>Figure 3: Achieved sum rate over time for all users.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: Comparison of the achieved sum rate for URLLC and eMBB users between the proposed method and [6].</figcaption></figure>

achieves superior data rates for URLLC users, demonstrating its ability to meet stringent latency and reliability requirements without sacrificing the performance of eMBB users.

## V. CONCLUSION

In this paper, we proposed a novel DRL-based framework for addressing the challenging coexistence of eMBB and URLLC services in NOMA-based networks. By employing a two-layer DQN structure, the framework dynamically optimizes both subband and power allocations, enabling a balance between the stringent latency and reliability requirements of URLLC users and the throughput demands of eMBB users. The proposed solution demonstrated superior performance compared to conventional methods, particularly in its ability to efficiently manage interference and flexibly adapt to the varying requirements of different services. This is achieved without introducing excessive complexity, making it highly scalable for large network environments. The convergence analysis confirmed the stability of the proposed DRL approach, underscoring its potential for practical deployment in 5G and beyond 5G networks.

## ACKNOWLEDGMENT

This work was supported by the ANR under the France 2030 program, grant "NF-PERSEUS: ANR-22-PEFT-0004" and the

Important Projects of Common European Interest on Micro-Electronics and Communications Technologies led by Orange "IPCEI ME/CT Orange".

## REFERENCES

[1] M. Shafi, A. F. Molisch, P. J. Smith, T. Haustein, P. Zhu, P. De Silva, F. Tufvesson, A. Benjebour, and G. Wunder, "5G: A Tutorial Overview of Standards, Trials, Challenges, Deployment, and Practice," *IEEE J. Sel. Areas Commun.*, vol. 35, no. 6, pp. 1201-1221, Jun. 2017.

[2] 3GPP R1-1612306, "On the hardware implementation of channel decoders for short block lengths," 3GPP TSG RAN WG1 Meeting #87, Nov. 2016.

[3] 3GPP R1-1700374, "Downlink multiplexing of eMBB and URLLC transmission," 3GPP TSG RAN WG1 NR Ad-Hoc Meeting, Jan. 2017.

[4] J. Tang, B. Shim, and T. Q. S. Quek, "Service Multiplexing and Revenue Maximization in Sliced C-RAN Incorporated With URLLC and Multicast eMBB," *IEEE J. Sel. Areas Commun.*, vol. 37, no. 4, pp. 881-895, Feb. 2019.

[5] P. Popovski, K. F. Trillingsgaard, O. Simeone, and G. Durisi, "5G Wireless Network Slicing for eMBB, URLLC, and mMTC: A Communication-Theoretic View," *IEEE Access*, vol. 6, pp. 55 765-55 779, 2018.

[6] E. N. Tominaga, H. Alves, R. D. Souza, J. Luiz Rebelatto, and M. Latvaaho, "Non-Orthogonal Multiple Access and Network Slicing: Scalable Coexistence of eMBB and URLLC," in *2021 IEEE 93rd Veh. Technol. Conf.*, Helsinki, Finland, Apr. 2021, pp. 1-6.

[7] J. Li and X. Zhang, "Deep Reinforcement Learning-Based Joint Scheduling of eMBB and URLLC in 5G Networks," *IEEE Wireless Commun. Lett.*, vol. 9, no. 9, pp. 1543-1546, May 2020.

[8] B. Di, L. Song, and Y. Li, "Sub-Channel Assignment, Power Allocation, and User Scheduling for Non-Orthogonal Multiple Access Networks," *IEEE Trans. Wireless Commun.*, vol. 15, no. 11, pp. 7686-7698, Nov. 2016.

[9] J. Doumit, M.-J. Youssef, C. Abdel Nour, J. Farah, and C. Douillard, "AoI Minimization in Mixed Traffic Full-Duplex Uncoordinated Communication Systems With NOMA," *IEEE Internet Things J.*, vol. 11, no. 2, pp. 2830-2841, Jul. 2024.

[10] R. Kassab, O. Simeone, and P. Popovski, "Coexistence of URLLC and eMBB Services in the C-RAN Uplink: An Information-Theoretic Study," in *2018 IEEE Glob. Commun. Conf. (GLOBECOM)*, Abu Dhabi, United Arab Emirates, Dec. 2018, pp. 1-6.

[11] A. Matera, R. Kassab, O. Simeone, and U. Spagnolini, "Non-Orthogonal eMBB-URLLC Radio Access for Cloud Radio Access Networks with Analog Fronthauling," *Entropy*, vol. 20, no. 9, 2018.

[12] E. J. dos Santos, R. D. Souza, J. L. Rebelatto, and H. Alves, "Network Slicing for URLLC and eMBB With Max-Matching Diversity Channel Allocation," *IEEE Commun. Lett.*, vol. 24, no. 3, pp. 658-661, Mar. 2020.

[13] J. Mei, K. Zheng, L. Zhao, Y. Teng, and X. Wang, "A Latency and Reliability Guaranteed Resource Allocation Scheme for LTE V2V Communication Systems," *IEEE Trans. Wireless Commun.*, vol. 17, no. 6, pp. 3850-3860, Jun. 2018.

[14] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, "Human-level control through deep reinforcement learning," *Nature*, vol. 518, pp. 529-533, 2015.

[15] J. Zhang, X. Tao, H. Wu, N. Zhang, and X. Zhang, "Deep Reinforcement Learning for Throughput Improvement of the Uplink Grant-Free NOMA System," *IEEE Internet Things J.*, vol. 7, no. 7, pp. 6369-6379, Jul. 2020.

[16] H. Yang, Z. Xiong, J. Zhao, D. Niyato, C. Yuen, and R. Deng, "Deep Reinforcement Learning Based Massive Access Management for UltraReliable Low-Latency Communications," *IEEE Trans. Wireless Commun.*, vol. 20, no. 5, pp. 2977-2990, May 2021.

[17] Y. S. Nasir and D. Guo, "Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks," *IEEE J. Sel. Areas Commun.*, vol. 37, no. 10, pp. 2239-2250, Oct. 2019.

[18] O. Naparstek and K. Cohen, "Deep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access," *IEEE Trans. Wireless Commun.*, vol. 18, no. 1, pp. 310-323, Jan. 2019.

[19] T. Tieleman, "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude," p. 26, 2012.