# Slotted Reinforcement Learning-based radio resource allocation in sliced 5G networks

Gauthier Meffe¹, Philippe Owezarski¹, and Pascal Berthou²

¹ LAAS-CNRS, Université de Toulouse, CNRS, LAAS, Toulouse, France
gauthier.meffe2@gmail.com, owe@laas.fr

² LAAS-CNRS, Université de Toulouse, CNRS, UPS, Toulouse, France
berthou@laas.fr

**Abstract.** 5G networks are designed for providing several different services as high speed Internet, low latency and M2M (Machine to Machine) communications. For enforcing such guaranteed services, slicing techniques are of essential importance to ensure isolation between resources allocated to each of these services, especially at the level of RAN (Radio Access Networks) and its time/frequency matrix. Given the scarcity of radio resources, this paper aims at proposing efficient radio resource allocation algorithms and mechanisms for 5G networks, avoiding resource wastes and enforcing slices isolation. The proposed solution highlights a new way of using reinforcement learning, and more specifically the Double DQN (Deep Q-Network) algorithm, based on a slotted approach for 5G resource allocations. The slotted use of Double DQN evaluation exhibits its benefits in terms of allocation performance and low latency.

**Keywords:** 5G, Radio Resource Allocation, Slotted Reinforcement Learning, DDQN

## 1 Introduction

5G networks are designed for providing several different services as high speed Internet, low latency and M2M (Machine to Machine) communications. For enforcing such guaranteed services, slicing techniques are of essential importance to ensure isolation between resources allocated to each of these services, especially at the level of RAN (Radio Access Networks) and its time/frequency matrix. Given the scarcity of radio resources, it is important to avoid any resource waste, as well as interferences.

This paper then aims at designing efficient radio resource allocation algorithms and mechanisms for 5G networks. This problem has been already significantly addressed, especially leveraging optimization techniques from the operational research domain, as integer or constraints programming. Indeed, allocating user requests in the time/frequency matrix is a typical optimization problem known as the knapsack problem [1]. Such optimization techniques leverage solvers that implement integer or constraints programming techniques. However, it takes hours, sometimes days for these solvers to provide the optimal results,
and often they even do not converge. Such latency is of course incompatible with
the mobile network management constraints. To avoid this latency issue, heuris-
tics are proposed, but they then lose the optimality of the provided allocation
scheme.

This placement of blocks in a matrix problem has recently been addressed using machine learning approaches, and especially reinforcement learning (RL) [2] [3] [4]. RL is a branch of machine learning that allows an agent to learn on the optimal decision to make based on sequential trials and errors made, receiving feedbacks (on a rewards form) on the performed actions in a given environment. One significant difference between the approaches leveraging optimization techniques from the operational research domain as MILP (Mixed Integer Linear Programming) or CP (Constraints Programming) and the ones based on reinforcement technique lies in the global allocation principle for the first, and the sequential allocation principle for the second. Indeed, MILP or CP know all resources to be allocated to user requests at the beginning and allocate them all in one time. This allocation result has been proved to be optimal. At the opposite, RL-based approaches allocate resources to users one by one, without considering what the next resource block is. RL-based resource allocation is then obviously suboptimal compared to MILP or CP-based allocation results. Existing heuristics addressing efficient resource allocation also leverage sequential resource blocks allocation principle. Their placement efficiency is then unfortunately suboptimal, but they provide results in a very short time compatible with low latencies requested by some users. Among the heuristics (leveraging sequential resource blocks allocation), the skyline algorithm has been proved to be optimal [1]. This paper then deals with improving the RL-based approach by slotting resource allocations principle. The approach then proposes to allocate several resource blocks corresponding to several user requests at the same time, instead of running several sequential allocation processes. Computing the allocation of several user requests in one time permits a better placement of resource blocks, what is impossible with a sequential approach that is blind of the requirements of the following resource blocks to be allocated. The objective is then to perform better than the best heuristic in terms of resource allocation results, while still enforcing the low latency requirements.

The rest of the paper is structured as follows: Section 2 details the time/frequency
resource allocation problem in the new framework of 5G networks. Section 3
presents the state of researches on the generic knapsack problem. Section 4 is
devoted to introducing the selected RL technique with regard to the alloca-
tion objectives and constraints. After having selecting the Double DQN (Deep
Q-Network) as a promising RL technique for 5G ressource allocation, section
5 describes the related Double DQN-based allocation agent design and imple-
mentation, presents its evaluation methodology, and analyzes and discusses the
obtained results. Finally, section 6 summarizes the contributions of the paper.
# 2 5G radio resources and problem description

This section covers the important 5G terminologies required for the exhibition of the RAN slicing problem. Mainly, the 5G radio resources structure is presented. Then, the RAN slicing enforcement problem is explained as well as the required allocations strategies.

## 2.1 5G background

In 5G system, the physical layer is more flexible with respect to the previous generations. Recall that radio resources in 4G are uniformly distributed over a time-frequency grid, i.e. the later is decomposed into resource blocks (RB) of 1 ms over 12 sub-carriers spaced by 15 kHz.

In order to fulfill the variety of services requirements, increase the network reliability and adapt to frequency range, 5G introduces different radio frames numerologies for sub-6 GHz, and above-6 GHz bands. In this article, we focus on the sub-6 GHz bands. The same developed approaches can be easily applied for bands above-6 GHz. Table 1 exhibits the different numerologies for sub-6 GHz bands. Each given numerology $\mu^3$ defines the time-frequency resource size in one Transmission Time interval (TTI), TTI=1ms. That is, a numerology $\mu$ refers to the sub-carrier spacing (SCS) in frequency domain and the slot duration in time domain. For instance, as depicted in fig. 1, for $\mu = 1$ the radio resource size is fixed to 0.5 ms over 12 sub-carriers spaced by 30 KHz. In general, the SCS scales by $2^\mu * 15kHz$ and the slot duration decreases with higher numerology ($\mu$). Such flexibility is essentially introduced as to achieve the diverse services requirements. For example, it is preferable to transmit latency sensitive services in shorter time interval with larger sub-carrier spacing, e.g. $\mu = 3$.

<table><caption>Table 1: 5G Radio frames numerologies for sub-6 GHz bands</caption><thead><tr><td>μ</td><td>SCS (kHz)</td><td>Slot duration (ms)</td></tr></thead><tbody><tr><td>0</td><td>15</td><td>1</td></tr><tr><td>1</td><td>30</td><td>0.5</td></tr><tr><td>2</td><td>60</td><td>0.25</td></tr></tbody></table>

In order to support the coexistence of the multiple numerologies on same carrier, the resources are structured in the so-called tiles [5]. The tile is the smallest subset of frequency and time resources allocated to a particular slice/service with same numerology $\mu$. Hence, for sub-6 GHz three tiles structures are tailored as shown in fig. 1. For instance, the tile structure for $\mu = 0$ is 1 ms over 12 subcarriers spaced by 15 KHz. Further, multiplexing over time and frequency is

<footnote n="3">³ 3GPP, TR 38.802, TR 38.804: Study on new radio access technology Physical layer aspects, Study on new radio access technology Radio interface protocol aspects. https://www.3gpp.org/</footnote>
required for the transmission of the different numerologies, e.g. over time, 3GPP imposes symbol alignment between tiles to insure orthogonality.

<figure><img src="image_1.png" /><figcaption>Fig. 1: Sub-6 GHz numerologies</figcaption></figure>

## 2.2 Problem formulation

The RAN slicing enforcement algorithms should then allocate resources in a way that maximizes the adjacency of the unallocated portion of resources, instead of having sparse unallocated resources in the time-frequency grid. This objective is illustrated in Figure 2. Two time-frequency resource grids are schematized as to explain the difference between an optimal (b) and sub-optimal (a) resource allocation for 4 slice requests in 5G context. Each of the 4 slices demands a different amount of resources what induces specific sizes and shapes of requested user blocks to be allocated in the matrix (corresponding to the different 5G numerologies). Even-though both allocations (a) and (b) satisfy the four slice requests during the allocation window $T$, it is clear that the allocation strategy in (a) is sub-optimal compared to (b). In fact, the resource allocation strategies as in (a) might lead to inefficient resource utilization. 5G proposes a variety of user blocks structures, and some allocation strategies induce a small sparse unallocated resources over the resource grid, as represented in Fig. 2 (a). Those small leaved resources are considered as wasted as they do not fit any possible user demand. Only the small continuous unallocated portion of resources can then be reused in an efficient manner, as different user requested block structures could fit in this portion. Thus, it enables the scalability requirement and increases the resource utilization efficiency.
<figure><img src="image_2.png" /><figcaption>(a) Sub-optimal allocation (small continuous unallocated area)</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>(b) Optimal allocation (large continuous unallocated area)</figcaption></figure>

<caption>Fig. 2: Optimal and sub-optimal resource allocation</caption>

## 3 Related work

Recently, researches have converged to enforce the RAN slicing from a resource allocation aspect. To overcome the static resources segmentation limitations, shared allocation strategies are proposed [6–8]. Most of the RAN slicing enforcement approaches leverage optimization techniques from the operational research domain, providing optimal resource allocation, but at the expense of unacceptable latency. For instance, B.Han et al. [8], Yang et al. [9] propose the use of Genetic algorithm to optimize resource management between heterogeneous slices with maximized long-term network utility. By introducing resources virtualization, Chang et al. [10] formulate the problem as a knapsack problem [11]. An algorithm is proposed to maximize the number of accepted slices with an efficient 5G resource partitioning. Papa et al. [12] solve the problem using a Lyapunov optimisation approach. Ojaghi et al. [13] propose a RAN slicing mechanism that uses the Mixed Integer Programming (MIP), and to cope with computation time, they propose a faster heuristic (SlicedRAN), unfortunately then losing the optimality aspect. Hossain and Ansari [14] deal with the slicing issue using Mixed Integer Non Linear programming (MINLP). All these previous works based on optimization techniques cannot be used in operational networks because of their huge latency. Indeed, solvers for computing all these optimization models takes hours or days to reach the optimal solution. And sometimes, they even do not converge, and then, never reach the optimal solution. The related heuristics aiming at solving the latency issue then do not provide the expected optimal solution in terms of resources placement.

Given these optimization model limits, some alternative works propose to use machine learning techniques and more specifically Reinforcement Learning (RL). Then, Zambianco and Verticale [15] also aim at optimizing the use of resources by limiting inter-slices interferences. For that, the issue is solved using Deep Reinforcement Neural networks (DRN). The solution, even if it does not reach the optimal solution, appears to be efficient comparatively to existing heuristics,
and provides low latency compared to optimization models. Khodapanah et al. [16], Boutiba et al. [4], Tian et al. [3], Abiko et al. [17], Li et al. [18], Kumar et al. [19], Sun et al. [20], Deng et al. [21], Sciancalepore et al. [22], Amonarriz-Pagola et al. [23] adopt a similar approach based on reinforcement learning. However, they all use a sequential allocation principle with the assumed corresponding allocation efficiency issues. The rest of the paper aims at exhibiting that a slotted version of DRL is meant to improve DRL efficiency.

# 4 Reinforcement algorithm

As indicated previously, a new trend for 5G resource allocation starts to investigate machine learning approaches in order to gain flexibility. RL is a machine learning method that allows an agent to learn how to make decisions by interacting with an environment. The agent aims at maximizing a cumulative reward by learning what action to trigger in various cases. It leverages on Action-Value Function (Q-value). The Q-value is an estimate of the future cumulative reward that the agent can expect by triggering a specific action in a given state (Fig. 3).

<figure><img src="image_4.png" /><figcaption>Fig. 3: Action-Value function</figcaption></figure>

Given the targeted 5G radio resource allocation problem, Q-Learning is of interest. Q-Learning is a RL algorithm that learns the value of actions independently from the policy followed by the agent. The agent updates its Q-value estimates according to rewards received and its future estimations. Deep Q-Network (DQN) [24] is an extension of Q-learning that uses deep neural networks to approximate the Q function. It aims at combining the perceptual capabilities of deep learning with the decision-making abilities of reinforcement learning. This approach has demonstrated excellent performances on complex tasks especially related to game playing that often have to solve similar problems as the one related to 5G resource allocation [11] [2]. The deep neural network in a DQN takes the states of the environment as an input and produces Q-values for all actions as outputs. The agent is free afterwards to select the action that maximizes the Q-value as well as the reward. The architecture is depicted on Fig. 4. Deep Q-Network (DQN) is a powerful method that combines reinforcement learning and deep neural networks for solving complex problems. Using mechanisms as experiments replay and neural networks, DQN successfully stabilizes and improves learning results, allowing the agent to learn efficient policies for various and dynamic environments.
Double Deep Q-Network (Double DQN or DDQN) [25] is a variant that improves the standard DQN algorithm aiming at correcting an overestimate of action values *Q* that can arise with reinforcement learning. Recently, an improved version of DDQN has been designed: the Dueling Double Deep Q Network (D3QN) [3]. It then can consider resource allocation at both the base station and the network slices. The contribution of this paper being to propose the slotted approach for DRL, and demonstrate its benefits for 5G resource allocation, considering all details of the 5G technology, especially the very complex ones of the physical layer, does not bring significantly useful information. In addition, to simplify the demonstration of the slotted-DRL approach gain, it is sufficient to consider the Double DQN algorithm working only with the sliced network level information.

<figure><img src="image_5.png" /><figcaption>Fig. 4: Deep Neural Network Architecture for DQN</figcaption></figure>

# 5 Evaluation

This evaluation section aims at demonstrating the benefits of the slotted approach for DRL-based resource allocation. The Double DQN algorithm has been selected as the DRL algorithm, as it appears as a promising solution for providing efficient/optimal allocation results, in real-time. This section evaluates the efficiency of the slotted Double DQN algorithm for allocating efficiently the list of resource blocks requested by users in the full time/frequency matrix of the 5G network. These results will also be compared with the ones obtained with the skyline algorithm, i.e. the optimal algorithm leveraging sequential resource block allocations.

## 5.1 Evaluation environment and methodology

This evaluation is based on a simulation of the time/frequency matrix where each cell corresponds to a resource that can be allocated to a user. Each user has specific requirements in terms of bandwidth and latency. User requests are then represented as bricks of different sizes that Double DQN has to optimally fit in
the full time/frequency matrix, i.e. avoiding having small groups of unallocated
cells in the matrix that are then very difficult to allocate for future requests.

Many key functions have been developed for handling this environment:

- **Environment initialization:** The initial time/frequency matrix is generated with null values for simulating fully unallocated resources.

- **user blocks placement:** Each user request is represented by a resource block to be allocated. This allocation is made by a placement algorithm based on Slotted Double DQN.

- **Rewards computing:** Rewards are computed based on the number of user requests placed in the matrix. If some blocks are not allocated the reward is decreased.

- Etc.

For all the computings, a platform consisting of a computers cluster has
been used<fnref n="4" />. It consists of several servers, including Nvidia GPUs, specifically
designed for intensive computings. These GPU are compatible with CUDA, a
parallel computing architecture that increases the performances of deep learning
algorithms. Using this computing platform is made possible thanks to program-
ming environments as TensorFlow and PyTorch, that are optimized for taking
advantage of GPU capabilities. TensorFlow has been specifically integrated with
CUDA, what makes the computing of deep learning algorithms more efficient
and faster.

## 5.2 Resource allocation efficiency

### Evaluation scenario and placement performance metrics

In order not to complexify the visual representation of results, this pa-
per presents an evaluation on a small scenario. On this running example, the
time/frequency matrix is limited to a size of 4x4 ressource blocks (RB). The list
of user requests is randomly generated. Each user request is represented by a
resource block of various sizes going from 1x1 to 4x4. Fig. 5 illustrates this by
showing 4 out of 16 possible block structures. The yellow block corresponds to
the smallest possible resource block in 5G at the time of this study, i.e. 15 kHz
x 0,25 ms. The green block corresponds to a user request having soft latency
constraints. The blue block correspond to the request of a user needing more
throughput with short latency. Finally, the red block represents the request of a
user needing high throughput without strong latency requests. Previous works
as the ones in [4] or [3] used a more complex model of the physical resources
of RAN. In this paper we argue that a simple model just considering time, fre-
quency and latency is sufficient for exhibiting the benefits of the slotted version
of Double DQN compared to other DRL or heuristic algorithms.

<footnote n="4">⁴ https://pfcalcul.laas.fr/</footnote>
<figure><img src="image_6.png" /><figcaption>Fig. 5: Examples of resource blocks corresponding to possible user requests</figcaption></figure>

The placement goal deals with maximizing the number of user requests placed in the matrix, minimizing unused spaces between allocated resource blocks, and avoiding any overlapping. The number of possible configurations in this case is $2^{4×4} × 16$. Indeed, each cell in the matrix can be allocated or empty, what gives 2¹⁶ possible configurations. Adding the 16 possible sizes for the blocks, the total is then 2¹⁶ × 16 possible configurations.

Fig. 6 illustrates one optimal placement of user requests on the time/frequency matrix, with all user requests on the left, and no unused RB between these blocks. To consider the case where the user requests do not completely fill the time/frequency matrix, the placement metrics used as reward for the Slotted Double DQN algorithm takes into account the absence of unused blocks between allocated ones, and the fact that the blocks are placed as left as possible.

<figure><img src="image_7.png" /><figcaption>Fig. 6: Example of optimal placement of user requests: blocks are allocated on the left of the 4x4 matrix without unused blocks in between</figcaption></figure>
At the beginning, the placement agent running the Slotted Double DQN algo-
rithm knows a list of N user blocks to be allocated. The state of the environment
includes the size of all user blocks and the placement indication: each user re-
quest is represented by $(x_i, y_i, \epsilon_i)$, where $\epsilon_i$ equals 1 if the block has already been
placed, 0 otherwise (cf. Fig. 7). This allows the agent to make better informed
decision, and to better anticipate future placements. Indeed, on the example of
Figure 7, the list provided to the agent has a size of N=6. It has then been
specifically built to provide the agent with all informations required for opti-
mally placing user blocks. The agent places user blocks one by one, but selecting
the best user block to place at each iteration with regard to the information
provided by the list of the N=6 considered user requests at this time.

$$
\begin{pmatrix} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix} + (x_1, y_1) + \cdots + (x_6, y_6) \longrightarrow \\ (0, \dots, 0, x_1, y_1, 0, \dots, x_6, y_6, 0)
$$

<caption>Fig. 7: Input vector (size 6) with a 4x4 matrix and several blocks.</caption>

With this state representation, the agent has to select the user block to be
placed at that time, and the place to be allocated for it. Table 2 schematizes
the list of possible actions, providing a few samples of them for the example
provided on Fig. 8.

<figure><img src="image_8.png" /><figcaption>Fig. 8: Example : Given 6 user blocks to be placed in an empty matrix, the agent selects action 53 (bloc 4 is first selected to be placed at the place number 6 - place in the matrix are numbered from 0 to 15 from left to right and top to bottom)</figcaption></figure>
<table><caption>Table 2: Examples of user blocks placement actions</caption><thead><tr><td>Action</td><td>Selected Block</td><td>Place of the block</td></tr></thead><tbody><tr><td>0</td><td>1</td><td>0</td></tr><tr><td>1</td><td>1</td><td>1</td></tr><tr><td>2</td><td>1</td><td>2</td></tr><tr><td>3</td><td>1</td><td>3</td></tr><tr><td>4</td><td>1</td><td>4</td></tr><tr><td>5</td><td>1</td><td>5</td></tr><tr><td>...</td><td>...</td><td>...</td></tr><tr><td>16</td><td>2</td><td>0</td></tr><tr><td>17</td><td>2</td><td>1</td></tr><tr><td>...</td><td>...</td><td>...</td></tr><tr><td>32</td><td>3</td><td>0</td></tr><tr><td>33</td><td>3</td><td>1</td></tr><tr><td>...</td><td>...</td><td>...</td></tr><tr><td>53</td><td>4</td><td>6</td></tr><tr><td>...</td><td>...</td><td>...</td></tr><tr><td>95</td><td>6</td><td>15</td></tr></tbody></table>

## 5.3 Training with several sets of users

How to train any machine learning algorithm still remains a significant piece of research work. Practically, it is commonly agreed that it is challenging to produce traces (gathering of real traces or generation of synthetic ones) of significant sizes, and representing exhaustively the many different situations that can be encountered in the real world. Reinforcement Learning has been partly selected in this work for not being concerned by such issue. Indeed, RL can leverage on very small training datasets, the model of RL algorithms being consolidated at the running stage thanks to the feedbacks obtained by the agent by the way of the rewards that take into account both the agent actions and their impact on the environment.

We then limit our training datasets to a small number of user requests lists. These lists are very basic, especially in the reduced case selected for illustrating the performances of the Slotted Double DQN algorithm for 5G resource allocation. The agent has been trained on 10 000 epochs, using a three layers neural network, sufficient for capturing the few features of our environment model (cf. Fig. 7). The neural network layers consists of respectively 512/512/180 neurons at the beginning. Having small training datasets and small neural networks will help to better analyze and understand the placement performance results, as well as explaining them.

The following method has been followed:

1. Creation of a training set consisting of a fixed number of user requests lists (N=10 for examples).
2. Training of our Slotted Double DQN model on this dataset.

3. Agent performance evaluation: A 100% success rate means that the agent successfully placed all user requests from all the lists.

4. If the success rate is not 100%, we increase the size of the neural network (adding neurons in the three layers) and repeat the process as long as it reaches an optimum.

Results of the different trials are summarized in Table 3.

<table><caption>Table 3: Success rate of the placement agent for several configurations of the Slotted Double DQN neural network and user requests lists. As a comparison, Slotted DDQN results are compared with the ones of standard DDQN and the Skyline algorithm</caption><thead><tr><th>Neural network configuration</th><th>Number of user requests lists</th><th>Slotted DDQN success rate</th><th>DDQN success rate</th><th>Skyline success rate</th></tr></thead><tbody><tr><td>512/1024/512</td><td>10</td><td>90%</td><td>77%</td><td rowspan="3">84%</td></tr><tr><td>1024/2048/1024</td><td>10</td><td>100%</td><td>79%</td></tr><tr><td>2048/4096/2048</td><td>10</td><td>100%</td><td>82%</td></tr><tr><td>512/1024/512</td><td>20</td><td>80%</td><td>62%</td><td rowspan="3">71%</td></tr><tr><td>1024/2048/1024</td><td>20</td><td>85%</td><td>65%</td></tr><tr><td>2048/4096/2048</td><td>20</td><td>80%</td><td>68%</td></tr><tr><td>512/1024/512</td><td>50</td><td>72%</td><td>48%</td><td rowspan="3">59%</td></tr><tr><td>1024/2048/1024</td><td>50</td><td>82%</td><td>53%</td></tr><tr><td>2048/4096/2048</td><td>50</td><td>84%</td><td>56%</td></tr></tbody></table>

These results exhibit that our slotted double DQN based placement agent is able to learn optimal placement strategies in some cases. For that, the neural network has to be complex enough to capture all possible configurations. It is however unfortunate that, even with the most complex neural networks that we tested, a success rate of 100% is not reached even when training our agent on our largest datasets.

Such results could appear as disappointing. We nevertheless compared the performance of our agent with the one of the skyline algorithm [1] that has been proved to be optimal for solving the Knapsack problem. Our Slotted Double DQN-based agent overpasses the skyline algorithm always reaching greater or equal success rates. This proves the limits of sequential allocation algorithms as they introduce many lost spaces between allocated blocks, at the opposite of the slotted approach that can select the best block among N that better fits the unallocated space. As expected, the standard version of DDQN that leverages a sequential approach for resource blocks allocation (not considering what the next blocks to allocate is) exhibits allocation efficiency lower than the ones of the Skyline algorithm (the optimal one leveraging the sequential allocation principle).
## 5.4 Benchmarking

In this section, the performances of our Slotted Double DQN agent are evaluated on the GPUs of our computing platform, especially using the capabilities of CUDA and TensorFlow.

For evaluating the performances, several training sessions have been run with the neural networks considered in this paper using different CPU and GPU configurations with various batch sizes.

The tested configurations are the following:

1. **CPU only**: Training using a single CPU: Intel 5218R 2.1G.

2. **Single GPU**: Training using a single GPU: RTX A6000.

<table><caption>Table 4: Training time in seconds for different batch sizes on CPU and GPU</caption><thead><tr><td>Batch size</td><td>CPU</td><td>GPU</td></tr></thead><tbody><tr><td>8</td><td>324</td><td>84</td></tr><tr><td>16</td><td>203</td><td>27</td></tr><tr><td>32</td><td>202</td><td>13</td></tr><tr><td>64</td><td>142</td><td>10</td></tr><tr><td>128</td><td>112</td><td>4</td></tr><tr><td>256</td><td>105</td><td>2</td></tr><tr><td>512</td><td>104</td><td>2</td></tr></tbody></table>

Results are shown in Table 4. They exhibit that working with GPU significantly reduces the training times, especially for large batch sizes. By optimizing our code specifically for GPU, we reduce the training time by a factor of 50 compared to the use of a CPU.

The Slotted Double DQN allocation computing times have been measured at run time. They appear to be very short whatever the tested value of N. Of course, N is necessarily small; the slot duration has to be short to respect latency requirements of users, then limiting the size of N. Obviously, the larger N, the better the allocation efficiency in terms of placement. Given the large number and large scope of experiments we run, the largest N value ever reached with realistic hypothesis was 50. In this worst case for allocation computing times, this computing time never exceeds 50 µs. This time is very short, and fully compatible with the latency requirements of possible user requests in 5G networks. It then validates the Slotted Double DQN as efficient enough in terms of computing time for addressing the task of resource allocation in 5G RAN.

## 6 Conclusion

In this paper, we addressed the problem of radio resource allocation in 5G networks. Based on the state of the art, it appears that mainly two different approaches are traditionally used:
- Operational research based techniques as MILP or CP that provide optimal placement solutions, but at the expense of very long computing times. As a result, it is generally proposed, together with these exact methods, some heuristics that provide approaching allocation results, but with significantly reduced computation times.

- Reinforcement Learning based algorithms that are, as heuristics, providing good allocation results, together with short computing times.

Analyzing this state of the art, it appears that operational research techniques know all resource requests of all users, and that they compute the optimal placement of resource blocks in one time, considering the requirement of all resource blocks to be allocated. At the opposite, heuristics or RL-based algorithms sequentially allocate resource blocks corresponding to a single user request, without taking into account what the next blocks to be allocated are.

Based on this statement, we propose to group N user requests in one slot. The slot size depends on the user requirements, especially in terms of latency. The slotted version of Double DQN then appears as a solution for taking advantage of the operational research approaches. We developed our Slotted Double DQN-based agent and evaluated its performances. Our agent appeared to provide an interesting solution, especially as it overpasses the optimal skyline placement algorithm, with limited and controlled latencies.

## Acknowledgment

We gratefully acknowledge the support of the PEPR 5G program of ANR in the framework of the France 2030 investment plan, specially of NF-MUST project under grant ANR-22-PEFT-0002.

## References

1. J. Jylanki, "A thousand ways to pack the bin - a practical approach to two-dimensional rectangle bin packing," 2010, available at https://fr.scribd.com/document/457472796/1000-ways-to-pack-the-bin-pdf.

2. M. Stevens and S. Pradhan, "Playing tetris with deep reinforcement learning," in *Stanford University Report*, 2024, available at https://cs231n.stanford.edu/reports/2016/pdfs/121_Report.pdf.

3. K. Tian, Y. Wang, D. Pan, and D. Yuan, "Drl-based dynamic resource configuration and optimization for b5g network slicing," *IEEE Access*, vol. 12, pp. 120864–120876, 2024.

4. K. Boutiba, M. Bagaa, and A. Ksentini, "Optimal radio resource management in 5g nr featuring network slicing," *Computer Networks*, vol. 234, 2023.

5. S. E. Elayoubi, S. B. Jemaa, Z. Altman, and A. Galindo-Serrano, "5g ran slicing for verticals: Enablers and challenges," *IEEE Communications Magazine*, vol. 57, no. 1, p. 28–34, Jan 2019.

6. M. Yan, G. Feng, J. Zhou, Y. Sun, and Y.-C. Liang, "Intelligent resource scheduling for 5g radio access network slicing," *IEEE Transactions on Vehicular Technology*, vol. 68, no. 8, p. 7691–7703, Aug 2019.
7. S. Mandelli, M. Andrews, S. Borst, and S. Klein, "Satisfying network slicing constraints via 5g mac scheduling," in *IEEE INFOCOM 2019 - IEEE Conference on Computer Communications*, Apr 2019, p. 2332-2340, paris, France. [Online]. Available: https://ieeexplore.ieee.org/document/8737604/

8. L. J. B. Han and H. D. Schotten, "Slice as an evolutionary service: Genetic optimization for inter-slice resource management in 5g networks, ieee access, vol. 6, no. 1," p, vol. 33, p. 137, 2018.

9. X. Yang, Y. Wang, I. C. Wong, Y. Liu, and L. Cuthbert, "Genetic algorithm in resource allocation of ran slicing with qos isolation and fairness," in *IEEE Latin-American Conference on Communications (LATINCOM)*, 2020.

10. C.-Y. Chang, N. Nikaein, and T. Spyropoulos, "Radio access network resource slicing for flexible service execution," in " in *IEEE INFOCOM 2018- IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)*. IEEE pp. 668-673, 2018.

11. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglu, D. Wierstra, and M. Riedmiller, "Playing atari with deep reinforcement learning," in *NIPS Deep learning Workshop*, 2013.

12. A. Papa, M. Klugel, L. Goratti, T. Rasheed, and W. Kellerer, "Optimizing dynamic ran slicing in programmable 5g networks," in *IEEE International Conference on Communications (ICC)*, 2019, pp. 1-7, shangai, China.

13. B. Ojaghi, F. Adelantado, A. Antonopoulos, and C. Verikoukis, "Slicedran: Service-aware network slicing framework for 5g radio access networks," in *IEEE Systems Journal*, 2021.

14. A. Hossain and N. Ansari, "5g multi-band numerology-based tdd ran slicing for throughput and latency sensitive services," in *IEEE Transactions on Mobile Computing*, 2021.

15. M. Zambianco and G. Verticale, "Spectrum allocation for network slices with inter-numerology interference using deep reinforcement learning," in *IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications*, 2020, pp. 1-7.

16. B. Khodapanah, A. Awada, I. Viering, A. N. Barreto, M. Simsek, and G. Fettweis, "Slice management in radio access network via deep reinforcement learning," *IEEE 91st Vehicular Technology Conference (VTC2020-Spring)*, pp. 1-6, 2020.

17. Y. Abiko, T. Saito, D. Ikeda, K. Ohta, T. Mizuno, and H. Mineno, "Flexible resource block allocation to multiple slices for radio access network slicing using deep reinforcement learning," *IEEE Access*, vol. 8, pp. 68 183-68 198, 2020.

18. R. Li, Z. Zhao, S. Q., I. Chih-lin, X. Yang, X. Chen, M. Zhao, and H. Zhang, "Deep reinforcement learning for resource management in network slicing," *IEEE Access*, vol. 6, pp. 74 429-74 441, 2018.

19. P. M. Kumar, B. Basheer, B. S. Rawal, F. Afghah, G. C. Babu, and M. Arunmozhi, "Traffic scheduling network slicing and virtualization based on deep reinforcement learning," *Cumputing Electrical Engineering*, vol. 100, 2022.

20. G. Sun, K. Xiong, G. O. Boateng, G. Liu, and W. Jiang, "Resource slicing and customization in ran with dueling deep q-network," *Journal of network computer applications*, vol. 157, 2020.

21. Z. Deng, Q. Du, N. Li, and Y. Zhang, "Rl-based radio resource slicing strategy for software-defined satellite networks," *IEEE 19th International Cconference on Communication Technology (ICCT)*, pp. 897-901, 2019.

22. V. Sciancalepore, X. Costa-Perez, and A. Banchs, "Rl-nsb: Reinforcement learning-based 5g network slice broker," *IEEE/ACM transaction on Networking*, vol. 27, no. 4, pp. 1543-1557, 2019.
23. I. Amonarriz-Pagola and J. A. Fernandez-Carrasco, "A reinforcement learning approach for network slicing in 5g networks," *IProceedings of Cybersecurity Conference (JNIC)*, pp. 1-7, 2023.

24. Z. Xiong, Y. Zhang, D. Niyato, R. Deng, P. Wang, and L.-C. WSang, "Deep reinforcement learning for mobile 5g and beyond: Fundamentals, applications, and challenges," *IEEE Vehicular Technology Magazine*, vol. 14, no. 2, pp. 44-52, 2019.

25. M. Shokrnezhad, T. Taleb, and P. Dazzi, "Double deep q-learning-based path selection and service placement for latency-sensitive beyond 5g applications," *IEEE Transactions on Mobile Computing*, vol. 23, no. 5, pp. 5097-5110, 2024.