# A Deep Q Network-Based Multi-connectivity Algorithm for Heterogeneous 4G/5G Cellular Systems

Juan Jesús Hernández-Carlón, Jordi Pérez-Romero, Oriol Sallent, Irene Vilà, Ferran Casadevall

► To cite this version:

Juan Jesús Hernández-Carlón, Jordi Pérez-Romero, Oriol Sallent, Irene Vilà, Ferran Casadevall. A Deep Q Network-Based Multi-connectivity Algorithm for Heterogeneous 4G/5G Cellular Systems. 18th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2022, Hersonissos, Greece. pp.34-43, 10.1007/978-3-031-08341-9_3. hal-05335649

HAL Id: hal-05335649

https://inria.hal.science/hal-05335649v1

Submitted on 28 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# A Deep Q Network-Based Multi-connectivity Algorithm for Heterogeneous 4G/5G Cellular Systems

Juan Jesús Hernández-Carlón(✉), Jordi Pérez-Romero, Oriol Sallent, Irene Vilà, and Ferran Casadevall

Department of Signal Theory and Communications, Universitat Politècnica de Catalunya (UPC)
Barcelona, 08034 Barcelona, Spain

{juan.jesus.hernandez,jordi.perez-romero, irene.vila.munoz}@upc.edu, {sallent,ferranc}@tsc.upc.edu

**Abstract.** Multi-connectivity, which allows a user equipment to be simultaneously connected to multiple cells from different radio access network nodes that can be from a single or multiple radio access technologies, has emerged as a useful feature to handle the traffic in heterogeneous cellular scenarios and fulfill high data rate and reliability requirements. This paper proposes the use of deep reinforcement learning to optimally split the traffic among cells when multi-connectivity is considered in a heterogeneous 4G/5G networks scenario. Obtained results reveal a promising capability of the proposed Deep Q Network solution to select quasi optimum traffic splits depending on the current traffic and radio conditions in the considered scenario. Moreover, the paper analyses the robustness of the obtained policy in front of variations with respect to the conditions used during the training.

**Keywords:** Multi-connectivity · Deep reinforcement learning · Deep Q Network · Heterogeneous networks

## 1 Introduction

With the advent of 5G Mobile Network Operators (MNOs) face further increase in network deployment heterogeneity with different cell types (e.g. macrocells, indoor and outdoor small cells) based on multiple Radio Access Technologies (RATs) (e.g., 2G, 3G, 4G and 5G New Radio (5G NR)), operating in different spectrum bands (e.g. sub 6 GHz bands used by all RATs and millimeter wave (mmW) bands used by 5G New Radio). In this context, Multi-Connectivity (MC) technology enables a User Equipment (UE) to be simultaneously connected to multiple nodes of the Radio Access Network (RAN), e.g. eNodeBs (eNB) operating with LTE and/or gNodeBs (gNB) operating with 5G NR [1, 2]. There is one master node (MN) responsible for the radio-access control plane and one, or in the general case multiple, secondary node(s) (SN) that provide additional user-plane links. In this way, a UE can aggregate the radio resources from multiple eNBs/gNBs, which allows efficiently achieving the 5G requirements of high
data rate and ultra-reliability. The literature has considered different problems in relation to MC, such as the resource allocation in [3, 4] or the traffic split [5–8].

This paper addresses the traffic split multi-connectivity problem in multi RAT scenarios by exploiting Deep Q Network (DQN) technique [9] to obtain a policy that allows optimally distributing the traffic of a UE across the different RATs and cells while fulfilling the QoS requirements and optimizing the bandwidth consumption of the UE, so that overload situations are avoided in the involved cells. Deep reinforcement learning techniques such as DQN are useful for optimizing dynamic decision-making problems that depend on a large number of input variables taking a wide range of possible values. This is the case of the MC problem formulated in this paper, for which a DQN solution is presented and assessed by means of simulations. In addition, the results presented in this paper pay particular attention to the capability of the solution to generalize the knowledge learnt during the training phase. In this direction, the robustness of the learnt policy is analyzed when the conditions experienced by the algorithm differ from the ones that were considered during the training.

The rest of the paper is organised as follows. Section 2 presents the system model and formulates the considered multi-connectivity problem. The proposed DQN-based solution is presented in Sect. 3 and different performance results are provided in Sect. 4. Finally, Sect. 5 summarises the conclusions.

## 2 System Model and Problem Definition

Let us consider a heterogeneous RAN where different UEs with multi-connectivity capabilities are camping. A given UE $u$ considers $M$ different RATs and $N$ different cells per RAT as candidates for the multi-connectivity. Then, let us denote as $A_u = \{C_{m,n}\}$ the set of candidate cells detected by the UE $u$. $C_{m,n}$ denotes the $n$-th cell of the $m$-th RAT with $n = 1, ..., N$ and $m = 1, ..., M$. It is worth mentioning that, due to the mobility of the UE, the specific cells that the UE detects in a given RAT may change with time. In this respect, it is assumed that the $N$ cells of a RAT correspond to the best $N$ cells detected by the UE at a certain time based on measurements averaged during a time window $\Delta T$.

Through the use of multi-connectivity, the traffic of the $u$-th UE is split across multiple RATs/cells of the set $A_u$. It is assumed that, at a certain time, the UE can be simultaneously connected to a maximum of $N_{max}$ cells among the $M \cdot N$ candidates. The multi-connectivity configuration for the $u$-th UE can be expressed as the $M \times N$ matrix $\mathbf{B} = \{\beta_{m,n}\}$ where $\beta_{m,n} \in [0,1]$ defines the fraction of total traffic of UE $u$ that is delivered through the $n$-th cell of the $m$-th RAT. Then, the objective is to find the optimal configuration $\mathbf{B} = \{\beta_{n,m}\}$ to be applied in a time window of $\Delta T$ s that allow ensuring the Quality of Service (QoS) requirements with minimum resource consumption and avoiding overload situations in the different RATs/cells. In this respect, it is assumed that the QoS requirements of the user $u$ are expressed in terms of a required bit rate $R_u$ (b/s) to be provided.
To formalize the problem, let us denote as $T_u(\mathbf{B})$ the total throughput or bit rate obtained by user $u$ during the last time window period $\Delta T$ with the multi-connectivity configuration $\mathbf{B}$. Let us also denote $a_{m,n}(\beta_{m,n})$ as the number of physical resource blocks (PRBs) in the $m$-th cell and $n$-th RAT assigned to the $u$-th UE to transmit the traffic corresponding to $\beta_{m,n}$. Considering that $b_{m,n}$ corresponds to the bandwidth of one PRB in the $m$-th cell and $n$-th RAT, the bandwidth allocated to the user $u$ in this RAT, denoted as $\gamma(\beta_{m,n})$, is given by:

$$ \gamma(\beta_{m,n}) = a_{m,n}(\beta_{m,n}) \cdot b_{m,n} \quad (1) $$

In addition, the total fraction of occupied PRBs in a RAT/cell accounting for all the UEs connected to that cell is denoted as $\rho_{m,n}(\beta_{m,n})$. Then, the considered problem to be solved for the $u$-th UE is formally defined as:

$$ \mathbf{B} = \underset{\mathbf{B}}{\operatorname{argmin}} \left[ \frac{1}{w_{\max}} \sum_{m=1}^{M} \sum_{n=1}^{N} \gamma(\beta_{m,n}) \right] \quad (2) \\ \text{s.t. } T_u(\mathbf{B}) \ge R_u, \rho_{m,n}(\beta_{m,n}) \le \rho_{\max} \forall m, n \\ \sum_{m=1}^{M} \sum_{n=1}^{N} \beta_{m,n} = 1 $$

where $w_{\max}$ is the maximum possible bandwidth to be assigned to the user $u$ and $\rho_{\max} \in [0,1]$ is the maximum threshold established to avoid overload situations in a cell.

Figure 1 depicts the architectural components to enforce the multi-connectivity configuration $\mathbf{B}$ in the network, obtained as a result of the above problem. The figure illustrates an example for the downlink traffic transmitted to a UE served by two cells of RAT $m=1$ (e.g., 5G). The cell $n=1$ is handled by the MN and the cell $n=2$ by the SN. The traffic between these cells is split at the Packet Data Convergence Protocol (PDCP) layer of the MN using dual connectivity feature. The multi-connectivity configuration is determined by an MC controller that takes as an input different measurement from the RATs/cells as it will be explained in Sect. 3. The output of the MC controller is the configuration $\mathbf{B} = \{\beta_{m,n}\}$ with the weights $\beta_{m,n}$ to be configured at the PDCP layer of the MN to split the traffic between cells 1 and 2. Finally, the Medium Access Control (MAC) scheduler in each 5G NR or LTE cell will allocate the necessary amount of PRBs $a_{m,n}(\beta_{m,n})$ to the UE to transmit the fraction of traffic $\beta_{m,n}$ corresponding to the cell. The specific design of the MAC scheduler is out of the scope of this work, but in general it will consider aspects such as the propagation and interference conditions observed by the UE, the QoS requirements, the amount of UEs in the cell, etc.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Architectural components of the considered approach</figcaption></figure>

## 3 DQN-Based Solution

A DQN approach is considered in this paper for solving the MC problem formulated in previous section. In this approach, the learning process is conducted dynamically by a DQN agent at the MC controller of Fig. 1 that makes decisions for the different UEs. The agent operates in discrete times with granularity equal to the time window duration $\Delta T$. These discrete times are denoted as $t, t + 1, \dots, t + k, \dots$. At time $t$ the DQN selects an action $a(t)$ that contains the MC configuration to be applied for a given UE in the next time window. The action selection is based on the current state at time $t$, denoted as $s(t)$ and on the decision-making policy available at this time. Then, as a result of applying the selected MC configuration, a reward signal $r(t + 1)$ is provided to the DQN agent at the end of the time window. This reward signal measures how good or bad was the last performed action and therefore it is used to improve the decision-making policy. The different components of this process are detailed in the following.

### 3.1 State, Action and Reward Specification

The state $s(t)$ is a vector that includes the following components for a given UE $u$:

* Requirements of UE $u$: $R_u$.

* Spectral efficiency per RAT/cell $\{S_{m,n}\}$ of UE $u$.

* Current configuration $\mathbf{B} = \{\beta_{m,n}\}$, which corresponds to the configuration applied at time $t - 1$.

* Bandwidth occupied by the UE $u$ in each RAT/cell $\{\gamma(\beta_{m,n})\}$.

* Fraction of total occupied resources in each RAT/cell $\{\rho_{m,n}(\beta_{m,n})\}$.

All the values $S_{m,n}$, $\gamma(\beta_{m,n})$ and $\rho_{m,n}(\beta_{m,n})$ are average values measured during the last time window of duration $\Delta T$, i.e. between discrete times $t - 1$ and $t$.

Each action $a(t) \in \mathcal{A}$ represents a matrix $\mathbf{B} = \{\beta_{m,n}\}$ that corresponds to the MC configuration to be applied during the next time window $\Delta T$. The action space $\mathcal{A}$ includes all the MC configurations and is defined by considering that the possible $\beta_{m,n}$ values are discretized with granularity $\Delta\beta$ and the aggregate of all $\beta_{m,n}$ values in matrix $\mathbf{B}$ equals 1. Moreover, the action space considers that the UE can be simultaneously connected to
at most $N_{max}$ cells of the $N \cdot M$ candidates, i.e. that at most $N_{max}$ values of $\beta_{m,n}$ can be different from 0.

The reward $r(t + 1)$ intends to measure how good or bad was the performance obtained by the last action $a(t)$ for the state $s(t)$ in relation to the target of the optimization. Then, considering the optimization problem (2), and that the last action $a(t)$ is given by MC configuration $\mathbf{B} = \{\beta_{m,n}\}$, the reward is defined as:

$$r(t+1) = \left(1 - \frac{1}{w_{max}} \sum_{m=1}^{M} \sum_{n=1}^{N} \gamma(\beta_{m,n}) \cdot \min\left(1, \frac{T_u(\mathbf{B})}{R_u}\right) \right. \\ \left. \cdot \prod_{\substack{m,n \\ \beta_{m,n} > 0}} \min\left(1, \frac{\rho_{max}}{\rho_{m,n}(\beta_{m,n})}\right) \right) \quad (3)$$

The first term in $r(t + 1)$ captures the total bandwidth assigned to the UE $u$ in all the cells/RATs, so the lower the amount of bandwidth assigned the higher will be the reward. The second term represents a penalty introduced when the achieved throughput $T_u(\mathbf{B})$ is lower than the minimum requirement $R_u$. The last term introduces a penalty for each cell/RAT in which the UE has transmitted traffic (i.e. $\beta_{m,n} > 0$) and the cell is overloaded. Note that the values of $\gamma_{m,n}(\beta_{m,n})$, $\rho_{m,n}(\beta_{m,n})$ and $T_u(\mathbf{B})$ correspond to the averages obtained during the time window $\Delta T$ between discrete times $t$ and $t + 1$.

## 3.2 Policy Learning Process

The DQN agent dynamically learns the decision-making policy $\pi$ used to select the different actions based on the rewards obtained from previous decisions. This is done by means of the DQN algorithm of [9] particularised to the state, action and reward signals presented above. In summary, the algorithm aims at finding the optimal policy that maximises the discounted cumulative expected reward by approximating the optimum action-value function with a deep neural network (DNN) denoted as $Q(s, a, \theta)$, where $s$ is the observed state, $a$ is one of the possible actions that can be selected and $\theta$ are the weights of the interconnections between the different neurons in the DNN. Given the DNN, the decision making policy consists in selecting the action $a$ with the highest value of $Q(s, a, \theta)$ for a given state.

The decision-making policy is updated progressively by modifying the weights $\theta$ based on the experiences gathered by the DQN agent. For this purpose, at a certain time $t$ the DQN agent observes the state of the environment $s(t)$ for a given UE and it triggers an action by selecting with probability $1 - \epsilon$ the action $a(t)$ with the highest value of $Q(s,a,\theta)$ and with probability $\epsilon$ a random action. As a result, the DQN agent gathers the obtained reward and the new state at time $t + 1$ and stores this experience (i.e., $s(t), a(t), r(t + 1), s(t + 1)$) in an experience dataset. The information collected in this dataset is then used to update the weights $\theta$ of the DNN using the expressions detailed in [9].

## 4 Performance Evaluation

This section evaluates the performance of the proposed solution by means of system level simulations.
## 4.1 Scenario Description

The considered scenario is a square area of 500 m × 500 m composed by four 5G NR cells and two LTE cells. The relevant parameters of the cells are presented in Table 1. The scenario assumes a non-homogeneous traffic distribution with MC-capable UEs moving at 1 m/s along the scenario and have an active session during the whole simulation duration with a required bit rate $R_u = 50$ Mb/s. The candidate cells of the UEs can connect to $M = 2$ RATs and $N = 2$ cell per RAT, and the maximum number of cells that the UE can be connected to using MC is $N_{max} = 2$. Additional background traffic is considered, with UEs generating Poisson session arrivals with aggregate generation rate 0.8 sessions/s and exponentially distributed session duration with average 120s. A background UE remains static during a session. 50% of the background UEs are randomly located inside a square hotspot of 250 m × 250 m centred at the middle of the scenario. The rest of background UEs are randomly distributed in the whole scenario. Background UEs connect to the RAT/cell with the highest Signal to Interference and Noise Ratio (SINR). To capture the different bit rates achievable by the two technologies, when a background UE is connected to LTE, its serving cell allocates the needed resource blocks to achieve a bit rate of 2.5 Mb/s, and when it is connected to 5G NR, the allocation is to achieve a bit rate of 40 Mb/s.

The DQN model parameters are detailed in Table 2. The DQN model has been developed in Python using the *TF-agents* library [10].

<table><caption>**Table 1.** Cell configuration parameters</caption><thead><tr><th>Parameter</th><th colspan="2">Value</th></tr></thead><tbody><tr><td>Type of RAT</td><td>LTE</td><td>5G NR</td></tr><tr><td>Cells position [x, y] m</td><td>[62, 250]<br>[437,250]</td><td>[187, 125] [187,375]<br>[312,125] [312,375]</td></tr><tr><td>Frequency</td><td>2100 MHz</td><td>26 GHz</td></tr><tr><td>Subcarrier separation</td><td>15 kHz</td><td>60 kHz</td></tr><tr><td>Nominal channel bandwidth</td><td>20 MHz</td><td>50 MHz</td></tr><tr><td>Number of available PRBs</td><td>100</td><td>66</td></tr><tr><td>Base station transmitted power</td><td>49 dBm</td><td>21 dBm</td></tr><tr><td>Base station antenna gain</td><td>5 dB</td><td>26 dB</td></tr><tr><td>Base station height</td><td>25 m</td><td>10 m</td></tr><tr><td>UE antenna gain</td><td>5 dB</td><td>10 dB</td></tr><tr><td>Overload threshold ρ<sub>max</sub></td><td>0.95</td><td>0.95</td></tr><tr><td>UE noise figure</td><td>9 dB</td><td></td></tr><tr><td>UE height</td><td>1.5 m</td><td></td></tr><tr><td>Path loss model</td><td colspan="2">Model of Sect. 7.4 of [11]</td></tr><tr><td>w<sub>max</sub></td><td colspan="2">95.04 MHz (corresponds to the case when MC is done with 2 cells of 5G NR)</td></tr></tbody></table>
<table><caption>**Table 2.** DQN algorithm parameters</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Initial collect steps</td><td>5000</td></tr><tr><td>Number of policy updates during learning</td><td>1e6</td></tr><tr><td>Experience Replay buffer maximum length (l)</td><td>1e5</td></tr><tr><td>Mini-batch size (J)</td><td>256</td></tr><tr><td>Discount factor(γ)</td><td>0.9</td></tr><tr><td>Learning rate (τ)</td><td>0.0003</td></tr><tr><td>ε value (ε-Greedy)</td><td>0.1</td></tr><tr><td>DNN architecture</td><td>Input layer: 17 nodes<br/>Two hidden layers: 100 and 50 nodes<br/>Output layer: 58 nodes</td></tr><tr><td>Time window (ΔT)</td><td>1 s</td></tr><tr><td>Granularity Δβ</td><td>0.1</td></tr></tbody></table>

## 4.2 Training Evolution

The training process of the DQN algorithm is performed by considering a MC-capable UE moving along the scenario following trajectories according to random walk and with required bit rate $R_u = 50$ Mb/s, while at the same time background UEs also generate traffic as explained in Sect. 4.1. The DQN agent decides the MC connectivity configuration of the UE and, based on the obtained rewards, the decision making policy is progressively updated as explained in Sect. 3.2. In order to illustrate this learning process, the policies that are obtained every 2500 weight updates (i.e. training steps) are applied to an evaluation scenario in which an illustrative MC-capable UE follows a specific trajectory of duration 400 s, starting from point $[X_1 = 50, Y_1 = 300]$ and following a straight trajectory up to the point $[X_2 = 450, Y_2 = 300]$ at 1 m/s. Figure 1 presents the evolution of the average reward obtained with the application of these policies as a

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Evolution of the average reward as a function of the training steps</figcaption></figure>
function of the number of training steps. The results of Fig. 2 show that as the number of training steps increase average reward values tend to increase until $40 \times 10^4$ training steps, when the average reward values stabilize.

## 4.3 Performance Evaluation of the DQN-Based Strategy

In order to assess the benefits brought by the proposed DQN-based MC strategy, this sub-section compares the performance obtained by the proposed approach against two benchmarking approaches, namely the *optimum strategy* and the *SINR-based strategy*. The former consists of applying an exhaustive search process to select in each time step the MC configuration with the maximum reward, while the later considers that all the traffic of a UE is served by the cell with the highest SINR.

The comparison is performed by simulating a UE of interest following one hundred different trajectories of duration 400 s in the evaluation scenario and applying in each time window the MC configuration according to each of the evaluated schemes. For the DQN approach, the results correspond to the policy learnt by the DQN agent after a training consisting of 1E6 policy updates according to the procedure of Sect. 3.2.

Figure 3 shows the obtained average reward for each one of the trajectories with all the considered strategies. It is observed that the DQN-based strategy achieves a performance very close to the optimum one in all the studied cases, which confirms the good behavior of the proposed approach. In turn, Fig. 3 also shows that the DQN-based strategy outperforms the classical SINR-based strategy in all the studied cases thanks to the better distribution of the traffic of the UE among the cells that avoids overload and enhances the obtained bit rate.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Average reward for different trajectories.</figcaption></figure>

## 4.4 Analysis of the Robustness of the Learnt DQN-Policy

This section aims at evaluating the robustness of the DQN-policy when it is applied under conditions that differ from the ones that were considered during the training. For this purpose, considering that the training process has been done with a required bit rate
$R_u = 50$ Mb/s, the following results assess the generalization capability of the learnt policy when it is applied to different $R_u$ values ranging between 15 and 65 Mb/s.

As a relevant metric for this assessment, a DQN-policy efficiency metric is considered, defined as the ratio between the reward of the DQN policy and that of the optimum strategy. Figure 4 depicts the DQN-policy efficiency as a function of the required bit rate $R_u$ value. The results of policy efficiency correspond to the average for the one hundred trajectories considered in the study. For the $R_u$ value of 50 Mb/s that was considered during the training it is observed in Fig. 4 that the efficiency of the original policy is around 95.75%. Then, it tends to decrease for higher and lower values of required bit rate. In fact, it is worth mentioning the effects of decreasing $R_u$, because even when the amount of required radio resources is less, the efficiency losses are higher. For example, the policy efficiency with $R_u = 15$ Mb/s is 13.42% less than with 50 Mb/s. From the red line in Fig. 4, we can realize that efficiency variations lower than 1% are observed when the $R_u$ value changes from 40 Mb/s up to 65 Mb/s, i.e. $-10\% + 15$ Mb/s with respect to the value used for training. This reflects that the learnt DQN policy is robust in front of variations of around 20–30% of the $R_u$ value used in the training.

Based on these results, another training process has been conducted with the same parameters of Table 2 but now changing the $R_u$ value during training. Specifically, $R_u$ at the beginning of the training is 50 Mb/s and then it is changed between 5 and 60 Mb/s with steps of $\pm 10\%$. Moreover, the number training steps has been increased up to 3E6 in order to account for a higher number of possible situations to learn. The green line in Fig. 4, shows the obtained efficiency with the new learnt policy (denoted as retrained policy) in comparison to the original policy. It is clearly observed that the new policy achieves a good efficiency for all the considered RBR values.

<figure><img src="image_6.png" /><figcaption>Fig. 4. DQN-Policy efficiency for different required bit rate (original vs retrained)</figcaption></figure>

# 5 Conclusion

This paper has presented a novel approach based on Deep Q- Network for splitting the traffic of a UE among cells when using multi-connectivity depending on the current
traffic and radio conditions experienced by the UE in the involved cells. The strategy
intends to minimize the bandwidth consumption, the overload situations in the cells and
enhancing throughput. The proposed strategy has been evaluated and compared against
the optimum case and against a classical SINR-based approach. Results have shown the
capability of the DQN agent to learn a quasi-optimal policy that in certain conditions
outperforms the SINR-based approach in up to 33% in terms of reward, obtaining as a
result better throughput performance with an optimized bandwidth assignment.

This paper has also analyzed the robustness of the learnt policy when being applied
with a required bit rate value different than the one that was considered during the training
stage. It has been observed that the learnt policy is able to work properly with variations
of the required bit rate of around 20–30% of the value considered in the training. In turn,
by conducting a training that considers a wider range of values of required bit rate, it is
possible to increase the performance of the obtained policy.

**Acknowledgement.** This paper is part of ARTIST project (ref. PID2020-115104RB-I00) funded by MCIN/AEI/10.13039/ 501100011033. The work is also funded by the Spanish Ministry of Science and Innovation under grant ref. PRE2018-084691.

References

1. Maeder, A. et al.: A scalable and flexible radio access network architecture for fifth generation mobile networks. IEEE Communications Magazine, November (2016). https://doi.org/10.1109/MCOM.2016.1600140CM

2. Dahlman, E., Parkvall, S., Sköld, J.: 5G NR the Next Generation Wireless Access Technology. Academic Press Elsevier (2018)

3. Yan, M., Feng, G., Zhou, J., Qin, S.: Smart multi-RAT access based on multiagent reinforcement learning. IEEE Trans. Veh. Technol. 67(5), 4539–4551 (2018). https://doi.org/10.1109/TVT.2018.2793186

4. Monteiro, V.F., et al.: Distributed RRM for 5G multi-RAT multiconnectivity networks. IEEE Syst. J., 13(1), March (2019). https://doi.org/10.1109/JSYST.2018.2838335

5. Gerasimenko, M., et al.: Adaptive resource management strategy in practical multi-radio heterogeneous networks. IEEE Access, February (2017). https://doi.org/10.1109/ACCESS.2016.2638022

6. Taksande, P.K., Roy, A., Karandikar, A.: Optimal traffic splitting policy in LTE-based het- erogeneous network. IEEE Wireless Communications and Networking Conference (WCNC) (2018). https://doi.org/10.1109/WCNC.2018.8377096

7. Zhang, B. et al.: Goodput-aware traffic splitting scheme with non-ideal backhaul for 5G-LTE multi-connectivity. IEEE Wireless Communications and Networking Conference (WCNC) (2019). https://doi.org/10.1109/WCNC.2019.8885728

8. Elias, J., Martignon, F. Paris, S.: Optimal split bearer control and resource allocation for multi-connectivity in 5G new radio. Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit), pp. 187–192 (2021). https://doi.org/10.1109/EuCNC/6GSummit51104.2021.9482505

9. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

10. Guadarrama, S., et al.: TF-agents: A library for reinforcement learning in TensorFlow (2018)

11. 3GPP TS 38.901 v16.1.0: Study on Channel Model for Frequencies From 0.5 to 100 GHz (Release 16) Dec. 2019