# A Transfer Reinforcement Learning Approach for Capacity Sharing in Beyond 5G Networks
###### Abstract  
The use of Reinforcement Learning (RL) techniques has been widely addressed in the literature to cope with capacity sharing in 5G Radio Access Network (RAN) slicing. These algorithms consider a training process to learn an optimal capacity sharing decision-making policy, which is later applied to the RAN environment during the inference stage. When relevant changes occur in the RAN, such as the deployment of new cells in the network, RL-based capacity sharing solutions require a re-training process to update the optimal decision-making policy, which may require long training times. To accelerate this process, this paper proposes a novel Transfer Learning (TL) approach for RL-based capacity sharing solutions in multi-cell scenarios that is implementable following the Open-RAN (O-RAN) architecture and exploits the availability of computing resources at the edge for conducting the training/inference processes. The proposed approach allows transferring the weights of the previously learned policy to learn the new policy to be used after the addition of new cells. The performance assessment of the TL solution highlights its capability to reduce the training process duration of the policies when adding new cells. Considering that the roll-out of 5G networks will continue for several years, TL can contribute to enhancing the practicality and feasibility of applying RL-based solutions for capacity sharing. R +
Footnote †: journal: Future internet  
In general, the practical application of RL-based solutions involves a training stage used to learn the optimal decision-making policy for a given environment, followed by an inference stage in which the learned policy is applied [16]. However, whenever a mismatch between the training and the inference environments occurs, the learned policy will no longer be optimal and can even lead to significant performance degradation. In such a case, a re-training process is needed to learn a new optimal decision-making policy. This mismatch is relevant in capacity sharing in the RAN because the operator will very often perform changes in the network topology, for example by deploying new cells to improve the coverage footprint or to increase the deployed capacity. Therefore, given the massive deployment of 5G and eventually 6G networks, addressing the AI/ML model re-training deserves careful attention for developing scalable and robust solutions that materialize the embedding of AI/ML into future communication networks. Carrying out a re-training process in RL can take a lot of time, as it typically involves acquiring a large number of state/action/reward experiences with the new environment. To address this challenge, Transfer Learning (TL) techniques are identified as key tools [17], since they allow accelerating the training process by reusing previous knowledge of a source task to perform a new target task. In this way, the re-training does not have to be done from scratch but can start from the previously learned policy and update it according to the new environment. With all the above, the main contribution of this paper is the proposal of a new TL approach for re-training an RL-based capacity sharing policy in multi-cell scenarios when expanding the RAN with new cells. To this end, we leverage the multi-cell RL-based capacity sharing solution proposed in our previous work [11; 12] that makes use of Multi-Agent Reinforcement Learning (MARL) with the Deep Q-Network (DQN) technique, and it is referred to here as the DQN-MARL algorithm. A key feature of this algorithm is that it associates one DQN agent with each slice, and this agent learns the policy that assigns the capacity to the slice in each of the cells in the scenario. Thus, it is a solution that jointly manages multiple cells. As discussed in [11], this approach brings substantial benefits with respect to single-cell decision-making processes, in which capacity sharing is managed independently at each cell.