# A Transfer Reinforcement Learning Approach for Capacity Sharing in Beyond 5G Networks
###### Abstract
The potential of TL to accelerate the re-training in RL-based capacity sharing solutions has been highlighted by some available works in the literature, where TL has shown to be effective in tackling different use cases. The authors in [18; 19] address drastic changes in traffic demand that the RL policy is not generic enough to adapt to, so they require a re-training. To support this re-training, the authors propose the use of the policy reuse TL technique. Instead of re-training the policy for the new situation from scratch, the ANN values and architecture of the old policy are reused as the initial policy in the new situation. Then, the transferred policy can be fine-tuned to the new scenario conditions, thus reducing the re-training time. The works in [20; 21] exploit TL to take a policy learned for a cell and then transfer it to be applied as the initial policy to another cell. Then, the initial policy is re-trained with specific data from this other cell. In [20], the policy reuse TL approach is used, while policy distillation is employed in [21]. In turn, refs. [22; 23] propose to exploit TL to accelerate the simultaneous training of multiple cells by learning a generalist policy that uses the experiences of multiple cells. Then, the knowledge of this generalist policy is transferred to learn local cell-specific policies, using the TL policy reuse technique. In this paper, TL is proposed to deal with the problem of the deployment of new cells, which, to the authors' best knowledge, has not yet been addressed in the literature. The deployment of new cells presents some challenges concerning RL-based capacity sharing solutions. On the one hand, there is the need to extend the intelligence of the RL-based capacity sharing solutions to manage the newly deployed cells. On the other hand, the deployment of a new cell will lead to changes in traffic demands and radio conditions of the neighboring cells (i.e., traffic demand will be redistributed among cells), leading to significant changes from a multi-cell scenario perspective. The TL approach proposed here addresses these two challenges for RL-based capacity sharing solutions where an RLagent jointly manages multiple cells, as in our work in [11]. For this kind of solution, the number of inputs and outputs of the ANN that constitutes the decision-making policy in RL depends on the number of cells. Therefore, a change in the number of deployed cells involves a change in the structure of this ANN. Thus, a novel aspect of the proposed TL approach in this paper is that it supports the transfer of weights between ANNs with a different number of inputs and outputs by using a TL technique referred to as inter-task mapping. Note that the fact that the proposed TL approach is designed for RL-based capacity sharing solutions where an agent deals with multiple cells is a distinctive aspect of this paper, since the works in [18; 19; 20; 21; 22; 23] have explored the use of TL for RL agents that make capacity sharing decisions for only a single cell. The results obtained in this paper illustrate the potential of the TL solution to accelerate the training process when increasing the number of cells, contributing to the practicality and feasibility of applying RL-based solutions for capacity sharing. In addition, the paper presents a functional framework for implementing the solution in the O-RAN architecture and discusses the deployment of the related O-RAN functionalities at the edge. The rest of the paper is organized as follows. Section 2 presents the system model and formally defines the problem addressed in the paper. Section 3 describes the proposed TL approach by first defining the states, actions, and rewards in the source and target tasks and then describing the proposed TL inter-task mapping approach. Afterwards, Section 4 provides the results, and Section 5 discusses the implementation considerations for the solution. Finally, Section 6 summarizes the conclusions extracted from the paper.