# A Transfer Reinforcement Learning Approach for Capacity Sharing in Beyond 5G Networks
## 4 Performance Evaluation
### Training Performance  
Figure 4 shows the evolution of the average aggregated reward per evaluation, \(R\), during the training of the policies for \(N^{\prime}=5\) cells with the _TRL_ and _Non-TRL_ modes. For both cases, the value of \(R\) increases abruptly at the beginning of the training and then stabilizes to a value of around 1.78. However, the value of \(R\) increases faster with the _TRL_ mode than with the _Non-TRL_ mode. Moreover, while \(R\) exhibits high fluctuations from the beginning until approximately \(200\times 10^{3}\) training steps with the _Non-TRL_ mode, these fluctuations are much smaller with the _TRL_ mode throughout the training. These results indicate that the proposed TRL approach allows for achieving a higher and more stable reward sooner during the training, thus reducing the training duration. To further assess the convergence of the policies obtained for _TRL_ and _Non-TRL_ modes, Figure 5 shows the standard deviation of \(R\), \(std(m,W)\), when considering a window of \(W=30\times 10^{3}\) training steps. In contrast to the evolution of \(R\), the value of \(std(m,W)\) decreases abruptly at the beginning of the training for both modes. After this initial decrease, the value of \(std(m,W)\) decreases slowly until stabilizing. Higher standard deviation values are obtained with the _Non-TRL_ mode due to the larger reward fluctuations that were observed in Figure 5. Accordingly, the standard deviation values for the _TRL_ mode decrease and stabilize much sooner, evidencing that the proposed TRL approach accelerates the training process. In fact, considering that convergence is achieved when the standard deviation \(std(m,W)\) is lower than \(std_{th}=0.01\), the training duration for the _Non-TRL_ mode is \(265\times 10^{3}\) training steps, while for the _TRL_ mode it is \(123\times 10^{3}\) training steps. This means that the use of the proposed TRL approach reduces the training duration by 54%. Figure 4: Average aggregated reward (\(R\)) during training. To evaluate the proposed TRL approach in a wider range of RAN deployments, the analysis of the training behavior has been extended to different RAN deployments that initially have \(N=1\ldots 7\) cells and are extended to \(N^{\prime}=2\ldots 8\) cells, respectively. The cells' configuration and SLA parameters in the different deployments are configured according to Table 1. For each deployment, results have been obtained when training the DQN-MARL capacity sharing solution according to the _TRL_ and _Non-TRL_ modes. The DQN configuration parameters are summarized in Table 3 for each number of cells, while the rest of the parameters not mentioned in this table are the same as those in Table 2. Note that the number of input and output layers for each number of cells in Table 3 equals the number of components of the state and the number of possible actions, respectively, which depend on the number of cells. Also, the number of initial training steps is increased with the number of cells to better explore the larger number of possible actions and states. The training performance for obtaining the policies for \(N^{\prime}=2\ldots 8\) cells has been assessed according to the methodology described in Section 4.2 with a window \(W=30\times 10^{3}\) training steps and a standard deviation threshold \(std_{th}=0.01\), as previously considered. Table 4 includes the training duration in time steps for the different deployments in both _TRL_ and _Non-TRL_ modes. For a given mode, the results show that the training durations for deployments with a low number of cells are similar (i.e., for \(N^{\prime}=2\) and \(N^{\prime}=3\), the duration is around 90 \(\times\) 10\({}^{3}\) training steps with the TRL mode and around 170 \(\times\) 10\({}^{3}\) training steps with the _Non-TRL_ mode). However, for a larger number of cells in the RAN deployment, the training duration increases, since the number of state components and possible actions increases substantially, as does the size of the ANN.