# A Transfer Reinforcement Learning Approach for Capacity Sharing in Beyond 5G Networks
## 3 Transfer Reinforcement Learning Approach
### Inter-Task Mapping Transfer Approach  
The state and action definitions for the source and target tasks differ in the number of components. Specifically, the state and action definitions in the target task incorporate the components \(\mathbf{s^{\Delta N}}\) and \(\mathbf{a^{\Delta N}}(d)\), respectively, with respect to the source task definitions, while the rest of the components are common. Based on this, only a partial mapping can be defined between the states/actions in the source and target tasks, because \(\mathbf{s^{\Delta N}}\) and \(\mathbf{a^{\Delta N}}(d)\) in the target task cannot be mapped to any component in the source task. This introduces a special requirement when selecting the TRL approach, as the majority of TRL solutions require a full mapping between the state, action, and reward components in both tasks. Considering this requirement, the TRL approach selected here is based on the _transfer_ via _inter-task mapping for policy search methods_ (TVITM-PS) in [25], which proposes a methodology that supports partial mappings between states and actions when transferring the weights of the ANN that represents the policy in the source task to build the ANN for the policy in the target task. The RL-based capacity sharing solution considered here in the source and target tasks relies on DQN-based agents [26] to learn the policy associated with each tenant. In particular, assuming that the state is \(\mathbf{s}\), the policy that selects the action \(\mathbf{a}\) for a tenant is given by:  
\[\pi=\underset{\mathbf{a}}{argmax}Q(\mathbf{s},\mathbf{a},\theta) \tag{3}\]  
where \(Q(\mathbf{s},\mathbf{a},\mathbf{\theta})\) is an ANN with weights \(\mathbf{\theta}\). The ANN architecture consists of an input layer composed of a neuron for each component of the state, \(L\) hidden layers (where layercontains \(H_{l}\) neurons), and an output layer with as many neurons as the number of possible actions in the action space. Each output neuron provides the value of \(Q(s,\,\mathbf{a},\,\mathbf{\theta})\) for each possible action \(\mathbf{a}\) in the action space. Considering that the neurons in the ANN are fully connected, \(\mathbf{\theta}\) is defined as \(\theta=\{\theta_{l,\rho,d}\ |\ l=1\ldots L+2,\,\forall o,\,\forall d\}\), where \(\theta_{l,\rho,d}\) corresponds to the weight between an origin neuron \(o\) of layer \(l\) and a destination neuron \(d\) of layer \(l+1\). Let us define \(\mathbf{\theta^{(S)}}\) and \(\mathbf{\theta^{(T)}}\) as the set of weights of the ANNs that represent the policy in the source and target tasks, and \(\theta^{(S)}_{l,\rho,d}\) and \(\theta^{(T)}_{l,\rho,d}\) as the individual weights in \(\mathbf{\theta^{(S)}}\) and \(\mathbf{\theta^{(T)}}\), respectively. Note that the ANNs in the source and target tasks present a different number of neurons in the input and output layers due to differences in the state and action definitions, but they have the same number of hidden layers and neurons in the hidden layers. This is illustrated in Figure 2, which shows the transfer operation between a source task with \(N=2\) cells and a target task with \(N^{\prime}=3\) cells, indicating the ANN associated with both tasks. As cell \(n=3\) is introduced in the target task, the input layer incorporates new neurons corresponding to the \(\mathbf{s^{AN}}\) component in \(\mathbf{s^{(T)}}\) (i.e., state \(\mathbf{s_{3}}\) for cell \(n=3\) in this case). Regarding the output layer, new neurons are incorporated in the ANN for all possible actions \(3^{N}\). Once the ANN in the target task is created, the weights \(\mathbf{\theta^{(T)}}\) are initialized based on those in \(\mathbf{\theta^{(S)}}\) of the source task as follows:  
\[\theta^{(T)}_{l,\rho,d}=\left\{ \begin{array}{ll}\theta^{(S)}_{l,\rho,d}& \left(l=1,\,\,o\,\,\varepsilon\,\,s^{(s)}\right)\,\,\text{or}\,\,(2<=l<=L)\\ \theta^{(S)}_{l,\rho,\rho(d)}&l=L+1\\ U(x_{1},x_{2})&\text{otherwise}\end{array} \right.