# A Transfer Reinforcement Learning Approach for Capacity Sharing in Beyond 5G Networks
## 3 Transfer Reinforcement Learning Approach
### Inter-Task Mapping Transfer Approach
\tag{4}\]  
The first condition in (4) applies in cases where the weights \(\theta^{(S)}_{l,\rho,d}\) are copied between the same pair of neurons in the source and target ANNs. This occurs in two cases. The first case is between the nodes in the hidden layers (i.e., \(2<=l<=L\)), where all the weights are copied. The second case is when transferring the weights between the input neurons corresponding to the components in \(\mathbf{s^{(T)}}\) that are common with \(\mathbf{s^{(S)}}\) (i.e., \(\mathbf{s^{N}}\) and \(\mathbf{s^{SLA}}\)) and the neurons of the first hidden layer. This is depicted in the example in Figure 2, where the \(\theta^{(T)}_{l,\rho,d}\) between the state components for \(\{\mathbf{s_{n}}\}_{n=1,2}\) are copied from \(\mathbf{\theta^{(S)}}\). Regarding the rest of the neurons in the input layer, since they correspond to the new cells (i.e., corresponding  
Figure 2: Illustrative example of the transfer of weights \(\mathbf{\theta^{(S)}}\) when the transfer is performed for a source task with \(N=2\) cells to a target task with \(N^{\prime}=3\). to \(\mathbf{s^{\text{AN}}}\)), all weights are set randomly as \(U(x_{1},x_{2})\), which denotes a uniformly distributed random variable in the range \([x_{1},x_{2}]\). This is considered in the third condition in (4) and illustrated by the red lines associated with the state component \(\mathbf{s_{3}}\) for the new cell \(n=3\) in Figure 2.  
The second condition in (4) corresponds to the transfer of weights between the last hidden layer (\(l=L+1\)) and the output layer (\(l=L+2\)). In this case, considering that the \(d\)-th output neuron in the target task corresponds to the \(d\)-th action, i.e., \(\mathbf{a^{\text{(T)}}}(d)\), and that the \(i\)-th output neuron in the source task corresponds to the \(i\)-th action, \(\mathbf{a^{\text{(S)}}}(i)\), the transfer of weights between the last hidden layer and the output layer is held according to the action mapping function \(\varphi(d)\) that maps each output neuron \(d\) in the target task to the output neuron \(i\) in the source task as follows:  
\[\varphi(d)=i\ |\ \mathbf{a^{\text{(S)}}}(i)=\mathbf{a^{\text{N}}}(d) \tag{5}\]  
which considers that the values of \(\mathbf{a^{\text{N}}}(d)\) components in action \(\mathbf{a^{\text{(T)}}}(d)\) of the target task are the same as those of \(\mathbf{a^{\text{(S)}}}(i)\) in the source task. An example of this mapping function is shown in Figure 2, where the weights between the last hidden layer and the output neuron in the target ANN corresponding to action \(\mathbf{a^{\text{(T)}}}(1)=\{\Delta,\,\Delta,\Delta\}\) are copied from those of source action \(\mathbf{a^{\text{(S)}}}(1)=\{\Delta,\,\Delta\}\), since the values of \(\mathbf{a^{\text{N}}}(1)=\{\Delta,\,\,\Delta\}\) of the target task are the same as \(\mathbf{a^{\text{(S)}}}(1)\). Note that this mapping implies that all actions in the target task with common values of \(\mathbf{a^{\text{N}}}(d)\) will be mapped to the same source task action, e.g., \(\mathbf{a^{\text{(T)}}}(1)=\{\Delta,\,\Delta\}\), \(\mathbf{a^{\text{(T)}}}(1)=\{\Delta,\,\Delta,0\}\) and \(\mathbf{a^{\text{(T)}}}(1)=\{\Delta,\,\Delta,-\Delta\}\) in Figure 2 are mapped to \(\mathbf{a^{\text{(S)}}}(1)=\{\Delta,\,\Delta\}\). The fact that random weights are set for the input neurons related to new cells (third condition in (4)) and that different output neurons are mapped to the same output neuron in the source ANN (second condition in (4)) implies that once the weights in the ANN of the source task are transferred to the ANN in the target task, they cannot be directly used for the target task. Therefore, a training stage of the transferred ANN is required after transferring the weights to tune the weights of the ANN in the target task. Figure 3 summarizes the overall process for obtaining the policy of a DQN agent in the target task following the proposed TRL approach.