## 5 Transfer Learning Actor-Critic Framework for Data Protection in Cognitive Radio Networks
### Transfer Learning Actor-Critic Algorithm  
Previous work proposed a POMDP-based approach to solving the problem in Equation (14) on the assumption that the SU already has information about the harvested energy model. In this paper, we introduce a new solution to the problem based on the actor-critic learning framework, which does not require the SU to already know the dynamics of energy harvesting. Instead, the SU determines those dynamics by directly interacting with the environment. A regular actor-critic model comprises three main elements: an actor (related to a learning policy), a critic (related to a learning value function), and the environment, as shown in Figure 5.  
At time step \(t\), the actor selects action \(a(t)\) based on the current state, \(s(t)\), and the policy, \(\pi(s(t))\), which is defined by using a Gibbs softmax function as follows [22]:  
\[\pi(s,a)=P(a(t)=a|s(t)=s)=\frac{e^{\theta(s,a)}}{\sum\limits_{a^{\prime}\in \mathbb{A}}e^{\theta(s,a^{\prime})}}, \tag{15}\]  
where \(\theta(s,a)\) is the tendency to select action \(a\) when the SU is in state \(s\). The final objective of this paper is now to find an optimal mode decision policy for the SU at the \(t\)th time slot, and the problem in Equation (14) can be rewritten as  
\[\pi^{*}(s)=\operatorname*{argmax}_{a\in\mathbb{A}}\left\{R(s,a)+\eta\sum \limits_{s^{\prime}\in\mathbb{S}}P(s^{\prime}|s,a)V^{*}(s^{\prime})\right\}, \tag{16}\]  
where \(P(s^{\prime}|s,a)\) is the transition probability from state \(s\) to state \(s^{\prime}\) after taking action \(a\). Figure 5: A regular actor–critic model. TD: temporal difference. After that, the SU transits into a new state, \(s(t+1)\), and receives an instant reward \(R(s(t),a(t))\). The critic evaluates the new state and computes a temporal difference (TD) error as  
\[\delta(t)=R(s(t),a(t))+\eta V(s(t+1))-V(s(t)). \tag{17}\]  
The critic uses the TD error to improve the estimate of the value function as well as the policy. The value function is updated as  
\[V(s(t))\gets V(s(t))+\alpha_{c}\cdot\delta(t), \tag{18}\]  
where \(\alpha_{c}\) is a positive parameter of the critic. The action resulting in a positive TD error is favorable, since the state value is better than expected. Hence, the probability of selecting action \(a(t)=a\) in state \(s(t)=s\) in the future should increase, and vice versa. Following that, the tendency to select this action is updated as  
\[\theta(s(t),a(t))\leftarrow\theta(s(t),a(t))+\alpha_{a}\cdot\delta(t), \tag{19}\]  
where \(\alpha_{a}\) is a positive parameter of the actor. Furthermore, we exploit the idea of transfer learning to increase the convergence speed to the optimal solution by making use of historical learning data, as depicted in Figure 6.  
The obtained information is transferred to the new actor-critic algorithm for real-time training in which the initialized value function is the same as the transferred function while the overall policy, \(\theta_{o}(s(t),a(t))\), for choosing an action at time step t is given as  
\[\theta_{o}(s(t),a(t))=\varepsilon(t)\theta_{l}(s(t),a(t))+(1-\varepsilon(t)) \theta_{n}(s(t),a(t)), \tag{20}\]  
where \(\theta_{l}(s(t),a(t))\) is the transferred policy; \(\theta_{n}(s(t),a(t))\) is the new policy, which will be updated in every time slot by using Equation (19); and \(\varepsilon(t)\) is the transfer rate, which will be reduced after each time step to gradually remove the effect of the transferred policy on the new one. The training process of the actor-critic learning framework for the SU to decide its operation mode is illustrated as follows. At the beginning of the \(t\)th time slot, the SU chooses an action according to policy \(\pi\) considering the sensing result and the remaining energy in its battery. The SU can decide to stay idle, \(a(t)=1D\), to save energy, or it can transmit the encrypted data, \(a(t)=TR_{Nk}\), to the FC. The immediate reward, \(R(s(t),a(t))\), and the next state, \(s(t+1)\), are updated at the end of the time slot based on the following cases.