# Applying Reinforcement Learning to Protect Deep Neural Networks from Soft Errors
## 3 Methodology
### Task IV: Deploying the Agent with Transfer Learning  
Although this DRL-based agent could protect the parameters in a specific layer, such a protection needs to become more effective across the DNNs. Considering the complexity of the state and action spaces, it is impractical to train such an agent for each layer in terms of time consumption and scalability. To illustrate these issues, we utilize a five-layer neural network as an example: the action space of bit masks contains \(2^{w}\) actions to protect weights within each layer; to obtain all potential optimal actions simultaneously following the methods in [50; 51], the action space increases from \(2^{w}\) to \(2^{5w}\). In addition, with an increase in the number of neurons in deeper layers, training the layer-wise DRL agent becomes more time-consuming due to the faults' need to be injected across a proportion of neurons. To cope with this issue, transfer learning provides a solution to accelerate the learning process by reusing the knowledge of a trained agent with similar scenarios [62]. Additionally, with the results from the fault injection service, the layer-wise resiliency of the DNNs can expose prior knowledge for designing the reward functions. Such prior knowledge could stimulate the convergence of training the agent [63]. As a common method of transfer learning, we adopt the reward shaping by formulating the MDP as follows [62; 64; 65]:  
\[\mathcal{M}=<\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma>\rightarrow\mathcal{ M}^{\prime}=<\mathcal{S},\mathcal{A},\mathcal{T},r^{\prime},\gamma> \tag{9}\]  
where \(\mathcal{M}\) and \(\mathcal{M}^{\prime}\) refer to the source and target domain for transferring the knowledge. To be specific, \(\mathcal{M}\) is the MDP, which is trained at the \(l\)-th layer in Section 3.3. \(\mathcal{M}^{\prime}\) is the MDP for the other layers. \(r^{\prime}\) refers to the newly shaped rewards, denoting to \(r^{\prime}=r^{l}+r_{e}\). We specify \(r_{e}\) as \(r_{e}^{r^{\prime}}\) regarding the layer index \(n^{\prime}\), where \(n^{\prime}\
eq l\). We define the reshaped reward \(r_{e}^{r^{\prime}}\) as Equation (10) for balancing the number of protected bits with the prior knowledge concluded from the fault injection service.  
\[r_{e}^{n^{\prime}}=-\tau\cdot(\sum_{i=0}^{w}\mathbf{I}(m_{i}^{n^{\prime}})- \sigma_{n^{\prime}}) \tag{10}\]  
where \(\tau\) refers to a coefficient to adjust the importance of the prior knowledge for training agents. \(m_{i}^{n^{\prime}}\) refers to the protected bit \(i\) in a specific layer \(n^{\prime}\). The number of protected bits estimated by the DQN-based agent is formulated as \(\sum_{i=0}^{w}\mathbf{I}(m_{i}^{n^{\prime}})\). By combining the knowledge from bit masks trained in the specific layers with the resiliency analysis, \(\sigma_{n^{\prime}}\) refers to an approximated number of protected bits. For example, \(\sigma_{n^{\prime}}\) could be larger in some non-trivial layers (e.g., the first layer of the DNNs) than others. This threshold is determined by the resiliency of each layer under fault injection, which helps reduce the search cost within the state space. When the number of protected bits estimated by the DQN-based agent exceeds the threshold defined by prior knowledge, the agent receives a penalty and is encouraged to continue exploring. Conversely, if the estimated number of protected bits is within the threshold, the agent receives a reward and is encouraged to exploit its current policy to optimize performance. Equation (10) implies that transfer learning encourages the agent to protect the layers regarding approximated protected bit numbers from prior knowledge. In particular, when attaining a similar performance, the agent receives a penalty if the amount of bit masks from the agent is more significant than the approximated one.