# Applying Reinforcement Learning to Protect Deep Neural Networks from Soft Errors
## 4 Case Study
### Ablation Study of Transfer Learning  
In Section 4.2, we find that training a layer-wise agent in MLP and ResNet-18 requires 31,668 s and 27,675 s. Obviously, this time cost is impractical to learn all the layers with the similar workload. In addition, with more neurons in the deeper layers, the training of the agents is more time-consuming. Therefore, regarding Equations (9) and (10), we use transfer learning by modifying the rewards and reusing the agent in Algorithm 1. We configure \(\tau=0.5\) to leverage the current and approximated bit masks. According to the bit masks \(\mathbf{m}_{mlp}^{2},\mathbf{m}_{res}^{1}\) learned from Section 4.2, we specify \(\sigma_{n^{\prime}}=2\) for the rest of the layers of the MLP. Considering the complexity of the ResNet-18, \(\sigma_{n^{\prime}}=3\) is used for the critical layers, which are those that exhibit significant performance degradation by fault injection simulations in Figure 5 (e.g., Layers 7, 8, and 9). \(\sigma_{n^{\prime}}=2\) is used for the rest of the parts of the networks (e.g., Layers 2, 3, and 4). To implement transfer learning in the remaining layers, we reload the parameters of the DRL-based agent trained by Algorithm 1 after the training concludes, as described in Algorithm 1. Next, the DRL-based agent with transfer learning continues the training with  
Figure 6: Snapshot during training the agents with their performance, rewards, and redundancy. The red boundaries within the figures denote the upper and lower values observed across various training episodes. reshaped rewards \(\sigma_{n^{\prime}}\) following Equation (10). The terminal states of this agent follow the condition defined in line 14-15 in Algorithm 1. In Figure 6b,e, we present the performance of bit masks by adopting transfer learning in Layer 1 and Layer 14 of the MLP and ResNet-18, which are sensitive layers to soft errors. Compared with Figure 6a,d, these agents spend fewer episodes to converge the training while still maintaining the performance of the DNNs. Moreover, these layers' protected bits are approaching even less than the bit numbers \(\sigma_{n^{\prime}}\) defined by prior knowledge. Figure 6c,f refer to adopting the transfer learning to train the agents for the robust layers shown in resiliency analysis. The results from these layers, including Layer 4 of the MLP and the Layer 3 of ResNet-18, present that the agents use a few episodes to attain acceptable bit masks. Although soft errors in these layers could slightly impact the performance, the bit masks still show superior results to the baseline. To this end, we conduct an ablation study in Figure 7 by comparing the adaptation of transfer learning with the training approach outlined in [50,51], where the layer-wise agents are trained to protect bits across the entire DNN simultaneously. In this figure, we also present the number and index of vulnerable bits by configuring different \(\sigma_{n^{\prime}}\) during transfer learning. Figure 7a,b present the comparison of time cost between using RL and transfer learning. By adopting transfer learning, the trend of training time consumption is significantly decreased. Compared to the RL-based method without transfer learning, the training time of the agent for the MLP is reduced by an average of 62.76%, decreasing from 72,170 s to 26,874 s, and for ResNet-16, it is reduced by 44.03%, decreasing from 652,225 s to 363,991 s. We observe that there is one exception in Figure 7b, where Layer 16 of ResNet-18 spends more time than the RL-based training method. The potential rationale behind this phenomenon lies in balancing the resiliency of the layers with the \(\sigma_{n^{\prime}}\) assigned by prior knowledge. Such a design attempts to spend more time to reach the desired performance as well as the optimal number of protected bits. In fact, the agent in this layer suggests that the bit masks protect 3 bits, while \(\sigma_{n^{\prime}}=2\). Such a conflict implies the consistency of the rationale. Figure 7c,d show the number of protected bits and their allocation in each layer.