# Applying Reinforcement Learning to Protect Deep Neural Networks from Soft Errors
## 5 Conclusions and Future Work  
This paper presents a dynamic error protection method by combining RL with transfer learning to protect DNNs from soft errors. The proposed method can dynamically and flexibly identify and address the vulnerable bits from each layer of the DNNs. By comparing with the current work, we summarize the contribution of the proposed methods as follows: (1) Compared to current work in functional fault injection [2, 5, 15, 24, 25], we introduce a simulated model for injecting faults within the weights of the DNNs, which enables RL-based methods in addressing vulnerable bits in the early function design phase. (2) Compared with conventional protection schemes that protect arbitrary bits, we train a layer-wise agent with the consideration of the resiliency revealed by the fault injection service. As a result, this agent dynamically selects vulnerable bits by balancing the performance and redundancy. (3) Compared with the related work by using RL-based agents in [50, 51], our work optimizes the time consumption by training a layer-wise agent instead of using global and local ones. By specifying an explicit reward mechanism, the proposed layer-wise agent intends to suppress the hyper-parameters' impact on the agent, avoiding configuring sophisticated reward functions in [50, 51]. (4) With the help of the prior knowledge acquired by layer-wise resiliency analysis, the proposed method accelerates the agent's convergence by adopting transfer learning. The proposed method performs flexibility to configure the number of protected bits regarding the prior knowledge. Moreover, the adoption of transfer learning enhances the potential to generalize the DRL-based agent to other DNN structures by refining the reshaped rewards. (5) In the experiments, ResNet-18, which is more complex than a simple MLP, still demonstrates significant efficiency, suggesting the potential for our framework to be extended to larger and more complex DNNs. Based on the above conclusions, we prepare to extend our work from the following aspects: (1) The implementation of protection schemes using bit masks generated by the DRL-based agent could be further strengthened by integrating it with specific hardware  
Figure 8: Performance between different protection schemes with soft errors. configurations to optimize the proposed framework. (2) The design of DRL and transfer learning can be further enhanced through various optimization techniques (e.g., knowledge distillation) to improve the training efficiency. For example, the hyperparameters within this framework can be fine-tuned in accordance with specific system requirements. Specifically, the correlation between the specification of numerical rewards and injected faults can be further parameterized to mitigate the coupling effects caused by the random fault injection presented in Task I. In addition, the algorithmic optimization can be further used in the framework to improve the training efficiency. For example, prioritized replay buffers offer an enhanced version of the standard replay buffer by sampling transitions that are less frequent or more informative, thereby improving the agent's performance. (3) Given the detailed and realistic evaluation from microarchitecture fault injection, combining the RL-based agent with a technical fault injection seems promising. Utilizing the simulator gem5 and its fault injector gemifi, we prepare to conduct a detailed evaluation of the RL-based agent, taking into account timing and resource constraints. (4) The proposed framework can be further integrated into the design of safety-critical systems to ensure functional safety. For example, the DRL-based agent can be deployed online or offline in the learning-enabled components of the ADS to improve reliability. Specifically, during the technical design for deploying these components in hardware (e.g., FPGA/ASIC), the proposed framework provides solutions to enhance functional safety by identifying and addressing vulnerabilities in the context of bit-level data.