# Applying Reinforcement Learning to Protect Deep Neural Networks from Soft Errors
## 1 Introduction  
With the advancement of Artificial Intelligence (AI), Deep Neural Networks (DNNs) are widely employed in sensor-based systems across various fields [1]. In these systems, DNNs usually are utilized in the perception of complex operational conditions by dealing with unstructured high-dimensional data. However, the extensive parameters within the DNN lead to the computational process that is difficult to interpret and explain, posing challenges for quality management of DNNs. In particular, the ability of DNNs to withstand potential functional faults is considered to be an important factor in the safety criticalsystems, such as Automated Driving Systems (ADSs) [2] and medical equipment [3]. As typical functional faults, soft errors caused by radiation, aging, device variation, and temperature [4] can affect a small proportion of the data within DNNs [5]. Consequently, they weaken robustness and threaten safety by leading unexpected behaviors and system failure. Some related work [4; 5; 6] shows soft errors such as bit-flips that could lead to the DNN misclassifying a critical object (e.g., a truck) as a trivial one (e.g., a bird). Based on the reasoning of related system hazards, industrial standards (e.g., IEC 61508 [7] and ISO 26262 [8] ) advise systematic approaches to guarantee functional safety by fault avoidance, fault tolerance, and fault detection [9]. In addition, soft errors--particularly bit-flips--can also act as a form of attack that threatens the security of distributed intelligent systems, such as the Internet of Vehicles (IoV) and Unmanned Aerial Vehicle (UAV) swarms [10]. For example, sensor data integrated into IoV systems is often used by DNNs to determine vehicle positions. An attacker could exploit this by flipping specific bits to falsify the input data [11]. Therefore, enhancing the security of these systems requires a thorough understanding of, and protection against, the impact of soft errors on DNNs. Conventional fault-tolerant techniques [12], such as replicating critical components via Triple Module Redundancy (TMR) [13] or inspecting vital outputs via symptom of neurons within DNN [14], pose challenges in efficiently and effectively achieving fault tolerance. Such challenges often arise from the difficulty in effectively addressing and identifying vulnerabilities within DNNs. For example, while recent studies suggest that certain layers of DNNs are more susceptible [2; 15], such findings may not provide sufficient granularity for redundancy design or checksum validation in certain safety-critical applications, particularly in embedded systems with AI techniques, where physical sizes and economic costs are limiting factors [16]. Current studies propose contributions from various strategies [4; 17], including optimizing DNN architectures, algorithms, microarchitecture, and hardware implementation to address these issues. These approaches enable DNN protection and reliability analysis across the functional and technical designs, as well as the Validation and Verification (V&V) process [18]. However, there remains a need for increased effort in protecting DNNs from soft errors by dynamically and efficiently identifying vulnerable bits. Meanwhile, the identification of vulnerable bits can play a critical role during functional design exploration within safety-critical systems (e.g., ADS) to support the design of necessary safety mechanisms that provide the protection [18; 19; 20]. Therefore, we propose a Reinforcement Learning (RL)-based method to protect DNNs from soft errors. Regarding soft errors impacting DNN performance, the RL-based agent dynamically selects and protects vulnerable bits by generating bit masks. To improve the efficiency of the proposed method, we adopt transfer learning across DNNs by utilizing the layer-wise resiliency acquired from fault injection simulation as the prior knowledge for reducing the training time. As a case study, we select different DNNs to test and validate the proposed approach.