# Applying Reinforcement Learning to Protect Deep Neural Networks from Soft Errors
## 3 Methodology
### Task III: Generating Bit Masks by a RL-Based Agent  
A conventional error protection and correction scheme (e.g., Hamming Code) usually protects a few arbitrary bits from soft errors [59]. Although such a protection scheme has been widely used, it is insufficient for soft errors with multiple vulnerable bits. Moreover, such protection schemes usually protect arbitrary bits without considering their significance in the context of DNN operations. However, given the extensive parameters of the DNNs, efficiently addressing the vulnerable bits across the DNNs is challenging due to the following reasons: (1) Soft errors can have varying impacts across the DNNs--specifically, errors occurring in different layers can lead to different levels of performance degradation and result in varying numbers of vulnerable bits. (2) With more neurons in the layers, analyzing and addressing vulnerable bits become inefficient due to the need to inject faults  
Figure 3: The development flow of the proposed fault injection service. The dash box represents the contribution of our proposed method. across a proportion of neurons. (3) With deeper networks, it is time-consuming to identify vulnerable bits by iterating layer by layer. To address these challenges, an RL-based solution is developed to dynamically and efficiently identify the vulnerable bits by interacting the agent and the environment. The backbone of the interaction is to create a Markov Decision Process (MDP) that contains the following elements [60; 61]: \(\langle\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma\rangle\). \(\mathcal{S}\) refers to the state space, and \(\mathcal{A}\) refers to a set of actions. \(\mathcal{T}(\mathbf{s}_{t+1}|(\mathbf{s}_{t},\mathbf{a}_{t}))\) describes transition dynamics from the current states \(\mathbf{s}_{t}\) to the next states \(\mathbf{s}_{t+1}\) at timestamp \(t\) by taking an action \(\mathbf{a}_{t}\). \(r\) refers to an immediate reward regarding consequences of a policy \(\pi(\mathbf{s},\mathbf{a})\) based on the state-action pairs \((\mathbf{s}_{t},\mathbf{a}_{t})\). A discount factor \(\gamma\in[0,1]\) is used to offset the impact of the immediate reward \(r\). To evaluate the long-term rewards \(R\) of current state-action pairs, we use the following function [61]:  
\[Q^{\pi}(\mathbf{s}_{t},\mathbf{a}_{t})=\mathbb{E}[R|\mathbf{s}_{t},\mathbf{a}_{t},\pi(\mathbf{s}_{ t},\mathbf{a}_{t})] \tag{3}\]  
The configuration of actions and states considers the following aspects to encourage the training of the RL agent: (1) It is impractical to implement the protection scheme with small memory granularity (e.g., each weight assigned to an individual bit mask) in the memory hierarchy. Therefore, the protection scheme should be flexible to be deployed in the context of DNNs. (2) The protection scheme should leverage the resiliency in terms of layers to decrease the protection cost (e.g., the protection number of bits). Thus, we protect the parameters (e.g., weights) by specifying a layer-wise bit mask. To be specific, we propose an RL-based agent by generating bit masks to protect specific bits in the \(l\)-th layer. Such a bit mask can be formulated as follows:  
\[\mathbf{m}^{l}=(m^{l}_{0},m^{l}_{1},\dots,m^{l}_{w-1})\in(0,1)^{w} \tag{4}\]  
where \(w\) refers to the number of bits requiring to store a parameter from the \(l\)-th layer. \(m^{l}_{i}=1\) refers to protect bit \(i\) of every weight in the layer \(l\). By selecting the protected bits of parameters (e.g., weights in a layer of the DNNs), the state space \(\mathcal{S}\) contains bit mask with different combinations, which dynamically determines the efficiency of the protection. As an example, Figure 4 shows that the weights of the DNNs store in different memory locations. By assigning the bit masks regarding the resiliency, it is efficient to protect the parameters of a layer from soft errors. As shown in Figure 1, the actions of the RL-based agent involve marking the bit positions within the masks \(m^{l}\) to explore and exploit the vulnerable bits.