# Applying Reinforcement Learning to Protect Deep Neural Networks from Soft Errors
## 3 Methodology
### Task III: Generating Bit Masks by a RL-Based Agent
We define the action \(\mathbf{a}^{l}\) as follows:  
\[\mathbf{a}^{l}=(a^{l}_{0},a^{l}_{1},\dots,a^{l}_{w-1}) \tag{5}\]  
Figure 4: An example to illustrate that bit masks protect weights from different layers by selecting vulnerable bits. where \(a_{i}^{l}\in\{0,1\}\) refers to the actions for configuring the bit masks to protect bit position \(i\) at layer \(l\). The primary reason for selecting binary representations of actions is to reduce the size of the state and action spaces. The size of the action space using this binary representation is \(|\mathcal{A}|\), denoted as \(2^{w}\), which is less than the agent sample's actions from a continuous distribution (e.g., a Gaussian distribution).As an example of generating bit masks via current actions, Figure 4 presents how the initial bit masks \(\mathbf{m}_{0}^{l}=(\mathbf{0})\) are modified by actions \(\mathbf{a}_{t}^{l}\) at step \(t\), where \(a_{1}^{l}\) and \(a_{5}^{l}\) are set to \(1\) to produce \(\mathbf{m}_{t}^{l}\). In addition, to encourage the agent to explore and exploit vulnerable bits in the next step, it receives rewards based on the policy \(\pi(\mathbf{m}_{t}^{l},\mathbf{a}_{t}^{l})\). We design the reward function by considering the following aspects: (1) The reward should encourage the generation of bit masks by the RL-based agent to protect the parameters of DNNs from soft errors. (2) The reward should balance the performance and redundancy cost by exploring and exploiting vulnerable bits. Therefore, the reward function designed for layer \(l\) at timestamp \(t\) is modeled as follows:  
\[r_{t}^{l}=\alpha\cdot f_{i}(\mathcal{P}_{0}-\hat{\mathcal{P}_{t}}^{p_{e}})- \beta\cdot f_{p}(\sum_{i=0}^{w}\mathbf{I}(m_{i}^{l})) \tag{6}\]  
where \(\mathcal{P}_{0}\) denotes the performance of the DNNs defined by the system requirements. \(\hat{\mathcal{P}_{t}}^{p_{e}}\) derived from Equation (2) refers to the performance after injecting faults with the fault probability \(p_{e}\). The functions \(f_{i}\) and \(f_{p}\) are scaling functions that normalize the performance and protected number of bits to facilitate the convergence of the DRL-based agent. Specifically, \(f_{i}(\cdot)\) is a function where the agent receives a greater reward when there is less discrepancy between the \(\mathcal{P}_{0}\) and \(\hat{\mathcal{P}_{t}}^{p_{e}}\). \(f_{p}(\cdot)\) represents that when more bit needs to be protected, the agent receives more penalty. When a specific bit is masked by \(m_{i}^{l}\), the indication function \(\mathbf{I}(m_{i}^{l})=1\). \(\alpha\), \(\beta\) refer to the coefficients to balance the performance and the number of masked bits. Based on the modeling of the design space and reward functions of the RL-agent, we conclude the states and actions are discrete with numerical rewards. Therefore, we adopt Deep Q-learning (DQN) [53], a model-free and off-policy algorithm, to approximate the optimal action-value functions \(Q^{*}(\mathbf{s},\mathbf{a})\), which is represented as follows:  
\[Q^{*}(\mathbf{s},\mathbf{a})=max_{\pi}\mathbb{E}[R_{t}|\mathbf{s}_{t}=\mathbf{s},\mathbf{a}_{t}= \mathbf{a},\pi(\mathbf{s},\mathbf{a})] \tag{7}\]  
where \(\mathbf{a}\) refers to the action for marking bits defined in Equation (5), and \(\mathbf{s}\) refers to the bit masks defined in Equation (4). To stable optimize and approximate the Q-value, the DQN utilizes the similar implementation in [53], where this Q-network contains a target network and a policy network with the same configurations and parameters. The weights of the target network are slowly updated by the learned policy network. We model this training process as Equation (8), which minimize loss function \(\mathcal{L}_{i}(\theta_{i})\) at each iteration \(i\):  
\[\mathcal{L}_{i}(\theta_{i})=\mathbb{E}_{\mathbf{s}\mathbf{a}\sim\rho(\cdot)}\Big{[}( r_{i}^{l}+\gamma\cdot Q(\mathbf{s},\mathbf{a};\theta_{i-1}^{\prime})-Q(\mathbf{s},\mathbf{a}; \theta_{i}))^{2}\Big{]} \tag{8}\]  
where \(\rho(\cdot)\) refers to the probability distribution of the state-action pairs.