# Applying Reinforcement Learning to Protect Deep Neural Networks from Soft Errors
## 4 Case Study
### Training the DRL-Based Agent with a Specific Layer  
From Figure 5, we observe that input and intermediate layers are significantly affected by the soft errors. Considering the extensive parameters in the intermediate layers (e.g., Layer 8 of the ResNet-18 contains more than 140,000 parameters), we select Layer 2 of the MLP and Layer 1 of the ResNet-18, which have fewer parameters compared to the intermediate layers, to serve as the environment for the DRL-based agent. The agent consists of two MLP networks as a target network and a policy network. Each of them contains three hidden dense layers with sizes of 64, 128, and 128, and the output neurons of the first two hidden layers are activated by Leaky Relu functions. The generated actions from the last layer is activated by a Relu function. We configure the initialized bit masks as \(\mathbf{m}_{0}=(\mathbf{0})\). The discount factor is set as \(\gamma=0.99\) with the learning rate \(l_{r}=0.001\). During the training of the agent, we use the Adam optimizer for stochastic gradient descent. We adopt the \(\epsilon-\)greedy algorithm to balance the efficiency of exploring and exploiting with different state-actions pairs. At the start of the training episodes \(M\), \(\epsilon\) could be configured with 0.5. After some iterations of the training episodes, we improve the efficiency to exploit the optimal actions by increasing the value of \(\epsilon\). We propose the requirements with \(\mathcal{P}_{0}\) and \(\sigma_{0}\) to indicate the threshold of the desired metrics of the DNNs. To converge the training of the DRL-based agent, we mark the terminal conditions for ending the loop of each episode as the bit masks satisfy the requirements instead of iterating the entire loop. Such an implementation could decrease the amount of training episodes and avoid the over-fitting. The overall training algorithm incorporated to the fault injection service is listed in Algorithm 1. When the performance and deviation in different episodes always meet the requirements (line 14 in Algorithm 1), then the agent is treated as converge. ```
0: Configurations of the agent \(\triangleright\) such as \(\gamma,l_{r},l\)
0: Tailored bit masks \(\mathbf{m}^{l}\) for a specific layer \(l\)
1:for\(k\gets 1\) to \(M\)do\(\triangleright\) Start episodes, \(M=200\)
2: Initialize the bit masks \(\mathbf{m}^{l}_{0}\)
3: Configure the fault parameters \(p_{c}\)\(\triangleright\) Equation (1)
4: inject faults into a specific layer \(l\)
5:for\(i=1\) to \(T\)do\(\triangleright\) Start iterations, \(T=500\)
6: Run a random seed \(p_{t}\)
7:if\(p_{t}\geq p_{c}\)then\(\triangleright\)\(\epsilon\)-greedy algorithm
8: Select a random action \(\mathbf{a}^{l}_{i}\)
9:else
10: Select an action \(\mathbf{a}_{i}\) by the DQN
11:endif
12: Execute \(\mathbf{m}^{l}_{i}\leftarrow\mathbf{a}^{l}_{i}\)\(\triangleright\) Equations (4) and (5)
13: Protect the weights with bit masks \(\mathbf{m}^{l}_{i}\)
14:if\(\mathcal{P}^{l}_{i}\geq\mathcal{P}_{0}\) and \(\sigma_{t}\leq\sigma_{0}\)then\(\triangleright\) Equation (2)
15: Break the iteration and start new episodes
16:else
17: Continue the iterations
18:endif
19: Optimize DQN by gradient descent
20:endif
21:endfor
22: Execute \(\mathbf{m}^{l}=\mathbf{m}^{l}_{i}\)
23:return \(\mathbf{m}^{l}\)
```  
**Algorithm 1** Training the DRL-based Agent for a Specific Layer  
A sample of training the agent with the MLP and ResNet-18 can be found in Figure 5(a),d. The upper and lower bounds of the red region in the performance's curves refer to \(\tilde{P}_{t}\pm\sigma_{t}\). All these tests run on the hardware configuration with Intel i7-9750H and NVIDIA Geforce GTX 1660 Ti. In these experiments, we configure \(r^{i}_{e}\) as a normal distribution and \(r_{b}=20\%\). We set \(f_{i},f_{p}\) as a scaling function with \(\alpha=1.15\) and \(\beta=1\)to normalize the output. By converging the training of the agent, we observe that the bit masks finally stabilize to maintain the performance by protecting a few bits of each weight.