# Applying Reinforcement Learning to Protect Deep Neural Networks from Soft Errors
## 3 Methodology
### Task I: Simulating Faults Within DNNs  
As previously mentioned, current functional fault injection methods typically manipulate parameters without considering memory resource constraints [2; 5; 15; 24; 25]. As a result, faults are usually injected into parameters as independent and identically distributed (i.i.d.) events, disregarding the occurrence of faults in the context of memory allocation. To address this issue, the proposed framework introduces an simulated memory model for fault injection within the DNN parameters. The design of this simulated model follows the memory hierarchy of DNN accelerators. Specifically, a DNN accelerator usually consists of the Processing Elements (PEs) to process the data and memory hierarchy to store the parameters of the PE [4; 5]. Compared with potential faults occurred in the PE, previous research [4; 5; 6; 32] shows that fetching data (e.g., parameters of DNNs) from the memory hierarchy could be more vulnerable due to the highly frequent operations. Therefore, we mainly focus on the fault occurring in parameters of DNNs within the memory. To inject these faults, we firstly present a schematic of a typical DNN accelerator, as shown in Figure 2.  
The memory hierarchy usually organizes extensive memory cells by words and entries so that each operation fetches a batch of parameters from specific entries. To characterize the operations from different entries, we allocate the parameters including weights and bias to this memory hierarchy. Such a hierarchy is analogous to the HBM2 memory, which is a common type of Dynamic Random Access Memory (DRAM) used in DNN accelerators [6; 26]. To be specific, the parameters of the target DNN are converted to a generic machine-readable format (e.g., floating bytes) and then stored in the simulated memory representing the memory hierarchy. To design this simulated model for supporting the faults to be injected, we configure the memory hierarchy, which is similar to the memory in [6]. This memory contains multiple entries to store batches of parameters as a set of continuous words. These words are strings of bytes converted from the parameters of DNNs. A batch of parameters from the same layer is stored in the same entry in sequence until all of the parameters are loaded or these entries are fully stacked. To define soft errors occuring in these units, we assign each entry \(i\) with an independent error distribution \(r_{e}^{i}\). Meanwhile, we use Bit Error Rate (BER) \(r_{b}\) to indicate the number of bit errors per entry. To this end, the fault probability \(p_{e}\) of the simulated memory for storing the parameters of DNNs can then be formulated as follows:  
\[p_{e}=\sum_{i=1}^{N_{e}}r_{e}^{i}\cdot r_{b} \tag{1}\]  
Figure 2: A simple structure of DNN accelerator. Depending on the DNN layer-wise characteristics (e.g., the dimensions), the parameters from the PE could be saved either in the same entry or different entries of the memory. Therefore, the fault rate \(p_{e}\) could impact the parameters (e.g., parameter #a and parameter #b) when they store in different entries. where \(N_{e}\) refers to the number of entries, determined by the number of parameters to be stored in the simulated model and the size specification for each entry. To this end, we present the development flow based on the proposed fault injection service shown in Figure 3 as follows: (1) creating a simulated model by configuring the size of entries and words; (2) extracting the weights from pre-trained DNN models; (3) allocating the weights sequentially based on the constraints of the simulated model; (4) injecting the soft errors by assigning fault parameters within Equation (1). Specifically, the fault injection simulation is configured by injecting bit-flips in the weight values of the DNNs, with a fault probability \(p_{e}\) affecting different memory entries.