# Comparing Application-Level Hardening Techniques for Neural Networks on GPUs
## 3 Proposed Evaluation Methodology  
This section introduces the evaluation methodology used in this work to assess a set of application-level HTs implemented on a representative collection of NNs from the edge AI domain. Figure 4 depicts the three main steps involved during the evaluation process: (i) HT implementation, (ii) reliability assessment through hardware-aware fault injection campaigns, and (iii) data analysis.  
The first step involves the implementation of the HTs on the NN architectures. For this purpose, a set of NN architectures are modified according to the HTs (e.g., changing the order of the layers or including an additional operation in the original NN architecture).  
This stage involves the implementation of the HT on a set of target NN architectures. To make such HT successful, it is necessary to consider two main aspects. The first one corresponds to the required changes or degree of modifications required to implement a given HT on an NN model. For example, an HT can involve replacing a ReLU activation function with a ReLU6 activation function. This modification is straightforward and can be implemented simply by substituting one layer for another. However, some strategies are more complex and require deeper modifications to the NN, such as adding layers that include custom operations, (e.g., a median filter or TMR operations). On the other hand, the HTs may involve only fine-tuning the model by using the pre-trained weights of the original architecture. In contrast, another HT implementation might involve training the NN using a hyperparameter configuration that significantly differs from the one used to train the original NN (e.g., the required number of epochs can increase with respect to the original NN training process).  
The second step consists of an experimental stage that conducts the reliability assessment to assess the implemented HT's reliability enhancement capabilities. Specifically, FI campaigns are conducted at the application level resorting to 2 hardware-aware error models: (i) WSBF and (ii) NBER.  
The last step in our evaluation procedure involves an evaluation that targets (i) the HT implementation impact and (ii) the effectiveness of the HTs as fault countermeasures. Remarkably, the implementations of the considered HTs require modifications in the NN architecture, which can increase inference time. Additionally, this new architecture may require an extra training phase, and there is a possibility that the new neural network may not reach the same level of convergence as the original architecture. For this reason,  
Figure 4: Hardening technique evaluation procedure.  
during the HT implementation impact evaluation, we evaluated the impact on the NN accuracy (quantitative assessment), the required effort to make HTs successful (qualitative evaluation) and the introduced overhead (overhead estimation), whereas in the faulty scenarios (during WSBF FI and NBER FI), we evaluated the NN accuracy degradation.  
Concerning the HT effectiveness study, we have conducted a quantitative assessment on the NN accuracy degradation recorded after the WSBF and NBER FI campaigns.