# Comparing Application-Level Hardening Techniques for Neural Networks on GPUs
## 3 Proposed Evaluation Methodology
### Data Analysis  
#### 3.2.1 HTs Implementation Impact Evaluation  
To conduct the study on the impact of the HT implementation, we introduced the effort of implementation (EoI) that contributes to better grade in an objective and qualitative way, the best HT to be implemented considering 2 main features: complexity and training adaptations. As shown in Table 1, we assigned an EoI score ranging from 0 to 10, which is based on the composition of two features. The first feature is called _complexity_, and it can range from 0 to 5. A higher score indicates that more topological modifications are required for the implemented HT design. We considered assigning a complexity from score 1 to 3 when the layers of the original NN topology are replaced with another one, while complexity is assigned a score of 4 or 5 when an additional layer is introduced within the original NN topology. _Training adaptations_ also can range from 0 to 5. This feature measures the extent of differences between the training process of the original NN architecture and the training process needed for the HT implementation, with higher values assigned for more significant modifications. A lower degree of training adaptations (scores of 1 or 2) indicates that the new neural network (NN) architecture only needs fine-tuning. This fine-tuning can either follow the original hyperparameter configuration (score = 2) or make the training process lighter than the original setup (score = 1). A score of 3 or higher is assigned when the new architecture requires retraining with a hyperparameter setup that may be similar (score = 3) or different (score > 3) from the original configuration. After setting up the new NN topology with the HT implementation, we evaluate the accuracy with Equation (3) according to [44]. \[Acc=\frac{number\_of\_correct\_predictions}{total\_number\_of\_predictions} \tag{3}\]  
where \(number\_of\_correct\_predictions\) indicates the number of elements that the NN correctly classified, while the \(total\_number\_of\_predictions\) represents the overall samples on which the NN made inferences. During our impact assessment on the HT implementation, we considered the computational overhead introduced in terms of increased algorithm execution time. Given the necessity for a near real-time response from the algorithm running on the device in focus for this work, it is crucial to quantify the computational overhead introduced by the HT implementation. Consequently, we evaluate the inference time overhead (ITO) with Equation (4). \[ITO=\frac{IT_{hard}-IT_{base}}{IT_{base}}\times 100 \tag{4}\]  
where \(IT_{hard}\) and \(IT_{base}\) are the execution times of the NN inference when the HT is implemented and when it is not implemented, respectively. Since the execution time depends on several factors (e.g., software libraries or the underlying hardware executing the NN inference), we created a technology-independent metric that estimates the inference time (IT), resorting to the count of the number of basic operations (i.e., comparison, addition, subtraction, multiplication, and division) that each HT adds, taking as reference the application level operations to the original NN workload. However, from the hardware perspective, each operation involves a different complexity in terms of bit-wise operations. For this reason, we evaluated the \(\Pi\) as a weighted average of the executed operations according to Equation (5). \[IT=\frac{(w_{comp}\cdot\#comps)+(w_{add}\cdot\#adds)+(w_{sub}\cdot\#subs)+(w_ {mul}\cdot\#muls)+(w_{div}\cdot\#divs)}{w_{comp}+w_{add}+w_{sub}+w_{mul}+w_{div}} \tag{5}\]  
where #comps, #adds, #subs, #muls, and #divs represent the number of executed comparison, addition, subtraction, multiplication, and division operations, respectively,