# Comparing Application-Level Hardening Techniques for Neural Networks on GPUs
## 5 Experimental Results
### HT Impact Assessment  
#### 5.1.1 Trading off EoI and Accuracy  
Table 4 shows the results of the qualitative and quantitative studies conducted on the HTs when implemented on the NN architectures under test. This study provides a detailed view of the impact of HTs on NN architectures immediately after their implementation, focusing on the effort involved and the accuracy of the resulting model. In particular, Table 4 illustrates the EoI as the composition of complexity and training adaptations, the accuracy of the resulting model and the accuracy of the model without any HT implemented. As an initial result, it is important to note that not all hardening techniques, when their implementation includes a training step, allow for the convergence of the training algorithm. For example, median filter requires a significant effort to be adapted. In fact, it does not guarantee training algorithm convergence for all NN architectures at an acceptable level of test accuracy. Specifically, the training of Mnasnet, Mobilenet, and Squeezenet converged to 35.40%, 54.50%, and 62.34% accuracy, respectively. Nowadays, most NNs used for Edge AI applications (e.g., Mnasnet, Mobilenet, and Squeezenet) involve an optimized version of the convolution algorithm, which resorts to the grouping strategy. This strategy allows the optimization of the RAM footprint and the inference time simultaneously by convolving only a group of channels with a corresponding group of kernels, thus avoiding the computation of redundant operations. A drawback of the grouping strategy is that the processed tensors are shaped according to the optimized convolution algorithm, which makes the implementation of median filter at the application level (requiring a window computing median and sliding along the tensor channel dimension) almost impractical. Similarly, Mnasnet's training algorithm converged to the accuracy of 35.40% when Swap ReLU was implemented. By observing the distribution of the Mnasnet intermediate feature, we have noticed a higher variance than the intermediate feature of other NN architectures (up to 10%). Consequently, the strict activation functions upper bounding to 6 makes the convergence of the Mnasnet training algorithm impractical. \begin{table}
\begin{tabular}{c c} \hline _Operation_ & _Weight_ \\ \hline Comparison & 1 \\ Addition & 2 \\ Subtraction & 2 \\ Multiplication & 3 \\ Division & 4 \\ \hline \end{tabular}
\end{table} 
Table 3: Assigned weights for each MAC operation. In Lenet5, the Ranger method achieves an EoI score of \(1/10\). This low score is attributed to the use of only three activation functions, which contributes to a complexity rating of \(1/5\). Additionally, Ranger does not require retraining and maintains the baseline accuracy of \(98\%\). Similarly, the Adaptive Clipper method also does not experience any drop in accuracy when applied to Lenet5. It has a relatively low EoI score of \(4/5\), reflecting a complexity of \(1/10\). This is due to the fact that the EoI score of \(1/10\) is not a significant function of \(1/5\). \(1/5\) and a training adaptation degree of \(3/5\). However, it is important to note that Adaptive Clipper requires retraining using the same setup as the baseline model. Fine-grain TMR also maintains its accuracy with no drop. Even though it can be implemented without additional training, the introduction of new layers results in a complexity rating of \(5/5\). In contrast, both Swap ReLU6 and median filter methods have higher EOI scores, with \(7/10\) and \(10/10\), respectively. These methods involve substantial changes to the neural network (NN) architecture, such as altering the order of sequential layers and adding new custom operations. Consequently, they are assigned complexities of \(4/5\) and \(5/5\), respectively. Regarding training adaptations, Swap ReLU6 requires only fine-tuning of the NN with fewer epochs than the original training, while the median filter necessitates retraining with a different hyperparameter configuration.