# Comparing Application-Level Hardening Techniques for Neural Networks on GPUs
## 5 Experimental Results
### HT Impact Assessment
#### 5.1.2 HT Inference Time Overhead  
While some hardening techniques demonstrate a certain degree of EOI and accuracy degradation, they may also result in an increased computation overhead, manifested as an increased inference execution time with respect to the NN architecture without HT incorporated. Thus, to extend the initial comparison of HTs during the study of the HT implementation impact, in Table 5, we report an estimation of the ITO based on the basic operations (i.e., comparison, addition, subtraction, multiplication, and division) that each HT requires when implemented on a specific NN architecture. In the first column, a weighted average of the absolute number of operations for the Baseline model is reported (i.e., the NN architecture without HT incorporated). The weighted average number of operations was estimated using equation Equation (5) considering the equivalent number of basic operations for every NN. Subsequently, the overhead with respect to the NN without HT implemented is computed using Equation (4). We can notice that Adaptive Clipper introduces a negligible overhead at inference time (i.e., less than 1%) for Lenet5, Resnet18, and Squeezenet, while for Mnasnet and Mobilenet V2, the overhead increases to 3.52% and 2.24%, respectively. Since Adaptive Clipper bounds the activation functions of the original NN architecture, the introduced overhead depends on the number of activation functions included in the original NN. On the other hand, Ranger shows a higher overhead than Adaptive Clipper because it includes additional checks after every functional block. In fact, regardless of the NN, the \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{_NN Architecture_} & \multirow{2}{*}{\
\begin{tabular}{c} _Weighted_ \\ _Average Number_ \\ _of Operations_ \\ \end{tabular} } & \multicolumn{5}{c}{_Inference Time Overhead_} \\ \cline{3-7}  & & _AC_ & \(R\) & _SR_ & _MF_ &
\begin{tabular}{c} _Fine-Grain_ \\ _TMR_ \\ \end{tabular} \\ \hline Lenet5 & 97,709.45 & 0.29\% & 2.29\% & 0.29\% & 2.29\% & 200\% \\ \hline Mnasnet & 1,007,184 & 3.52\% & 7.54\% & - & - & 200\% \\ \hline Mobilenet V2 & 1,787,803.62 & 2.24\% & 1.93\% & 2.24\% & - & 200\% \\ \hline Resnet18 & 6,992,621 & 0.30\% & 0.65\% & 0.30\% & 0.65\% & 200\% \\ \hline Squeezenet & 1,776,322.87 & 0.88\% & 2.16\% & 0.88\% & - & 200\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Inference time overhead estimate based on the number of the additional MAC operations when the HT is implemented. In this table, the HT names are abbreviated as follows: Baseline (BL), Adaptive Clipper (AC), Swap ReLU6 (SR) median filter (MF). overhead of Ranger increases by at least 2\(\times\) with respect to the overhead introduced by Adaptive Clipper. In the case of Swap ReLU6, swapping the positions of batch normalization and ReLU activation functions does not add any overhead. However, the bounding introduced by the activation function results in an overhead similar to that caused by Adaptive Clipper. Furthermore, the main idea behind implementing the median filter involves sorting a sliding window of length \(n\). For all implementations of median filter, n is always set to 3, and the sorting operation is performed for each convolutional and linear layer. Consequently, the expected median filter is \(\approx 9\times\) larger than Ranger HT overhead. The optimized implementation of the median operation, as proposed by the authors of [13], achieves a high degree of parallelization that reduces the overhead to approximately 3\(\times\) that of Ranger HT. Finally, fine-grain TMR reaches a significant overhead (i.e., \(\approx 200\%\)) since it replicates each operation 3 times (neglecting the overhead due to the majority voting step as it would have introduced a negligible overhead with respect to the triple computation of the same operation). More in general, our study indicates that range-restriction-based HTs, specifically Ranger and Adaptive Clipper, exhibit the best balance between implementation effort and NN accuracy.