# Comparing Application-Level Hardening Techniques for Neural Networks on GPUs
## 5 Experimental Results
### Fault Mitigation with Respect to WSBFs  
Figure 5 allows us to compare the effectiveness of different HTs with respect to the WSBF error model. Figure 5 reports the classification of the WSBF according to their impact on the NN outcomes. The results indicate that due to the redundancy introduced by the fine-grain TMR (independent NN kernel copies), this strategy masks the impact of the injected WSBF. Therefore, we opted not to report the results of the corresponding FI campaigns for this case. In general, the experimental results show that according to the NN architecture on which the HT is implemented, the effectiveness of the hardening mechanism can quantitatively differ. In Lenet5, Adaptive Clipper outperforms the other HTs, reducing the baseline critical SDC percentage from 3.31% to 0.85% by transforming most of the SDCs in masked faults. The significant increase of 58.23% in Masked faults leads to a decrease of 55.78% in safe SDCs as well. In contrast, the critical SDCs have been decreased to 0.97%, 1.84%, and 1.60% by Ranger, Swap ReLU6, and median filter. A less pronounced enhancement in reliability can be observed when the NNs with a larger model size is considered. For example, in Mnaset, the critical SDC percentage is reduced by 1.17% with the implementation of Ranger with respect to the Baseline. A more sensible hardening effect can be noticed when Adaptive Clipper is implemented on Mnasnet by decreasing the number of critical SDCs by 0.5% with respect to the Baseline. The same weak beneficial effect is reflected by the sensible increase in masked faults (from a Baseline value of 31.41% to 36.31% and 32.95% for Ranger and Adaptive Clipper, respectively). Similarly, Mobilenet V2 shows limited reliability enhancement effects; in fact, it achieves the highest critical SDC percentage with Swap ReLU6. The Baseline Mobilenet V2 architecture involves ReLU6 activation consisting of activation functions bounded to 6. Consequently, the result demonstrates the ineffectiveness of the step in the Swap ReLU6 on the Mobilenet V2 technique, in which activation functions and batch normalization layers are switched in position. The distribution of faults is very similar to the Baseline when Ranger and Adaptive Clipper are implemented, reporting a negligible decrease in the critical SDCs of 0.01% and 0.20%, respectively. The negligible reliability enhancement effect for Ranger and Adaptive Clipper is explained by the bounding to 6 of the Baseline NN architecture, which leads to computed bounds for the new activation functions that are constantly around 6 regardless of the NN residual block. On the other hand, Resnet18 shows better performance in terms of resiliency, specifically when Swap ReLU6 and median filter are implemented. In fact, a significant reduction in critical SDC percentage (9.87%in Baseline) is registered to 3.52% and 1.06% with Swap ReLU6 and median filter, respectively. The increase in Masked fault percentage is noticeable as well, going from 1.04% in Baseline to 32.38% for Swap ReLU6 and 40.33% for median filter. The results highlighted the importance of an additional training step when an HT is implemented on Resnet18, and it is not based on range restriction. However, the effect of Adaptive Clipper and Ranger is limited both in the percentage of critical SDCs (8.87% and 8.68%, respectively) and in masked faults (1.48% and 0.89%, respectively). Our results indicate that the HTs implemented on Squeezenet show a negligible decrease in the critical SDC percentage (from 11.13% to, at most, 11.01% with Ranger). Figure 6 reports further analysis at the bit level when using the WSBF model. This analysis helps to better visualize how the severity of soft errors is mitigated by HTs reporting the obtained accuracy for soft errors injected in different bit locations. Overall, this plot confirms the lack of clear patterns for ranking all HTs regardless of the NN architecture.