# Comparing Application-Level Hardening Techniques for Neural Networks on GPUs
## 4 Experimental Setup  
We conducted a series of hardware-aware FI campaigns (i.e., WSBF and NBER), examining the 5 SW-implemented HTs introduced in Section 2.3. The effectiveness of the different HTs was evaluated on five of the most representative NN architectures: MnasNet, MobileNet V2, ResNet18, LeNet5, and SqueezeNet. The first three NNs were trained and tested on the Cifar10 [40] dataset, LeNet5 on the MNIST [39] dataset, and SqueezeNet on the STL10 [41] dataset. The choice of the NN-dataset pairings was guided by the original articles that introduced the tested NN architectures, with the exception of Resnet18. MobileNet V2, MnasNet, and ResNet18 all utilize skip connections, but they employ them in different ways. MobileNet V2 and MnasNet incorporate skip connections within an inverted residual block, while ResNet integrates them in a residual block. Testing these three neural network architectures on the same dataset allows us to evaluate the impact of faults when HTs are implemented across these different architectures. As stated in Section 2.3, implementing Adaptive Clipper, Swap ReLU6, and median filter HTs requires an additional step of retraining or fine-tuning according to the implementation methodology. The original HT implementation, as it is described by the corresponding articles, shows a hyperparameter configuration that is suitable only for the NN architectures under test. For this reason, we adopted a trial-and-error approach to the find the hyperparameter configuration that allows the model to converge to an acceptable level of accuracy (at most 10% of degradation with respect to the NN version without the hardening). required by the baseline neural network architecture; the hyperparameters are selected through a trial-and-error approach The description of the original implementation of HTs does not explicitly specify, when training is required, whether the new NN architecture needs to undergo fine-tuning or a complete training step. For this reason, we initially attempted to fine-tune the model using a number of epochs ranging \([0,original\_number\_of\_epochs]\), where \(original\_number\_of\_epochs\) indicates the number of epochs originally needed to train the NN without applying the HT. In case the model does not successfully converge, a complete training step is required, and the number of epochs ranges \([original\_number\_of\_epochs,2\times original\_number\_of\_epochs]\). During the hyperparameter tuning stage, (i) the number of epochs required for training convergence is increased until the model does not reach an acceptable validation accuracy score, (ii) the NN weight optimization algorithm always refers to the one used to train the original NN architecture, (iii) the initial lr ranges within \([\frac{original\_lr}{2},original\_lr]\) when fine-tuning is required and within \([original\_lr,2\times original\_lr]\) when re-training is required. It is reduced if the training process ends with a steady training loss and an accuracy score below 74%, (iv) the scaling factor for each epoch multiplies the current lr, and it is kept constant. Finally, we reported the best hyperparameter configuration in Table 2 for each architecture under test, along with details on whether the neural network architecture requires retraining from scratch. Table 2 illustrates that although Swap ReLU6 does not need retraining when deployed on Lenet5, Mnasnet, and Mobilenet V2, the number of epochs remains consistently close to the number of epochs utilized for baseline convergence (i.e., 16, 100, 100, respectively). Conversely, as the Squeezenet architecture lacks a batch normalization layer, it is sufficient to fine-tune the baseline architecture for 2 epochs. In contrast, Adaptive Clipper demands retraining, regardless of the chosen NN architecture and with the identical baseline training configuration. In the end, despite the median filter's advantageous impact on fault tolerance, it necessitates many epochs for model convergence. For example, Resnet18 requires retraining and 198 epochs to be trained.