# Comparing Application-Level Hardening Techniques for Neural Networks on GPUs
## 2 Background  
### Neural Networks  
In Edge artificial intelligence (AI) applications, convolutional neural networks are typically employed to solve tasks, such as image classification, object detection, or real-time signal processing, using convolutional operations as basic functional blocks. Every network design consists of a specific arrangement and set of layers. As shown in Figure 1, a possible convolutional neural network organization can involve the execution of (i) the convolutional layer, (ii) batch normalization layer, and (iii) activation function. The convolutional layer performs the dot product \(\underline{w}\cdot\underline{x}+\underline{b}\) where \(\underline{w}\) is the matrix containing the layer weights (i.e., kernel), \(\underline{x}\) is the patch of the input whose shape matches the kernel size and, \(\underline{b}\) is the bias vector. Batch normalization is used to enhance the stability of the training process by normalizing the inputs to each layer. Lastly, activation functions are generally used to make the convolutional neural networks able to learn/non-linear input/output relationships. For example, the rectified linear unit (ReLU) activation function applies the ramp function \(ReLU(\underline{x})=max(0,\underline{x})\). Building upon this function, several other ReLU-like functions have been developed, e.g., SELU [42], GELU [43], ReLU6 [36], andTanH. NNs designed for Edge AI are built upon the convolutional block, focusing on optimizing weight model size for greater efficiency in resource-constrained environments. The following models are five of the most representative NN architectures used in Edge AI applications. LeNet, as one of the pioneering convolutional NNs, is relatively simple and lightweight, making it suitable for tasks with lower complexity [34]. It includes three convolutional layers, each employing 3 \(\times\) 3 kernels with a padding of 1. The number of filters increases from 32 to 128 across these layers. Following each convolutional layer, a  
Figure 1: Basic convolutional block. 2 \(\times\) 2 MaxPooling operation is applied with a stride of 2, progressively reducing the spatial dimensions. The fully connected layers comprise 256, 128, and 10 neurons each. MobileNet was originally designed for mobile and embedded vision applications [36]. Its architecture is built around inverted residual blocks and linear bottlenecks, which help reduce space complexity. In its topology, MobileNetV2 starts with an initial standard convolutional layer with 32 filters, followed by a series of 19 residual bottleneck layers. The inverted residual design incorporates shortcut connections, also known as skip connections, that bypass the inner bottleneck layers. This design enhances the propagation of gradients and reduces computational overhead. The bottleneck layers utilize an expansion factor, usually set to 6, which increases the number of channels before depthwise convolution is applied. The bottleneck layers allow the model to capture richer feature representations with minimal additional complexity. Additionally, the architecture uses ReLU6 as its activation function and includes a linear bottleneck to maintain low-dimensional features, ensuring high efficiency for mobile applications. MnasNet is a convolutional NN specifically designed for mobile devices, incorporating real-world latency and accuracy as key objectives, thus making it suitable for Edge AI applications [35]. The architecture features a modular design that includes inverted residual blocks. The process begins with a 3 \(\times\) 3 convolution, followed by batch normalization and ReLU activation to process the input image effectively. Within the feature extraction block, multiple inverted residual blocks are stacked together. Each inverted residual block consists of three phases: an expansion phase, which increases the number of features; a depthwise convolution phase, which performs spatial filtering; and a projection phase, which reduces dimensionality. ResNet offers a good trade-off between depth and complexity [37].