# Fine-Grained Fault Sensitivity Analysis of Vision Transformers Under Soft Errors
## 3 Results
### Results and Analysis
#### 3.2.2 Layer-Wise  
In this section, the importance of individual layers in different ViT models is evaluated by introducing the same BER. Each layer in a ViT model consists of multiple sub-blocks, as outlined in Table 1, such as the attn.qkv, attn.proj, mlp.fc1 and mlp.fc2. According to the settings of BER, we randomly selected weights from the weight matrices of these sub-blocks, such as \(W_{qkv}\), \(W_{proj}\), \(W_{FC1}\), and \(W_{FC2}\) and injected bit-flip faults to the chosen weights; then, we ran the model and determined the classification accuracy with errors. This scenario was repeated 10 times for each layer from the first to the last, and the average vulnerability factor was used to quantify the sensitivity of each layer to weight perturbations. The results are shown in the bar chart in Figure 5, revealing several noteworthy observations. The bar charts represent the vulnerability factors obtained via fault injection for two model variants, while the dotted line plots show the results from the identity replacement experiment. For B16-384-augreg and B32-384-augreg, both of which consist of 12 layers, the vulnerability factors remain significantly high, with an average of 5.376 and 4.339, respectively, particularly in the early (Layers 1-2) and middle stages (Layers 3-9). In contrast, L16-384-augreg and L32-384-orig, with 24 layers, show much lower vulnerability factors. However, the first few layers (Layers 1-2) of these models exhibit relatively higher vulnerability factors than the other layers. These observations indicate that certain layers within the model contribute differently to its overall accuracy. Some layers are more critical than others. To validate this assumption, we conducted an experiment in which the selected layers of the model were replaced with torch.nn.Identity(), a placeholder identity operator that simply returns its input without any transformation, in order to assess the contribution of each layer to the overall inference accuracy. As shown in the dotted line plot in Figure 5, the identity experiment results exhibit a similar trend to that observed in the layer-wise fault injection experiments, where the first layer and the layers close to the output (Layers 8-9) show notably higher sensitivity. To gain a deeper understanding of how sub-blocks within each layer (e.g., attn.qkv, attn.proj, mlp.fc1, and mlp.fc2 affect its vulnerability), we conducted targeted fault injection experiments on each sub-block individually. The fine-grained vulnerability analysis of individual sub-blocks is shown in Figure 6. For the B16-384-augreg model, the attn.proj in the initial layer, along with the mlp.fc1 and mlp.fc2 components in Layers 4 through 9, exhibit a higher contribution to layer-wise vulnerability. For the L32-384-orig model, the attn.proj module, as well as both mlp.fc1 and mlp.fc2, demonstrate notable sensitivity in the first layer. Consistent with observations from the B16-384-augreg model, the MLP components exhibit greater sensitivity in the intermediate and deeper layers. The early layers in a ViT primarily focus on capturing local and low-level features, such as fine-grained textures, edges, and local patterns, from the input image. As the input is divided into non-overlapping patches and projected into an embedding space, these \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **1\% Accuracy Loss** & **5\% Accuracy Loss** & **10\% Accuracy Loss** \\ \hline B16-224-augreg & \(4.062\times 10^{-6}\) & \(2.015\times 10^{-5}\) & \(3.675\times 10^{-5}\) \\ B16-384-augreg & \(4.062\times 10^{-6}\) & \(1.650\times 10^{-5}\) & \(3.008\times 10^{-5}\) \\ B32-224-augreg & \(6.062\times 10^{-6}\) & \(3.008\times 10^{-5}\) & \(5.484\times 10^{-5}\) \\ B32-384-augreg & \(1.650\times 10^{-5}\) & \(5.484\times 10^{-5}\) & \(10.000\times 10^{-5}\) \\ L16-384-augreg & \(5.484\times 10^{-5}\) & \(1.823\times 10^{-4}\) & \(2.721\times 10^{-4}\) \\ L32-384-orig & \(2.015\times 10^{-5}\) & \(1.000\times 10^{-4}\) & \(1.823\times 10^{-4}\) \\ \hline \hline \end{tabular}
\end{table}