# Fine-Grained Fault Sensitivity Analysis of Vision Transformers Under Soft Errors
## 4 Conclusions  
This paper presents a fine-grained vulnerability analysis of various ViT variants using two kinds of bit-flip fault injection methods. The analysis was conducted from four perspectives: model-wise, layer-wise, type-wise, and head-wise. The model-wise experimental results show that model size and patch size may influence the error resilience of ViT models. Different models' BER thresholds can span several orders of magnitude. For layer-wise, type-wise, and head-wise experiments, the results can be summarized as follows: (1) Due to its crucial role in extracting both local and global features from input patches, the first layer exhibits high sensitivity to faults for both ViT base and large models, with errors in this layer leading to substantial degradation in model inference accuracy. (2) In the middle and later stages of the ViT model, the mlp.fc1 and mlp.fc2 sub-blocks \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Model Name**} & \multirow{2}{*}{**Layer 1, Head X *} & \multicolumn{2}{c}{**Vulnerability**} & \multicolumn{1}{c}{**Mean Attention**} \\  & & **Factor** & **Distance** \\ \hline \multirow{5}{*}{B16-384-augreg} & 7 & 5.1096 & 157.72 \\  & 4 & 3.9056 & 2.15 \\  & 10 & 2.6086 & 197.26 \\  & 1 & 2.6037 & 151.55 \\  & 0 & 2.0893 & 12.50 \\ \hline \multirow{5}{*}{L16-384-augreg} & 14 & 1.1614 & 63.31 \\  & 8 & 1.1354 & 202.58 \\ \cline{1-1}  & 15 & 1.1342 & 201.59 \\ \cline{1-1}  & 6 & 1.1042 & 148.32 \\ \cline{1-1}  & 13 & 1.0834 & 167.17 \\ \hline \hline \end{tabular}
* Sorted in descending order based on vulnerability factor.  
\end{table} 
Table 6: The MAD of Top-5 sensitive heads. Figure 10: Size of attended area by head and network depth. No error bars are shown because the result is computed from static model weights without repeated runs: (**a**) B16-384-augreg model; (**b**) L16-384-augreg model. within each layer play a dominant role in the overall computation and representation learning. Faults in these sub-blocks in the middle and later stages of the ViT model can result in a considerable reduction in inference accuracy. (3) In contrast to intuitive expectations, attention heads responsible for capturing both local and global contextual information from image patches equally contribute to the model's inference process. Given the high sensitivity of the first transformer, patches, fault-tolerant design techniques such as error correction codes (ECCs), triple modular redundancy (TMR), or selective use of higher-precision data formats (e.g., FP16 instead of Int8) can be employed to enhance reliability. Additionally, robustness-aware training strategies, including noise injection or adversarial perturbation targeting the first layer, may improve its fault resilience during inference. In future work, we intend to utilize these findings to inform the development of more efficient and fault-tolerant hardware accelerators specifically designed for ViT models. Additionally, we aim to develop theoretical models and mathematical formulations to better explain and predict the fault sensitivity patterns observed in this study. Our proposed fault sensitivity analysis and mitigation strategies for ViTs are particularly applicable to safety-critical systems, where robustness against hardware faults is essential. For instance, in autonomous vehicles, aerospace guidance systems, and industrial robotics, even minor computation errors can lead to severe system failures. The ability to identify and harden vulnerable components of ViT models could enhance the deployment of deep learning systems in these domains by improving their reliability and fault tolerance. Data curation, J.H. ; funding acquisition, Y.L. and Y.Y. ; methodology, J.H. and X.L. ; project administration, J.H. ; writing--original draft, J.H. ; writing--review and editing, Y.L. and C.X. All authors have read and agreed to the published version of the manuscript.