# Fine-Grained Fault Sensitivity Analysis of Vision Transformers Under Soft Errors
## 2 ViT Model and Fault Resilience Evaluation Platform  
### ViT Model  
The transformer architecture [9] overcomes the challenge of capturing non-local dependencies faced by traditional NLP models by introducing the self-attention mechanism. Inspired by the huge success of the transformer in NLP, ViT is designed to apply the original transformer architecture for image classification tasks as closely as possible [12]. Leveraging scalable NLP transformer architectures and their efficient implementations, ViT achieves classification accuracy that surpasses state-of-the-art (SOTA) performance. A classical ViT architecture consists of three main components: patch embedding, transformer encoder, and MLP head. The transformer encoder is composed of alternating layers of multi-headed self-attention (MSA) and MLP blocks, as shown in Figure 1. Compared to the original transformer, layer norm (LN) is applied before the MSA and MLP block for better performance. To receive 2D images, ViT changes the input image \(\mathbf{x}\in\mathbb{R}^{H\times W\times C}\) into a sequence of flattened 2D patches \(\mathbf{x}_{p}\in\mathbb{R}^{N\times(P^{2}\cdot C)}\), where \((H,W)\) is the height and width of the input image, \(C\) is the number of channels, \((P,P)\) is the resolution of each image patch, e.g., \(16\times 16\) in ViT original paper, and \(N=HW/P^{2}\) is the number of patches. ViT flattens each patch and uses a trainable linear projection to map it to the \(D\) dimension, which is the constant latent vector size used in the transformer through all of its layers:  
\[\mathbf{I}=(\mathbf{x}_{p}^{1}\mathbf{E};\mathbf{x}_{p}^{2}\mathbf{E};\dots; \mathbf{x}_{p}^{N}\mathbf{E}),\quad\mathbf{E}\in\mathbb{R}^{(P^{2}\cdot C)\times D} \tag{1}\]  
ViT attaches a learnable embedding(\(\mathbf{x}_{class}\)) to the sequence of embedded patches, which is used at the output of the encoder to serve as the image representation. Position embeddings are added to the patch embeddings to keep the image positional information:  
\[\mathbf{z}_{0}=[\mathbf{x}_{class};\mathbf{I}]+\mathbf{E}_{pos},\quad\mathbf{ E}_{pos}\in\mathbb{R}^{(N+1)\times D} \tag{2}\]  
The embedded patches (\(\mathbf{z}\)) will then pass through LN and MSA. The self-attention (SA) relies on a scaled dot-product attention, which operates on a query matrix \(\mathbf{Q}\), a key matrix \(\mathbf{K}\), and a value matrix \(\mathbf{V}\). For each element in an input sequence \(\mathbf{z}\in\mathbb{R}^{N\times D}\), \(\forall\mathbf{i}\mathbf{T}\) computes a weighted sum over all values \(\mathbf{v}\) in the following sequence:  
\[[\mathbf{Q},\mathbf{K},\mathbf{V}]=\text{LN}(\mathbf{z})\mathbf{U}_{\text{gtk }},\quad\mathbf{U}_{\text{gtk}}\in\mathbb{R}^{D\times 3D_{h}} \tag{3}\]  
\[\text{SA}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{V}\times softmax(\frac{ \mathbf{K}^{\top}\mathbf{Q}}{\sqrt{D^{h}}}) \tag{4}\]  
MSA is an extension of SA in which it calculates \(k\) self-attention operations in parallel and projects concatenated outputs to keep each layer with the same size:  
\[\text{MSA}(\mathbf{z})=[\text{SA}_{1}(\mathbf{z});\text{SA}_{2}(\mathbf{z}); \dots;\text{SA}_{k}(\mathbf{z})]\mathbf{U}_{msa},\quad\mathbf{U}_{msa}\in \mathbb{R}^{k\cdot D_{h}\times D} \tag{5}\]  
After the residual addition, the input of the MLP is  
\[\mathbf{z}_{\ell}^{\prime}=\text{MSA}(\text{LN}(\mathbf{z}_{\ell-1}))+ \mathbf{z}_{\ell-1},\quad\ell=1\dots L \tag{6}\]  
MLP is composed of two fully connected layers and a nonlinear activation function between them. After the residual normalization and addition, the output of the encoder is as follows:  
\[\mathbf{z}_{\ell}=\text{MLP}(\text{LN}(\mathbf{z}_{\ell}^{\prime}))+\mathbf{ z}_{\ell}^{\prime},\quad\ell=1\dots L \tag{7}\]  
Figure 1: Typical ViT architecture. which has the same dimension as the input, so it can be passed to the input of the next encoder directly.