# Fine-Grained Fault Sensitivity Analysis of Vision Transformers Under Soft Errors
## 3 Results
### Results and Analysis
#### 3.2.4 Head-Wise  
Multi-head attention projects the input into multiple subspaces, allowing each head to learn different feature patterns, such as local edge details and global shape structures. Since different attention heads independently focus on different parts of the input, the model can learn more comprehensive representations. To further investigate the resilience of different attention heads, we performed fault injections on different attention heads separately. Figure 9 shows that different attention heads exhibit varying degrees of sensitivity, particularly in the first layer. This observation implies that certain heads are functionally more significant, potentially playing a pivotal role in the model's inferential processes.  
To further investigate and understand how different heads in different layers affect the models' accuracy, we calculated the mean attention distance (MAD) [12] of the three most vulnerable attention heads in each layer. The MAD refers to the average spatial distance (pixels or patches) between a given query patch and the key patches it attends to in the  
Figure 8: Layer-wise vulnerability factor analysis of Q, K, V projection matrices in the self-attention mechanism. Base models (B16/B32) exhibit clear vulnerability peaks in the V matrix at Layer 1, while large models (L16/L32) show low and uniform sensitivity across all layers: (a) B16-384-augreg (error bar up to 0.0065); (b) B32-384-augreg (error bar up to 0.0072); (c) L16-384-augreg (error bar up to 0.0693); (d) L32-384-orig (error bar up to 0.0093)self-attention mechanism. It quantifies how \"far\" the model's attention head looks when aggregating information across the input image.  
The results are shown in Figure 10. In this figure, the gray dots represent the MAD values of all original attention heads, while the red dots indicate the MAD values of the top three most sensitive heads. We can infer that the top three most sensitive head covers both have large and small attention distances in each layer. A key factor underlying the success of ViTs is the incorporation of self-attention, which allows the model to capture information across the entire image patches, even in the lowest layers. This phenomenon indicates that attention heads responsible for capturing both local and global contextual information from image patches play equally important roles in the model's inference process. Table 6 presents the MAD values of the five most sensitive attention heads in the first layer across the two models.  
These findings offer valuable guidance for both architectural refinement and accelerator fault-tolerant design. For instance, MAD can be used as a heuristic to selectively harden  
Figure 9: Head-wise vulnerability factor analysis for different ViT models. Each line represents a different attention head, showing the relative sensitivity to injected faults at each layer: **(a)** B16-384-augreg (error bar up to 0.0148); **(b)** B32-384-augreg (error bar up to 0.0111); **(c)** L16-384-augreg (error bar up to 0.0195); **(d)** L32-384-orig (error bar up to 0.0178).  
or replicate attention heads that span broader distances, which may be more functionally significant. Moreover, model pruning or quantization strategies should be MAD-aware to avoid disproportionately impacting attention diversity and spatial coverage. By incorporating MAD into the design loop, more resilient and computation-efficient ViT architectures may be achieved.