Title: Interpreting vision transformers via residual replacement model
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision transformer

mechanistic interpretability

sparse autoencoder

circuit discovery

replacement model

spurious correlation
Abstract: How do vision transformers (ViTs) represent and process the world? This paper addresses this long-standing question through the first systematic analysis of 6.6K features across all layers, extracted via sparse autoencoders, and by introducing the residual replacement model, which replaces ViT computations with interpretable features in the residual stream. Our analysis reveals not only a feature evolution from low-level patterns to high-level semantics, but also how ViTs encode curves and spatial positions through specialized feature types. The residual replacement model scalably produces a faithful yet parsimonious circuit for human-scale interpretability by significantly simplifying the original computations. As a result, this framework enables intuitive understanding of ViT mechanisms. Finally, we demonstrate the utility of our framework in debiasing spurious correlations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the first comprehensive analysis of Vision Transformer (ViT) features across all layers using sparse autoencoders (SAEs), and introduces the \"residual replacement model\" framework for interpretable circuit discovery. The authors manually annotate 6.6K features across ViT, CLIP, and DINOv2 models, revealing a progression from low-level visual patterns to high-level semantic concepts. Their residual replacement model bypasses attention complexity by focusing on the residual stream, enabling scalable circuit discovery through attribution patching. The framework is validated on spurious correlation debiasing tasks, demonstrating practical utility beyond interpretability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- manual annotation of 6.6K features across all ViT layers represents the largest systematic study of transformer vision features to date, providing valuable empirical insights about feature evolution\n- the residual replacement model cleverly sidesteps attention mechanism complexity while maintaining faithfulness, addressing a key scalability challenge in transformer interpretability\n- the edge-based circuit discovery method and gradient correction techniques (extending LibraGrad) show solid technical innovation with measurable improvements in faithfulness metrics\n- application to spurious correlation debiasing is useful, achieving performance close to data-driven methods like SpuFix using only single-image analysis\n- three-metric evaluation framework (faithfulness, completeness, causality) provides robust validation of circuit quality\n- authors promise to release code, SAE weights, and the full annotated dataset\n\n**Weaknesses:**\n\n- despite user studies and cross-checking, the manual feature annotation remains inherently subjective and may not capture all relevant feature types or miss subtle concepts\n- while more efficient than naive approaches, the O(f) gradient computation faces challenges with very large feature dictionaries, potentially limiting applicability to future larger models\n- the paper lacks theoretical analysis of when and why the residual replacement approximation works well, relying primarily on empirical validation\n- beyond spurious correlation debiasing, the practical utility demonstration is somewhat narrow - additional downstream applications would strengthen the case (even though for interpretability immediate use cases are often not a given)\n\nQUESTIONS:\n- how do the feature evolution patterns and circuit structures vary across different ViT architectures (e.g., ViT-L, ViT-H) and patch sizes? \n- can you provide insight on when the residual stream approximation accurately captures the full model behavior? Under what conditions might this approach fail?\n- how sensitive are the conclusions to the specific SAE hyperparameters and training procedures? \n- bypassing attention is a design choice and improves scalability, does this limit the understanding of attention mechanisms' specific contributions to ViT behavior? what insights about attention's role in ViT computation are lost?\n\nLIMITATIONS:\nThe authors adequately discuss technical limitations in §6, including SAE constraints and the qualitative nature of interpretability evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a detailed mechanistic analysis of Vision Transformers (ViTs) by adapting interpretability techniques from language models. The authors train Sparse Autoencoders (SAEs) on the residual stream of each layer across three pretrained ViT variants (supervised ImageNet, DINOv2, and CLIP), extracting 6.6K manually annotated features. They then discover meaningful circuits in the SAE latent space (rather than the ViT latent space). The results suggest that ViT features evolve from low-level patterns (colors, edges) in early layers to high-level semantic concepts (object parts, complete objects) in later layers, similar to CNNs. The authors present a number of case studies of circuits (curve detectors, position detectors, and a classification circuit). Additionally, the authors explore some practical applications of their work, using their framework to mitigate spurious correlations in ImageNet classifiers.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The early and middle layers in vision models are underexplored in mechanistic interpretability, so the paper’s exploration of these layers in ViTs is an important contribution. \n2. The paper presents some interesting case studies; for example, I found the Granny Smith circuit to be very interesting. The breakdown of feature types across layers presented in Figure 2b was also very interesting. \n3. The circuits discovered by the authors appear to be significantly more faithful compared to a number of baselines. \n\n**Weaknesses:**\n1. **Overstated novelty.** The paper greatly overstates its methodological novelty. The way the paper is written suggests that the residual replacement model is a novel methodology. However, it is nearly identical to Marks et al. (2024), which first trains SAEs on a language model’s layers, then identifies sparse circuits via causal importance in the SAE latent space (rather than the model’s latent space). I feel as though the authors overstated the novelty of their approach and neglected to sufficiently acknowledge the parallels to Marks et al. (2024); it is more accurate to say that the authors applied existing sparse feature circuit methodology from LLM interpretability to ViTs than to say the authors introduced a novel method. I would strongly recommend that the authors reframe the paper as an adaptation of existing NLP interpretability techniques to vision models, as the current framing is misleading.\n2. **Limited scope of the analysis.** The paper claims to present a “comprehensive, large-scale analysis” of features in ViTs. However, the authors opt for human annotation of features over an automated interpretability method; as a result, only 200 features per layer are labeled, which is a very small subset. This is highly likely to bias the circuits that are discovered, since important features may lack an annotation. The annotation methodology is also unclear; I can’t tell 1) how these 200 features were selected, 2) how the annotations were assigned, or 3) whether or how these annotations were validated. The human participants in section 2.2 are also not adequately described. For the analysis to be systematic, these methodological details should be described clearly; for it to be large scale, then perhaps the authors should employ automated interpretability techniques. \n3. **Lack of comparisons to existing methods.** It isn’t clear that the method is an improvement over existing vision interpretability techniques (e.g. Rajaram et al., 2024) since no comparisons to existing methods are made. \n4.  **Clarity.** The paper is missing some concrete technical details that I think would make the methodology and results much more clear. For example, in section 3.2, it wasn’t clear what the authors meant by “replacing a circuit;” what concretely is being done to remove the circuit? What is an empty circuit? Figures 5 and 7 include nodes labeled as “error,” but this doesn’t appear to be explained. \n\n**Citations:**\n\n[ 1 ] Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., & Mueller, A. (2024). Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647.\n\n[ 2 ] Rajaram, A., Chowdhury, N., Torralba, A., Andreas, J., & Schwettmann, S. (2024). Automatic discovery of visual circuits. arXiv preprint arXiv:2404.14349.\n\nQUESTIONS:\n1. Do the authors plan to release the SAE weights or an interactive interface for exploring features and circuits? This would be a very useful contribution and would help to increase the significance!\n\n2. Is there a reason why the authors use human annotations over some sort of automated interpretability method? If the goal was to provide a comprehensive analysis of all ViT layers, then automated interpretability would be the more natural choice since human annotations don’t scale very well. I also wasn’t sure why “gold standard” (in reference to the use of human annotations) was in quotes (line 99); is this in reference to a particular paper, or is this the author’s personal opinion?\n\n3. How were the 16 participants described in Section 2.2 selected? Were they recruited from e.g. Mechanical Turk, or were they colleagues of the authors? Were they compensated? It’s important to clarify these details when including results from human participants. \n\n4. How do the authors contextualize the Granny Smith circuit in relation to the results in Figure 2b? For example, shape features appear to be the most prevalent in layer 2, but shape features don’t appear in the Granny Smith circuit until layer 6.\n\n5. Faithfulness and completeness of the circuits are high, but the causality seems quite low (only ~50%). How might the authors explain this discrepancy?\n\n6. Did the authors try intervening on circuits in meaningful ways, e.g. trying to change Granny Smith classification to another class by intervening on e.g. the shape feature?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how Vision Transformers (ViTs) represent and process visual information by extracting 6.6K features across all layers and multiple ViT variants (ViT, CLIP, and DINOv2) using sparse autoencoders. The authors introduce a residual replacement model, which replaces ViT computations with interpretable features in the residual stream. Their findings show that similar to CNNs, ViTs evolve from detecting low-level patterns to capturing high-level semantics, and include specialized features for encoding curves and spatial positions. The residual replacement model produces a simplified yet faithful representation of ViT circuits, significantly improving human interpretability. To evaluate these circuits, the authors apply three key metrics: faithfulness, completeness, and causality. Finally, they demonstrate the practical use of their framework in effectively debiasing spurious correlations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is clearly written and easy to follow.\n2. To address the technical challenges of applying replacement models to ViTs, particularly the scalability of edge importance estimation, the authors employ the Jacobian-vector product trick. Given that ViTs are prone to noisy gradients, they further extend the LibraGrad method to enhance the faithfulness of the discovered circuits compared to those derived using the vanila gradient.\n3. I appreciate that the authors have evaluated the discovered circuits using the criteria of faithfulness, completeness, and causality.\n4. The insights into curve and position circuits, along with the demonstrated practical use of the proposed replacement model in debiasing spurious correlations are interesting.\n\n**Weaknesses**\n\n1. The novelty of the proposed residual replacement model is somewhat limited, as similar approaches have already been explored in the NLP community [a, b]. The authors primarily extend this replacement philosophy to Vision Transformers, with the main contribution being their handling of domain-specific challenges, particularly scalability and noisy gradients.\n\n[a] Marks et al. “Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models”\n\n[b] Antropics’s Circuit Tracing, https://transformer-circuits.pub/2025/attribution-graphs/methods.html \n\n2. In Table 1, some values related to the causality analysis are missing. Could you please clarify the reason for their omission?\n3. To evaluate the quality of the learned SAE features and assign semantic labels, the authors rely on human annotations. This dependence on manual labeling makes the proposed framework less scalable and more difficult to apply to new models and architectures.\n\nQUESTIONS:\nPlease refer to the second point in **Weaknesses**.\n\nLIMITATIONS:\nThe limitations are discussed in Section 6 of the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors try to open up the layers of Vision Transformers via sparse autoencoders and residual replacement model. They try to highlight some of the key understandings when it comes to interpreting the features in a network – lower level features are represented in earlier layers, while more complicated features are built up by combining the earlier features in higher layers.\n\nSTRENGTHS AND WEAKNESSES:\nThe study is important for interpretability and explainability, in terms of showing how the model interprets the individual objects in its layers, going from low level features like the lines to higher level features like complex arches. The 640 visualizations per layer  with 16 participants seems quite extensive. \nThey have set out to create a gold-standard feature analysis through human interpretations. \n\nThe authors try to pin their work on the fact that they can explain the human alignment of ViTs (and subsequently other networks). However, in the current form, the paper misses to demonstrate a clear alignment with human/biological vision. \nThe authors miss a few crucial details/reasoning as well – how were the averages of activations taken to reduce the dimensionality of explanations? How were certain feature selected in the layers? Why only those features were selected/shown?\nIn the validation subsection, why did the authors only select faithfulness, completeness and causality as the metrics for evaluation?\nThe authors also don’t say much about how the numbers were reached for the 3 metrics in table 1. \nThe authors talk about the qualitative analysis of a discovered circuit, but don’t clarify much on how this circuit is identified. It might be good to have some background for the reader. Please also clarify the link of this with the residual replacement model. \nIn Figure 5, are the layer and feature numbers manually selected? If so, how? \nI am particularly pleased with the legend indicating the round fruit, round object, error, etc. in each of the layers. But the authors missed mentioning about how these were selected? Is it purely by looking at the activations or can this be automated for scalability?\nThe authors mention a lot of metrics (PCA, SAE, mAUC, etc.) but don’t compare their representations using t-SNE which seems to be quite standard in the field. \nI am also not too sure how backprop and other gradient based methods fit into the framework for visualization. Can the presented method/framework be scaled for different arbitrary models?\n\nQUESTIONS:\nHow were the annotations done? The authors mention large-scale human annotations of the of the ViT features, but do not mention anything about how the studies were conducted? How were the humans selected? How were the responses filtered? Was there an IRB approval? How were the human subjects’ annotations evaluated? How was the quality of annotations assessed? \nCan the authors also clarify the training details from a hardware standpoint? How long did the experiments take? What CPU/GPU configs were used?\nThe authors are also encouraged to release the code and datasets in the supplementary materials to the public. \nIn lines 126-127, the authors mention that the detector activates only when the corner is well-aligned with the patch grid. Do they imply that a feature activates a layer/neurons only when it is well aligned? Have the authors observed a gabor filter-like pattern with higher activations in the center and lower activations towards the edges? \nThe authors cover feature categorization by layers – low level features in early layers and high level features in higher layers are more prominent. Have the authors tried comparing their study with other human vision aligned works like the Extreme Image Transforms (EITs) and Minimal recognizable configurations (MIRCs)? EITs can also help with the spurious correlations and interpretability, functioning similar to humans and confirmed by psychophysics studies. \nThe reader might benefit from knowing about concept discovery in GAN Dissection by Bau et al, ICLR 2019.\n\nLIMITATIONS:\nYes, the authors do mention limitations. I am not sure how this framework can be scaled to other models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}