Interpreting vision transformers via residual replacement model
Authors: Jinyeong Kim
Junhyeok Kim
Yumin Shim
Joohyeok Kim
Sunyoung Jung
Seong Jae Hwang
Venue: NeurIPS 2025 poster ()
Keywords: vision transformer
mechanistic interpretability
sparse autoencoder
circuit discovery
replacement model
spurious correlation
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: How do vision transformers (ViTs) represent and process the world? This paper addresses this long-standing question through the first systematic analysis of 6.6K features across all layers, extracted via sparse autoencoders, and by introducing the residual replacement model, which replaces ViT computations with interpretable features in the residual stream. Our analysis reveals not only a feature evolution from low-level patterns to high-level semantics, but also how ViTs encode curves and spatial positions through specialized feature types. The residual replacement model scalably produces a faithful yet parsimonious circuit for human-scale interpretability by significantly simplifying the original computations. As a result, this framework enables intuitive understanding of ViT mechanisms. Finally, we demonstrate the utility of our framework in debiasing spurious correlations.

Interpreting vision transformers via residual replacement model
Jinyeong Kim'
Junhyeok Kim
Yumin Shim
Joohyeok Kim
Sunyoung Jung
Seong Jae Hwang!
Yonsei University
{jinyeong1324,
timespt, seongjae)
httpss///gihtb...mm/rbatttttt
Abstract
How do vision transformers (ViTs) represent and process the world? This paper addresses this long-standing question through the first systematie analysis 6.6K features across all layers, extracted via sparse autoencoders, and by introducing the residual replacement model, which replaces VIT computations with interpretable features in the residual stream. Our analysis reveals not only a feature evolution from low-level patterns to high-level semantics, but also how ViTs encode curves and spatial positions through specialized feature types. The residual replacement model scalably produces a faithful yet parsimonious circuit for human-scale in- terpretability by significantly simplifying the original computations. As result, this framework enables intuitive understanding of ViT mechanisms. Finally, we demonstrate the utility of our framework in debiasing spurious correlations.
Introduction
Understanding the internal mechanisms of vision models has been long-standing challenge. Two fundamental questions drive this pursuit: how do vision models represent the world, and how do they process these representations make predictions? While considerable progress has been made in interpreting convolutional neural networks (CNNs) (+S], vision transformers (ViTs) [TI remain substantially more difficult to understand, despite their widespread application [92 ViTs are more "polysemantic" than CNNs 20-22) and their architecture introduces additional complexity through attention mechanisms
multiple unrelated concepts within a single neuron-
which
obscure the flow of information.
Recent advances in mechanistic interpretability (MI) for transformer-based language models (LMs) offer tools for addressing these obstacles. In particular, dictionary learning methods such as sparse autoencoders (SAEs) (55 effectively disentangle polysemantic neurons into monosemantic "features." Building on these features, one can construct a replacement model, an interpretable surrogate in which neurons of the modules are replaced with these features 40). Since these features are sparsely activated, the model's key computational mechanisms, known as "circuits," can be identified in parsimonious and interpretable manner 41
However, extending the replacement model framework to ViTs presents several unique challenges. First, unlike in LMs, feature representations in ViTs are still poorly understood. Although a few studies have begun to explore ViT features (4)) they typically focus only on the late layers and present limited qualitative examples. As result, comprehensive and systematic understanding of
TThese authors contributed equally to this work. Correspnnding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Original Model
a. Residual Repiacement Model
b. Interpreting VIT Features
D0P
000
SAE
P O
Wi con bergeeecale anatyede features layors to intemallyy repiesented.
0MM00
DO0
000
Discovering Circuits
preor
000
000
O
wctiere the inow serncinge VIT. enstling scalatiee analyds of mecheriim krme.
Foand
smah
D0D
d. Debiasing Spurious Correlation
000
000
We dctlis the model by masking out spuricudly commisted sted fealunes e.g. miligpee lerdency machassiy grafti images treight
000
000
SAE
0000
Figure 1: Overview. a. The residual replacement model replaces the original computations of ViT with SAE features in the residual stream. b. We interpret features to understand what the ViT represents. c. We discover parsimonious circuits that explain the underlying mechanisms of the ViT. d. We apply the residual replacement model to debias spurious correlations.
VIT features across all layers is still lacking, yet such an understanding is crucial for constructing inter- pretable replacement models. Second, naively applying existing techniques to ViTs leads to complex circuits that are infeasible to interpret, due to the nature of the visual domain. In particular, whereas most MI studies in LMs use short textual inputs [51H54], ViTs innately operate on hundreds of image tokens. Consequently, modeling token-to-token interactions via attention modules, is commonly done in LMs, produces excessively large circuits that hinder human-scale interpretability 2
To address the first challenge, we present the first comprehensive analysis of ViT features across all layers and multiple model variants, based on large-scale human annotations of a total of 6.6K features (Figure [b). Remarkably, we find that early-layer features, though seemingly uninterpretable, are fact consistently interpretable at the patch level. Next, we uncover a developmental pattern in VIT features: as depth increases, they gradually evolve from low-level attributes such color to more semantic concepts such as object parts. Moreover, we identify feature types that were not previously well documented in ViTs, including curve detectors and position detectors, which are known to emerge in CNNs 58 and LMs [59), respectively. We prove that these features collectively cover all angles and spatial positions, contributing to a holistic understanding of the image. Overall, these findings significantly advance the understanding of how ViTs represent visual information.
To tackle the second challenge, we propose the residual replucement model, a framework that substantially simplifies circuits by focusing on the residual stream (Figure Tb. c). The residual stream serves as "communication channel" in transformers (20]. where each module reads from and writes directly to. By tracking interactions between features along the residual stream, we can comprehensively audit how information evolves across layers, bypassing the complexity of attention modules [T.. We further reduce the dimensionality of the explanation by aggregating token-to- interactions, as many tokens redundantly encode similar features. By overcoming additional technical obstacles, we improve both the faithfulness and scalability of circuit construction, achieving up to a 1.6x increase in faithfulness with only a few seconds of computation. Ultimately, our framework enables a clear and scalable interpretation of how VITs internally process visual information.
Our residual replacement model can be utilized not only as interpretation tool to explain the internal mechanisms of ViTs, but also as an actionable tool for practical applications (Figure It). We demonstrate its wide utility in debiasing spurious correlations in ImageNet classifiers with simple human-in- in-the-loop edits. Our contributions are summarized as follows:
We perform comprehensive, large-scale analysis of features across all layers and multiple VIT variants, unveiling how ViTs internally structure and represent information ($)
We propose residual replacement model, an interpretable framework that explains the mech- anisms of ViTs parsimoniously by focusing on the residual stream. It is faithful, simple, and scalable {$3)
We dissect VIT's via residual replacement model, validating its practical utility in discovering and mitigating spurious correlations ($4).
We will release our code, SAE weights, and datasets, including the full set of 6.6K manually annotated SAE features, to support and accelerate future research in ViT interpretability.

---

2 Systematic Feature Analysis
While SAEs have been widely used in LMs, their application to VIT' has also recently gained atten- tion 50). However, most existing studies focus primarily on the later layers and rely heavily on qualitative visualizations, limiting systematic understanding of features throughout the entire model. Some approaches devise automated interpretation methods, which typically use models such as CLIP to project the image into semantic space [22 60- (2] Yet these approaches depend on predefined concept vocabularies and struggle to capture subtle concepts beyond CLIP's representational capacity. As result, it remains unclear whether features in ViTs are interpretable across all layers, and what types of features emerge at different depths.
2.1 Preliminaries
Sparse Autoencoders (SAEs) We begin the section by formulating SAEs. SAEs have emerged as popular approach for extracting interpretable features from polysemantic representations [25 35 due to their effectiveness and scalability [33 34 45 47). SAEs decompose polysemantic neurons into a set of "features" " under a sparsity constraint, thereby promoting disentanglement. Concretely, TopK SAE 34 [33, a widely used variant, maps representations IR to sparse features x E R and reconstructs the original representations as follows:
z = Topk W enc Dpre)), = W dec + Upre:
where W ene E R/xd W dec E R and bre Rd are learnable weights. Training the SAE involves minimizing the reconstruction error e = x at and the sparsity constraint is directly enforced by selecting the top-k elements in z. We refer each element 2; the activation of the i-th feature.
Training We train TopK SAEs on the residual stream of each layer in the supervised VIT (hereafter simply ViT) [9]. CLIP [0)), and DINOv2 [T] to comprchensively understand the features emerging from diverse training paradigms. TopK SAEs have two primary hyperparameters: the number of features f and the sparsity level k. Since there is no established consensus on appropriate values for these hyperparameters in ViTs, we sweep over range of configurations and extend the scaling laws of (34] to ViTss For each layer, we select SAEs whose fraction of variance unexplained is below 0.15. See $C for further details, results, and analysis on the training process.
2.2 Systematic Feature Analysis
Overview To systematically understand how ViTs represent visual information, we conduct a large-scale feature analysis across all layers of ViT, CLIP, and DINOv2 through human interpretation, which is regarded as the gold standard. To support this, we newly design a visualization that mimics feature visualization LMs 33 41 64). While common the vision domain to visualize only the maximally activated image for a feature [4 48-50 65 we find that including the maximally activated patch and class significantly enhances interpretability (see ID.T). Using this visualization, we ① verify that features are more interpretable than raw neurons, @ annotate and categorize features across all layers and models, and @ summarize key findings discovered through the process.
① We evaluate the interpretability of SAE features and raw neurons in a blind setting. Specifically, we sample 640 visualizations per layer, randomly selected from both features and neurons, and rate each sample's interpretability as 'Yes (1)', 'Maybe (0.5)', or 'No (0)' with a brief explanation, following the protocol of 30 3I 35) To reduce potential bias from authors familiar with MI, we additionally conduct a user study with 16 participants who are not experts in this field. As shown in Figure? 2b. both groups consistently judged features to be significantly more interpretable than raw neurons across all layers, confirming the effectiveness of SAFs. Layer-wise analysis reveals that interpretability is generally higher in the carly and late layers than in the middle ones; we find that features in the middle layers often represent more abstract and less human-interpredia interpretable concepts that require substantial cognitive effort to interpret, such as the subtle space between two objects or geometric constructs like vanishing points, echoing similar observations in LMs 42
2 Next, we annotate 200 features per layer across all layers of the three models, resulting in a total of 6.6K annotated features. To ensure reliability, we also group similar features into categories, and interpretations are refined through iterative cross-checking, mitigating subjectivity and possible labeling errors. A summary of the results for ViT is shown in Figure 2b. We observe a general progression from low-level visual attributes (e.g., color, line) early layers to high-level semantic concepts in later layers. This layerwise evolution aligns with general trends observed in CNNs,
3

---

Postiooel
LIM03 (perical
LOW59
11 Nianted Ine]
1229100 Doorcoest ine)
LO45 (Topes L shape)
LSM052 (ooss)
Use
Proporton
Todue
LS
LS4709 (dotted
Backgroan
LSM842
(tackground)
0.2
Semertt
LS#17 Top of objects)
LOCAS8 (brigtaress
LSB15E Strchendi
L1OKGE Itescoafi
25 bonal
Color Texture
Line
Semantti
Mistellnnous
Figure 2: Systematic Feature Analysis. a. Average interpretability scores of neurons and SAE features across all layers. Features are consistently more interpretable than neurons. b. Feature categorization by layer. Low-level features (e.g.. color) emerge in early layers, while high-level features (e.g.. objects) become more prominent in later layers. c. Example features from different categories.
highlighting feature universality across architectures 22 66 For results on DINOv2 and CLIP, as well as detailed descriptions of each feature, see D
@ Through this process, we gain a broad understanding of how ViTs represent visual information within their internal layers. In the early layers, ViTs predominantly detect localized, patch-level patterns. For instance, L0#5, a detector for L-shaped corners (second row of Figure ( activates only when the corner is well-aligned with the patch grid and fails to respond when the same corner is split across multiple patches. As we progress deeper into the network, ViTs increasingly capture complex semantic concepts by incorporating broader image context. For example, L5#155 consistently detects the foreheads of various animals despite variations in appearance (fourth row of Figure (2), indicating that ViTs learn generalizable and abstract semantic representations.
From analogical perspective, many features in ViTs resemble those in CNNs and LMs. Like CNNs (55 66 67], ViTs learn universal features such as edges, curves, and low-frequency textures. Meanwhile, ViTs exhibit pnsition-sensitive features (first row of Figure 2b), driven by positional encodings, similar to LMs [59]. To further investigate these analogies, we investigate established
findings on (1) curve detectors in CNNs (55) and (2) position detectors LMs (59 in ViT.
2.3
Case Studies: Curve Detectors and Position Indicators
Curve Detectors Curve detectors (55 are a well- known example of features in CNNs that track articulable visual properties. With the help of the manual feature inspection in $2.2] we could assess their universality in ViTs. We reproduce radial taning curves (Figure 33,, which visualize feature's activation as function of the curve's second angle [55] We find that each curve detector
12,
L24579
2207
12
L28566
275
layer of the VIT consistently activates for a specific angle. Taken together, they collectively span the angular space, indicating form of rotational equivariance
Figure 3: Curve Detectors
Position Detectors Following Voita et al. [59], we au-
LOMB3
LOWIA
LOMIS
LO0229
LON16
LOWIZ
tomatically identify position detectors in VIT. The aver- age activation of each feature is shown in Figure 4 (left). Features early layers tend activate on specific rows, columns, or patches, whereas those in deeper layers ex- hibit more diffuse patterns, likely due the mixing of positional information through the attention mechanism. Moreover, when aggregating the activations of all identified position detectors, we observe that they together form a complete coverage of suggesting that ViT maintains complete representation
L2X807
L2M11
L24999
12504
L2473
128502
Layer
Figure 4: Position Detectors
entire image patch space (FigureF of positional information throughout the network. We will revisit these features after constructing the residual replacement model and further analyze the "cireuits" they compose ( (34)

---

3 Residual Replacement Model
Through 2 we verify that features in the residual stream across all layers exhibit highly interpretable representations. We now "connect the dots" between layers (i.e., link features across the network) by constructing the residual replacement model to interpret the entire decision- making process of ViTs.
3.1 Formulation
The residual replacement model is a surrogate for ViT' in which the residual stream at each layer is substituted with interpretable features extracted via SAEs (Figure D. Inspired by replacement models LMs [4] [42], we formalize this model as a directed acyclic graph G. where each node corresponds to feature (or error term <), and each edge denotes connection between features in adjacent layers, reflecting how information flows across the network. As discussed in T targeting the residual stream for interpretability has the advantage of preserving the original information flow without loss, while enabling a more parsimonious surrogate by bypassing the complexity of attention modules. We also average the activations of the feature across the tokens to reduce the dimensionality of the explanation, as many tokens encode similar features and their interactions yield limited information. This choice not only speeds up computation, but also provides [ 22 77. which are more human buman-friendlyy
Definition
CNN-like explanations
We estimate the importance of each edge in the residual replacement model by using attribution patching [4] 75, scalable approximation of interventional causal effects 76-79 Given the activations along an edge (u, d), its importance can be approximated as:
Estimating Causal Effects
Iu + d) = V am Vud (u (.a--0 =²'),
(2)
where is the objective of interest, and u' is the median activation of across the dataset. We define m as the logit of the target class, normalized by subtracting the mean logit across all classes [80]. Intuitively, Equation (2) estimates the effect of on the model's output mediated only through d.
Discovering Circuits Now, we define circuit as a subgraph C C G that captures the key internal mechanism of the model. We identify such circuits through the following procedure:
1. In the final layer I., we select the top-k most important nodes V.. with respect to the target class.
2. In the preceding layer L - 1, we select the top-k most important nodes VL-1 based on the im- portance of their edges to the already selected downstream nodes in L, i.c., using the aggregated score Edev, I(u d).
3. This process is repeated recursively until the leaf node is reached.
Through this procedure, we obtain a of nodes prediction through selected downstream connections. In contrast to prior works [41 nodes based on their total outgoing importance all downstream nodes (i.e., Ea I(u d)), our method identifies features based on the strength of their specific causal edges to already relevant downstream nodes. We find that this edge-based discovery results in circuits with stronger causality. whereas the prior node-based approach dilutes causality by ignoring the downstream context ($3.2).
{VeJt= that causally influence the target
42). which select
Other Technical Challenges We address two technical challenges in implementing the residual replacement model. ① Scalability of Edge Importance Estimation. Unlike typical MI studies in 51
12 81) which handle only a handful tokens, ViT's process hundreds of patch tokens per image. Therefore, naïve Jacobian computation for estimating edge importance requires O(T x f) backward passes per layer, where T is the number of tokens and f is the number of features. This makes edge importance estimation computationally intractable. However, because we only need the aggregated importance across tokens, we reduce the cost to O(f) using the Jacobian- vector product trick. This choice yields a ~200 speed-up, reducing computation few seconds per image. @ Noisy Gradients. ViTs are known to suffer from noisy gradients (84] which significantly compromise the reliability of gradient-basee we extend LibraGrad 885 86), a method that corrects gradients by pruning and scaling backward paths, to intermediate representations and SAEs. This results in up to 1.6x improvement in circuit faithfulness over the vanilla gradient. Please refer to for further details and design choices.
LMs 41
42
estimation. To mitigate this issue,
5

---

Table Circuit Evaluation. We evaluate the faithfulness, completeness, and causality of the circuits using 1,500 randomly sampled images from the ImageNet validation set.
Faithfulness (%)
1- Completeness (%)
Causality (%)
Unit
Strategy
VIT
DINOv2
CLIP
DINOv2
CLIP
VIT DINOv2 CLIP
Neuron
Random Circuit
24.9
24.7
25.0
81.0
82.6
87.7
Nañve Circuit
23.0
25.8
29.2
95.2
94,9
97.1
Edge-based Discovery
26.3
26.7
30.2
92.4
93.7
97.1
Gradient Correction
61,4
42.8
32.4
94.2
94.4
95.4
Feature
Random Circuit
30.2
27.5
27.0
78.1
90.1
84.4
35.6
33.9
31.1
Naive Circuit
64.9
58.9
50.7
94.2
99.0
99.3
46.9
42,2
40.3
7
Eddgebbsed Discovery
74.2
64.1
52.2
93.4
98.8
99.2
48.6
43.4
43.0
Gradient Correction
94.1
85.1
82.3
99.6
99.8
99.7
54.5
54.8
53.8
3.2
Validation
Metrics
We evaluate the circuit C using three criteria: faithfulness, completeness, and causality. Faithfulness quantifies how well the circuit accounts for the model's behavior:
mm(C m(2)
[C, 9: mg) - m)))"
where G is the full model graph and Ø is the empty circuit.
Completeness, in contrast to faithfulness, measures the necessity of the circuit. Completeness is defined as f(G \ C), which captures how much performance is lost when C. is removed from the full model. To remove the circuit C from the residual stream, we ablate the selected features and replace them with their median activation values across the dataset. Since lower values indicate that more necessary features are captured by C, we report
completeness for convenience.
While faithfulness and completeness are widely used [1 51 87 89], they only reflect the final model performance and do not ensure that the circuit reflects the model's internal computation. To address this, we also measure causality, which assesses whether the connections in C' reflect true causal influence. Specifically, downstream nodes at layer E' f, computed [V,l Edev, d
ablate the nodes layer É and observe changes in the activations of
Ad where Ad denotes the reduction
in activation of downstream node d after ablation. We average this value across all layers & and all downstream nodes Ve. We only compute causality for feature circuits, since the causality metric meaningful only when the activations of the nodes that constitute a circuit are guaranteed to be non-negative.
Since the circuit C depends on the number of selected features k, evaluate each metric across different values of and report the area under the curve (AUC) 84 89 90). Additionally, to prevent overestimation of AUC, we clamp metric values the range 1]. For instance, while faithfulness is expected to lie within this range, it may exceed 1 in practice [87 89). However, since our goal is to identify circuits that best reflect the model's original behavior, such values are undesirable and thus capped at 1. For more details on the metrics, please refer to H..3
Results Table T shows the performances of circuits discovered using different strategies. For context, we also include the performance of random circuits, constructed by randomly selecting nodes. The results summarized follows:
C Contributions of Edge-based Discovery and Gradient Correction. To assess the contributions of bdge- discovery and gradient correction, we define the circuit discovered via node-based strategy with vanilla gradients as the naïve circuit, and measure improvements relative to it. As shown in the 6th-8th rows of Table T both components consistently improve the circuit's faithfulness and causality.
@ Neuron Circuit vs. Feature Circuit. We compare the feature circuit with the neuron circuit, where polysemantic neurons are used as the unit nodes instead of features. As shown in the 4th and 8th rows of Table I the feature circuit consistently outperforms the neuron circuit across all metrics, while also offering greater interpretability. This result indicates that SAFs effectively decompose dense neuron activations into sparse, disentangled features, yielding more parsimonious circuit.

---

LG158
Input Image
Granny smuth:
green
round object
1: L331443
L6A2676
L.415663
1641:538
granny
eITOr
Figure 5: Granny Smith Circuit. Each node and edge is shaded according to its importance, with darker tones indicating higher importance. Node symbols represent the interpretation of each feature.
3.3 Interpreting ViTs via Residual Replacement Model
In contrast to LMs [5],54,, there has been little understanding of how ViTs function end-to-end, primarily due to the large number of tokens and complex attention mechanisms. However, the residual replacement model yields a parsimonious circuit that aggregates coken-to-token interactions and skips the attention modules. This enables a
interpret the decision-making process of VITs at scale. To illustrate this, we present a qualitative analysis of a discovered circuit in ViT-B/16. We then analyze the circuits quantitatively, focusing the trends of feature similarity across layers.
Qualitative Analysis: Granny Smith Circuit How does a ViT recognize an image of a "Granny Smi th"t belonging the "Granny Smith" class? One might expect the model to detect attributes such as the fruit's green color and round shape before identifying it as a Granny Smith th. Indeed, this confirmed in practice.
As illustrated in Figure we interpret the top-3 feature circuit to explain how the VIT processes a "Granny Smith' image. We find that the activated features categorized into three types: green color, round shape (or round fruits), and the specific concept of Granny Smith. In the early layers (layers 0 to 5), the model primarily attends to green color features, which are low-level visual cues. By layer 6, round shape detectors emerge, which evolve into more specific features like round fruits by layer 8. Finally, in layer 9, highly specific feature that activates only for Granny Smi th appears, combining both color and shape information. This feature is maintained through to the final layer, where it plays dominant role in the model's 's classification decision. Overall, the residual replacement model reveals a clear progression from low-level features to high-level concepts, culminating in the specific identification of Granny Smith.
Additionally, we observe that many features connected by high-importance edges exhibit high cosine similarity in their feature vectors [9I] [92] This finding validates that similar features across layers indeed preserved through the residual connections 95) in images, beyond statistical correlation between features [9 97). See for further details and additional analysis.
We analyze how feature circuit similarity evolves across layers to better understand the ViT's general decision-maie maaking process. Specifically, we compute the Adjusted Dice Score betweee nodes in each layer of the top-100 feature circuits for intra-class image pairs (and inter-class pairs as a baseline), as shown in Figure 6 Across all training objectives, we observe an increase in feature similarity for intra-class images in the later layers, while inter-class similarity remains low. This result suggests that semantically similar images converge to shared internal representations in deeper layers 198 99
Quantitative Analysis
DINOv2 CUP
Intro-class Inter-class
0.4
Aqusted Scom
0.3
0.0
Figure 6: Trend of Feature Similarity
We subtract the expected Dice Score from randomly selected node sets obtain the Adjusted Dice Score.
7

---

3.4 Case Studies Revisited: Curve Circuit and Position Circuit
The residual replacement model is useful not only for understanding the final predictions of ViTs, but also for interpreting internal feature- to-feature interactions. By simply changing the objective of interest m from the logit the tar- get class to the activation of specific feature, we can analyze how the model processes an image to activate that feature. We revisit two table feature types, curve detectors and position detectors (,.)), understand which upstream features contribute to their activation and how they, in turn, influence downstream features.
LOAB2
L14298
L24579
L34801
LO4104
L14341
L2471
LO47B
L1#63
L24423
LONE
L1#E
L2HÉ
Curve Circuit Figure shows the circuit for L3#801, feature that detects upper-half curves (~). Remarkably, the resulting circuit is mech- anistically similar to those found in InceptionVI differences. In layer color contrast features combine to form line detector (L1#2999,, mirroring the Contrast - Line circuit described in [7]] These line detectors then combine to form a curve detector (L24579), consistent with the Line - Curve circuit reported [56). Finally, the curve detectors and line detectors combine to produce the more complex curve detector L3#801. Overall, this progression illustrates that ViTs can construct curve detectors from line detectors, and subsequently build more complex curve detectors through their composition.
Figure 7: Curve Circuit (L3#8801)
56
66
despite substantial architectural
Position Circuit Figure 8 shows the circuit for L34509, feature that activates on the left side of an object. As an- ticipated in $2.3 broader position detectors such as L1#239 emerge through the composition sharper, lower-level po- sition detectors (e.g., L0#80 and LO#116), These features contribute to the model's 's ability detect the left side of objects, in conjunction with additional nodes such as the er- ror term L.2#E. This circuit indicates that position detectors provide spatial information to other features, facilitating the model's s understanding of object layout and arrangement.
LOA80
L14239
L2#448
L3#509
L0#169
LONE
L1#E
L2AE
Figure Position Circuit (L3#509)
Application: Debiasing Spurious Correlations
Vision models often struggle with spurious correlations- that frequently co-occur with target class but are not causally related to it [05) For instance, in ImageNet [106] the class "f 'freight car" often appears with graffiti, leading the model to mistakenly associate the presence of graffiti with the class "freight car". We can leverage the residual replacement model to iden- tify and intervene such spurious correlations.
L942371
L10N1534
freight oae
Discovering Circuits with Spurious Features First, we investigate whether the top-3 feature circuits identified by the residual replacement model can reveal spurious correlations in seven ImageNet classes previously reported exhibit such correlations [109]. For each class, we in- terpret the circuit of single image containing spurious feature and identify the spurious node within the circuit. We find that the residual re- placement model consistently discovers the criti- cal spurious features. For example, as illustrated in Figure 9 the "freight car" feature in the final layer (L.1##1534) is composed of three up- stream features: graffiti (L9#2371), container
L992729
L10R1238
LB42190
L94553
Ingut imago
LaVE
L10NE
Figure 9: freight car Circuit

---

(L9#2729), and wheel (L9#553). This suggests that the ViT makes predictions compositionally based on object parts and places considerable weight on spurious features (graffiti).
Debiasing Spurious Feature Circuits We then ablate this spurious feature (e.g., L9#2371 in Figure 9 to assess whether the model be- comes less dependent on it after intervention. Following the evaluation protocol of Neuhaus al. [09]. we report (I) the accuracy of the in- tervened model on the original ImageNet dataset and (2) the AUC for distinguishing between im- ages that contain only the spurious feature (without the target class) and those that contain the actual target class. Successful debiasing should improve the AUC while maintaining the model's original accuracy. For reference, we also include SpuFix [109] as upper bound, which analyzes the entire training set each category to identify and mitigate spurious correlations in a data-driven manner. As shown in Table 2 our intervention effectively reduces reliance spurious correlations with minimal loss in overall accuracy. Remarkably, the performance ablating a single feature from a single image is close to that of SpuFix, suggesting that the identified spurious feature generalizes across images within each class. Overall, we find that the residual replacement model not only identifies spurious features but also enables effective debiasing of the model's 's internal mechanism.
Table 2: Debiasing Spurious Correlations
Accuracy
mAUC
Original ViT Intervened ViT SpuFix (Upper bound) 0.848
0.849
0.854
0,848
0,904
0.917
5 Related Work
To interpret neural networks, various methods have been proposed, including input attribution, concept discovery, and circuit discovery. Input attribution methods explain model predictions by assigning importance to inputs [6 90 [101337 However, these methods can be unreliable [2] 138 and
only provide "where" the model focuses, not ""waat" the model learns [139]. To address this, concept discovery and circuit discovery have emerged as promising directions for interpreting the internal representations and mechanisms of neural networks in a numan-unnderstandable manner.
Concept Discovery Concept-based interpretability methods aim to identify and understand the concepts learned by neural networks. Early approaches focused on supervised settings, where human-annotated data was used to define desired concepts [12] More recently, unsuper- vised approaches have enabled open-ended discovery of concepts directly from the model [143
146].
These methods typically extract concepts using techniques such as PCA (or SVD)
147
152
NMF 139 147 153], or clustering methods
[98 157] Among these, SAES 225
32 35
40
[58
159 and their variants [38] 39 94] have emerged as particularly effective and SAEs are also being increasingly applied to vision models (43550].
scalable tools
[33 34
Circuit Discovery Olah et al. introduced the concept of a circuit as subgraph of neural network and investigated various types of circuits in Inception [66 67 7 Since then, many
strategies have been proposed to discover circuits in vision models 6 7 99 163) including automated discovery methods for CNNs ( More recently, circuits have also been identified in
164 165]. However, since neurons or coarse units such as attention heads are often polysemantic 21 66), the resulting circuits can be difficult to interpret. To address this, recent work on LMs has focused on discovering circuits in replacement models, where neurons in the original model are substituted with interpretable features [36 In this paper, we extend the concept the replacement model to ViTs by overcoming various challenges, ultimately introducing the residual replacement model. We also provide more detailed discussion of related works in $B
LMs 20 54
6 Conclusion
We present the first comprehensive interpretation of features across all layers of ViTs and introduce the residual replacement model, a scalable framework for constructing interpretable feature circuits in ViTs. Through large-scale human annotation, we show that ViT features are broadly interpretable and follow clear progression from low-level visual cues to high-level semantic concepts across layers. The residual replacement model enables faithful extraction of meaningful circuits within seconds, providing insight into how ViTs form intermediate representations, such as curves and positions, and

---

ultimately arrive at their predictions. Furthermore, we demonstrate that our framework can be used to identify and mitigate spurious correlations ViTs, offering a practical tool for debugging the model.
Limitation and Future Work While SAEs significantly improve feature interpretability. recent studies have identified several limitations 44 [7I] We use the TopK SAE variant due to its popularity [34]. but we expect that improved versions of SAEs could further enhance feature quality (29 32 172) Given the qualitative nature interpretability, we validate our interpretations through a user study and cross-checking ($2.2), with additional examples provided in the Appendix. The manual annotation process inherently involves subjectivity, and it is unavoidable that certain subtle concepts may be missed. While automating feature interpretation for vision models holds promise for scalable and less subjective analysis, it remains considerably more challenging than in the language domain. An exciting direction for future work is to leverage feature annotation data to advance and evaluate automatic interpretability methods, Lastly, although the residual replacement model offers a parsimonious representation of the circuit, it only indirectly captures the roles of attention and feed-forward modules via gradient- bbased methods, and thus does not explicitly explain their functions. While interpreting these modules introduces additional complexity, believe that uncovering their internal mechanisms is also promising direction toward a more fine-grained understanding of how ViTs function 173 176)
Acknowledgments and Disclosure of Funding
This work was supported in part by the HTTP RS 2020-I120136,, NRF 25-2024-00345806, NRF RS- RS-2023-0021901 and NRF RS-2023-002620. The authors would like to thank Chanmin Kim, Dahye Kim, Deogyong Kim, Heejae Chon, Hyunah Ko, Inhu Kim, Jaeseung Baek, Jeongeun Lee, Junseong Lee, Kihyuk Lee, Kyungwon Park, Sangmin Lee, Yena Kim, Yeonu Jeong, Youngjae Kang, Youngjoo Lee for participating in our user study.
(AI Research Hub Project). IITP
10

---

References
[1] Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi:
1012391/dissiil.00000 https://distill.ubbbtt puw/2011//uuldiigg 8/builling-
[2] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov. and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/diisiil.0002222
[3] Yulong Wang. Hang Su, Bo Zhang, and Xiaolin Hu. Interpret neural networks by identifying critical data routing paths. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8906-880 5906-8914. IEEE, 2018.
[4] Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is neuron? arXiv preprint a805: 12233, 2018.
[5] Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. Advances in Neural Information Processing Systems, 1115922-59333 2020.
[6] Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Christian Rupprecht, Seong Tae Kim, and Nassir Navab. Neural response interpretation through the lens of critical pathways. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13523-13533. IEEE Computer Society, 2021.
[7] Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Scbastian Bosse, Thomas Wiegand, Wojciech Samek, and Sebastian Lapuschkin. From attribution maps to human understaaatt explanations through concept relevance propagation. Nature Machine Intelligence, 5(9): 1006-1019, 2023.
[8] Achyuta Rajaram, Neil Chowdhury, Antonio Torralba, Jacob Andreas, and Sarah Schwettmann. Automatic discovery of visual circuits. arXiv preprint arXiv:2404. 14349, 2024.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[10] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[11] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pages 1-31, 2024.
[12] Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC 2021 -32nd British Machine Vision Conference, 2021.
[13] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou. Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervise vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[14] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696-712. Springer, 2022.
[15] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 34916, 2023.
[16] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.

---

[17] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 1188731--77566, 2024.
[18] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chun- yuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38-55. Springer, 2024.
[19] Fernando Pérez-Garchs Harshita Sharma, Sam Bond-Taye Taylor, Kenza Bouzid, Valentina Sal- vatelli, Maximilian Ilse, Shruthi Bannur, Daniel Castro, Anton Schwaighofer, Matthew Lungren, et al. Exploring scalable medical image encoders beyond text supervision. Nature Muchine Intelligence, pages 2025.
[20] Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds. Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http::/transformer-cireee pitpeersrrrrruussssp/222222 /fframework/nddee..nnn
[21] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravee, Zac -Ddddd, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209. 10652, 2022.
[22] Maximilian Dreyer, Jim Berend, Tobias Labarta, Johanna Vielhaben, Thomas Wiegand, Sebas- tian Lapuschkin, and Wojciech Samek. Mechanistic understanding and validation of large ai models with semanticlens. arXiv preprint DeXiiv:5501.03990 2025.
[23] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0477 2014.
[24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
[25] Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. arXiv preprint arXiv:2103. 15949, 2021.
[26] Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders. In A/ Aligument Forum, volume 8, pages 15-16, 2022.
[27] Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2023.
[28] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023.
[29] Dan Braun, Jordan Taylor, Nicholas Goldowsky-Din and Lee Sharkey, Identifying functionally important features with end-to-end sparse dictionary learning. Advances in Neural Information Processing Systems, 37:107286-107325 2024.
[30] Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404. 16014. 2024.
[31] Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint 24X 2024.
12

---

[32] Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410, 2024.
[33] Adly Templeton. Scaling monosemanticity. Extracting interpretable features from claude 3 sonnet. Anthropic. 2024.
[34] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04090 2024.
[35] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Son- nerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at arXiv:2408.. 1.Xiv:2408.0514777 2024.
gemma 2. arXiv preprini
[36] Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. arXiv preprint arXiv:2402. 12201, 2024.
[37] Charles O' and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. arXiv preprint arXiv:2405. /2522, 2024.
[38] Xuyang Ge. Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Auto- matically identifying local and global circuits with linear computation graphs. arXiv preprint arXiv:2405. 13868, 2024.
[39] Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances Neural Information Processing Systems, volume 37, pages 24375-244 Curran Associates, Inc., 2024. URL https htt.:attpps//prrcceednnggs neurips cc/ cctpppprccpppppppppppppppppppp papp/_frpeess_sss_p_p files/ pdf
[40] Lucy Farnik, Tim Lawson, Conor Houghton, and Laurence Aitchison. Jacobian sparse au- toencoders: Sparsify computations, not just activations. arXiv preprint arX1v:2502.18147, 2025.
[41] Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:24033 19647, 2024.
[42] Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi. T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https: /ttps:///rrass ormer- cir crruitttts........ pub/pu2bp2222 10225/attriie /tt//aattiiioiooooo www.riittoooonnnoeeeneaeeeeeeeee
html
[43] Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-namee Task- agnostic concept bottlenecks via automated concept discovery. European Conference on Computer Vision, pages 444-461. Springer, 2024.
[44] Hyesu Lim, Jinho Choi, Jacgul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. arXiv preprint arXiv:2412.05276. 2024.
[45] Harrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos Derpanis. Universal sparse autoencoders: Interpretable cross-moded concept alignment. arXiv preprint arXiv:2502.0377 2025.
[46] Samuel Stevens, Wei-Lun Chao, Tanya Berger-Wolf, and Yu Su. Sparse autoencoders for scientifically rigorous interpretation of vision models. arXiv preprint arXiv:2502.06753 2025.
13

---

[47] Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models. arXiv preprint arXiv:2502.. /2892, 2025.
[48] Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting clip with hierar- chical sparse autoencoders. arXiv preprint arXiv:2502.20578, 2025.
[49] Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models. arXiv preprint arXiv:2504.028211 2025.
[50] Sonia Joseph, Praneet Suresh, Ethan Goldfarb, Lorenz Hufe, Yossi Gandelsman, Robert Graham, Danilo Bzdok, Wojeiech Samek, and Blake Aaron Richards. Steering clip's vision transformer with sparse autoencoders. arXiv preprint arXiv:2504.0872 2025.
[51] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint urXiv:211..0593,, 2022.
[52] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in pre-trained language model. Advances in Neural Information Processing Systems, 36:76033-76060, 2023.
[53] Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09455, 2023.
[54] Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, and Yonatan Belinkov. Position-aware automatic circuit discovery. arXiv preprint arXiv:2502.04577, 2025.
[55] Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, 5(6):=00024-003, 2020.
[56] Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 6())e00022-0066 2021.
[57] Liv Gorton. The missing curve detectors of inceptionvl: Applying sparse autoencoders to inceptionvi early vision. arXiv preprint S0VV:220060366622 2024.
[58] Liv Gorton. Group crosscoders for mechanistic analysis of symmetry. arXiv preprint arXiv:2410.24184, 2024.
[59] Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. In Findings of the Association for Computational Linguistics ACL 2024, pages I 1288-1301, 2024.
[60] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. arXiv preprint 11111 2022.
[61] Tuomas Oikarinen and Tsui- Weng. Clip-dissect: Automatic description of neuron repre- sentations in deep vision networks. arXiv preprint arXiv:2204, 10965, 2022.
[62] Yong Hyun Ahn, Hyeon Bae Kim, and Seong Tae Kim. Www: A unified framework for explaining what, where and why of neural networks by interpretation of neuron concepts. In 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10968-10977. IEEE, 2024.
[63] Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv:1312.50 2013.
[64] Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. URL http:::t//s/ee wwtp:/wwwwwww....a.oo.. org Software available from neuronpedia.org
14

---

[65] Julien Colin, Lore Goetschalckx, Thomas Fel, Victor Boutin, Jay Gopal, Thomas Serre, and Nuria Oliver. Local vs distributed representations: What is the right basis for interpretability? arXiv preprint arXiv:241 I. .03993, 2024.
[66] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview of early vision in inceptionvl. Distill, 5(4):c00024-003 2020.
[67] Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. High-low frequency detectors. Distill, 6(1):e000240005, 2021.
[68] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999. PMIR, 2016.
[69] Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 359-858. IEEE, 2018.
[70] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rota- tion equivariant cnns for digital pathology. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 210-218, 2018.
[71] Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh. Naturally occurring equivariance in neural networks. Distill, 5(12)xc00244-004 2020.
[72] Robert-Jan Bruintjes, Tomasz Motyka, and Jan van Gemert. What affects learned equivariance in deep image recognition models? In 2023 IEEE/CVF Conference Computer Vision and Puttern Recognition Workshops (CVPRW), pages 4839-4847. IEEE Computer Society, 2023.
[73] Neel Nanda. Attribution patching: Activation patching at industrial scale. URL: https://www. 2023.
neelnanda. hat.:/ansscc-nteeereeebbbiaarrrr
[74] Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. arXiv preprint 1tt.:23300.00445 2023.
[75] János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp An efficient and scalable method for localizing 1lm behaviour to components. arXiv preprint (rXiv:2403.00745. 2024.
[76] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388- 2020.
[77] Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373-392. 2022.
[78] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems. 15:77259-77372 2022.
[79] Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metries and methods. arXiv preprint arXiv:2309. 16042, 2023.
[80] Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. arXiv preprint S.3iv:22004152555 2024.
[81] Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, Ben Thompson, Sam Zimmerman, Kelley Rivoire. Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of a large language model. Transformer Circuits Thread, 2025. URL https:// transformer www.rorrrerrrcuttt.... www.ciis.ccctceeeeeeeeeeeeeeeeee pwwp://///eeoeeee/e// http:///tribuuiion graphs/biolog.. html
[82] David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In International conference on machine learning, pages 342-350, PMLR, 2017.
15

---

[83]
Ann-Katinn Dombrowski, Christopher J Anders, Klaus-Robert Müller, and Pan Kessel. To- wards robust explanations for deep neural networks. Pattern Recognition, 121:108194, 2022.
[84] Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Attnirp: Attention-aware layer-wise relevance propagation for transformers. In International Conference on Machine Learning, pages 135-168. PMLR, 2024.
[85] Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualiza- tion. Advances in neural information processing systems, 32, 2019.
[86] Faridoun Mehri, Mahdieh Soleymani Baghshah, and Mohammad Taher Pilehvar. Libragrad: Balancing gradient flow for universally better vision transformer attributions. arXiv preprint arXiv:2411.16760, 2024.
[87] Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit faithfulness metrics are not robust. arXiv preprint arXiv:2407.08734, 2024.
[88] Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv preprint arXiv:2403.17800 2024.
[89] Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fiouto-Kaufman, Tal Haklay, Michael Hanna, al. Mib: A mechanistic interpretability benchmark. arXiv preprint 2rXiv:2504.13/5, 2025.
[90] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation black-box models. arXiv preprint
2018.
[91] Daniil Laptev, Nikita Balagansky, Yaroslav Aksenov, and Daniil Gavrilov. Analyze feature flow to enhance interpretation and steering language models. arXiv preprint arXiv:2502.03032. 2025.
[92] Nikita Balagansky, Ian Maksimov, and Daniil Gavrilov. Mechanistic permutability: Match features across layers. arXiv preprint arXiv:2410.07656, 2024.
[93] Tim Lawson, Lucy Farnik, Conor Houghton, and Laurence Aitchison. Residual stream analysis with multi-layer saes. arXiv preprint arXiv:2409.0441 2024.
[94] Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christo- pher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024.
[95] Davide Ghilardi, Federico Belotti, Marco Molinari, and Jachyuk Lim. Accelerating sparse autoencoder training via layer-wise transfer learning in large language models. In Proceedings of the 7th BlackboxNIPP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 530-550, 2024.
[96] Junxuan Wang, Xuyang Ge, Wentao Shu, Qiong Tang, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Towards universality: Studying mechanistic similarity across language model architectures. arXiv preprint arXiv:2410.06672, 2024.
[97] Daniel Balcells, Benjamin Lerner, Michael Oesterle, Ediz Ucar, and Stefan Heimersheim. Evolution of sae features across layers in 1lms. arXiv preprint arXiv:2410. 0886699,,, 2024.
[98] Johanna Vielhaben, Dilyara Bareeva, Jim Berend, Wojeiech Samek, and Nils Strodthoff. Beyond scalars: Concept-based alignment analysis in vision transformers. arXiv preprint arXiv:2412.06639 2024.
[99] Matthew Kowal, Richard Wildes, and Konstantinos G Derpanis. Visual concept connectome (vec): Open world concept discovery and their interlayer connections in deep models. In 2024 IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10895-10905. IEEE, 2024.
16

---

[100] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trainee traaned cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International conference learning representations, 2018.
[101] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456-473, 2018.
[102] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9558 2020.
[103] Luke Oakden-Rayner, Rayner, Jared Dunnmon, Gustavo Cameiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pages 151-159, 2020.
[104] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In International Conference on Learning Representations, 2021.
[105] Katherine L Hermann, Hossein Mobahi, Thomas Fel, and Michael C Mozer. On the foundations of shortcut learning. arXiv preprint 1rXiv:2310.162. 2023.
[106] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. pages
Ieee, 2009.
[107] S Singla and S Feizi. Salient imagenet: How to discover spurious features in deep learning. In International Conference on Learning Representations (ICLR), 2022.
[108] Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard imagenet: strong spurious cues. Advances in Neural Information Processing Systems, 35:10068-10077, 2022.
Segmentations for objects with
[109] Yannic Neuhaus, Maximilian Augustin, Valentyn Boreiko, and Matthias Hein. Spurious features everywhere-large-seee detection of harmful spurious features in imagenet. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 20235-20246, 2023.
[110] Yongjie Wang, Tong Zhang, Xu Guo, and Zhiqi Shen. Gradient based feature attribution in explainable ai: A technical review. arXiv preprint IxXi:22433100155 2024.
[111] Shichang Zhang, Tessa Han, Usha Bhalla, and Himabindu Lakkaraju. Building bridges, not walls aavlnsad interpretability by unifying feature, data, and model component attribution. arXiv preprint arXiv:2501. 18887, 2025.
[112] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818-833. Springer, 2014.
[113] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.
[114] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In 2017 IEEE International Conference on Computer Vision (ICCV), pages IEEE, 2017.
[115] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in Neural Information Processing Systems, 30, 2017.
[116] Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: An information-theorrtio perspective on model interpretation. International Conference on Machine Learning, pages 883-892. PMLR, 2018.
17

---

[117] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2950-2958 IEEE, 2019.
[118] K Simonyan, A Vedaldi, and Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR, 2014.
[119] Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: I4:X:v::41...0 wet...422.... 2.6866, 2014.
[120] Ramprasaath R Selvaraju, Abhishek Das. Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint arXiv: 16100450, 2016.
[121] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328. PMLR, 2017.
[122] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning. pages 3145-3153, PMIR, 2017.
[123] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint arXiv: 1706:17000000 1706:00865, 2017.
[124] Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22(104):1-54, 2021.
[125] S Bach, A Binder, G Montavon, F Klausehen, KR Müller, and W Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7)=e0130140, 2015.
[126] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renor- malization layers. In Artificial Neural Networks and Machine Learning-ICAN 2016: 25th International Conference on Artificial Neural Networks, Barcelona, Spain, September 6-9, 2016, Proceedings, Part 11 25, pages 63-71. Springer, 2016.
[127] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojeiech Samek, and Klaus- Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern recognition, 1-22,, 2017.
[128] Elena Voita, David Talbot, Fedor Moiseev, Rico Semnrich, and Ivan Titov. Analyzing multi- self-attention: Specialized heads do the heavy lifting, the rest can be pruned. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, 2019.
[129] Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samek, and Klaus- Robert Müller. Layer-wise relevance propagation: an overview. Explainable Al: interpreting, explaining and visualizing deep learning, pages 193-209, 2019.
[130] Zhengxuan Wu and Desmond C Ong- On explaining your explanations of bert: An empirical study with sequence classification. arXiv preprint arXiv:2101.00196, 2021.
[131] Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Glaus-Robbrr Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. In International Conference on Machine Learning, pages 435-451, PMLR, 2022.
[132] Leila Arras, Bruno Puri, Patrick Kahardipraja, Sebastian Lapuschkin, and Wojeiech Samek. A close look at decomposition--aasel xai-methods for transformer language models. arXiv preprint 2502 15886, 2025.
18

---

[133] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1190-4197. 2020.
[134] Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. Globenc: Quantifying global token attribution by incorporating the whole encoder layer in transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 238227,, 2022.
[135] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
[136] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi. Franco Turini, and Fosca Giannotti. Local rule-based explanations black box decision systems. arXiv preprint arXiv:1805. 2018.
[137] Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Steven Wu, and Himabindu Lakkaraju. Towards the unification and robustness of perturbation and gradi- ent based explanations. In International conference on machine learning, pages 110-119. PMLR, 2021.
[138) Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. Explainable Al: Interpreting, explaining and visualizing deep learning. pages 267-280, 2019.
[139] Thomas Fel,, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Remi Cadenc, and Thomas Serre. Craft: Concept recursive activation factorization for explain- ability. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2711-2721. IEEE Computer Society, 2023.
[140] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677. PMLR, 2018
[141] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 3319-3327. IEEE, 2017.
[142] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference Computer Vision (ECCV), pages 119-134, 2018
[143] Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems, 36:54805-54815 2023.
[144] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Inter- preting clip with sparse linear concept embeddings (splice). Advances in Neural Information Processing Systems, 37:84298-8848 84328, 2024.
[145] Hengyi Wang, Shiwei Tan, and Hao Wang, Probabilistic conceptual explainers: Trustworthy conceptual explanations for vision foundation models. In International Conference on Machine Learning, pages 51 51502-51522. PMLR, 2024.
[146] Robin Hesse, Jonas Fischer, Simone Schaub-Meyer, and Stefan Roth. Disentangling poly- semantic channels in convolutional neural networks. In The First Workshop on Mechanistic Interpretability for Vision, 2025.
19

---

[147] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-basse based explanations for emn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1162--11690, 2021.
[148] Mara Graziani, An-phi Nguyen, Laura O' O'Mahony, Henning Müller, and Vincent Andrearezyk. Concept discovery and dataset exploration with singular value decomposition. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML. 2023.
[149] Mara Graziani, Laura O' Mahony, An-Phi Nguyen, Henning Müller, and Vincent Andrearezyk. Uncovering unique concept vectors through latent space decomposition. arXiv preprint arXiv:2307.0691. 2023.
[150] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint vxXiv:2310006824 2023.
[151] Oskar Hollinsworth, Curt Tigges, Atticus Geiger, and Neel Nanda. Language models linearly represent sentiment. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 58-87, 2024.
[152] Jing Huang. Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluat- ing interpretability methods on disentangling language model representations. arXiv preprint arXiv:2402. 17700, 2024.
[153] Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert, Ben Egan, Swee Kiat Lim, and Chris Olah. Visualizing weights. Distill, 6(2):e00024-007 2021.
[154] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. Advances in Neural Information Processing Systems, 32, 2019.
[155] Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff Multi-dimennional concept discovery (med): A unifying framework with completeness guarantees. arXiv preprint 11111:2301.11911, 2023.
[156] Laura O' Mahony, Vincent Andrearezyk, Henning Müller, and Mara Graziani. Disentangling neuron representations with concept vectors. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3770-3775. IEEE, 2023.
[157] Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojeiech Samek, and Sebastian Lapuschkin. Pure: Turning polysemantic neurons into pure features by identifying relevant circuits. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2212-8217. 2024.
[158] Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. arXiv preprint arXiv:2406. 17759, 2024.
[159] Kola Ayonrinde, Michael T Pearce, and Lee Sharkey. Interpretability as compression: Recon- sidering sae explanations of neural activations with mdl-saes. arXiv preprint arXhv:2410.1119, 2024.
[160] Fuxun Yu, Zhuwei Qin, and Xiang Chen. Distilling critical paths in convolutional neural networks. arXiv preprint
2018.
[161] Chris Hamblin, Talia Konkle, and George Alvarez. Pruning for feature-preserving circuits in cnns. arXiv preprint arXiv:2206.0166 2022.
[162] Sayed Mohammad Vakilzadeh Hatefi. Maximilian Dreyer, Reduan Achtibat, Thomas Wiegand, Wojeiech Samek, and Sebastian Lapuschkin. Pruning by explaining revisited: Optimizing attribution methods to prune cnns and transformers. arXiv preprint arXiv:2408. /2568, 2024.
[163] Yifan Wang, Yifei Liu, Yingdong Shi, Changming Li, Angi Pang, Sibei Yang. Jingyi Yu, and Kan Ren. Discovering influential neuron path in vision transformers. arXiv preprint arXiv:2503.m 203002503309046,, 2025.
20

---

[164] Arthur Conmy, Augustine
Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonoo Towards automated eircuit discovery for mechanistic interpretability. Ad- vances in Neural Information Processing Systems, 36: 0116668-663355 2023.
[165] Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. Advances in Neural Information Processing Systems, 37:18506- 18534, 2024.
[166] Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:2312.092210, 2023.
[167] David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409. 2024.
[168] Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis. arXiv preprini uuXii:2502.04555 2025.
[169] Gonçalo Paulo and Nora Belrose, Sparse autoencoders trained on the same data learn different features. arXiv preprint arXiv:2501.16615, 2025.
[170] Subhash Kantamneni, Joshua Engels. Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? a case study in sparse probing. arXiv preprint arXiv:2502. 16681, 2025.
[171] Sai Sumedh R Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry. arXiv preprint arXiv:2503.01822 2025.
[172] Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. arXiv preprint arXiv:2503. 2025.
[173] Martina G Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. Advances in neural information processing systems, 36:40030-40041, 2023.
[174] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint ar Ar3v 130.,888966688, 2023.
[175] Yossi Gandelsman, Alexei Efros, and Jacob Steinhardt. Interpreting clip's image representa- tion via text-based decomposition. arXiv preprint arXiv:2310.055116 2023.
[176] Sriram Balasubramanian, Samyadeep Basu, and Soheil Feizi. Decomposing and inter- preting image representations via text in vits beyond clip. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural In- formation Processing Systems, volume 37, pages 1104--80776. Curran Associates, Inc., neur cc/ wwtppccccpppppppcceeeeeeceeeepee paper_ files/panne pass//pper/222///lee pdf
2024. URL
[177] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention modell explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 397-406 2021.
[178] Yao Qiang, Deng Pan, Chengyin Li, Xin Li, Rhongho Jang, and Dongxiao Zhu. Attcat: Explaining transformers via attentive class activation tokens. Advances in neural information processing systems, 35:5052-5064, 2022.
[179] Junyi Wu, Bin Duan, Weitai Kang, Hao Tang, and Yan Yan. Token transformation matters: Towards faithful post-hoe explanation for vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[180] Haiyan Zhao, Fan Yang, Bo Shen, Himabindu Lakkaraju, and Mengnan Du. Towards un- covering how large language model works: An explainability perspective. arXiv preprint arXiv:2402. 10688 2024.
21

---

[181] Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety review. arXiv preprint arXiv:2404.14082, 2024.
[182] Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. A primer on the inner workings of transformer-based language models. arXiv preprint arXiv:2405.00208, 2024.
[183] Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint urXi:2440.002646, 2024.
[184] Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang, Bo Tang, Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey, arXiv preprint arXiv:2409.03752, 2024.
[185] Maithra Raghu, Thomas Unterthiner, Simon Korbllihh, Chiyuan Zhang, and Alexey Doso- vitskiy. Do vision transformers see like convolutional neural networks? Advances Neural Information Processing Systems, 34:12116-12122 2021.
[186] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting the second-order effects neurons in clip. arXiv preprint arXiv:2406.043444 2024.
[187] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprini 151.0435,,, 2015.
[188] Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega. Joseph Bloom, et al. Open problems in mechanistic interpretability. arXiv preprini arXiv:2501. 16496, 2025.
[189] Trenton Bricken, Siddharth Mishra-Sharma, Jonathan Marcus, Adam Jermyn, Christopher Olah, Kelley Rivoire, and Thomas Henighan. Stage-wise model diffing. Transformer Circuits Thread, 2024.
[190] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2 (11);e7, 2017.
[191] Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? visual exploration. arXiv preprint arXiv:2212.06727 2022.
[192] Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom Rousseau, Rémi Cadène, Lore Goetschalekx, al. Unlocking feature visualization for deep network with magnitude constrained optimization. Advances in Neural Information Processing Systems, 36:37813-37826, 2023.
[193] Judy Borowski, Roland Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain enn activations better than state-of-the-art feature visualization. arXiv preprint arXiv:2010. 1222000..226000 /2606 2020.
[194] Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un) reliability of feature visualizations. In International Conference on Machine Learning, pages 15294-15330. PMLR, 2024.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims and contributions are summarized in the abstract, introduction, and Figure
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work is discussed in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This is not a theoretical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [ess]
Justification:
We described our implementation details and training results in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: We will release our code, SAE weights, and datasets to support future research in interpretability.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/nsps.cc/public//guu guwdes/ddee/// CodeSubmeeeeeeeee wwweesuubissiooPpoe ssiocyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details are described in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We report error bars Figure 2 to support statistical significance. While some error bars or standard deviations are omitted in the main text for clarity, additional details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification: We provide the compute resources, memory, and time of execution in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics and our research conforms
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not expect harmful societal impacts from our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Since work does not involve harmful datasets models, it poses foreseeable risk misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Ys]]
The data and models used in this work are explicitly mentioned and properly cited.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, taperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our dataset and source code along with detailed documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We describe the details of our study and annotation process in the Ap-
pendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
28

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Method development in this research does involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non otandtdddard components.
Please refer to our LLM policy (https: for what should or should not be described.
29

---

A Additional Related Work
To provide further context for our work, we additionally review related studies in the areas of input attribution methods and mechanistic interpretability in vision transformers.
Input Attribution Methods Input attribution methods aim identify the most important input features for a model's prediction 86 99, [ICH137]. These methods are used to understand model behavior and detect potential biases. Among various input attribution methods, gradient- methods, such as Integrated Gradients [ and Layer-wise Relevance Propagation (LRP) [25 127], are widely used for their scalability and efficiency. In addition, since attention mechanisms in transformer-bat based models can offer interpretable features, attention-based methods have also been proposed [33 [34 177. Input attribution techniques have been extended to interpret internal components of the model, such neurons or attention heads example, attribution patching estimates the causal effect a component by replacing its activation with baseline value using gradients [73]. Recent works have proposed combining Integrated Gradients with attribution patching to improve the quality of estimation [41 88]. In this work, extend LibraGrad [86]. theoretically grounded, gradient-based input attribution method, to estimate the importance of each SAE feature, which significantly improves the faithfulness of the interpretation.
73 74
76
78
88 [64], For
Mechanistic Interpretability in Vision Transformers Mechanistic interpretability (MI) is an emerging research field dedicated to interpreting the inner workings of neural networks by reverse- engineering their computations into human-interpretable mechanisms [2 180- [84). In particular, to understand the mechanisms of vision transformers (ViTs), early studies employed representation analysis, linear probing, and the logit lens to examine the representations learned by ViTs [173 174. [85]. More recent works have also sought interpret neurons and attention heads in ViTs using natural language descriptions [175] [766 186). In addition, with the rise of sparse autoencoders (SAEs), there has been growing interest in interpreting VIT features through the lens of SAES 50] While the aforementioned approaches explain how individual components of ViTs function, they do offer a comprehensive, end-to-end understanding of the model from input output. To address this, we propose the residual replacement model, which characterizes the entire process by which ViTs transform input images into predictions.
30

---

B Discussion
Towards Interpreting Vision Models We find that interpreting vision models is more challenging than interpreting language models (LMs) for several reasons.
First, the features learned by vision models are often difficult to describe in natural language, making it challenging to provide intuitive explanations. Therefore, we alternatively describe the features using maximally activating images patches. For certain types of features, such curve detectors and position detectors, we additionally provide human-interppeetable patterns to aid understanding. Nonetheless, it remains difficult to provide concise natural language descriptions for some features, whereas maximally activating images or patches often reveal clearer patterns. Developing more effective interpretation strategies, particularly those that leverage visual demonstrations, remains an important direction for future work.
Second, the number of patch tokens in vision models is significantly larger than the number of tokens typically used in mechanistic interpretability studies of language models [51-533] 78 [64 [65]. This makes it challenging token-to-token interactions. To address this, we aggregate patch tokens, enabling the interpretation of spatially independent feature interactions. While many token-to-token interactions may be redundant due to the semantic similarity between neighboring patches, some interactions can still play important roles. For instance, understanding how the [CLS] token interacts with other tokens could provide valuable insights into the model's behavior [173] We leave the investigation of such interactions to future work.
scalably identify circuits in vision models, especially when considering
Finally, unlike mechanistic interpretability studies in language models where tasks are typically predefined (e.g., indirect object identification [51], greater-than [52]. subject-verb agreement [4I,, vision models often lack clearly defined tasks. Although one can average circuits across images with the same class label, such averaging does not guarantee meaningful or interpretable circuits, even images from the same class may rely on different pathways for prediction. An interesting direction for future work is to develop methods for identifying meaningful circuits corresponding implicitly defined tasks in vision models.
Comparison with Related Work Olah al. (2) attempted reverse-engineer the computations of CNNs by analyzing the weights of Inception and identifying how features in early layers compositionally combine to form higher-level features in deeper layers, resulting in what they called "circuits." " For example, they identified curve detector features that are equivariant to transformations, demonstrating how different features. such as various angles of lines in earlier layers, combine to form a curve detector (55] 56 66, 71]. Our residual replacement model enables similar analysis of features and circuits in ViTs by observing the model's residual stream. Furthermore, we find that similar features and circuits emerge in ViTs, despite their distinct architectures. From the perspective of the universality hypothesis 2 188 this work suggests that CNNs and ViTs may share similar mechanisms for processing visual information, although the representations in the early layers of ViT' tend to more patch-wise compared to those CNNs.
To mechanistically interpret large language models (LLMs), Marks et al. proposed a method for identifying sparse feature circuits, where the nodes correspond to SAE features. They demonstrated how LLMs achieve subject-verb agreement across relative clauses by identifying the specific circuits responsible for this behavior. Concurrently, Ameisen al. 42 proposed constructing attribution graphs within a replacement model equipped with cross-layer transcoder (CLT)) [38 39 189] Lever-
aging the replacement model, they identified various circuits in LLMs, including those responsible for multi-step reasoning and arithmetic [81) Inspired by these works, we aim to construct similarly interpretable sparse circuits in ViTs. However, unlike LLMs, ViTs operate on a much larger number of tokens, making it infeasible to identify circuits at scale using the same methodology. Therefore, we simplify the process by focusing on the residual stream of ViTs 20), which enables us to identify circuits in a more parsimonious and scalable manner.
31

---

C
Training Sparse Autoencoders
C.1
Training
Formulation We apply a Topk SAE (4 to each layer of the residual stream in ViTs. As shown in Equation (1), the Topk SAF maps residual stream representation
Rd sparse representation E IR by selecting the top-k features from W ex bpe), where f denotes the number of sparse features and & is the sparsity level (i.e., the number of selected features). Following Gao et al. [34,, we train the TopK SAE with a combination of two loss functions: reconstruction loss and an auxiliary loss. The reconstruction loss Lreon measures the discrepancy between the original input æ and its reconstruction 2' produced by the decoder:
frecon
For normalized evaluation, we report the Fraction of Variance Unexplained (FVU), defined as:
x a|3
FVU
(5)
which allows comparison across layers and models regardless of the input scale 99 97 The
auxiliary loss Lanx encourages the SAE to represent residual information using dead (inactive) features. Specifically, we define the reconstruction error where #dad = TopK (ReLU W enc ca bprc) )))) The auxiliary loss is then defined as:
& and estimate it via é = W dee dend,
Laxx = ||e éll
(6)
The total loss given by:
C Crecon + OeLaux:
(7)
where a a hyperparameter controlling the weight of the auxiliary loss. While the reconstruction loss ensures faithful input reconstruction, the auxiliary loss discourages feature inactivity (dead feature) and promotes more effective utilization of the sparse feature set.
Implementation Details We trained the TopK SAE using the Adam optimizer with a learning rate of 2 x 10-4, + 0.9, and 3.2 0.999. The model was trained for 50 epochs with a batch size of 512. Hyperparameters are set as follows: = 1/32, kaux 256. Decoder normalization is applied at each training step, and the input x is normalized to have zero mean and unit variance. Each SAE was trained using an RTX 3090 GPU for approximately 12-14 hours per model. We conducted a hyperparameter sweep over the sparsity level & and the expansion rate R f //d to investigate scaling behavior. We used the following backbone models: supervised ViT-B/16 2. DINOv2 ViT-B/14 (with
register tokens) I1 174,, and CLIP ViT-B/16 [00] Training and evaluation are performed on the ImageNet-1K IK (106] training/validation dataset, respectively. For each model, we collect all tokens ([CLS), patch tokens, and register tokens (if present)) from each layer to train the TopK SAE.
C.2 Analysis
Scaling Law To understand the scaling behavior of the TopK SAE in ViTs, we extend the scaling law analysis of Gao al. 34 to vision transformers. Specifically, we fit a joint scaling law of the reconstruction loss with respect to the number of latent features f and the sparsity level & using the following functional form:
(8) where o, 3z. By, 7: C. and r) are the parameters to be fitted. We find that this scaling law fits well across all models and layers, explaining at least 99% of the variance. The fitted losses are visualized as contour plots in Figures 10 to [2] where the x- and y-axes correspond to the expansion rate R- = f/d and the sparsity level k. respectively. We additionally mark the contour line corresponding to FVU = 0.15, which indicates the threshold below which the reconstruction loss is considered acceptable. Our chosen hyperparameters are denoted with a cross. Moreover, Figure [ shows the FVU 0.15 contour line across layers. We observe that deeper layers generally require higher sparsity or more latent features to reach the same reconstruction performance, suggesting that they encode more complex information. An exception is found in the penultimate and final layers, which require fewer features, possibly due to their proximity to the output and reduced representational complexity.
L(f,k) = exp (a + 8g log(k) + By log(f) + + log(k) log(f)) + exp (C + 1log(k))),
32

---

Layer
Layer
Layer 2
182
160
160
64-
64-
66
16
1.29
5.20
0.00
1.29
3.20
Layer
Laver
Laver
222
256
206
160
100
160
04-
1.00
Leyer
Layer
Layer
234
256
256
100
128
32-
32-
1.0
24000
1.50
120
120
32-
32
16
5.00
88
2.00
Figure 10: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion
rate R = f f/d and the sparsity level & (VIT). Dotted line indicates the contour line corresponding to FVU 0.15. The cross indicates the chosen hyperparameters for the SAE.
33

---

234
Leyer
256
Layer
256
Layer
100
160
160
04
64
'8
1.00
Layer 3
Layer
Layer 5
162
64-
64-
66
16:
044
640
Laver
Laver
Laver
66
96
16
10
E00
1.00
1.00
1.0
100
238
100
120
123
Figure 11: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion rate R f/d and the sparsity level & (DINOv2). Dotted line indicates the contour line corresponding FVU 0.15. The cross indicates the chosen hyperparameters for the SAE.
34

---

Layer
Layer
Layer 2
162
160
160
64-
64-
64
18
3.29
3.20
3.20
810
Layer
Layer
200
250
250
100
160
160
125
32-
32-
2.00
015
2:00
8.00
5.00
Laver
Laver
1000
100
1.00
10
Layer
Layer
256
256
100
380
32-
16
833
2:80
833
1.00
2.02
-2
Figure 12: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion
rate R = f/d and the sparsity level & (CLIP). Dotted line indicates the contour line corresponding to
FVU 0.15. The cross indicates the chosen hyperparameters for the SAE.
35

---

VIT
DINOV2
CLIP
256
256
256
160
160
160
64
66
18
5.00
4.0
1835
5.00
Figure 13: FVU = 0.15 contour line across layers. Dotted lines are extrapolated from the scaling law.
Activation Analysis To better understand the features learned by the TopK SAE, we analyze the activation patterns of the features. For each latent feature, compute its activation frequency (i.e.. how often the feature is selected) and its average activation value [44]. As shown in Figures [1] to T6] we find a strong positive correlation between activation frequency and average activation value: features with higher values tend to be activated more frequently, indicating their greater importance. Additionally, we observe that intermediate layers often exhibit bimodal distribution in both activation frequency and value. This suggests that they specialize into two groups of features: those that are consistently informative and those that are only selectively used.
36

---

Layer
Layer
Layer
Log 10 Mever Acctroetton
Log 10
Long
Log 10 Activation Frequency
Log 10 Activation Frequency
Log 10 Activation Frequency
Layer
Layer
Layer
Log 10 Mewen Accrovstio
Lo0 10 Mewen
Luo
Log 10 Activation Frequency
Log 10 Activation Frequency
Log 10 Activation Frequency
Layer
Layer 7
Layer
Log 10 Heven
100
Log 10 Activation Frequency
Log 10 Activation Frequency
Log 10 Activation Frequency
Layer
Layer 10
Log 10 Meren Aeitewetoo Walee
Loo 10
Log 10 Activation Frequency
Log 10 Activation Frequency
Figure 14: Activation frequency mean activation value of the TopK SAE features (VIT).
37

---

Layer
Layer
Layer
Log 10 Meeen Actrosston Valee
Log
Log 10 Activation Frequency Layer
Log 10 Activation Frequency
Log 10 Activation Frequency
Layer
Layer
Log 10 Hean Keitweetoo
Log 10
30
Log 10 Activation Frequency
Log 10 Activation Frequency
Log 10 Activation Frequency
Layer
Layer 7
Layer
Log 10
Logg 50
Log
Log 10 Activation Frequency
Log 10 Activation Prequency
Log 10 Activation Frequency
Layer
Layer 10
Log 10 Meren Neitewitoon Vatee
Log
Log 10 Activation Frequency
Log 10 Activation Frequency
Figure Activation frequency and mean activation value of the TopK SAE features (DINOv2).
38

---

Layer
Layer
Layer
Log 10 Newen
Log
Log Activation Frequency
Log 10 Activation Frequency
Log 10 Activation Frequency
Layer
Layer
Layer
Log 10 Weeer Acchoostooo Wilee
Log 10
Leg 10 Activation Frequency
Log 10 Activation Frequency
Log 10 Activation Frequency
Layer
Layer
Layer
Log 10
Log 10
Log 10 Activation Frequency
Log 10 Activation Frequency
Log 10 Activation Frequency
Layer
Layer 10
Log 10 Mere Keitewiton Value
Lo0 10
Log 10 Activation Frequency
Log 10 Activation Frequency
Figure 16: Activation frequency mean activation value of the TopK SAE features (CLIP).
39

---

D Systematic Feature Analysis
D.I
Visualization
In vision models, visualizing maximally activated images is common method for interpreting learned features 45 48, [65]. However, we observe that this approach often insufficient for understanding the features learned by the TopK SAE, particularly in early layers. Early-layer features frequently activate in only a small number of patches (sometimes just a single patch) within an image, rendering the maximally activated image uninformative. To address this, we adopt visualization strategy inspired by language models, where maximally activated tokens are commonly used to visualize the maximally activated patches for each feature, which provides a more localized and interpretable view. Additionally, we provide class label and logit lens interpretation, which are especially informative for late-layer features. Throughout the analysis in $2.2. we employ visualizations such as Figures 17 and T8 to illustrate the characteristics of the learned features.
interpret features [33 64]. Specifically,
D.2 User Study Details
31 35 to compute the interpretability scores of SAE features and neurons. To reduce potential author bias. recruited 16 participants evaluate interpretability following the same procedure. The participants were publicly recruited through an online community platform, and consisted undergraduate and graduate students with no specialized knowledge in mechanistic interpretability. Following the same scoring process as the authors, participants were shown feature visualizations ($D.I) and asked evaluate whether each feature was interpretable by choosing from 'Yes', 'Maybe', or Noo.. Each participant spent approximately two hours on the task and was compensated with minimum wage.
In 2 we follow the protocol of 30
D.3 Categorization
Definition of Categories To systematically analyze the features learned by SAE, we categorize them into several groups based on their characteristics. Since feature categorization is inherently subjective, different researchers may interpret the same feature differently. To ensure consistency in our analysis, we define clear definitions for each category as below, and then categorize each feature using its visualization (DD.I)
Line: In cases where linear curvilinear structure is captured within patch.
Shape: In cases where a distinct geometric shape (e.g., circle or rectangle) is observed within patch.
Color: In cases where consistent coloration is observed, abrupt color transitions detected, or other color-related features are present.
Texture: In cases where recurring texture or repetitive pattern is present.
Semantic: In cases where a consistent high-level semantic concept is observed across images,
even if not casily described by single word.
Object: In cases where a consistent, concrete, and identifiable object entity is captured (e.g.,
sky. ground, dog, dog's nose).
Background: In cases where the entire area of the image excluding the primary object is captured.
Positional: In cases where fixed spatial location within the image consistently captured, independent of the image's content.
Miscellaneous: In cases where there is a clear visual commonality among patches, yet the pattern does not align well with any of the predefined categories above.
Polysemantic: In cases where multiple distinct groups of patches or images each capture different semantic or visual attributes.
Uninterpretable: In cases where no common or interpretable features can be identified at the patch or image level.
*We also consider specialized feature visualization methods that optimize the input image to maximize the activation of a specific feature [190 [92]. However. this approach has been shown to be less effective than using maximally activated images and tends produce less interpretable results in ViTs 193 [94]
40

---

Class most promoted
Maximally Activated Images
gpazele
bee jetter
NUvisz
lown mower
bel peozer
jensy
mittary ueffem
twhippe:
must turtle
butble
Italian greyhound
0.0
042
0.0
0.64
6.5
0.06
Maximally Activated Patches
2
7
10
Figure 17: Visualization Example (1).
Class most prometed
Maximally Activated Images
dial_telephoon
plyy phona
boosted
iron
Shoa
del Belgene
telephonee
Jelephate
Jelestore
teleshone
lide bluw her
carttnn
snow, Jespard
microwase
alace
000
082
0.64
0.60
0.00
6la
aie
Maximally Activated Patches
2
salephone
telephnne
tnlephane
teleshore
beleshone
7
10
Figure 18: Visualization Example (2).
41

---

VIT
DINOV2
CLIP
1.6
1.0
1.0
0.8
08-
0.8-
Propotinn
0.6
0.4
0.0
0.0
0.0
tional
Color
Line
Shape
Background
Object
Figure 19: Category Proportions for ViT, DINOv2, and CLIP.
More Results on Categorization We provide additional results on the categorization of features learned by SAE in DINOv2 and CLIP. As shown Figure 19 the features learned by DINOv2 and CLIP exhibit patterns similar to those observed in ViT 2.2
D.4 Case Studies Details
Curve Detectors To generate radial tuning curves [55], create synthetic images in which each patch contains a curve with varying angles and curvatures. For each curve detector, we compute the maximum activation of the feature across all patches in each image and then take the maximum across all images sharing the same angle. Finally, we plot the maximum activation as function of the curve angle, as shown in Figure of the main
we compute the mutual information between variables: the activation of feature and the position of patch within the image. The mutual information defined as:
Position Detectors To automatically identify position detectors 59
fru (pos)
1 -frh (pos)
I(act,pos) = T
Σ pos=1
fropo))
log
+ +111 fr((pos) log 1 frn
(9)
frn
where fr(mos) denotes the activation frequency of feature n position pos, frn is the overall activation frequency of feature n, and T is the total number of patches in the image. We identify position-sensitee features by selecting those with mutual information exceeding threshold, i.e., T(act, pos) т. For the analysis in the main text, we use r = 0.05, although range of reasonable thresholds yields similar results due the clear separation between position detectors and other features. To visualize the position detectors, plot the average activation of each feature across spatial positions, as shown Figure of the main text.
D.5 More Feature Examples
We present additional examples of features learned by the TopK SAE in Figures 20 to 31 These examples include various types of features, such as ripple patterns, V shapes, horizontal lines, hands, and watermarks. We optionally visualize the maximally activated patches when doing so aids in interpreting the feature.
42

---

L3#10 (vertical line)
L3#790 (horizontal line)
Line
Kesserr
L1#1711 (circle)
Shape
L244133 (V shape)
Shape
Figure 20: More VIT Feature Examples (Line and Shape).
43

---

L0#660 (red)
Color
L1#88 (slanted stripe pattern)
Texture
LO#71 (ripple pattern)
Texture
L9M69 (curly hair)
Texture
Figure 21: More ViT Feature Examples (Color and Texture).
44

---

L1#32 (bottom of image)
itional
L5#845 (background)
Background
L5#784 ledge of facial features)
Sema ntic
L5#59 (bottom edge of vehicles)
Sema ntic
Figure 22: More ViT Feature Examples (Positional, Background, and Semantic).
45

---

L9#833 Ihammer head)
о bject
L1043 (cat)
Object
L10#50 (hand)
Object
L6#5
(Chinese waterman rmtak)ark)
Miscellaneous
Figure 23: More VIT Feature Examples (Object and Miscellaneous).
46

---

LO#13 (slanted line)
Line
THE 1E
L1#58 (slanted line)
Line
TV
L1#51 (blue)
Color
L4x27 (text)
Miscellaneous
Tar Cagarl
BIG
Figure 24: More DINOv2 Feature Examples (Line, Color, and Miscellaneous).
47

---

L2#131 (circle)
S Shape
L3452 (<shape)
Shape
L2#57 (mesh pattem)
Texture
L7#4 (broccoli-like texture)
Texture
Figure 25: More DINOv2 Feature Examples (Shape and Texture).
48

---

L2#26 (vertical axis)
itional
Apuadd
Apstabo
L1499 (left edge of objects)
Semanti ntic
L6477 (above the eyes of animals)
Semantic
L7#84 (plants in front of animals)
Sema ntic
Figure 26: More DINOv2 Feature Examples (Positional and Semantic).
49

---

L8#18 (background)
Background
L849 (paws)
Object
L8#199 (nose)
Object
L9#182 (fire)
Object
Figure 27: More DINOv2 Feature Examples (Background and Object).
50

---

LO#85 (slanted line)
Line
L0#156 (light green)
Color
L2#95 (wavy shape)
Shape
W
Eamumh
L4#1,05 (cross shape)
Shape
Figure 28: More CLIP Feature Examples (Line, Color, and Shape).
51

---

(checkerboard pattern)
Texture
L10#1199 (dotted fur)
Texture
L6A16 (2')
Miscellaneous
11
12
12
10
61
3
8
4
3
6
5
2 OZ.
L8453 ('ED)
Miscellaneous
L
ed
REC
AMERICANS
Cullamo
TEL
Ie Dankl Donnns den git far the Living
IR Hil
COULEVARD STOP
Figure 29: More CLIP Feature Examples (Texture and Miscellaneous).
52

---

L1#112 (left edge)
itonal
L8#125 (object being touched hand/paw)
Semantic
L9496 (person/animal on the right)
Sema ntic
L10#76 (edge between adjacent people/animals]
Semanta ntic
Figure 30: More CLIP Feature Examples (Positional and Semantic).
53

---

L1#125 (background)
Background
L8#111 (baby face)
Object
L9#125 (mask)
Object
L10#187 (strawberry)
O Object
Figure 31: More CLIP Feature Examples (Background and Object).
54

---

Table 3: Circuit Evaluation for Various Node Selection Rules. We evaluate the faithfulness and
completeness of the circuits using 1,500 randomly sampled images from the ImageNet validation set.
Faithfulness (%)
Completeness (%)
Rule
VIT
DINOv2
CLIP
VIT
DINOv2
CLIP
top-k
94.1
85.1
82.3
99.6
99.8
99.7
lop-p
95.0
85.4
82.9
99.7
99.7
99.7
threshold
93.6
84.6
82.2
99.6
99.8
99.7
Table 4: Comparison Cross-Layer Attribution (CLA) Algorithm [
We evaluate the faithfulness,
completeness, and causality of the circuits using 1,500 randomly sampled images from the ImageNet
validation set.
Faithfulness (%)
- Completeness (%)
Causality (%)
Strategy
ViT
DINOv2
CLIP
ViT
DINOv2
CLIP
ViT
DINOv2
CLIP
Random
30.2
27.5
27.0
78.1
90.1
84.4
35.6
33.9
31.1
CLA Ours
56.3
41.3
36.7
92.5
93.8
93.5
57.2
54.8
55.1
94.1
85.1
82.3
99.6
99.8
99.7
54.5
54.8
53.8
E Residual Replacement Model
E.1
Design Choices
Selection Strategies In $ we select the top-k features per layer based on the importance of their edges to the selected downstream nodes Ve+1. We also consider two alternative selection strategies: (1) selecting the top p-percent of features in each SAE, and (2) selecting features until the cumulative importance the selected features exceeds a threshold, i.e., Σnev, Edey. I(u - d) > T. As
shown in Table 3 these alternative selection rules yield no significant differences in the faithfulness or completeness of the resulting circuits.
8 proposed the Cross-Layer Attribution (CLA) method to identify important neurons in CNNs. Rather than evaluating each neuron's 's contribution to the output logits, their approach emphasizes interactions across internal layers to determine neuron importance, constructing what they term functional connectivity graph. Since Rajaram et al. [ did not release a public codebase, we re- re-implemented their method based on Algorithm (CLA) in their paper and compared it against ours. The results are presented in Table 4
Comparison to Prior Work Rajaram
Our method substantially outperforms CLA in both faithfulness and completeness, highlighting its effectiveness. For causality, CLA attains slightly higher scores, likely because it is specifically designed identify nodes that directly affect subsequent layers. Although this design naturally boosts causality, it overlooks the role of features in determining the final output, which is crucial for constructing faithful and complete circuits. In contrast, our method explicitly accounts for the influence of features on the final output, resulting in higher faithfulness and completeness.
E.2 Technical Challenges Details
Scalability of Edge Importance Estimation Let T' be the number of tokens in the input image, and let fu and fa denote the number of features in the upstream and downstream SAES, respectively. Let u, and d, represent the upstream and downstream features of the (-th token. Since aggregate features across all tokens, each node defined as the average over all tokens, i.e., u Ei= u and d = ET_dd de
The importance of the edge (u, d) computed as:
T
I(u d= Σ Iuu. dd)
)
V d, m Vu,d; dj (u, u!).
i=1
55

---

This requires computing the Jacobians for T token pairs per edge, resulting in a total of ff. x f' feature pairs. In practice, this would require O(T fa) backpropagation steps, which becomes computationally infeasible for large T'
To mitigate this, apply the Jacobian- vectorr product trick efficiently compute the Jacobian of the downstream features with respect to the upstream features:
T ΣΣ i=1 5=1
T
T Σ
d. m Vu, d (w: - u/)
(V dj 1=1
(u; u().
==1
Note that Va, m is treated formulation reduces the complexity to O(fa) backward passes, significantly accelerating computation compared to the naïve approach. As a result, we can compute the full edge importance in a few seconds for single image using single RTX A6000 GPU.
constant during computing the Jacobian of dj with respect to u, This
Noisy Gradients Unlike language models, ViTs often suffer from noisy gradients, making gradient- based interpretations less stable (82 To address this issue, adopt LibraGrad [86]. recent method designed to stabilize gradients. They found that some modules modern transformer architectures, such as attention mechanisms and layer normalizations, can disrupt the gradient flow, leading to noisy gradients. LibraGrad mitigates this issue by applying a gradient pruning technique, which removes the noisy gradients while preserving the informative ones. An important thcoretical property of LibraGrad is that satisfies wewlGGrad-completee model output into the exact contributions from each input feature along with bias term:
[85], meaning it decomposes the
m(a: b) Vam((: b)" Vom(a: b)
where a is the input feature, is the bias term, and m (a: b) is the model output. This completeness property enables us to assign an explicit contribution
each input feature, akin to Layer-wiss wise Relevance Propagation (LRP) [84] [27] 129 [32). We extend this decomposition to intermediate features, allowing us to quantify the contribution of each SAE feature to the final prediction through gradient-based analysis.
However, extending FullGrad-completenes to SAE features requires careful handling of the SAE's normalization step. In standard SAEs, the input representations are normalized by dividing by the standard deviation during encoding and re-scaled during reconstruction. This normalization step introduces nonlinearity that can break the completeness property. Concretely, given an input æ e R. the SAE computes the normalized representation as:
æ ((
a
where u(x) and (xx) are the mean and standard deviation of the input, respectively. Similarly, during reconstruction, the SAE applies a re-scaling step:
o x
u(x),
where a W d + 6pc is the reconstructed representation. To preserve FullGrad-completeness. we block the backpropagation path through the standard deviation normalization parameters o (xx). Under this modification, the contribution of the SAE features and the error term can be faithfully computed using gradient- based decomposition.
Finally, while the bias term in FullGrad reflects the impact of higher-order interactions between intermediate features [33, we do not consider this term in our main analysis. Incorporating such interactions remains an interesting direction for future work.
E.3
Metrics Details
The most commonly used metrics for evaluating discovered circuits are faithfulness and completeness, as established in prior works [41 [1 87 [9]. In addition, we introduce causality as a complementary metric to provide further insight into circuit quality.
For the AUC computation, we evaluated metrics at different sparsity levels by varying where corresponds to (0, 0,001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5. 1) times the number of features in the layer with the largest SAE dimensionality 889 The AUC then obtained by aggregating the
results over these A values.
56

---

E.4 Feature Similarity Analysis
To further investigate the Granny Smith circuit in $3.3 (Figure of the main text), we conduct two simple analyses to evaluate the similarity and continuity of features within the circuit.
First, test whether features are preserved across layers through the residual stream. For each feature in a given layer, we identify the feature in the next layer whose decoder vector has the highest cosine similarity with it. We then check whether this most similar feature is included in the eircuit. This analysis helps verify if the circuit retains geometrically similar features across layers 991 92].
Second, we examine whether these cosine-similar pairs correspond to strong connections by inspecting whether the bighest-weighted edges in the circuit connect features with high cosine similarity. This allows us assess whether semantic similarity is aligned with edge importance.
Our results show that, for all layer pairs, the next-layer feature with the highest cosine similarity is consistently included in the circuit. Moreover, in all layers except for 0-1, 3-4, and 8-9, the highest- weighted edges connect to the features with the greatest cosine similarity. In the cases of layers 0-1 and 8-9, the error term appears to play a significant role in constructing the next-layer feature, which may explain the lack of direct feature preservation. For layers 3-4, the edge weights are relatively low, suggesting that the next-layer features are formed through compositional combination of multiple upstream features. For example, L3#128 and L3#1438 are both connected to L4#58 with similar edge weights. While their individual cosine similarities with L4#58's s decoder vector are moderate (0.44 and 0.48, respectively), their combined vector achieves a similarity of 0.62. Considering that the maximum cosine similarity between any single L3 feature and L4#58 is 0.68, this suggests that L3# 1283 and L3#1438 jointly contribute to the construction of L4#58.
Overall, by combining the residual replacement model with feature similarity analysis, we can identify which features are preserved across layers through the residual stream and which features are compositionally combined to form higher-level representations 97]
E.5 More Circuit Results
We provide additional qualitative results on the circuits in ViT (Figures 35), DINOv2 (Figures 36 to 39p, and CLIP (Figures 440 to (43). To aid intuitive understanding of the features, we visualize the maximally activated patches for each feature in the first row and the corresponding maximally activated images in the second row. The first column shows the input image along with its maximally activated patches and activation values.
E.6 More Curve Circuits and Position Circuits
We also provide additional examples of curve circuits and position circuits in DINOv2 and CLIP. We find that the circuits in DINOv2 and CLIP behave similarly those in ViT. For curve circuits (Figures 44 and (45). lines at different angles are compositionally combined to form curve detectors. For position circuits (Figures 46 and 47). position detectors in early layers are combined to construct more complex detectors in deeper layers. For example, in Figure 46 various vertical position detectors and other features combine form a top-and- non-botoo background detector (L4#1 103). In Figure 47] we observe that a bottom position detector (L3#1515) and an object detector (L3#419) combine to form bottom background detector (L4#70)) We can also find that the object detector (L3#419) is influenced by color detector (L2#84).
E.7 Debiasing Spurious Correlations
In this section, we provide additional details on the debiasing procedure. We construct the top-3 feature circuits for seven ImageNet classes: hummingbird, freight car, koala, fireboat, hard disc. gondola, and racket, which are known exhibit spurious correlations with frequently co-occurring features. For each class, we manually identify one spurious feature within the circuit and ablate Specifically, we ablate the following features: L9#1210 (bird feeder for hummingbird), 1.9#3711 (graffiti for freight car), 199#1369 (eucalyptus for koala), 1.911688 (water jet for fireboat), 1,9#2867 (label for hard disc), 1.9#077 (house/river for gondola), and 1.9#855 (tennis court/player for racket).
57

---

Teapot circuit in ViT. The circuit reveals that the model initially attends to the grayish tone of the teapot (1) and its specular highlights (L11241) in the lower layers. As it progresses through the intermediate layers, the model gradually captures the overall shape of the teapot (see the heatmap of the input image in the higher layers, information about the teapot's s handle (L842785) and body (L8#2560) is integrated. leading to the model's 's prediction of a teapot.
Figure 32:
1.3#208 - L4#258 -> 15#1374 + L6#6633. In the
Figure 33: Street sign circuit in VIT. As the input progresses through the carly and intermediate layers, the model gradually develops an understanding of the text written the street sign. For instance, at L0#140, the characters are perceived as a stripe pattern; at L1#42, they are interpreted as combination of diagonal lines; and at L2#10, model begins to recognize the characters as alphabetical and Arabic numerals. This understanding of the text is subsequently integrated with the wide board feature (L8#3063), allowing the model to grasp the overall appearance of the signboard (L.91332 1321). Finally, this representation is combined with the signal light features (L9#2068, L10# 1492) to yield the prediction of traffic light.
composition of curves. Starting from L3, the
1444
64
Figure 34: switch circuit in ViT. Up to the intermediate layers, the model captures information about the areas surrounding the switch-boo laterally (L24253, L3#439) and vertically (L2#13, L3#1275). This information is integrated to form representation of the switch's 's rounded shape (L6#2676. L7#40). Beginning from layer 5, the model starts recognizing the Roman numerals written on the switch (L5#855,, L6# L6#1429). 1429). The information combines with the switch's 's round shape (L7#40, L8#63) leading the model to predict the presence of a switch.
Figure 35: Tabby circuit in VIT. Up layer 7, the model captures features related to fur texture (L1#16, L2#516) and the cat's whiskers (L2#658, L3#1 102, L4#637, L5#1062). In layer 8, both the whiskers (L8#2319) and the cat's cat's eye (L8#1155) 156) significantly contribute to the model's ss ability to identify the cat's 's face in layer 9 (L9#2381), Notably, the detection of the cat's 's stripes (L9#377) and animal fur (I9#2651)) in layer 9 plays integral role in forming the final representation of tabby cat (L10#404).
58

---

Figure 36: Typewriter keyboard circuit in DINOv2. In the early layers, the model interprets the spaces between keys either as lines (e.g. L0#98, L1#69) or as textures (e.g., L2# 1+136 136, L34904), In the middle and later layers, the model captures the round and repetitive shapes of the keys (L5#88, L6#1350), leading to the recognition keyboard (L8#455, L9#1213). By further detecting the characters engraved the keys (L9#1091, L10#275), the model arrives at the prediction of typewriter.
Figure 37: Scale circuit DINOv2. Through the subgraph LI#131 + L2#296 + L3#806, the model captures the white, monotone color of the scale. Simultaneously, it interprets the text printed on the scale via the subgraph L1#132 + L2#244 + L3#40 - L4#1531 - L5#1114. Notably, in layer 6, the model appears to separately recognize the logo text (L6#1373) and the numerical values on the dial (L6#1 177). In the later layers, the structural form of the scale (L8#744, L9#1 106, L10#1332) is integrated with the textual and numerical information (L.10#1 135), ultimately leading the model to predict a scale.
CORLS
Figure 38: Hare circuit DINOv2. In the early layers, the model detects the hare's 's eye as either a texture or distinct shape (L1#1, L2#9, L3#1268), while simultaneously capturing the texture of the hare's ' fur (L2##199, L3#1 145). In the middle layers, the eye is interpreted more semantically as an animal eye (L4#316. L5#1225) and the model begins to represent the overall body of the hare (L4#633,, L5#6). the subsequent layers, the model exhibits increasingly fine me-ggrinee understanding of the hare's 's face by attending to the eye (L6#609), the area below the eye (L6# (L01), the region between the eye and nose (L6#209), and the snout (L9#696). In the final layer, the activation of the hare's 's face (1.10#113 137) and the animal's whiskers (1.10#734) culminates in the model's prediction of a hare.
Figure 39: Goose circuit in DINOv2. In the early layers, the model captures the goose's coloration by separating it into white (L0#60), gray (L0#178) 18)), and edge regions (L2#224), In the middle layers, the model detects the left (L3#1 1268) and right (L3#1 1711 sides of the goose's s eye as distinct shapes, which are subsequently integrated in later layers form semantic understanding of an animal eye [L4#316, L5#1225, L6# 16#1)3).. In layers 7 and 8, the model attends to the goose' neck (L7#1471) cheek (L7#338), body (L8#1017), and beak (L8#983), ultimately leading the prediction of goose.
59

---

Figure 40: Cellular telephone circuit in CLIP. In the early layers, the model focuses on the blue, uniform color of the cell phone (L3#1179)). In the middle layers. identifies the keypad of the device (L6#688), and subsequently recognizes it as a button-equipped device (L7#518). This understanding is further refined as the model begins to interpret it time-displaying device (L8##200 and communication device [L9#1163, L10#1382). ultimately leading to the prediction of a cell phone.
Figure 41: Tennis ball circuit in CLIP. In the early layers, the model captures the background of the tennis ball (e.g., L0#214, L1#17) as well as the texture of the ball (e.g., L2#41, L33838). In the intermediate layers, it detects the edges both sides of the tennis ball (L4#703) and the lower edge (L4#1 (46)). leading to representation of a round object (L5#540). This understanding is further refined into multiple round objects (L6#0), and in the subsequent layers, the model recognizes the object as a ball (L7##1 1L7#139, L8#274). By identifying the characteristic stripes on the ball (see the maximum activating patches in L9#957), the model ultimately predicts tennis ball.
Figure 42: Wall clock circuit in CLIP. In the early layers, the model focuses the background of the wall clock (L0#101, L0#157, L L1#2)3). In the middle layers, it begins to recognize the numbers on the clock (L6#178), followed by the detection of the tick marky in the subsequent layer (L7#1248). The model then forms a representation of a round-shaped clock (L8#876), eventually activating general clock features (L9#1246, L10#568), which leads to the final prediction of wall clock.
Figure 43: Oscilloscope circuit in CLIP. the early and middle layers, the model recognizes the background of the oscilloscope (e.g., 10#2100 L1#236, 12#1111 11). In the later layers, it identifies the oscilloscope's buttons (1.7#518), display (1.##1472), and body (I.8#456). Subsequently, the red line connected to the oscilloscope (1.9#1396) is interpreted as a wire (1333)33). This understanding, com- bined with the oscilloscope-speciif features (L104361), leads the model to predict an oscilloscope.
60

---

LO#10
L1#167
L2#45
L3#87
VE
L0#154
L1#70
L2+42
LO#35
L1#57
L2#251
N
LO#62
L1#130
L2#240
LO#E
L1#E
Figure 44: Curve Circuit (L.3#87) for DINOv2.
L0R12
L1#72
L24708
L3F141
LG#369
L1#12
L2#693
LOWGS
L1#354
L2#593
LO4305
L14229
L2#398
A
LONE
L1#E
L2#É
Figure 45: Curve Circuit (L.3#141) for CLIP.
LO#74
L1#86
L2#312
L3#655
L4H 1203
L0#17
L 11#17
L2#1 1 16
L3H 1 231
LO#E
L1#E
L2#3500
L3#1
Figure 46: Position Cireuit (L4##2033) for DINOv2.
L24749
L4970
L1#382
L2#84
L3e419
L1#E
L2#E
L3#E
Figure 47: Position Circuit (L.4#70) for CLIP.
61


Reviews:
[{"review_id": "pLD3JVUztw", "reviewer": "Reviewer_PvQ3", "summary": "In this paper, the authors try to open up the layers of Vision Transformers via sparse autoencoders and residual replacement model. They try to highlight some of the key understandings when it comes to interpreting the features in a network – lower level features are represented in earlier layers, while more complicated features are built up by combining the earlier features in higher layers.", "questions": "How were the annotations done? The authors mention large-scale human annotations of the of the ViT features, but do not mention anything about how the studies were conducted? How were the humans selected? How were the responses filtered? Was there an IRB approval? How were the human subjects’ annotations evaluated? How was the quality of annotations assessed? \nCan the authors also clarify the training details from a hardware standpoint? How long did the experiments take? What CPU/GPU configs were used?\nThe authors are also encouraged to release the code and datasets in the supplementary materials to the public. \nIn lines 126-127, the authors mention that the detector activates only when the corner is well-aligned with the patch grid. Do they imply that a feature activates a layer/neurons only when it is well aligned? Have the authors observed a gabor filter-like pattern with higher activations in the center and lower activations towards the edges? \nThe authors cover feature categorization by layers – low level features in early layers and high level features in higher layers are more prominent. Have the authors tried comparing their study with other human vision aligned works like the Extreme Image Transforms (EITs) and Minimal recognizable configurations (MIRCs)? EITs can also help with the spurious correlations and interpretability, functioning similar to humans and confirmed by psychophysics studies. \nThe reader might benefit from knowing about concept discovery in GAN Dissection by Bau et al, ICLR 2019.", "limitations": "Yes, the authors do mention limitations. I am not sure how this framework can be scaled to other models.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The study is important for interpretability and explainability, in terms of showing how the model interprets the individual objects in its layers, going from low level features like the lines to higher level features like complex arches. The 640 visualizations per layer  with 16 participants seems quite extensive. \nThey have set out to create a gold-standard feature analysis through human interpretations. \n\nThe authors try to pin their work on the fact that they can explain the human alignment of ViTs (and subsequently other networks). However, in the current form, the paper misses to demonstrate a clear alignment with human/biological vision. \nThe authors miss a few crucial details/reasoning as well – how were the averages of activations taken to reduce the dimensionality of explanations? How were certain feature selected in the layers? Why only those features were selected/shown?\nIn the validation subsection, why did the authors only select faithfulness, completeness and causality as the metrics for evaluation?\nThe authors also don’t say much about how the numbers were reached for the 3 metrics in table 1. \nThe authors talk about the qualitative analysis of a discovered circuit, but don’t clarify much on how this circuit is identified. It might be good to have some background for the reader. Please also clarify the link of this with the residual replacement model. \nIn Figure 5, are the layer and feature numbers manually selected? If so, how? \nI am particularly pleased with the legend indicating the round fruit, round object, error, etc. in each of the layers. But the authors missed mentioning about how these were selected? Is it purely by looking at the activations or can this be automated for scalability?\nThe authors mention a lot of metrics (PCA, SAE, mAUC, etc.) but don’t compare their representations using t-SNE which seems to be quite standard in the field. \nI am also not too sure how backprop and other gradient based methods fit into the framework for visualization. Can the presented method/framework be scaled for different arbitrary models?", "was_revised": true, "final_justification": "Based on the discussion with authors during the rebuttal phase, i am sticking to my original rating of the paper. Deferring to AC for the final call on ethics review.", "rebuttal": "We thank the reviewer for their constructive feedback and helpful observations. Our detailed responses to the comments are outlined below. If there are any remaining concerns, please inform us, and we will address them promptly.\n\n# Q1: Annotation Process Details\n> How were the annotations done? … How was the quality of annotations assessed?\n\nDetails regarding the annotation process can be found in Appendix D. As additionally stated in Line 116, the annotations were manually performed by the authors to provide accurate and ground-truth labels for the SAE features. Note that *these GT labels are not used to evaluate our SAE features or circuit*, and thus the annotation process is unrelated to any bias that can favor our method.\nTo ensure the quality of the annotations, we conducted cross-checking among multiple authors. Furthermore, to quantitatively assess annotation quality, we conducted an additional user survey. We recruited five individuals who were not involved in the authorship of the paper and presented them with a total of 220 feature visualizations along with their corresponding annotations. Each participant was asked to evaluate whether the provided annotation accurately described the visualization, responding with “Yes,” “Maybe,” or “No.” The results showed that 81.4% of responses were “Yes,” while only 6.82% were “No,” indicating that the annotations are generally reliable.\n\n# Q2: Hardware Details & Code and Dataset Release\n> Can the authors also clarify … supplementary materials to the public\n\nWe apologize for not providing hardware-specific details in Appendix C.1 regarding the training setup. Each sparse autoencoder (SAE) was trained using an RTX 3090 GPU for approximately 12–14 hours per model. We will include this information in the final version of the paper.\nAs for the dataset and code release, due to file size limitations, the supplementary material includes only a subset of the dataset, although the complete code is provided. We plan to publicly release the full dataset along with all associated code upon publication.\n\n# Q3: Meaning of line 126-127\n> In lines 126-127, the authors mention … lower activations towards the edges?\n\nIn Lines 126–127, we state that “the detector activates only when the corner is well-aligned with the patch grid.” This statement indicates that the activation of the L-shaped feature varies sensitively depending on the location of the L shape within the patch. As a related example, please recall that a curve feature (Figure 3) activates most strongly at a specific angle. Similarly, the L-shaped feature also shows the highest activation when the L shape is located at a specific position, and the activation decreases rapidly when there is an offset from that location.\n\n# Q4: Existence of Gabor Filter-like Features\n> Have the authors observed  … lower activations towards the edges?\n\nIn this study, we focus primarily on investigating curve features and positional features. This is because these features are well-known and representative in the contexts of CNNs [55-58] and language models [59], respectively. However, we find the reviewer’s suggestion regarding the presence of Gabor-like filters in ViTs to be a highly interesting and promising research direction. We appreciate this insightful suggestion, and we plan to explore it as part of our future work.\n\n# Q5: Comparison with Human Vision Aligned Works\n> The authors cover feature categorization by layers … concept discovery in GAN Dissection.\n\nOur study does not aim to investigate *the similarity between ViTs and the human perceptual system*, but rather to *thoroughly understand how Vision Transformers process images*. To make this clearer, we will revise the introduction section to better reflect the primary objective of our work.\nThat said, we appreciate the reviewer’s suggestion. Exploring the alignment between ViTs and human perception seems a fascinating research direction. We also thank the reviewer for pointing us to the concept discovery work in GANs, which we will consider incorporating into future research.\n\n# W1: Missing Clear Alignment of ViTs with Human Vision\n> The authors try to pin their work on … clear alignment with human/biological vision.\n\nWe apologize for the reviewer’s confusion caused by our terminology. As also mentioned in our response to Q5, our work primarily aims to gain a clear understanding of how Vision Transformers process images, rather than to evaluate their alignment with the human perceptual system.\nIn our paper, we used the term “human-aligned” interchangeably with “human-interpretable.” And our definition of “human-interpretable features” follows the criteria established in [30, 31, 35], as noted in Lines 107–108. For example, Figure 2c presents features that meet this definition of human interpretability.\nTo prevent further confusion, we will revise the terminology in the final version of the paper to make our intended meaning more explicit and consistent.\n\n# W2: Missing Details: Average of Activations & Feature Selection\n> The authors miss a few crucial details … Why only those features were selected/shown?\n\nAs mentioned in Lines 170–171, the feature activation is averaged along the token dimension. Each of the $T$ image patches has an $f$-dimensional feature activation, forming a $T \\times f$ matrix. We compute the mean across the token dimension, resulting in an $f$-dimensional vector, $\\mathcal{A}_L$. The top-$k$ important nodes within $\\mathcal{A}_L$ (denoted as $\\mathcal{V}_L$ in the paper) constitute a single layer of the residual replacement model. This design choice of averaging along the token axis simplifies interpretation by removing the need to analyze complex token-wise interactions.\nAdditionally, the reviewer asked about the feature selection process. This corresponds to *how the top-$k$ important nodes are selected for each layer*. We compute the importance of each node using Equation (2) and select the top-$k$ nodes accordingly.\nThe reviewer also enquired why we limit our analysis to only these selected features. Analyzing the top-$k$ important features enables us to capture the dominant circuits operating within the ViT, which is a common practice in circuit discovery studies [46, 78].\n\n# W3: Choice of Metrics\n> In the validation subsection … for the 3 metrics in table 1.\n\nAs described in Lines 213–214, the most commonly used metrics for evaluating discovered circuits are *faithfulness* and *completeness*, as established in prior works ([37], [51], [87–89]). In addition to these, we introduce *causality* as a complementary metric to provide further insight into the quality of the circuits.\nThe reviewer also asked how the values in Table 1 were computed. The faithfulness and completeness values reported in Table 1 follow the standard evaluation protocol proposed in [89]. Additionally, the causality values are computed using the same framework as in [89], with the only difference being that the metric is replaced from Equation (3) to the one described in Line 218. We will include these details in the Appendix to provide a more thorough explanation.\n\n# W4: Details on Circuit Identification Process\n> The authors talk about the qualitative analysis … can this be automated for scalability?\n\nThe circuit refers to a subgraph newly constructed by extracting the key components from the residual replacement model. As described in Lines 164–167, the residual replacement model is a directed acyclic graph composed of all SAE features. Then we construct a *circuit* by selecting $k$ most important features from the graph using our algorithm (Line 181). As explained in W2, *no manual selection is involved in this process.*\nOur goal in Section 3.3 is to understand how the ViT recognizes the concept of “Granny Smith.” To this end, we formulate a hypothesis in Lines 247–249: the ViT first detects a “round fruit” feature, then a “green” feature, and combines them to recognize “Granny Smith.” To evaluate this hypothesis, we manually inspect the features included in the discovered circuit and categorize them into interpretable groups such as “round fruit” and “green.” This inspection supports the validity of our hypothesis.\nWe would like to emphasize again that the feature selection process is fully automated, with no human intervention. The only manual step is interpreting the features selected by the algorithm. Automating this interpretation phase is beyond the scope of the current work, but as discussed in Section 6, it represents an important direction for future research.\n\n# W5: No Comparison of Representations Using Other Metrics\n> The authors mention a lot of metrics … which seems to be quite standard in the field.\n\nWe would like to clarify that PCA and SAE were mentioned in the Related Work section not as metrics for comparing representations, but rather as examples of dictionary learning methods. Additionally, mAUC is designed to measure the degree of spurious correlation and is therefore not an appropriate metric for comparing SAE representations.\nMeanwhile, t-SNE is primarily used for visualization purposes and is not generally considered a suitable method for comparing representations. However, if the reviewer could share a paper or methodology demonstrating how t-SNE can be effectively used for representation comparison or evaluation, we would consider incorporating it into the final version of the paper.\n\n# W6: Generalizability of Our Method on Arbitrary Models\n> I am also not too sure how … fit into the framework for visualization.\n\nOur circuit discovery method can, in principle, be applied to any model that allows the computation of importance as defined in Equation (2). In other words, as long as the model supports backpropagation, our approach is applicable. Furthermore, as described in Lines 194–201, the cost of our method increases *linearly* with the size of the dictionary, demonstrating its potential for generalization across different architectures."}, {"review_id": "1nWL8Nqyfx", "reviewer": "Reviewer_Dv9E", "summary": "This paper explores how Vision Transformers (ViTs) represent and process visual information by extracting 6.6K features across all layers and multiple ViT variants (ViT, CLIP, and DINOv2) using sparse autoencoders. The authors introduce a residual replacement model, which replaces ViT computations with interpretable features in the residual stream. Their findings show that similar to CNNs, ViTs evolve from detecting low-level patterns to capturing high-level semantics, and include specialized features for encoding curves and spatial positions. The residual replacement model produces a simplified yet faithful representation of ViT circuits, significantly improving human interpretability. To evaluate these circuits, the authors apply three key metrics: faithfulness, completeness, and causality. Finally, they demonstrate the practical use of their framework in effectively debiasing spurious correlations.", "questions": "Please refer to the second point in **Weaknesses**.", "limitations": "The limitations are discussed in Section 6 of the main paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is clearly written and easy to follow.\n2. To address the technical challenges of applying replacement models to ViTs, particularly the scalability of edge importance estimation, the authors employ the Jacobian-vector product trick. Given that ViTs are prone to noisy gradients, they further extend the LibraGrad method to enhance the faithfulness of the discovered circuits compared to those derived using the vanila gradient.\n3. I appreciate that the authors have evaluated the discovered circuits using the criteria of faithfulness, completeness, and causality.\n4. The insights into curve and position circuits, along with the demonstrated practical use of the proposed replacement model in debiasing spurious correlations are interesting.\n\n**Weaknesses**\n\n1. The novelty of the proposed residual replacement model is somewhat limited, as similar approaches have already been explored in the NLP community [a, b]. The authors primarily extend this replacement philosophy to Vision Transformers, with the main contribution being their handling of domain-specific challenges, particularly scalability and noisy gradients.\n\n[a] Marks et al. “Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models”\n\n[b] Antropics’s Circuit Tracing, https://transformer-circuits.pub/2025/attribution-graphs/methods.html \n\n2. In Table 1, some values related to the causality analysis are missing. Could you please clarify the reason for their omission?\n3. To evaluate the quality of the learned SAE features and assign semantic labels, the authors rely on human annotations. This dependence on manual labeling makes the proposed framework less scalable and more difficult to apply to new models and architectures.", "was_revised": true, "final_justification": "Although some of my concerns were addressed during the rebuttal, my main concern, the limited novelty of the proposed framework, remains. Therefore, I am keeping my original score.", "rebuttal": "We thank the reviewer for their constructive feedback and helpful observations. Our detailed responses to the comments are outlined below. If there are any remaining concerns, please inform us, and we will address them promptly.\n\n# W1: Limited Novelty\nAs the reviewer rightly pointed out, our work is indeed heavily inspired by circuit discovery methods from the NLP community, such as [a] and [b]. Our contribution does not lie in proposing a fundamentally new circuit discovery method, but rather in addressing the technical challenges that prevent the direct application of these NLP-based approaches to Vision Transformers (ViTs) (Line 194-206 and Appendix B).\n\nHowever, we would like to emphasize that overcoming these technical challenges is non-trivial and forms a key part of our contribution.\n\n1. **Scalability.** By introducing a simple yet effective idea, aggregating tokens, we were able to accelerate circuit discovery by 200×, making large-scale analysis of ViTs practically feasible. Notably, our method maintains constant computation time regardless of the increase in ViT’s patch size. At the same time, this aggregation leads to easier and more concise interpretations of ViT behavior.\n\n2. **Noisy Gradients.** Furthermore, by incorporating Libragrad, a recent method designed to stabilize gradients, we improved the faithfulness of ViT circuits by approximately 1.6×. We believe this is a substantial improvement and meaningfully support future efforts toward more accurate ViT interpretation.\n\nThese details are described in Appendix E.2, and we kindly encourage the reviewer to refer to that section.\n\n# W2: Missing Causality Values in Table 1\nThank you for your thoughtful question. The causality metric is meaningful only when the activations of the nodes that constitute a circuit are guaranteed to be non-negative. In the definition of causality, $\\Delta d$ refers to the reduction in activation of downstream nodes. Thus, the intent of this metric is to measure the causal influence of upstream nodes by averaging the degree of reduction across all downstream nodes. Therefore, this metric only makes sense when node activations are guaranteed to be non-negative.\nHowever, while it is common practice to enforce non-negativity in sparse autoencoder features, neuron activations in general neural networks are typically allowed to take on negative values. As a result, the causality scores of neurons would not be meaningful. For this reason, we chose not to measure causality values in Table 1.\n\n# W3: Limited Scalability of Our Method Due to Manual Annotations\n\nWe agree with the reviewer that the reliance on manual annotations is a limitation of our current approach. However, we believe that manual annotation was an appropriate and necessary choice for our study, for the following three reasons:\n\n1. First, the primary objective of our study is not to *automate feature interpretation*, but to *achieve accurate interpretations of features* in order to better understand how Vision Transformers (ViTs) process information. In this context, we believe that manual annotation is more suitable than relying on potentially unreliable automated methods.\n\n2. Second, our annotations can serve as valuable ground-truth labels for future research on automating feature interpretation in ViTs. Given that research on automatic interpretation of ViT features is still in its early stages, we believe our annotations offer a meaningful foundation for further progress in this direction. We would also like to highlight that reviewers Y8uq and PvQ3 explicitly recognized our manual annotation effort as a strength of the paper, underscoring its contribution to interpretability analysis.\n\n3. Third, while recent studies have begun to explore automation for late-layer features, there is currently little to no research on automation for early-layer features. We believe this is because early-layer features tend to activate very locally within a small number of patches, making them particularly difficult to interpret automatically. However, through manual annotation, we were able to reliably interpret even such challenging cases."}, {"review_id": "ngJdmXahQF", "reviewer": "Reviewer_ru17", "summary": "The paper presents a detailed mechanistic analysis of Vision Transformers (ViTs) by adapting interpretability techniques from language models. The authors train Sparse Autoencoders (SAEs) on the residual stream of each layer across three pretrained ViT variants (supervised ImageNet, DINOv2, and CLIP), extracting 6.6K manually annotated features. They then discover meaningful circuits in the SAE latent space (rather than the ViT latent space). The results suggest that ViT features evolve from low-level patterns (colors, edges) in early layers to high-level semantic concepts (object parts, complete objects) in later layers, similar to CNNs. The authors present a number of case studies of circuits (curve detectors, position detectors, and a classification circuit). Additionally, the authors explore some practical applications of their work, using their framework to mitigate spurious correlations in ImageNet classifiers.", "questions": "1. Do the authors plan to release the SAE weights or an interactive interface for exploring features and circuits? This would be a very useful contribution and would help to increase the significance!\n\n2. Is there a reason why the authors use human annotations over some sort of automated interpretability method? If the goal was to provide a comprehensive analysis of all ViT layers, then automated interpretability would be the more natural choice since human annotations don’t scale very well. I also wasn’t sure why “gold standard” (in reference to the use of human annotations) was in quotes (line 99); is this in reference to a particular paper, or is this the author’s personal opinion?\n\n3. How were the 16 participants described in Section 2.2 selected? Were they recruited from e.g. Mechanical Turk, or were they colleagues of the authors? Were they compensated? It’s important to clarify these details when including results from human participants. \n\n4. How do the authors contextualize the Granny Smith circuit in relation to the results in Figure 2b? For example, shape features appear to be the most prevalent in layer 2, but shape features don’t appear in the Granny Smith circuit until layer 6.\n\n5. Faithfulness and completeness of the circuits are high, but the causality seems quite low (only ~50%). How might the authors explain this discrepancy?\n\n6. Did the authors try intervening on circuits in meaningful ways, e.g. trying to change Granny Smith classification to another class by intervening on e.g. the shape feature?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The early and middle layers in vision models are underexplored in mechanistic interpretability, so the paper’s exploration of these layers in ViTs is an important contribution. \n2. The paper presents some interesting case studies; for example, I found the Granny Smith circuit to be very interesting. The breakdown of feature types across layers presented in Figure 2b was also very interesting. \n3. The circuits discovered by the authors appear to be significantly more faithful compared to a number of baselines. \n\n**Weaknesses:**\n1. **Overstated novelty.** The paper greatly overstates its methodological novelty. The way the paper is written suggests that the residual replacement model is a novel methodology. However, it is nearly identical to Marks et al. (2024), which first trains SAEs on a language model’s layers, then identifies sparse circuits via causal importance in the SAE latent space (rather than the model’s latent space). I feel as though the authors overstated the novelty of their approach and neglected to sufficiently acknowledge the parallels to Marks et al. (2024); it is more accurate to say that the authors applied existing sparse feature circuit methodology from LLM interpretability to ViTs than to say the authors introduced a novel method. I would strongly recommend that the authors reframe the paper as an adaptation of existing NLP interpretability techniques to vision models, as the current framing is misleading.\n2. **Limited scope of the analysis.** The paper claims to present a “comprehensive, large-scale analysis” of features in ViTs. However, the authors opt for human annotation of features over an automated interpretability method; as a result, only 200 features per layer are labeled, which is a very small subset. This is highly likely to bias the circuits that are discovered, since important features may lack an annotation. The annotation methodology is also unclear; I can’t tell 1) how these 200 features were selected, 2) how the annotations were assigned, or 3) whether or how these annotations were validated. The human participants in section 2.2 are also not adequately described. For the analysis to be systematic, these methodological details should be described clearly; for it to be large scale, then perhaps the authors should employ automated interpretability techniques. \n3. **Lack of comparisons to existing methods.** It isn’t clear that the method is an improvement over existing vision interpretability techniques (e.g. Rajaram et al., 2024) since no comparisons to existing methods are made. \n4.  **Clarity.** The paper is missing some concrete technical details that I think would make the methodology and results much more clear. For example, in section 3.2, it wasn’t clear what the authors meant by “replacing a circuit;” what concretely is being done to remove the circuit? What is an empty circuit? Figures 5 and 7 include nodes labeled as “error,” but this doesn’t appear to be explained. \n\n**Citations:**\n\n[ 1 ] Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., & Mueller, A. (2024). Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647.\n\n[ 2 ] Rajaram, A., Chowdhury, N., Torralba, A., Andreas, J., & Schwettmann, S. (2024). Automatic discovery of visual circuits. arXiv preprint arXiv:2404.14349.", "was_revised": true, "final_justification": "The authors successfully addressed my concerns in their rebuttal. My initial score was strongly influenced by the work's similarity to Marks et al. (2024), leading me to judge that the work was lacking in novelty (Weakness 1). However, upon reading the author's rebuttal as well as the thoughts of the other reviewers, I now understand that the technical innovations related to implementing this method for ViTs are significant enough to warrant acceptance.  \n\nAs for Weakness 2 (\"Limited scope of the analysis\"), I still find the scope of the analysis to be relatively small. However, the authors promised to scale down their claims regarding the scope, and they clarified that 200 features per layer is actually a meaningfully large sample. I believe that the work would benefit from scaling to even more features, but the current number of features is a good start.\n\nWith respect to Weakness 3 (\"Lack of comparisons to existing methods\"), the authors kindly supply a comparison to an existing method that I mentioned, showing significant improvements. \n\nAs for Weakness 4 (\"Clarity\"), the authors promised to revise the problematic sections I pointed to.\n\nAll in all, I find the rebuttal to be satisfying enough to raise my suggestion to Borderline Accept.", "rebuttal": "Thank you for the detailed and thoughtful review. First, we apologize for the lack of clarity in the writing, missing comparisons and incomplete results that may have made the reading uncomfortable. Your feedback has been extremely helpful in improving our work, making many parts more clarified or concrete. We provide a detailed response to the reviewer's comments below. If there are any remaining concerns, please inform us, and we will address them promptly.\n\n# W1: Overstated Novelty\nWe appreciate the reviewer’s important feedback. We did not intend to overstate the novelty of our work, and we apologize if it was perceived that way. We will revise the paper to clarify this point. However, we would like to clarify two key aspects:\n\n1. Our work is deeply inspired by Marks et al. (2024), and we have made efforts to give appropriate credit in the paper. We discuss its relationship to our work in Appendix B and cite it in the Introduction and Related Work as a key reference. However, we recognize that our writing style may have unintentionally overstated the novelty. We will revise Sections 1, 3, and 5 to more accurately reflect our dependence on and connection to Marks et al. (2024).\n\n2. Additionally, we would like to emphasize that our work is a non-trivial extension of Marks et al. (2024) to ViTs, rather than a simple re‑implementation. Specifically, we address several computational and technical challenges that arise in adapting their approach to ViTs, as also noted by Reviewers Dv9E and Y8uq in their Strengths sections. In the revised version, we will explicitly acknowledge Marks et al. (2024) as the inspiration for our work and more clearly articulate how our method extends their approach to the ViT setting.\n\n# W2: Limited Scope of the Analysis\n* **Small number of interpreted features**: We acknowledge that the number of features interpreted in our analysis is not extremely large. However, as shown in Figure 2b, sampling 200 features per layer is sufficient to capture the distribution and trends of feature types across layers. This sample size gives meaningful insights into the overall feature distribution in the model. To the best of our knowledge, this is the first work to interpret features and identify trends across all layers of ViTs. However, we recognize that the scale of our analysis may have been overstated in the current manuscript. We will revise the wording in the main paper to more accurately convey this point.\n\n* **Bias in circuit discovery**: We would like to clarify that the discovered circuits do not require features to be interpreted in advance (i.e., the process is independent of the feature annotation). As demonstrated in Section 3.3, features within circuits can be interpreted post hoc without excessive effort. Therefore, the discovered circuits are not biased by the specific features selected for annotation.\n\n* **Annotation methodology**: We randomly sampled 200 features per layer and assigned them to the authors such that each feature was annotated by at least two people. To ensure consistency, we iteratively refined the category codebook through discussion (Appendix D.2). In cases where the two annotators disagreed on the category or name of a feature, a third author (not involved in the initial annotation) provided an independent judgment. If all three annotators disagreed, we classified the feature as \"uninterpretable.\"\n\nTo further validate annotation quality, we conducted an additional survey with five participants who were not involved in the paper. Each participant evaluated whether the provided annotation accurately described the visualization, responding with “Yes,” “Maybe,” or “No.” The results showed that 81.4% of responses were “Yes,” while only 6.82% were “No,” indicating that the annotations are generally reliable.\n\nWe apologize for not describing this process more clearly in the original submission and will revise the paper to provide a more detailed explanation. For details on participant recruitment, please refer to our response to Q3. For discussion on automated interpretability techniques, we kindly refer the reviewer to our response in Q2.\n\n# W3: Lack of Comparisons to Existing Methods\nWe thank the reviewer for suggesting a comparison with existing methods. Since Rajaram et al. (2024) does not release a public codebase, we re‑implemented their method based on Algorithm 1 (CLA) in their paper and compared it with ours. The results are shown below:\n\n| Strategy | | Faithfulness | | | 1 - Completeness | | | Causality | |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| | ViT | DINOv2 | CLIP | ViT | DINOv2 | CLIP | ViT | DINOv2 | CLIP |\n| Random | 30.2 | 27.5 | 27.0 | 78.1 | 90.1 | 84.4 | 35.6 | 33.9 | 31.1 |\n| CLA | 56.3 | 41.3 | 36.7 | 92.5 | 93.8 | 93.5 | 57.2 | 54.8 | 55.1 |\n| Ours | 94.1 | 85.1 | 82.3 | 99.6 | 99.8 | 99.7 | 54.5 | 54.8 | 53.8 |\n\nOur method substantially outperforms CLA in both faithfulness and completeness, demonstrating its effectiveness. Regarding causality, CLA achieves slightly higher scores, likely because it is specifically designed to identify nodes that directly influence subsequent layers. While this design inherently boosts causality, it does not account for the importance of features in the final output, which is critical for producing faithful and complete circuits. In contrast, our method explicitly considers the effect of features on the final output, leading to higher faithfulness and completeness.\n\n# W4: Clarity\nWe apologize for the lack of clarity in describing certain technical details.\nBy *removing* a circuit, we mean replacing its feature values with their median activation values to ablate the circuit’s influence. An empty circuit is one that contains no features (hence its faithfulness is 0 and completeness is 1) [37]. The error term in Figures 5 and 7 is defined in Line 87 and represents the portion of the representation not explained by the SAE [42].\nWe will revise the paper to clarify these points and add more detailed explanations for better readability. We appreciate the reviewer’s attention to these issues.\n\n# Q1: Weight/Interface Release\nWe will release all trained SAE weights. We also agree with the reviewer’s suggestion to provide an interactive interface for visualizing features and circuits. To this end, we are developing a user‑friendly tool that enables users to view feature descriptions by hovering over features within the circuit visualization. We plan to make this interface publicly accessible via an online link.\n\n# Q2: Automated Interpretability Techniques\nWe are grateful for the reviewer’s insightful question. First, we are the first to find that jointly interpreting logits, patches, and activated images is crucial for understanding the features. As a result, we were able to discover position detectors, line detectors, L‑shape detectors, etc. This process requires comprehensive visual inspection, which is challenging to automate with existing automated interpretability techniques (e.g., [22, 60, 61]). Therefore, we adopted human annotation to establish a reliable initial interpretation of features.\n\nBuilding upon this validated foundation, the natural next step is to pursue scalability, leveraging automated interpretability techniques to extend the analysis to a larger scale and to other models. Such automation would enable broader generalizability while preserving the rigor established in our human‑annotated analysis. We plan to explore these automated approaches in future work.\n\nRegarding the phrase \"gold standard\", we did not intend any personal implication; rather, it was meant to refer to a generally accepted standard in the field. We apologize for any confusion this may have caused and will remove the quotation marks in the revised version.\n\n# Q3: Human Participation in Section 2.2\nWe apologize for not providing sufficient detail regarding participant recruitment for the interpretability survey. Participants were recruited via a public community platform and compensated for their time. They were undergraduate or graduate students without prior experience in computer vision or mechanistic interpretability. We will include these recruitment details in the revised manuscript. We appreciate the reviewer’s suggestion to clarify this.\n\n# Q4: Granny Smith Circuit (Figure 5) vs Statistics (Figure 2b)\nAs noted by the reviewer, shape features are most frequently observed in layer 2, yet the Granny Smith circuit contains no shape features from that layer. While shape features are statistically prevalent in layer 2, their inclusion in any given circuit depends on the specific input image. Specifically, we infer that this is because Granny Smith has relatively simple shapes that do not require complex shape combinations at low‑level representations, and thus such features do not necessarily emerge in the early layers. Some circuits in Appendix E.4 do contain shape features (e.g., L2#10 in Figure 24, L2#430 in Figure 34).\n\n# Q5: Relatively Low Causality Score\nWe appreciate the reviewer’s thoughtful observation. Compared to faithfulness and completeness, causality is indeed a more difficult metric to achieve a perfect score. Causality measures the reduction in downstream activations across all layers $\\ell^\\prime \\gt \\ell$ when upstream nodes in layer $\\ell$ are ablated.\n\nFor example, with $\\ell = 1$ and $\\ell' = 10$, it is challenging to make activations in layer 10 to zero by ablating layer 1 nodes, since the signal must propagate through many intermediate layers and inevitably diffuses. This difference in nature explains why achieving perfect causality is inherently more difficult than achieving high faithfulness or completeness.\n\n# Q6: Intervention on the Circuit\nWe thank the reviewer for suggesting this promising direction. Intervening on circuits to observe changes in model behavior could be highly relevant for research in model safety and actionable interpretability. We will explore this avenue in future work."}, {"review_id": "qpNZBYLAfZ", "reviewer": "Reviewer_Y8uq", "summary": "This paper presents the first comprehensive analysis of Vision Transformer (ViT) features across all layers using sparse autoencoders (SAEs), and introduces the \"residual replacement model\" framework for interpretable circuit discovery. The authors manually annotate 6.6K features across ViT, CLIP, and DINOv2 models, revealing a progression from low-level visual patterns to high-level semantic concepts. Their residual replacement model bypasses attention complexity by focusing on the residual stream, enabling scalable circuit discovery through attribution patching. The framework is validated on spurious correlation debiasing tasks, demonstrating practical utility beyond interpretability.", "questions": "- how do the feature evolution patterns and circuit structures vary across different ViT architectures (e.g., ViT-L, ViT-H) and patch sizes? \n- can you provide insight on when the residual stream approximation accurately captures the full model behavior? Under what conditions might this approach fail?\n- how sensitive are the conclusions to the specific SAE hyperparameters and training procedures? \n- bypassing attention is a design choice and improves scalability, does this limit the understanding of attention mechanisms' specific contributions to ViT behavior? what insights about attention's role in ViT computation are lost?", "limitations": "The authors adequately discuss technical limitations in §6, including SAE constraints and the qualitative nature of interpretability evaluation.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- manual annotation of 6.6K features across all ViT layers represents the largest systematic study of transformer vision features to date, providing valuable empirical insights about feature evolution\n- the residual replacement model cleverly sidesteps attention mechanism complexity while maintaining faithfulness, addressing a key scalability challenge in transformer interpretability\n- the edge-based circuit discovery method and gradient correction techniques (extending LibraGrad) show solid technical innovation with measurable improvements in faithfulness metrics\n- application to spurious correlation debiasing is useful, achieving performance close to data-driven methods like SpuFix using only single-image analysis\n- three-metric evaluation framework (faithfulness, completeness, causality) provides robust validation of circuit quality\n- authors promise to release code, SAE weights, and the full annotated dataset\n\n**Weaknesses:**\n\n- despite user studies and cross-checking, the manual feature annotation remains inherently subjective and may not capture all relevant feature types or miss subtle concepts\n- while more efficient than naive approaches, the O(f) gradient computation faces challenges with very large feature dictionaries, potentially limiting applicability to future larger models\n- the paper lacks theoretical analysis of when and why the residual replacement approximation works well, relying primarily on empirical validation\n- beyond spurious correlation debiasing, the practical utility demonstration is somewhat narrow - additional downstream applications would strengthen the case (even though for interpretability immediate use cases are often not a given)", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and helpful observations. Our detailed responses to the comments are outlined below. If there are any remaining concerns, please feel free to inform us, and we will address them as soon as possible.\n\n# W1: Concern on Manual Annotations\nAs the reviewer correctly pointed out, we acknowledge that the use of a manual annotation process introduces the possibility of subjectivity. In addition to cross-checking among annotators, we made an effort to maintain objectivity by continuously referring to the definition of categories described in Appendix D.2, throughout the annotation process. Nonetheless, we recognize that it is inevitable that some subtle concepts may still be missed. We will make sure to state this limitation more clearly in our limitation section.\n\n# W2: Challenges with Large Feature Dictionaries\nWe appreciate the reviewer for the insightful comment. As rightly pointed out, the computational cost of our approach increases with the size of the dictionary. However, we would like to note that this is not a limitation unique to our method. All existing circuit discovery approaches [46, 78] inherently incur higher computational cost as the number of features increases.\nHowever, by removing token-wise interactions, we were able to reduce the circuit discovery time by approximately 200×. As a result, the time required to discover a circuit is now only a few seconds, which we believe is sufficiently scalable for current settings. While our method does scale with the number of features, it has the key advantage of being independent of the number of tokens. We believe that this property gives our approach greater potential for application to future, larger-scale models.\n\n# W3 & Q2: Discussion on Approximation of Residual Replacement Model\n\nWe thank the reviewer for this insightful comment. While empirical validation using faithfulness and completeness is a common practice in mechanistic interpretability, we agree that a more detailed analysis of the approximation in the residual replacement model would be beneficial. We first discuss the approximation theoretically, and then empirically analyze the conditions under which it performs more accurately.\n\n### 1. Theoretical Discussion\n\nTo construct a circuit, we approximate each feature’s contribution to the final output using the modified gradient method LibraGrad [86]. This approach ensures that the output logit $f(x)$  can be decomposed into contributions from each feature $z_i$ and bias terms $b_j$ (i.e., FullGrad-Completeness):\n$$\nf(x) = \\sum_{i=1}^{F} z_i \\nabla_{z_i} f(x) + \\sum_{j=1}^{M} b_j \\nabla_{b_j} f(x),\n$$\nwhere $x$ is the input image, $F$ is the number of features in a layer, and $M$ is the number of bias terms between that layer and the end of the model.\n\nAs described in [85], bias terms capture the importance of interactions between features, which may also provide valuable information when estimating each feature’s contribution to the final output. However, incorporating bias terms into circuit construction requires further theoretical investigation, as it may demand additional interpretive steps. If interactions among early‑layer features in later layers are substantial, the approximation may not fully capture the true contribution of each feature to the output.\n\nMoreover, gradient‑based methods are inherently less accurate than direct causal intervention, since their purpose is to approximate the effect of such interventions. If the model does not exhibit smooth gradients, this approximation may degrade in accuracy. We will include these considerations in the revised version of the paper. We also consider analyzing the accuracy of the approximation more rigorously in future work.\n\n### 2. Empirical Analysis\n\nFollowing the reviewer’s suggestion, we examined the conditions under which the discovered circuits best replicate the behavior of the full model. Specifically, we measured the faithfulness score of our circuits for each input image as a proxy for approximation accuracy. We then qualitatively analyzed the 100 images with the highest faithfulness scores and the 100 with the lowest, yielding the following observations:\n\nFirst, we found that images in which the main object occupies a large portion of the image frame tend to exhibit higher faithfulness. For instance, one image with a large soap bubble filling most of the image was included in the top 100, whereas another soap bubble image with multiple other dominant objects, such as people, appeared in the bottom 100.\n\nSecond, we observed that faithfulness tends to be higher when the object belongs to a class with distinct, easily recognizable features. For example, in the ImageNet dataset, dog classes are subdivided by breed (e.g., \"Scotch Terrier\" and \"Border Terrier\"), which makes them often difficult to distinguish one from another. Images from such fine-grained dog classes were frequently found in the bottom 100.\n\n# W4: Limited Practical Utility\nWe agree with the reviewer’s insightful comment. While our study primarily focuses on analyzing the features and decision process within the ViT, we acknowledge that we did not provide sufficient discussion on how our findings could be practically applied to downstream tasks.\nOur study aims to discover circuits that reveal the decision process of the ViT. We believe that this line of work could potentially be applied to investigating the underlying causes of model hallucination and, ultimately, to correcting such behaviors. We view this as a promising direction and plan to explore it further in future work.\n\n# Q1: Question on Feature Evolutions Across Different Patch Sizes\nIn this study, we prioritized analyzing the patterns from different training strategies (i.e., (ViT, CLIP, DINOv2), as we believed this to be more important than analyzing differences caused by the number of patches. Since this analysis has yielded several meaningful insights, we are planning to investigate how the number of patches affects the emergence of features for future work.\n\n# Q3: Robustness Across Different SAE Hyperparameters\nWe appreciate the reviewer for the insightful question. To examine the sensitivity of our conclusions to different SAE hyperparameters, we conducted two additional experiments:\n(1) reducing the SAE dictionary size $d$ by half across all layers, and\n(2) reducing the value of $k$ (the number of selected features) by half across all layers.\n\nThe results of these experiments are summarized in the tables below:\n\n- **Table 1 in our paper**\n| Method | Faithfulness (ViT) | Faithfulness (DINOv2) | Faithfulness (CLIP) | 1-Completeness (ViT) | 1-Completeness (DINOv2) | 1-Completeness (CLIP) | Causality (ViT) | Causality (DINOv2) | Causality (CLIP) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Random Circuit        | 30.2 | 27.5 | 27.0 | 78.1 | 90.1 | 84.4 | 35.6 | 33.9 | 31.1 |\n| Naive Circuit         | 64.9 | 58.9 | 50.7 | 94.2 | 99.0 | 99.3 | 46.9 | 42.2 | 40.3 |\n| +Edge-based           | 74.2 | 64.1 | 52.2 | 93.4 | 98.8 | 99.2 | 48.6 | 43.4 | 43.0 |\n| +Gradient Correction  | 94.1 | 85.1 | 82.3 | 99.6 | 99.8 | 99.7 | 54.5 | 54.8 | 53.8 |\n\n---\n\n- **Low R**\n| Method | Faithfulness (ViT) | Faithfulness (DINOv2) | Faithfulness (CLIP) | 1-Completeness (ViT) | 1-Completeness (DINOv2) | 1-Completeness (CLIP) | Causality (ViT) | Causality (DINOv2) | Causality (CLIP) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Random Circuit        | 43.1 | 28.7 | 28.1 | 55.8 | 85.3 | 82.2 | 21.2 | 25.0 | 24.9 |\n| Naive Circuit         | 77.4 | 60.8 | 62.6 | 85.7 | 98.9 | 99.1 | 27.6 | 32.2 | 32.9 |\n| +Edge-based           | 86.8 | 66.9 | 66.3 | 85.1 | 98.5 | 99.0 | 30.0 | 33.8 | 35.6 |\n| +Gradient Correction  | 99.2 | 88.1 | 85.1 | 99.2 | 99.6 | 99.6 | 35.9 | 45.6 | 43.5 |\n\n---\n\n- **Low K**\n| Method | Faithfulness (ViT) | Faithfulness (DINOv2) | Faithfulness (CLIP) | 1-Completeness (ViT) | 1-Completeness (DINOv2) | 1-Completeness (CLIP) | Causality (ViT) | Causality (DINOv2) | Causality (CLIP) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Random Circuit        | 30.9 | 27.5 | 27.2 | 77.4 | 90.5 | 84.2 | 38.6 | 37.9 | 33.6 |\n| Naive Circuit         | 64.1 | 54.7 | 49.7 | 92.4 | 98.8 | 99.1 | 51.0 | 46.6 | 43.5 |\n| +Edge-based           | 73.6 | 59.0 | 50.3 | 91.2 | 98.5 | 98.9 | 52.8 | 47.2 | 46.5 |\n| +Gradient Correction  | 93.7 | 84.2 | 81.1 | 99.6 | 99.8 | 99.7 | 58.3 | 60.3 | 56.4 |\n\n\nAs shown above, the overall trends are preserved across various SAE hyperparameters. These results suggest that our findings are robust to the choice of $d$ and $k$, further supporting the reliability of our conclusions.\n\n# Q4: Limited Understanding of Attention Mechanism\nAs the reviewer correctly noted, our circuit offers only a limited understanding of the attention mechanism. Since interactions between different tokens occur exclusively through the attention module, it remains difficult to precisely determine which patches are interacting with each other. This limitation restricts our ability to fully understand how ViTs collect global information (e.g., via the [CLS] token) via attention.\nHowever, incorporating the attention module into the circuit discovery process presents significant challenges in identifying a parsimonious circuit. For this reason, we deliberately chose to bypass the attention module in order to extract meaningful insights from the ViT in a tractable manner. Discovering circuits that explicitly include the attention mechanism remains an important direction for our future work."}]

Author rebuttal:

Meta-review: This paper used sparse autoencoder to interpret features learned by vision transformers. The methodology was borrowed from Marks et al. (2024) for LLMs. However, some technical challenges were addressed to apply this to ViTs, which adds to the technical contribution of this paper. The reviewers generally find this direction relevant and the results interesting. After some clarifications and additional experiment results in the rebuttal, the reviewers generally agreed on acceptance. The authors are recommended to follow the suggestions and modify the text to clearly state the contributions in the context of previous work, and to add details in terms of human studies raised by Ethics Reveiwers.
Decision: Accept (poster)