Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP
Authors: Sriram Balasubramanian
Samyadeep Basu
Soheil Feizi
Venue: NeurIPS 2024 poster (2024)
Keywords: vision
interpretability
explainability
Primary area: interpretability_and_explainability
TL;DR: We show how to do representation decomposition and interpretation if your ViT != CLIP

Abstract: Recent work has explored how individual components of the CLIP-ViT model contribute to the final representation by leveraging the shared image-text representation space of CLIP.  These components, such as attention heads and MLPs, have been shown to capture distinct image features like shape, color or texture. However, understanding the role of these components in arbitrary vision transformers (ViTs) is challenging. To this end, we introduce a general framework which can identify the roles of various components in ViTs beyond CLIP. Specifically, we (a) automate the decomposition of the final representation into contributions from different model components, and (b) linearly map these contributions to CLIP space to interpret them via text. Additionally, we introduce a novel scoring function to rank components by their importance with respect to specific features.
Applying our framework to various ViT variants (e.g. DeiT, DINO, DINOv2, Swin, MaxViT), we gain insights into the roles of different components concerning particular image features. These insights facilitate applications such as image retrieval using text descriptions or reference images, visualizing token importance heatmaps, and mitigating spurious correlations. We release our [code](https://github.com/SriramB-98/vit-decompose) to reproduce the experiments in the paper.

Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP
Sriram Balasubramanian Department of Computer Science University of Maryland, College Park srirane aarrrmmbcc... umd. edu
Samyadeep Basu
Department of Computer Science University of Maryland, College Park sbasul 2@umd.
Soheil Feizi
Department of Computer Science University of Maryland, College Park sfeizi@cs. umd. udd
Abstract
Recent work has explored how individual components of the CLIP-ViT model contribute to the final representation by leveraging the shared image-text represen- tation space of CLIP. These components, such attention heads and MLPs, have been shown to capture distinct image features like shape, color or texture. However, understanding the role of these components arbitrary vision transformers (ViTs) is challenging. To this end, we introduce a general framework which can identify the roles various components in ViTs beyond CLIP. Specifically, we (a) auto- mate the decomposition of the final representation into contributions from different model components, and (b) linearly map these contributions to CLIP space to interpret them via text. Additionally, we introduce novel scoring function to rank components by their importance with respect to specific features. Applying our framework to various ViT variants (c.g. DeiT, DINO, DINOv2. Swin, Max VIT), we gain insights into the roles of different components concerning particular image features. These insights facilitate applications such image retrieval using text descriptions or reference images, visualizing token importance heatmaps, and mitigating spurious correlations. We release code reproduce the experiments at ltttt:::/// //tpthh////ttbbb http://rrrrr---//ee--eeeeooppppe
Introduction
have emerged as powerful image encoders, becoming the preferred architecture for modern image foundation models. However, the mechanisms by which these models transform images into representation vectors remain poorly understood. Recently, Gandelsman al. [IT] made significant progress this question for CLIP-ViT models with two key insights: (i) They demonstrated that the residual connections and attention mechanisms of CLIP-ViT enable the model output to be mathematically represented vectors over layers, attention heads, and tokens, along with contributions from MLPs and the CLS token. Each vector corresponds the contribution of a specific token attended to by a particular attention head in a specific layer. (ii) These contribution vectors exist within the same shared image- text representation space, allowing the CLIP text encoder interpret each vector individually via text.
Vision transformers and their variants 10
22
7
33
17
32
sum of
Extending this approach to other transformer-basee based image encoders presents several challenges. Popular models like DeiT (32), DINO-VIT 7 22), and Swin [17 lack a corresponding text encoder to
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Contributions
Represeniaion
REP DEcDDooPOOS
Residual Trans.
COMPAMGN
Joint token-component decomposition
fo 6
6
Image-based Image retrieval
fieal 1isd
Residual Trans.
CLIP Text Encoder
patterm
"beach""
Image
"person"
Toceetoon
"stried pattem"
Figure 1: (Left) Workflow: The first step (REPDECOMPOSE) is to decompose a representation z into contributions from its model components e; after being transformed by residual transformations like LayerNorm, linear projections, resampling, patch merging and so on. The second step (COMPALIGN) aligns each contribution CLIP space using set of linear maps fu- fi..... fn on the corresponding contributions Co, Ci...., We can then interpret these aligned contributions using the CLIP text encoder. (Right) Applications of our method: specific component using joint token-component decomposition close matches of the reference image (on top) with respect to a given image feature like pattern, person, or location
(a) Visualizing contributions of each token through
(b) Retrieving images that are
interpret the component contributions. Additionally, extracting the contribution vectors corresponding to these components is not straightforward, as they are often not explicitly computed during the forward pass of the model. Other complications include diverse attention mechanisms such as grid attention, block attention (in Max and windowed/shifted windowed attention (in Swin), as well various linear transformations like pooling, downsampling, and patch merging applied to the residual streams between attention blocks. These differences necessitate a fresh mathematical analysis for each model architecture, followed by careful application of necessary transformations to the intermediate output of each component to determine its contribution to the final representation. To address these challenges, we propose our framework (described in Figure] identify roles of components in general ViT's.
First, automate the decomposition of the representation by leveraging the computational graph created during the forward pass. This results a drop-in function, REOMESOEE that can decompose any representation into contributions vectors from model components simply by calling it on the representation. Since this method operates on the computational graph, is agnostic to the specifics of the model implementation and thus applicable
variety of model architectures.
Secondly, we introduce an algorithm, COMPALIGN, to map each component contribution vector to the image representation space of CLIP model. We train these linear maps with regularizations so that these maps preserve the roles of the individual components while also aligning the model's image representation with CLIP's image representation. This allows us to map each contribution

---

vector from any component to CLIP space, where they can be interpreted through text using a CLIP text encoder.
Thirdly, we observe that there is often no straightforward one-to-one mapping between model components and common image features such shape, pattern, color, and texture. Sometimes, single component may encode multiple features, while multiple components may be required to fully encode single feature. To address this, we propose a scoring function that assigns an importance score to each component-featur pair. This allows us to rank components based on their importance for a given feature, and rank features based on their importance for a given component.
Using this ranking, we proceed to analyze diverse vision transformers such as DeiT, DINO, Swin, and Max ViT, in addition to CLIP, in terms of their components and the image features that they are responsible for encoding. We consistently find that many components in these models encode the same feature, particularly in ImageNet pre-trained models. Additionally, individual components in larger models Max Vit and Swin do not respond to any image feature strongly, but can encode them effectively in combination with other components. This diffuse and flexible nature of feature representation underscores the need for interpreting them using a continuous scoring and ranking method as opposed labelling each component with a well-defined role. We are thus able to perform tasks such as image retrieval, visualizing token contributions, and spurious correlation mitigation by carefully selecting or ablating specific components based on their scores for given property.
2 Related Work
Several studies attempt to elucidate model predictions by analyzing either a subset of input example
through heatmaps 27 30 31 [8] or subset of training examples [13 23 24) Nevertheless,
empirical evidence suggests that these approaches are often unreliable in real scenarios 14 3
These methods do not interpret model predictions in relation the model's internal mechanisms, which is essential for gaining deeper understanding of the reliability of model outputs.
Internal Mechanisms of Vision Models: Our work is closely related to the studies by Gandelsman et al. [1] and Vilas et al. 34],, both of which analyze vanilla ViTs in terms of their components and interpret them using either CLIP text encoders pretrained ImageNet heads. Like these studies, our research can be situated within the emerging field of representation engineering (36] and mechanistic focus on interpreting individual neurons understand vision models' internal mechanisms. However, these methods often fail to break down the model's output into its subcomponents, which is crucial for understanding model reliability. Shah et al. [29) examine the direct effect of model weights on output, but do not study the fine-grained role of these components in building the final image representation. Balasubramanian and Feizi [ focus on expressing CNN representations as a sum of contributions from input regions via masking.
interpretability (6. Other works [2 21
Interpreting models using CLIP:
Many recent works utilize CLIP 25 to interpret models via text. Moayeri et al. [19 align model representations to CLIP space with a linear layer, but it is limited to only the final representation and can not be applied to model components. Oikarinen and Weng 20 annotate individual neurons in CNNs via CLIP, but their method cannot be extended easily to high-dimensional component vectors. COMPALIGN is related to model stitching in which one representation space is interpreted in terms of another by training a map between two spaces 2 [6
3 Decomposing the Final Image Representation
Recently, Gandelsman al. [1I decomposed ECLS. tins the final [CLS] representation of the CLIP's image encoder as a sum over the contributions from its attention heads, layers and token positions, as well as contributions from the MLPs. In particular, they observe that the last few attention layers have a significant direct impact on the final representation. Thus, this representation can be decomposed
as: ZCLS. iin #CLS, init + El CLMLP + Ele Em= ΣN CLA.A, where L., H, N correspond to the number of layers, number of attention heads and number of tokens. Here, Cr,h,a denotes the contribution of token t though attention head h in layer 1, while CLMLP denotes the contribution from the MLP in layer 1. Due to this linear decomposition, different dimensions can be reduced by summing over them to identify the contributions of tokens or attention heads to the final representation. While this decomposition is relatively simple for vanilla ViTs, it cannot be directly used for general ViT architectures due to use of self-attention variants such window attention, grid attention, or block
3

---

Algorithm REPDECOMPOSE
Input: f. the final node (denoting the function that output the representation 2) in the computational
graph G N Output: {e' 1==1. N direct contributions from various components (indexed by j) in the model after graph traversal
function REPDECOMPOSE(f)
f(z1, 22...., kn)
if f is non-linear then
return [z]
Cannot decompose further b f is linear
else
Let = fi...), = S2(... Ja(...)...,,..- Sn(e... N; Vi, {c. j=1 = REPDECCOMOOS((F) > 2: 2; -ij= of (c. are component contributions )
Then, = -[E G. Ej G..... Ej. c
Or, = ====E==1 E (c)) > exists since f distributes over inputs due linearity
return [[
attention, combined with operations such pooling or patch merging the residual stream. The final representation may also not just be single #CLS, inn but N Σ Zi,6m in even & Σ #CLS, or some combination of the above.
3.1 REPDECOMPOSE: Automated Representation Decomposition for ViTs
We thus seek a general algorithm which can automatically decompose the representation for general ViT's This can be done via recursive traversal of the computation graph. Suppose the final representation z can be decomposed into component contributions C₁, such that z = Er, Ci,-- Here each C corresponds to the contribution of a particular token through some model component i. For convenience, let C, Σ. Ct Then, if given access to the computational graph of x, we can identify potential linear components Ci by recursively traversing the graph starting from the node which outputs in reverse order till hit non-linear node. The key insight here is that the output of any node which performs a linear reduction (defined as a linear operation which results in a reduction in the number of dimensions) is equivalent to sum of individual tensors of the same dimension the output. These tensors can be collected and transformed appropriately during the graph traversal to obtain list of tensors Ci,r, each members the same vector space as the representation z. This kind of linear decomposition is possible due to the overwhelmingly linear nature of transformers. The high-level logic of REPDECOMPOSE is detailed in Algorithm please refer to Algorithm in the appendix or the code for more detailed description. We also illustrate the operation of the algorithm on an attention-Ne MLP block in the appendix.
In practice, the number of components quickly explodes as there are very large number of potential component divisions for given model. To make analysis and computation tractable, we restrict it to only the attention heads and MLPs with no finer divisions. We also constrain REPDECOMPOSE only return the direct contributions of these components contribution c; is the direct contribution of component i to z, and does not include its contribution via downstream component j. Additionally, the token in Ci,r present in the input of the component i, and not the input image. In principle, REPDECOMPOSE could return higher order terms such as Cj which the contribution of model component via the downstream component j. A full understanding of these higher order terms is essential to get a complete picture of the inner mechanism of a model, however we defer this for future work.
the output. This means that the
4 Aligning the component representations to CLIP space
Having decomposed the representation into contributions from relevant model components, we now aim to interpret these contributions through text using CLIP by mapping them to CLIP space.
Formally, given that we have a set of vectors {c; J such that Σ e; = 2, the final representation of model, we require set of linear maps f. such that the sum of S((.) = ECLiP, the final

---

Embedding Source
ImageNet pretrained
One map only
COMPALIGN
COMPALIGN
(A = 0)
wardrobe medicine cabinet window shade
gyromitra home theater
bookcase
filing cabinet
snorkel
snorkel
TEXTSPAN's
drumstick
red wolf
bakery
top 10 descriptions of random component
desk barbershop
Samoyed
barbershop
bathtub
muzzle
microwave oven bassinet
dining table
refrigerator
bookstore
red wolf
library
dining table
disc brake
gyromitra
shoji screen
medicine cabinet
dining table
shoji screen
bathtub
park bench
sink
Norwich Terrier
dining table
tusker
window screen
bookstore
Match rate Cosine Distance
0,08
0.155
0.185
0.23
0.18
0.17
Table I: Comparison of different methods to map the representation space of ImageNet- Ik pre-trained DeíT-B/16 CLIP image representation space. The green colored texts are exact matches with the 10p-10 descriptions obtained from the imagenet pretrained embeddings, while the orange colored texts are approximate matches. The match rate is the average fraction of exact matches across all components, while cosine distance is the average cosine distance between the CLIP representations and the transformed model representations on ImageNet
representation of the CLIP model. Once have these CLIP aligned vectors, proceed to interpret them via text using CLIP's text encoders.
However, from an interpretability standpoint, a few additional constraints the linear maps are desirable. Consider component contribution c; and two directions u, v belonging to the same vector space C, which represent two distinct features, say shape and texture. Let us further assume that the component's dominant role is to identify shape, and thus the variance of the projection of c; along is higher than that of v. We want this relative importance of features to be maintained in fc.)... Additionally, we also want any two linear maps fi and f to not change the relative norms of features in components c; and Cy. We can express these conditions formally follows:
1. Intra-component norm rank ordering: For any two vectors u, v and a linear map fa such that ||u|| < ||l|] we have [[f( s [+)]l
Inter-component norm rank ordering: For any two vectors u, v and linear maps fa. such have [u)]l s (Ifr(w))l
that |||||| < |0||,
Theorem 1. Both of the above conditions together imply that all linear maps fa must be scalar multiple of an orthogonal transformation, that is for all i, ft kI for some constant k. Here, is the identity transformation.
The proof is deferred to appendix We can now formulate a novel alignment method, COMPALIGN. map contributions of model components to CLIP space. COMPALIGN minimizes loss function over | to obtain good alignment between model representation space and CLIP space:
1 (E FCUP + f |
L(fA)N Be
The first term of the objective is the alignment loss, which is the average cosine distance between the CLIP representation #CLPP and the transformed model representation Σ, f. (c.). quantifies the directional discrepancy between the two vectors. The second term is the orthogonality regularizer which imposes a penalty if the linear maps fa are not orthogonal, ensuring that the Si adhere closely to the specified conditions. We can now train fi using the above loss function on ImageNet-1k, lk. The training label-free and can be done even over unlabeled datasets. We obtain žCLPP from the CLIP image encoder and NN N from running REPDECOMPOSE the final representation of the model.

---

DeiT
0.8
CLIP
ImageNet accuracy
DINO
0.6
DINOv2
SWIN
0.5
MaxVit
02-
42 Number of layers ablated (starting from the last layer)
10 12
24 28 20
26
52 54 56
Figure 2: Ablation results for various different image encoders. The top-1 ImageNet accuracy is plotted as the layers of the model are increasingly ablated away, starting from the last layer up till the first layer. The circles the plot represent the endpoints of blocks, the definition of which varies across model architectures. For the vanilla ViT variants, a block is an attention MLP pair, while for SWIN, it is pair of windowed/shifted windowed attention and an MLP. For Max this might either be a grid/block attention-MLP pair, or an MBConv block.
Ablation study: We now conduct ablation study on COMPALIGN. The first naive alignment method is the case where all f; are the same linear map f. without constraints on f. similar to [19]. The second method is version of COMPALIGN with À = O, where all ft are different but not trained with the orthogonality regularizer. To compare these methods, we first get "ground truth" description for each model component by using the TEXTSPAN [|I] algorithm the class embedding vectors from the ImageNet pre- pre-trained head. TEXTSPAN retrieves those class embedding vectors along which variance of the component output is maximized, thus yielding descriptions of each component in terms of the top 10 most dominant ImageNet classes. We then use COMPALIGN and the baselines to map the representations CL.IP space, and apply TEXTSPAN ImageNet class vectors to label each model component. We can then compare the descriptions this yields with the "ground truth" text description for each head. The results, shown in Tab. [ indicate that COMPALIGN's TEXTSPAN descriptions have the most matches to the ImageNet pre-trained descriptions, followed by COMPALIGN with À 0 and the naive single map method. This trend is similar in the average cosine distance between the CLIP representations and the transformed model representations.
CLIP embedded
5 Component ablation
To identify the most relevant model layers for downstream tasks, we progressively ablate them and measure the drop in ImageNet classification accuracy. Ablation involves setting layer's contribution
to its mean value over the dataset. We use the following models from Huggingface's 's t timm [35
repository: (i) DeiT (ViT-B-16) [32], (ii) DINO (ViT-B-16) 16) M (iii DINOv2 (VYT-B-14)
22 (iv)
Swin Base (patch size 4, window size 7) 17. (v) Max VIT Small 33 along with (vi) CLIP
(ViT-B-16) [9] from open_cllle clip [3. DeiT, Swin, and Max are pretrained on ImageNet with a
supervised classification loss, DINO on ImageNet with self- superviedd loss, DINOv2 on LVD-142M with self-supervised loss, while CLIP is pretrained on a LAION-22 2B subset with contrastive loss.
In Fig. 2 we see that for models not trained on ImageNet (CLIP and DINOv2), removing the last few layers quickly drops the accuracy zero. In contrast, models trained ImageNet experience more gradual decline in accuracy, reaching zero only after more than half the layers are ablated. This trend is consistent across both sulf-super (DINO) and classification-supeeee models encode useful features redundantly
(DeiT, SWIN,
Max ViT) models. This suggests that ImageNet-traing across layers for the classification task. Additionally, larger models with more layers, such as Max Vit, show significantly more redundancy, with minimal accuracy impact from ablating the last four layers. Conversely, the first few layers all models contribute little to the output. Therefore, our analysis in the subsequent sections is focused on the last few layers of each model.

---

Forest
Most relevant
Least relevant
Figure 3: Top-3 images retrieved by DeiT components for "forest" and "beach" ordered according to their relevance for the attribute "location". Each column here corresponds to the images returned by the sum of contributions of components, column i corresponds components C3s. C3s+1 C3++2- large fraction of components which can recognize the "location" feature are sorted correctly by the scoring function
Feature-based component analysis
We now analyze the final representation terms finer components like attention heads and MI.Ps, focusing the last few significant layers. We limit decomposition to 10 layers for DeiT, DINO, and DINOv2, but 12 layers for SWIN and 20 layers for Max Vit due to their greater depth and redundancy across components. We accumulate contributions from the remaining components in single vector Cinits expressing
Cinit Σ C₁ where N + is the total number of components including Ciit- Here, N = 65 for DeiT. DINO, and DINOv2; N = 134 for SWIN, and N = 156 for Max Vit.
We then ask if it is possible attribute feature-specific role to each component using algorithm such as TEXTSPAN [.]. These image features may be low-level (shape, color, pattern) or high-level (such as location, person, animal). However, such roles not necessarily localized to single component, but may be distributed among multiple components. Furthermore, each individual component by itself may not respond significantly to particular feature, but it may jointly contribute to identifying feature along with other components. Thus, rather than rigidly matching each component with a role, aim devise scoring function which can assign score each component feature pair, which signifies of how important the component is for identifying a given image feature. continuous scoring function allows us to select multiple components relevant the feature by sorting the components according to their score.
We devise this scoring function (described in the appendix in Alg. B3 by looking at the projection of each contribution vector C; onto vector space corresponding to certain feature. Suppose we have a feature, "pattern", that we want attribute the components. We first describe the feature in terms of an example set of feature instantiations, such "spotted", "striped"", "checkered",, and so on. We then embed each of these texts to CL.IP space, obtaining a of embeddings B. We also calculate the CLIP aligned contributions fc(),) for each component i over an image dataset (ImageNet- 1k validation split). Then, the score simply the correlation between projection of fa(c;) and the projection of E, fc.)) onto the vector space spanned by B. Intuitively, this measures how closely the component's 's contribution correlates with the overall representation. The
Model
Feature ordering
Component ordering
DeiT
0.531
0.684
DINO
0.714
0.723
DINOv2
0.716
0.703
SWIN
0.628
0.801
Max Vit
0.681
0.849
Table 2: Spearman's rank correlation between the orderings induced by CLIP score and component score averaged over a selection of common features
scores obtained for each component and feature can be used to rank the components according to its importance for given feature obtain a component ordering, or to rank the features according to its importance for a specific component get a feature ordering.
7

---

pattern
Figure 4: Top-3 images retrieved by the most significant components for various features relevant to the reference image (displayed on top). The models used are (from left to right) DINO, DeiT, and SWIN. More exhaustive results can be found in appendix H
6.1 Text based image retrieval
We can now use our framework to identify components which can retrieve images possessing certain feature most effectively. Using the scoring function described above, we can identify the top k components {c,) which are the most responsive to given feature p. We can use the cosine similarity of Σ f((e)) the CLIP embedding of an instantiation Sp of the feature p to retrieve the closest matches in ImageNet-Ik lk validation split. In Fig. 3 we show the top 3 images retrieved by different components of the DeiT' model for the location instantiation "forest" and "beach" when sorted according to the component ordering for the "location" feature. As the component score decreases, the images retrieved by the components grow less relevant. Also note that a significant fraction of components capable of retrieving relevant images. This further confirms the need for continuous scoring function which can identify multiple components relevant to a feature.
To quantitatively verify our scoring function, we devise the following experiment. We first choose set of common image features such color, pattern, shape, and location, with representative set of feature instantiations for each (details in appendix B). The scoring function induces a component ordering for each feature p and feature ordering for each component i. We then compute the cosine similarity sim;,s, = cos(S (cx),, CLIP, where CLII is the CLIP text embedding of 8p. We can compare this to the cosine similarity SiMCLIP.s, = (CCLPC CLIP) where #CUIP is the CLIP image representation. The correlation coefficient between sim,,,, and SiMCLIP,,, over an image dataset can be viewed as another score which is purely function of how well the component can retrieve images matching Sp as judged by CLIP. Averaging this correlation coefficient over sp for given p yields "ground truth" proxy for our scoring function. We can measure the Spearman rank correlation (which ranges from to 1) between the component (or feature) ordering induced by our scoring function and the ground truth and average it over features (or components). In Tab. 2 we observe that the rank correlation is significantly high for all models for both feature and component ordering. The individual rank correlations for component orderings for common features can be found in Tab. 4
6.2 Image based image retrieval
We can also retrieve images that are similar to reference image with respect to specific feature. To this, we first choose components which are highly significant for the given feature while being comparatively less relevant for other features. Mathematically, for feature E P. the set all relevant features, we want to choose component with score 8i,p such that the quantity miny' e P Si,p sipp is maximised. Intuitively, want components which have the highest

---

TUT
DEOCY
Voaa Tind
Figure 5: Visualization of token contributions as heatmaps for two example images for the DeiT model. The relevant feature and the head most closely associated with the feature is displayed on the bottom of the heatmap, while the feature instantiation displayed on the top. The layer numbering starts from the last layer (which has index '00"). The regions highlighted in red contribute positively to the prediction, while blue regions contribute negatively. More results in appendix
gap between Si.p and Si,p' where p' can be any other feature. We can then select a set of such components Cş by sorting over the score gap, and sum them to obtain feature-specific image representation 2p = Liec, C; Now, we can retrieve any image x' similar in feature p to a reference image x by computing the cosine similarity between z, and 2p. which are the feature-specific image representations for x' and z. We show a few examples for image based image retrieval in Fig. 4 Here, we tune
ensure that it is not so small that the retrieved images do resemble the reference image at all, and not so large that the retrieved images are overall very similar to the reference image. We see that the retrieved images are significantly similar to the reference image with respect to the given feature. but not similar overall. For example, when the reference image is handbag with a leopard print, the "pattern" components retrieve images of leopards which have the same pattern, while the "fabric" components return other bags which are made of similar glossy fabric. Similarly, for the ball with spiral pattern on we retrieve images which resemble the spiral pattern in the second row, while they resemble the shape in the third row.
Note that this experiment only involves the alignment procedure for computing the scores and thereby selecting the component set Cs. The process of retrieving the images is based on Zp which exists in the model representation space and not CLIP space. This shows that the model inherently has components which (while not constrained to a single role) are specialized for certain properties, and this specialization not a result of the CLIP alignment procedure.
6.3 Visualizing token contributions
As discussed in Section 3.1 contribution from component can be further decomposed as a sum over contributions from tokens, C; =, Ci For any particular CLIP text embedding vector corresponding a realization of some feature p, we have u fi())) = E. u fi(Ct,). We can visualize this token-wise score u f((c).) as heat map to know which tokens are the most influential with respect to u. We show the heat map obtained via this procedure Fig. for two example images for the DeiT model. The components used for each heat map correspond the feature being highlighted and are selected using the scoring function we described previously. We can observe that the heatmaps are localized within image portions which correspond to the text description. We also compare our method against zero-shot segmentation methods for ImageNet classes such GradCAM 28 and Chefer et al. (8] and find that our method outperforms the baselines (see Appendix J).
Model
Worst group accuracy
Average group accuracy
name
DeiT
0.733
0.815
0.874 + 0.913
CLIP
0.507 + 0.744
0.727 + 0.790
DINO
0.800 + 0.911
0.900 + 0.938
DINOv2
0.967 + 0.978
0.983
0.986
SWIN
0.834 + 0.871
0.927 + 0.944
Max Vit
0.777 0.814
0.875 0.887
Table Worst group accuracy and average group accuracy for Waterbirds dataset before and after intervention for various models (for- mat is before - after)
6.4 Zero-shot spurious correlation mitigation
We can also use the scoring function to mitigate spurious correlations the Waterbirds dataset (6 in zero-shot manner. Waterbirds dataset is synthesized dataset where images of birds commonly

---

found in water ("waterbirds") ") and land ("landbirds") are cut out and pasted on top of images of land and water background. For this experiment, we regenerate the Waterbirds dataset following Sagawa et al. [26) but take care to discard background images with birds and thus eliminate label noise. We select the top 10 components for each model which are associated with the "location" feature but not with the "bird" class following the method we used in Sec. 6..2 We then ablate these components by setting their value to their mean over the Waterbirds dataset. In Tab. 3 we observe a significant increase in the worst group accuracy for all models, accompanied with an increase in the average group accuracy as well. The changes in all four groups can be found appendix K in Tab. 6
7 Conclusion
In this work, propose VIT component interpretation framework consisting of an automatic decomposition algorithm (REPDCCOMPOS) to break down the model's final representation into component contributions and method (COMPALiGN) text-based interpretation. We also introduce a continuous scoring function to rank components by their importance in encoding specific features and to rank features within component. We demonstrate the framework's 's effectiveness in applications such as text- based and image- retrieval, visualizing token-wise contribution heatmaps, and mitigating spurious correlations in zero-shot manner.
map these contributions to CLIP space for
Acknowledgments
This project was supported in part by a grant from an NSF CAREER AWARD 1942230, ONR YIP award
ARO's Early Career Program Award 310902-00001, (DARPA/ RED), HR001 19S0026 (DARPA/ GARD), Army Grant No. W91 INF2120070 the NSF award CCF2212458, NSF Award No. 2229885 (NSF Institute for Trustworthy Al in Law and Society, TRAILS), an Amazon Research Award and an award from Capital One.
HR001 EIR001120900 12090132
Author contributions
Sriram Balasubramanian conceived the main ideas, implemented the algorithms, conducted the experiments, and contributed to writing the paper. Samyadeep Basu contributed to the writing and provided essential advice on the presentation and direction of the paper. Soheil Feizi offered critical guidance on the presentation, writing, and overall direction of the paper.
References
[1] S. Balasubramanian and S. Feizi. Towards improved input masking for convolutional neural networks. In 2023 JEEE/CVF International Conference Computer Vision (ICCV), pages 1855-1865, 2023. doi: 10.1 1.1109//CCV5510002223..00
[2] Y. Bansal, P. Nakkiran, and B. Barak. Revisiting model stitching to compare neural representa- tions, 2021.
[3] S. Basu, P. Pope, and S. Feizi. Influence functions in deep learning are fragile. CoRR, abs/2006. 14651, 2020. URL hattps://arxiv/ org//bss//00...... 14651
[4] D. Bau, Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba. Understanding the role of individual units in deep neural network. CoRR, abs/2009.05041, 2020. URL https / //arxiv.. www.orggbbba.o.o abs//09.. 05041
[5] Bricken, A. Templeton, Batson, B. Chen, Jermyn, T. Conerly, Turner, C. Anil, Deni- son, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield- Dodds, A. Tamkin, K. Nguyen, B. McLean, E. Burke, Hume, Carter, Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary learn- ing. Transformer Circuits Thread, 2023. http://haaoommmmmeccceiiimp///2m fcatures/index.hum. html.
10

---

[6] N. Cammarata, S. Carter, G. Goh, C. Olah, M. Petrov, L. Schubert, C. Voss, B. Egan, and S. K. Lim. Thread: Circuits. Distill, 2020. doi: 10.23915/distil.000022
[7] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers, In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[8] H. Chefer, S. Gur, and L. Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 782-791, June 2021.
[9] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-imann learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 281 2818 08-2899,2202 2023.
[10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image worth 16x16 words:
Transformers for image recognition at scale, 2021.
[11] Y. Gandelsman, A. A. Efros, and J. Steinhardt. Interpreting CLIP's image representation via text- based decomposition. In The Twelfth International Conference on Learning Representations, 2024. URL httphtts//ppeereeiieee ott.:/oorr.....oooooooooooaaaaaa
met/forumm//
[12] G. Goh, N. C. f, C. v. Carter, M. Petrov, I.. Schubert, A. Radford, and C. Olah. Mul- timodal neurons in artificial neural networks. Distill, 2021. doi: (0.23915/distill.oom http://disiillppbb22222 l/mulimodal--neuron
[13] G. Ilharco. M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong. J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL hhttps://doi. htt::://doo.o/////// org/10. 10. 5281/2enode 5143773 you use this software, please cite it as below.
[14] P.-. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. Schüttt, S. Dähne, D. Erhan, and B. Kim. The (un)reliability of saliency methods, 2017.
[15] W. Koh and Liang. Understanding black-box predictions via influence functions, 2020.
[16] K. Lene and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence, 2015. URL htttttttts///iiii wttp://rrggggaa//11.55000000
[17] Z. Liu, Y. Lin, Y. Cao, H. Hu, Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[18] M. Lundberg and S. Lee. A unified approach to interpreting model predictions. CORR, fts//105.07874, 2017. URL. httpp:///// wtt.://orrr/abba//...0.
[19] M. Moayeri, K. Rezaei, M. Sanjabi, and S. Feizi. Text-to-concept (and back) via cross-model alignment, 2023.
[20] T. Oikarinen and T.-W. Weng. CLIP-dissect: Automatic description of neuron representations in deep vision networks. In The Eleventh International Conference on Learning Representations, 2023. URL httpps::/oppereeiieeeeeee
[21] C. Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Yc, and A. Mordvint- sev. The building blocks of interpretabblity. Distill, 2018. doi: 11223915ddisttl.00000
[22] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.- Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, Labatut, A. Joulin, and Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
11

---

[23] S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry. Trak: Attributing model behavior scale. 2023.
[24] G. Pruthi, F, Liu, M. Sundararajan, and S. Kale. Estimating training data influence by track- ing gradient descent. CoRR, abs/2002.08484, 2020. URL http:://aaxiv..or///aaaaaa 08484
[25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[26] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2020.
[27] R. R. Selvaraju, A. Das, Vedantam, M. Cogswell, D. Parikh, and D. Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. CoRR, abs/1610.02391, 2016. httt:httt//aaaeet org/ abaaabas/b/1 02391
[28] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Com- puter Vision 128(2)336-355, Oct 2019. ISSN 1573- 1405. doi: 10110077111223-009--0222277
http:/xxddooo.r... http://t:/dxx..oo doi org 10070 011-1260----9-1-----7777
[29] H. Shah, A. Ilyas, and A. Madry. Decomposing and editing predictions by modeling model computation. In Forty-first International Conference on Machine Learning, 2024. URL
[30] D. Smilkov, N. Thorat, B. Kim, F. B. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, has/1706.0322.. 2017. http:///:/aaxii org/abs//7000 03825
[31] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. CoRR, abs/1703.01365, 2017. URL http::/arxiiy..
http:/aaivv.orgaab///aaaa/a/a//a org//bs////.....b. 01365
[32] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention, 2021.
[33] 7. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li. Maxvit: Multi-axis vision transformer. ECCV, 2022.
[34] M. G. Vilas, T. Schaumlöffel, and G. Roig. Analyzing vision transformers for image classification in class embedding space. In Advances in Neural Information Processing Systems. volume 36, pages 40030-40041, 2023. URL URL httpt:tttt:://tttt /ttp wwwr cccpppppcccppppr..pppppp files/ http:///ww.//...///oo//ooo/aaaa.
pdf
[35] R. Wightman.
Pytorch image models.
httpp:///giihuu..coorrrrgg
pytorch- Ertage-oodels 2019.
[36] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks. Representation engineering: A top-down approach to transparency, 2023. URL htt.attpp//aarrii..gg/a/////////
12

---

A Limitations
Our analysis is limited in several ways which hope to address in future work. Firstly, similar to Gandelsman et al. []] we only consider the direct contributions from the last few layers, and do look at the indirect contributions though other components. Secondly, we limit ourselves to decomposition only over attention heads and tokens, while convolutional blocks are not decomposed even if they might admit one. Furthermore, it is still unclear if we can identify certain directions or vector subspaces in the model component contributions which are strongly associated with a certain property. We believe that detailed analysis of higher order contributions with a more fine-grained decomposition may be key for addressing these challenges.
B Implementation details
Feature instantiations: We use the following features and corresponding feature instantiations. They chosen arbitrarily:
1. color: "blue color", "green color", "red color", "yellow color", "black color", "white color"
2. texture: "rough texture", "smooth texture", "furry texture", "sleck texture", "slimy texture",
"spiky texture", "glossy texture"
3. animal:
"camel", "elephant", "giraffe", "cat", "dog", "zebra", "cheetah"
4. person: "face", "head", "man", "woman", "human", "arms", "legs"
5. location: "sea", "beach", "forest", "desert", "city", "sky", "marsh"
6. patterm: "spotted pattern", "striped pattern", "polka dot pattern", "plain pattern", "checkered
pattern"
7. shape:
"triangular shape", "rectangular shape", "circular shape", "octagon" "linen", "velvet", "cotton", "silk", "chiffon"
8. fabric:
The aligners are trained with learning rate 3 x 10-4, A 1/768 using the Adam optimizer (with default values for everything else) for upto an epoch on ImageNet validation split. Hyperparameters were loosely tuned for the DeiT model using the cosine similarity and then fixed for the rest the models. We may achieve better performance with more rigorous tuning. The number of components & used for the image- based image retrieval experiment was tuned on an image-by-iman basis. is approximately around for larger models like Swin or Max Vit, and around 3 for the rest.
Hyperparameters:
metric,
Computational resources: The bulk of computation is utilized to compute component contributions and train the aligner. Most of the experiments in the paper were conducted on single RTX A5000 GPU, with 32GB CPU memory and compute nodes.
13

---

C
More detailed pseudocode for REPDECOMPOSE
Algorithm 2 REPDECOMPOSE
Input: 2, the final representation output by the model and the final node in the computational graph.
z.ʃ the function that outputs node
Output: A tree consisting of component contributions c, such that components Eest 2. The
structure of is nested list where each list represents level in the tree
function REFDECOOMOSS(2)
is __nnlineaarr..11 then
return [=]
else if is (z. then
Function unary linear
2o <- z.parents()
lg + REPDECOMPOSE(22)
if is_ f then
to.a f- unbind(to)
Unbinds each E along the reduction dimension }> Returns fa such that LeEto, fa(c) z. (z0)
fa - decomp(z..))
return map(fa. t0,w)
b Maps each to fc(c)
else
return map( z.ʃ. )
t (c) Maps each element
else
bo 2. is binary
20, 21 f- z. z.parents()
Get the parents of in the graph (inputs to z.function) fa.0: fá. SEREOMOOOEESDDRRPPEEEOOOOSSSSLL REPDECOMPOSE(Z) binary(2.n
to, 11 + RE#DECOMPOSE(ZO))
Returns fa.0, fa.1 such that: return [map(fa,. to), map(fa1, f1)l b Eesto fao(e) + Eeet, fa,(c) = 2.1 21)
14

---

D Stepwise breakdown of the operation of RepDecompose on a vanilla attention-MLP block
Matrmal
Softmax
Div
LaverNorm
212
Matmaul
Concat 1
Ln mis.
LayerNorm
Linear EW do.))
Liner 100 2n))
Greki
LayerNorm
Figure 6: Illustration simple attention-mly -llp block. Intermediate tensors marked as 2,, non-linear nodes in orange, nodes where a tensor is reduced along a dimension of interest (tokens, attention head, etc) are marked by green borders
To illustrate the workings of our algorithm, we describe the steps that the RepDecompose algorithm takes simple attention-mip block of vanilla VIT transformer. Please refer to Figure 6 for the variable names in the following explanation.
First, mark (with green borders in the figure) the computational nodes in which the contributions of the components get reduced. For the tokenwise contributions, this is the 'matmul' operation, while for the attention head contributions, is the 'concat' operation. We also detach the graph at the input of each component to stop the algorithm from gathering only the direct contributions and not any higher-order contribution terms arising from the interaction between multiple components. Let the RepDecompose function be denoted by d(.) which takes representation and returns an array of contributions. Here, n, wherever appears, is the number of contributions in the decomposition of the input. The map(ʃ, d(=)) operation applies f to every contribution vector in d(=). At each step, ensured that the sum all contribution vectors/tensors the RHS is equal to the vector/tensor that is being decomposed in the LHS via the distributive property for linear transformations. Then:
1. d(=) = map(Ar: (x ) d(z1)) (LayerNorm linearized as in Gandelsman et al [1], n here the number of contributions in d(z1))
2. d(z1) ((==),,(zz))
3. d(22) map(Az.xWi + 51 d(21)) (n here is the number of contributions in d(21)
4. d(21) [zu] (stops when it hits non-linear node)
5. d(za) (d(=s), d(z))
6. d(ze) map(Xr.x:. + Ax d(z))) (n here is the number of contributions in d(27)
7. d(27) [[zrropa(w)) for u] for d(2x)] (Concatenation of tensor along dimen- sion can be expressed as a sum of zero-paddes tensors)
8. d(2x) [lue for (u, v) zip(U.cols, V.rows)] for U E d(2o) for V e d(210)] (via the distributive property for matrix multiplication)
9. d(za) = [zo] (stops when it hits a non-linear node)
d(=11)) (n here is the number of contributions in d(zz1) 10. d(=10) map(An.z W.. b.
d(=11 map(Ar. (), d(212)) (n here is the number of contributions in d(212) ) 12. d(z12) [212] = [2s] (Stopped since the comp graph is detached. not the algorithm would
return higher-order terms.)
15

---

E Proof of Theorem 1
Proof. From the first condition on intra-component rank ordering, for any two vectors u, v and linear map Si, if |||||| I || then [ - [[) We first show that fa is a scalar multiple of an isometry.
If ||ll ||||| + 0, then both |u|| < |||||| and ||||| < |za]l. This implies that [ S [[)
and [[)( - (w)ll Therefore, [[][(u)]] [|. when ||e]]] ||]|]- Given the input space of the transformation as U, we choose a unit vector Munir E U. Let's assume [[ (Xmia)))) )| = c. With the above result,
the following equality
u |uitl obtain the following:
Talll
fu(
fs
Il C,
|
|
Therefore:
[l((w))] = c||u||
(2)
Thus, the linear transformation f. is scalar multiple of an isometry. Now consider two linear
maps fi and fj such that [[f(())] = ci]lu]] and [| = =cyl cy||ul]. From the second condition on inter-component rank ordering, for any two vectors then (x)]l & happen when (u) = cl|u|] for some constant for all vi.
and linear maps fi, fj. if ||al]| I ||w|]]
(l,f(()) (x)]l- This implies that u = v. [[ [)]l = [[f( However, this can only
With this, let's denote ft preserve the inner product between two vectors u and v. First we prove that isometries preserve the inner product, which we will then use to prove the orthogonality of fi Given two vectors u and v, their inner product can be expressed as the following:
isometry. One of the general property of isometries are that they
- v
1
(1l u + v1l2 + 1 w113)
(3)
An isometry by definition preserves the norm of the vectors i.e. (z))l ||//|| and [ || = ||0|] Due to this property, we can express the following relations:
[| v)]l || v/|,
(4)
and
IIf(uu - v)]l = 1|u- v/|,
(5)
We can express fi(u)² ((()) the reduction from 33.
1 su()) ((v) 4 (11((()) (1(1)) + - fi(w))13), 1
(6)
fl4)² f(()) (11fi(u x) (1 -))1
Next we substitute the relations from (4. and Eq. 5 to Eq. 7 to obtain the following inner product preservation property:
fu(u)² f((v)
7 (1lu (1 w/12 u| vll?) uT
(8)
Next we use the inner product preservation property to prove the orthogonality of L as follows:
fi(u) (((v) cr
TT -2
(10)
From 10 we infer the orthogonality of f which leads to the following result:
f - 2I=
(11)
16

---

Scoring function
Algorithm 3 Scoring function for attributing properties to components
Input: Z, the image representation output by the model over n images with dimension d (shape:
d); the contribution of particular component of interest (shape: d); B, the set of k feature vectors that represent a given feature (shape: x d)
Output: A score that signifies the importance of the component for the given feature
function COMPATTRIBUTECC Z, B)
B orthogonalize((3
2z - ZB
sC + CBT
courelation_coefffiientte dim=0)
return mean(r)
17

---

G Text-based Image retrieval
Model name
Color
Texture
Animal
Person
Location
Pattern
Shape
DeiT
0.679
0.563
0.774
0.596
0.818
0.597
0.764
DINO
0.663
0.657
0.781
0.742
0.833
0.680
0,706
DINOv2
0.751
0.614
0.875
0.714
0.857
0.597
0.510
SWIN
0.795
0.720
0.904
0.780
0.912
0.760
0.739
Max Vit
0.872
0.832
0.911
0.828
0.901
0.803
0.797
Table 4: Spearman rank correlation for various common properties
18

---

Image-based Image retrieval
calle
Dalteen
shage
Figure 7: Top-3 images retrieved by the most significant components for various relevant properties for DINO
color
cdler
GANTTIER
pattern
thoce
Figure 8: Top-3 images retrieved by the most significant components for various relevant properties for SWIN
19

---

coter
pahten
Figure 9: Top-3 images retrieved by the most significant components for various relevant properties for DeiT
color
patremm
shappe
Figure 10: Top-3 images retrieved by the most significant components for various relevant properties for Max ViT
20

---

cdeer
coler
patter
shape
Figure 11: Top-3 images retrieved by the most significant components for various relevant properties for DINOv2
coler
shape
Figure 12: Top-3 images retrieved by the most significant components for various relevant properties for CLIP
21

---

I
Property visualization via token decomposition
camel
desert
triangle
layen 00, attn headed
layen31, Jthn head.10
layen91, attn head:00
striped
personn head:
poottem head:
Nocation heed:
Figure 13: Visualization token contributions for CLIP
camel
desert
triangle
layer 30, 617
Iayen attrchead:00
leyen:03, attnchedd:09
beach
'persann ipattarm attruh
lsyen:44, attruhead:44
layen: sttruheddi0
Figure 14: Visualization of token contributions for DINO
22

---

camel
triangle
"animal head layendd,, attry hwedh11, layen 53, atthchoadddd
"ocortoon head laywntts, hwad-122 loyen attrcheed
'shappe head: leyentai, alin heath:00 loyen:4, sttcceedd.03
man
striped pattem
beach
head: layen153, altry heed::0) attrCheadd heeddddd
layent elthy headt layen:3, attChheadl0e alto
leyentb altr heedt:00,
Ryen0S, heeds!t
Figure 15: Visualization of token contributions SWIN
camel
triangle
"animal' head layends, attn head-100 layen 00, attn_heedeeee
"ocation' head layendo, atin head:1l, leyen:d6, attn pradhesd
"shapee head: layends, atin head 13 lbyen htsnheeddd3
man
striped pattem
beach
head:
Nocartinn head: leyenith, alin heedii2) layerrde, hwadiol
loyeni23, allir heedl01, attnchead layer 02, headi10
layerit attry heudi22, attrChead layenr2l,, alli hwadlo
Figure 16: Visualization of token contributions for Max Vit
23

---

J Zero-shot segmentation
Defl
DINO
MaxVit mloU 0.4237
SWIN
Algorithm
mloU
mAP
pixAce
mloU
mAP 0.7237
pixAce 0.7136
mloU 0.4338
mAP 0.7620
Decompose
0.7719
0.5291
0.8305
0.7577
0.4863
0.8111
0.7163
Chefer
0.7870
GradCamm Jam
0.6533
0.4625
0,7129
0.7045
0.4309
0.7481
0.4732
0.1705
0.4243
0,5973
0,2360
0.5365
Table 5: Zero-shot segmentation results for different algorithms and models. Chefer at al 's code does
not support Max VTT and SWIN models.
24

---

K
Zero-shot spurious correlation mitigation
Model name
Waterbird in water
Waterbird in land
Landbird in water
Landbird in land
DeiT
0.985 + 0.971
0.733 + 0.815
0.787 + 0.886
0.991 7 0.980
CLIP
0.920 + 0.814
0.507 + 0.746
0.534 0.744
0.948 - 0.857
DINO
0.985 + 0.944
0.800 > 0.911
0,832 - 0,943
0.982 + 0.956
DINOv2
0.994 + 0.989
0.967 > 0.988
0.971 0.978
1.000 0.997
SWIN
0.989 + 0.989
0.834 + 0.871
0.893 - 0.923
0.994 - 0.994
Max Vit
0.959 + 0.942
0.796 - 0.814
0.777 - 0.832
0.970 - 0.961
Table 6: All group accuracies on the Waterbirds dataset before and after component ablation
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our paper introduces a framework to interpret the internal components of vision transformers via text. This is aptly discussed and presented in the abstract, introduction and the main paper. Our framework is supported by empirical and theoretical findings in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have added separate Limitations section in the appendix of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification:
We have provided a proof for one Theorem in our paper, in the Appendix
section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have added all the experimental details and the corresponding hyper- parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We include the code used in the supplementary
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ mripss.cc/ublliccgui.eee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The data, hyperparameters, optimizer and other relevant experimental detail have been presented in the main paper, appendix, and code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The empirical results are reported using a suitable precision so that variable factors do not have significant effect the numbers.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: These details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp::/neurrpps: httppp///aaaaaaa.cc/ppaaall//att
Answer: [Yes]
Justification: Our paper provides a framework to understand the internal mechanisms of vision transformers. It does not have any ethical issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work focuses on understanding the internals of neural networks, and thus does not have any direct impact on society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not pre-train models or release a datasct in this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the models and datasets have credited and cited properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "GLMSXmhYjH", "reviewer": "Reviewer_5CRC", "summary": "The paper proposes a novel method for evaluating the contribution of individual components in arbitrary vision transformers and mapping these contributions to the CLIP space for text interpretation. To avoid rigid matching of each component, the paper introduces a continuous scoring function for component-feature pairs. The effectiveness of the proposed method is demonstrated through token-wise contribution heatmaps, retrieval tasks, and the spurious correlations of the Waterbirds dataset.", "questions": "Could you explain why ImageNet pretrained models encode useful features redundantly?\n\nHow did you accumulate the contributions from the lower layers into a single contribution $c_{init}$?\n\nWhy does component ordering tend to have a higher correlation than feature ordering, particularly for SWIN and MaxVit, as shown in Table 2?", "limitations": "The authors have adequately addressed the limitations of the method. It would be beneficial to see an analysis of how other components and blocks contribute to the model.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a novel method for interpreting arbitrary vision transformers. It successfully interprets the contribution of each component through text, which could help in understanding the decision-making mechanisms of ViTs.\n\nThe paper presents a comprehensive set of experiments, including text-based image retrieval, image-based image retrieval, token contribution heatmaps, and zero-shot spurious correlation analysis. Additionally, ablation studies are conducted to demonstrate the utility of COMPALIGN and the rationale for focusing on the last few layers of the ViT. These experiments robustly validate the effectiveness of the proposed method.", "weaknesses": "The paper does not include comparisons with any baseline methods. Although previous methods may not extend to arbitrary Vision Transformer models, a comparison with specific models would be beneficial.\n\nThe paper lacks an analysis of failure cases, leaving it unclear how well the proposed method can be applied in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We are encouraged that you found our work novel with comprehensive experiments. We reply to your questions below:\n\n> The paper does not include comparisons with any baseline methods. Although previous methods may not extend to arbitrary Vision Transformer models, a comparison with specific models would be beneficial.\n\n Thank you for your suggestion. We have now added results of our experiments on image segmentation with competitive zeroshot segmentation baselines like GradCam and Chefer et al's method in the global rebuttal. Chefer et al's methods are not equipped to work with SWIN and MaxVit, in which case the cells have been left blank.\n\n> The paper lacks an analysis of failure cases, leaving it unclear how well the proposed method can be applied in real-world scenarios.\n\nSome limitations are discussed in the Limitations section in the appendix. When considering the failure cases for real world applications such as spurious correlation mitigation, the most significant is the case when both the spurious features and core features happen to be represented in the same component(s). In this case, it might be necessary to decompose the component contribution a second time into higher order components. However, as we mention in the Limitation section, we do not investigate this possibility currently.\n\n> Could you explain why ImageNet pretrained models encode useful features redundantly?\n\nWe tentatively offer the following hypothesis. ImageNet pretrained models (and its components) presumably learn many features that are most useful for datasets in-distribution with ImageNet. Since we conduct our experiments on the ImageNet validation split, it is likely that these models have many more features useful for ImageNet classification compared to other models such as CLIP or DINOv2. It seems that for non ImageNet pretrained models, much of these useful features are concentrated in the last layers. Thus, when performing layer ablation, it so happens that the accuracy drops more quickly when compared to ImageNet pretrained models.\n\n> How did you accumulate the contributions from the lower layers into a single contribution ?\n\nReferring to the figure and the explanation in the global rebuttal, we see the operation of RepDecompose on a single attention mlp block of a vanilla transformer. Here, $z_5$ contains the contributions from all the previous layers. RepDecompose appropriately applies the suitable transforms (in this case, only the LayerNorm) and obtains the contribution of $z_5$ to the final representation $z$. In this case, the contributions from the lower layers are automatically accumulated into $z_5$\n\n> Why does component ordering tend to have a higher correlation than feature ordering, particularly for SWIN and MaxVit, as shown in Table 2?\n\nThere are some components whose direct contribution is not useful for encoding any feature at all, although they may have useful indirect effects. When ordering the components according to their ability to retrieve a specific feature, these components are predictably at the bottom. However, for any component, there is no feature which is predictably the most (or least) dominant. Thus, the component ordering problem is inherently a somewhat easier problem, and our proposed scoring function is better at ordering components as compared to features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KFrxx9gAvJ", "reviewer": "Reviewer_oPeN", "summary": "The paper analyzes the direct contributions of individual layers and attention heads in vision models. Based on a similar idea to Gandelsman at al. that was applied to CLIP, this method decomposes the final representations of other models into individual contributions. To interpret these contributions, the paper proposes a method that translates the representations into the CLIP joint space. \nThe interpretations introduce additional application - spurious cues reduction, visualizing token contributions analysis and image-based image retrieval.", "questions": "Are individual heads spatialized mostly on specific tasks (e.g. counting/textures/colors) as shown in Gandelsman et al.?\n\nWhat is the effect of using different CLIP models when doing the COMPALIGN? Is there any trend about how interpretable the models are,  given larger CLIP models?\n\nWhat are the top text descriptions found by TextSpan/Scoring function for each mapped head? The paper will benefit from significantly more qualitative results.", "limitations": "The limitations of the paper are discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is very clear and intuitive for readers who are familiar with TextSpan.\n\nThe presented matching algorithm and scoring algorithms are novel and allow us to interpret the hidden state of new vision models.\n\n The spurious cue reduction method is convincing and shows that the interpretation of the different components is grounded.\n\n There is a convincing qualitative and quantitative ablation study for COMPALIGN and the need for lambda. \n\nI believe this paper presents a new opportunity for research into the mechanistic interpretability of non-CLIP models. I will recommend its acceptance if my concerns will be addressed.", "weaknesses": "A brief explanation of the TextSpan algorithm can be useful to understand the paper without the need for reading Gandelsman et al.\n\nIt is not clear that the presented scoring function is more useful than using TextSpan - a qualitative and quantitative comparison between the two approaches can provide a more convincing case for the introduction of this approach.\n\nWhile the presented approach uses CLIP, it doesn't use the main advantage of this model - the fact that full complex sentences can be encoded to the same joint image-text space. The current experiments use very limited sets of descriptions for scoring and interpreting the individual components. The paper will greatly benefit from more qualitative and quantitative results that make use of large-scale text sets that include image descriptions (just like the ones presented in Gandelsman et al.). As the scoring algorithm can use these datasets instead of the small set of concepts, more text descriptions might introduce a new understanding of the different components. \n\nLow match rate - it's not clear what is the reason behind the low match rate. One possibility is that CLIP is not expressive enough. It will be useful to show how does the match rate behaves for different CLIP models.\n\nIt is not clear why the representations should be mapped to CLIP image features first, as opposed to directly mapping them to a sparse linear combination of words in CLIP (i.e. via sparse autoencoder). An comparison to this approach can be very useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your favorable review. We are glad that you found our work novel, clear and convincing. We address your questions below:\n\n> A brief explanation of the TextSpan algorithm can be useful to understand the paper without the need for reading Gandelsman et al.\n\nWe agree and apologize for the oversight. We will add this in the appendix in the final version of the paper.\n\n> It is not clear that the presented scoring function is more useful than using TextSpan - a qualitative and quantitative comparison between the two approaches can provide a more convincing case for the introduction of this approach.\n\nThank you for this point. We do not claim that our scoring function is superior to TextSpan - the motivation and use cases of the two are somewhat different. TextSpan is intended to automatically discover the roles of various components by mining text descriptions from a text bank. The text descriptions thus obtained are sometimes coherent and point to a well-defined role for the component, but at other times all over the place, making the attribution of a role to the component difficult (see the appendix of Gandelsman et al). Thus, TextSpan is a more unsupervised method which tries to understand the model's components on its own terms.\n\nOur scoring function, on the other hand, is useful for identifying components which have a role in representing some externally specified property. Since the property is specified manually, it is unlikely that there will be a specific component which corresponds to this property. This is the reason we may need to select multiple components corresponding to a particular property. Thus, we need a way to sort and select components which encode a given property, which is where our scoring function comes into picture.\n\nLeaving the motivation aside, TextSpan and our scoring function are closely related in a technical sense. Both use the variation of the component contributions with respect to a subspace determined by the text encoder. We also use TextSpan to validate CompAlign's mapping functions in Table 1.\n\n> While the presented approach uses CLIP, it doesn't use the main advantage of this model ... As the scoring algorithm can use these datasets instead of the small set of concepts, more text descriptions might introduce a new understanding of the different components.\n\nThis is an interesting idea. We do not know if ImageNet trained/ self supervised models can learn to encode complex object relations (as present in \"the dog chases after the cat\") in the absence of text supervision. In fact, even CLIP models suffer from limited understanding of compositionality and are prone to treat the text prompt as a bag of words. However, we would have to defer this for future work owing to lack of space.\n\n> Low match rate - it's not clear what is the reason behind the low match rate. One possibility is that CLIP is not expressive enough. It will be useful to show how does the match rate behaves for different CLIP models.\n\nThere may be multiple reasons for this:\n1. The match rate shown in the table is the exact match rate, and the approximate matches are not included in the metric.\n2. CLIP text embeddings of the ImageNet classes are significantly different from the embeddings from the ImageNet trained fully conncected layer. \n\nNote that even with the low match rate, we are able to determine components which perform a specific role well enough to do zero shot spurious correlation mitigation and image-based image retrieval.\n\nWe tried two CLIP models, ViT-B-16 and ViT-L-14, as the source of $z_{\\text{CLIP}}$ for the experiments and we found that the bigger model (ViT-L-14) yielded better cosine similarity and match rate. We would need to conduct more experiments before we can definitively conclude this.\n\n> It is not clear why the representations should be mapped to CLIP image features first, as opposed to directly mapping them to a sparse linear combination of words in CLIP (i.e. via sparse autoencoder). A comparison to this approach can be very useful.\n\nThe primary reason for this is ease of training, as in this manner, our method can be used on any image dataset without accompanying text descriptions. It also uses a simple cosine distance loss rather than a more complicated loss like contrastive loss. Directly mapping the $c_i$ to the text encoder space also has issues due to the well-known \"modality gap\" between CLIP image and text representations.\n\nIn some experiments that didn't make it into the paper, we trained the maps $f_i$ using a contrastive loss on the MS-COCO dataset. The loss decreased slowly, and the resulting $f_i$ were unsatisfactory in terms of match rate and cosine similarity. One potential reason for this is that the supervision offered by the contrastive loss over text descriptions is much weaker than the cosine similarity with CLIP image features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5RKgJsp4fm", "reviewer": "Reviewer_N9ji", "summary": "This paper proposed a novel representation decomposition method for general ViT models. Then, with aligning the component representations to CLIP space, the decomposed contribution vectors can be interpreted through text using CLIP text encoder. Moreover, a scoring function is also proposed to assign an importance score to each component-feature pair. Multiple different types of ViT are analyzed.", "questions": "1. Could the flowchart of REPDECOMPOSE and COMPALIGN also illustrated by a more intuitive figure? The algorithm in Sec3.1 and Sec4 and the corresponding descriptions are not very clear. \n2. How to explain that with some models, the top-3 images retrieved by the most significant components for a specific property are unrelated to that property? For example, in Figure 10, with DINOv2 and “color” as the property, the retrieved images are just similar to the object in the target image, while not including similar color things.", "limitations": "Results are mostly exhibited by limited visualizations and a lack of strong evaluation metrics to support the effectiveness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This work proposes a general method to analyze various ViT models by decomposing the final representation into interpretable components. \n2. Multiple applications are conducted and also demonstrate that the decomposition and interpretation are effective, such as image retrieval and visualizing token contributions", "weaknesses": "1. Lack of qualitative or quantitative comparison with the related works, such as the previous image representation decomposition method mentioned in Sec.3.\n2. As for the applications of the decomposed components, there is a lack of a quantitative evaluation of the text or image-based image retrieval performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the favorable review. We answer your questions below:\n\n> Lack of qualitative or quantitative comparison with the related works, such as the previous image representation decomposition method mentioned in Sec.3.\n\nWe would like to point out that our work can be viewed as an extension or generalization of the work from Gandelsman et al, which was specific to CLIP models. Our method, on the other hand, works on models with various architectures trained with different methods. \n\nWe now include a comparison of our method with competitive baselines like Gradcam and Chefer et al's method in the global rebuttal. Note that some of the baselines are not equipped to work on architectures like SWIN or MaxVit, in which case we leave it blank.\n\n> lack of a quantitative evaluation of the text or image-based image retrieval performance.\n\nWe have provided a quantitative evaluation of the text based image retrieval performance in Table 2 and 4 in the paper. In the paper, this is meant as a validation of our overall method as we compare the ability of various components to retrieve images wrt a certain property as measured by the CLIP score ranking.\n\n> Could the flowchart of REPDECOMPOSE and COMPALIGN also illustrated by a more intuitive figure? The algorithm in Sec3.1 and Sec4 and the corresponding descriptions are not very clear.\n\nWe have added a step-by-step explanation of the execution of RepDecompose on one attention-mlp block in the global rebuttal, and we will add this in the camera-ready version should the paper be accepted. We will also revisit the CompAlign section and elaborate the algorithm further. In essence, the purpose of CompAlign is to map (or \"align\") the contributions to the shared CLIP space via an approximately orthogonal map. \n\n> How to explain that with some models, the top-3 images retrieved by the most significant components for a specific property are unrelated to that property? \n\nIt is possible that in some cases, we do not find any component which predominantly encode the property of interest. In these cases, retrieving images based on these components may also return images which are similar in dimensions other than the specified property. \n\nWe hope that your concerns with the paper are addressed with this, if there are still some remaining issue please do let us know. We request you to consider increasing the score if you are satisfied with our answers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NdhdFuKjtD", "reviewer": "Reviewer_FTxd", "summary": "This paper introduces a framework for identifying the roles of various components in Vision Transformers (ViTs), extending beyond CLIP models. The framework consists of two key components: RepDecompose, which automates the decomposition of representations into contributions, and CompAlign, which maps these contributions into text space for intuitive interpretation. Additionally, the authors propose a novel scoring function to rank component importance. The framework's efficacy is demonstrated through applications in vision-language retrieval and mitigating spurious correlations in a zero-shot manner.", "questions": "All following questions are related to weakness points. \n1. Given the assumptions presented in Section 3.1, how generalizable is the RepDecompose algorithm to ViT architectures that may not strictly adhere to these assumptions? \n2. While image retrieval and token visualization provide insights into model behavior, have you considered more advanced downstream tasks that could demonstrate the practical benefits of your framework?\n3. Have you conducted experiments to show how manipulating or removing specific components based on your RepDecompose and CompAlign observations affects model performance on specific tasks?\n4. How might your framework be extended to not only understand but also guide the improvement of ViT models? Are there plans for future work in this direction?", "limitations": "The discussion on limitations about the paper is relatively thorough. The points raised in the weaknesses section further articulate limitations that should be considered. There is NO discussion on societal impact.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "[**Writing**] The paper exhibits exceptional clarity and coherence in presenting its concepts. The accompanying figures serve as effective visual aids, significantly enhancing the comprehensibility of the discussed ideas.\n\n[**Methodology**] The proposed method offers researchers a valuable tool for conducting in-depth analyses of vision transformers, providing clearer insights into their internal mechanisms. This unified analysis framework represents a significant step forward in enhancing the interpretability of complex, black-box models.\n\n[**Empirical Evaluation**] The inclusion of comprehensive ablation experiments is a notable strength of this paper. These studies collectively offer a clear demonstration of the proposed algorithm's capabilities in interpreting the roles of different components within ViT models, providing crucial insights into model behavior and design choices.", "weaknesses": "[**Ambiguity in Introduction**] The description of insights from previous work [10] in line 22 lacks clarity. The statement \"a sum of vectors over layers, attention heads, and tokens, along with contributions from MLPs and the CLS token\" is ambiguous, as these components might have different dimensions and cannot be simply added. This lack of clarity compromises the self-contained nature of the manuscript, necessitating readers to refer to external sources for full comprehension.\n\n[**Generalizability**] While the authors present RepDecompose as a general algorithm for ViT decomposition and analysis, the additional assumptions outlined in Section 3.1 raise questions about its broader applicability. For instance, (1) The concept of \"direct\" contributions implies that the model must incorporate residual connections for the algorithm to consider earlier layers, (2) The linearity assumption restricts the analysis to linear components only, and (3) The reduction of c_{i,t} to c_{i} through direct summation relies on a strong underlying assumption as well. These constraints may limit the algorithm's applicability across diverse ViT architectures\n\n[**Limited Practical Applications**] While the paper makes strides in model understanding, it falls short in demonstrating practical applications of these insights. The current experiments, such as image retrieval and token visualization, primarily serve to validate the understanding process and can be considered preliminary downstream tasks. The manuscript would benefit from ablation studies that demonstrate how manipulating or removing specific components based on RepDecompose and CompAlign observations can enhance or reduce certain model capabilities. Such experiments would more convincingly illustrate the practical utility of the proposed framework in model improvement and optimization.\n\n[**Structural and Writing Issues**]\n* Figure 1 is never referenced in the main text.\n* Line 14: Missing space between \"features.\" and “These”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your favorable review. We are happy to hear that you found our work clear and coherent, with comprehensive evaluation. We answer your questions below:\n\n> Ambiguity in Introduction\n\nThank you for the feedback. We will take care to expand and rephrase the introduction to remove this ambiguity. We have also added a step-by-step explanation of the execution of RepDecompose on one attention-mlp block in the global rebuttal. The reason why we can add multiple contributions together is because RepDecompose automatically applies the transformations to the output of each component ($z_i$ in the figure) so that the contributions $c_i$ are in the same vector space as $z$. \n\nPlease let us know if there are any other points in the introduction that were unclear or ambiguous.\n\n> Generalizability\n\nIt is true that for our algorithm to work, all non-linear components must be short-circuited via a residual/skip connection in the model architecture. Fortunately, this is almost always the case in modern neural networks, even in modern CNN architectures like ConvNext. This is because the skip connections are added to ensure that any \"dead neurons\" in the non linearities does not affect gradient flow. This means that it is now a safe assumption that non-linearities can be \"skipped\" using residual connections. It is also true that only the linear parts of the model can be decomposed, but modern ViTs are overwhelmingly linear in nature with very few non-linearities in practice. Therefore, we expect our method to generalize well across multiple modern architectures.\n\n> The reduction of c_{i,t} to c_{i} through direct summation relies on a strong underlying assumption \n\nThe \"underlying assumption\" being referred to here is not very clear to us. Since the attention is linear in the OV circuit, it can be further decomposed as a sum over contributions from tokens. This holds as long as there are no non linearities in the OV circuit. Please do clarify further if we misunderstood you.\n\n> Limited Practical Applications\n\nThis is a relevant point, thank you for bring this up. In our paper, we demonstrate a practical application for mitigating spurious correlations by ablating the output of a specific set of attention heads, identified using our method. You can refer to Section 6.4, where we show improvements in both worst-case and average group accuracy (see Table 3) for the Waterbirds dataset. This result illustrates that manipulating or removing specific components, as guided by our framework, can enhance model robustness. We have also added results on zero-shot segmentation comparing it with other competitive baselines such as GradCAM and Chefer et al's method. Our method outperforms the baselines whenever they are applicable for a given model.\n\nOur interpretability framework is primarily validated through downstream applications in text-based and image-based image retrieval, as demonstrated in Sections 6.1 and 6.2. Our findings however suggest that practical tasks like image-conditioned image retrieval based on specific features (e.g., patterns or locations) can potentially be performed in a zero-shot manner using any vision encoder, rather than requiring a specialized model. While achieving state-of-the-art results in this task is not the primary focus of our work, we believe our results lay the groundwork for unlocking various capabilities in vision encoders, all without the need for additional training.\n\n> Structural and Writing Issues\n\nWe thank the reviewer for pointing out these issues. We will correct them in the final version of the paper.\n\n> Have you conducted experiments to show how manipulating or removing specific components based on your RepDecompose and CompAlign observations affects model performance on specific tasks?\n\nOur experiments on mitigating spurious correlations show that removing specific components helps in mitigating spurious correlations. We expect that our framework can also be applied to do model unlearning in an efficient manner by finetuning or removing specific components, however we leave this for future work.\n\n> While image retrieval and token visualization provide insights into model behavior, have you considered more advanced downstream tasks that could demonstrate the practical benefits of your framework?\n\n> How might your framework be extended to not only understand but also guide the improvement of ViT models? Are there plans for future work in this direction?\n\nThank you for raising this important question. Beyond model understanding, in our paper we demonstrated the potential applications of our framework in spurious correlation mitigation, image/text-based image retrieval, and zero-shot segmentation. We believe our framework can also be applied to address issues related to harmful biases or copyrighted image information within the network. By using probe text descriptions, it is possible to identify model components that encode such biases or infringement information, which can then be ablated to develop safer vision models in a post-hoc way. However, we note that these applications, while impactful, are beyond the scope of the current paper and represent promising directions for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbc1PYPhFQ", "reviewer": "Reviewer_HytW", "summary": "The authors extend the idea proposed in [10] of decomposing CLIP’s image representation and interpreting the decomposed components via text, to models other than CLIP via learning a set of mapping functions, one for each decomposed component, to map the representation of a model to be analysed (e.g., DINO) to the CLIP space, such that they can perform text-retrieval on the decomposed component with the CLIP text encoder, and assign the component a textual label. They further automate the decomposition process by making use of the computation graph of the model, proposing an algorithm called RepDecompose to achieve this. This allows for a more flexible decomposition for transformer models, especially models which are not based on the plain ViT architecture. They also score the decomposed components according to their relevance with a text such that they can select several components for a given textual feature. Applications such as image-text retrieval, image-image retrieval, token importance visualization, and mitigating spurious correlations on Waterbirds dataset are shown.", "questions": "If the authors have a different opinion on W1 and W2, I would like to hear it, supported by experimental evidence.  \n\nI understand that rebuttal time is limited, so I encourage the authors to address as much as possible from W3-W4-W5. \n\nThe minor weaknesses are comments which do not affect my decision.", "limitations": "Limitations are addressed in the supplementary material. Considering that interpretability is very important for several critical applications, weakness W1 (if not addressed) would mislead users into incorrect interpretations.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The authors are tackling a very important direction of interpreting models other than CLIP, with text. Text-based interpretation has lately (to some big extent) been limited to CLIP models only, due to their strong image-text alignment which allows for human-friendly textual interpretation. However, computer vision is absolutely not just about CLIP, and there are an abundance of different models that are not trained for alignment, and that are used everywhere: in the same and similar research areas, in other research areas, and in industry. Therefore, we are presented with a serious bottleneck in interpreting models which are not trained for alignment, via text. I myself as a person working in this field, have previously questioned the direction this paper is proposing. Therefore, I am happy that the authors took this direction, and I believe it will have a large impact on the community. \n\n- Figure 2 in a nice analysis, and the outcomes are interesting. It is compelling to see the mean-ablation experiments across different models trained with different datasets and supervision. \n\n- RepDecompose algorithm allows us to apply decomposition to a Transformer regardless of the type of attention module design (assuming RepDecompose is valid – more on this in the weaknesses).", "weaknesses": "I would like to congratulate the authors for their work. Unfortunately, there are flaws that should be addressed. I will take the time to explain those in detail:\n\nMajor Weaknesses: \n\n[W1] In my opinion, the authors are just interpreting CLIP features rather than the other models such as DeIT, DINO…etc. The problem with learning to map features from any model such as DINO, to the CLIP space (e.g., via a linear layer here) essentially means interpreting what the CLIP vision encoder learns. I will take some time to explain this: First let’s simplify the problem: assume we have a ViT model (e.g., DINO model) we want to interpret, and the CLIP vision encoder. Given an image, we want to train a one-to-one mapping from the features of the last layer of DINO (v_dino) to the features of the final projected global representation of CLIP (v_clip), which serves as the ground-truth for the mapper. If we assume a perfectly learned mapping function which maps v_dino to v_clip, then what we get out of the mapper is essentially v_clip. This is equivalent to just running the image through the CLIP vision encoder and obtaining v_clip (the ground-truth) directly. This means that what is interpreted after the mapping is v_clip rather than v_dino, and conveys what CLIP learns rather than what DINO learns. This simplification extends to mapping a component from any model (e.g., DINO), denoted as c_i, to v_clip. The authors put no restriction on preserving the semantics of c_i. There are no experiments to proof that the mapped representation still encodes what the original model (e.g., DeIT, DINO) learns. Since this is the topic of the paper, the authors should have addressed this before proceeding with any other experiment. Moreover, the regularization component in Theorem 1 has no effect in preserving the component role, as it decreases the cosine score by only 0.01 according to Table 1, which means it has a negligible effect (here I assume decrease is better because we should assume that the transformed representation is in the CLIP space, but should also be different from CLIP features for the input image). Therefore, these linear mappers could learn anything such that the addition of their output becomes the original CLIP representation v_clip, and the original semantics of c_i is lost. In summary, there is no restriction to keep the original semantics of c_i when mapping it to CLIP space, which means the interpretation is for CLIP’s representation rather than the models beyond CLIP. This deviates away from the title of the paper. \n\n[W2] The authors did not show the validity of RepDecompose algorithm. The authors should first obtain the representation of the components for some plain ViT model by applying the simple decomposition as in [10] which is simple to apply in case of a plain ViT. Then they should run RepDecompose on that same plain ViT, and show that indeed, the representations are the same and equivalent. The authors have tried to partially address this via comparing the match rate with that from [10], as shown in Table 1. It is shown that the match rate is extremely low (0.18), which disproves the validity of the algorithm. \n\nModerate Weaknesses: \n\n[W3] In Figure 3, the authors show that as the component score decreases, the retrieved images grow less relevant. This should not be validated qualitatively only. The authors should validate this quantitatively, for example by reporting the Area-Under-Curve, where the x-axis represents the number of components being removed, and the y-axis representing the similarity score between the image and text. This should also be compared across the different models analysed, and also compared with baseline CLIP: Can the global image representation of the baseline CLIP (denoted as Z_CLIP by the authors) still do a good job in this? Or do we really need its components to do this task? Again, this should be shown quantitatively. Finally, the authors showed an experiment in Table 2 where they report the correlation coefficient between (Z_CLIP, text) and (sum of most highly-ranked components, text), but the problem here is that if the authors consider (Z_CLIP, text) as the ground-truth, this means that they assume Z_CLIP can perform this retrieval task. In that case, what is the point of retrieval using the component outputs of the model as the vision source? If I understand correctly, the authors would like to perform property-based retrieval to validate the effectiveness of the components and the features they are responsible for encoding. But setting the ground-truth as Z_CLIP does not validate this, because if we reach the ground-truth score, it means we can perfectly do the retrieval via Z_CLIP.  \n\n[W4] There are no quantitative experiments on visualizing token contributions. For example, the work of [10] conducts zero-shot segmentation experiments and compares with other methods, but the authors do not perform such analysis. How do the different models analysed perform on zero-shot segmentation? How do they compare with existing methods in the literature and to [10]? \n\n[W5] The pseudo-algorithm in Page 4 is not clear and there are a lot of unexplained steps. Moreover, the authors mention they stop the recursion and traversal when they “hit a non-linear node”. The MLP module in the transformer involves running each token individually through the MLP, and therefore there is no linear operation in the MLP. such as summation. How do you traverse through this? This causes the algorithm to stop, and the parent nodes (here the attention heads) cannot be obtained since the traversal stops. What is meant by a binary z.f? In general, the algorithm should be understood to the point that readers are able implement it on their own (general idea is not enough – and listing the algorithm without explaining it is also not enough). I would recommend the authors to enhance the understanding via a diagram example for one attention-mlp block of the transformer, and explain clearly each step written in the algorithm. That is a very important part I was expecting to see in the supplementary material. Readability and a clear understanding is a vital part of any research work. \n\nMinor Weaknesses:\n\n- I do not see the scoring function as a contribution. In essence the scoring function is the dot product between the mapped representation f(c_i) and the encoded text concept (e.g., stripes), which comes for free when finding the component's matched text concept (since the text concept assigned to a component is the one with the highest similarity score).\n\n- Line 93: The work of [10] clearly shows that the MLPs have no effect and are mean-ablated, and thus it should be mentioned that the MLP term is ablated (replaced by the mean value).  \n\n- Texts in Table 1 should be differed to other sections or to the supplementary material. These are qualitative examples rather than quantitative…. It is not a good practice to mix the two in one table. \n\n\nIn summary, the paper requires significant changes and additional experiments to be ready for NeurIPS. So my decision will be, sadly enough, to reject this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments on our paper, we appreciate the feedback. We are glad that you find the direction of our work important and interesting. However, we disagree with your characterization of our work on multiple points. We answer your concerns below:\n\n> [W1]  In my opinion, the authors are just interpreting CLIP features ... \n\nWe disagree with this comment. It is true that by mapping to CLIP space and interpreting using the CLIP text encoder, we are \"constrained\" to using concepts that CLIP has learnt well. This does not mean, however, that this method of interpretation is ungrounded. Our results from the experiments on image-based image retrieval and zero-shot spurious correlation are evidence against this. In both these experiments, the mapping functions $f_i$ are only used for selecting the components to be used or ablated based on some specific property, and not used in performing the task itself (like image retrieval and inference on data with spurious correlations). If it were true that the property encoded in $f_i(c_i)$ is different from $c_i$, we would not observe property relevant images being retrieved or any significant improvement in worst group accuracy. \n\n> The authors put no restriction on preserving the semantics of c_i. ..\n\nWe discuss this in Section 4 Theorem 1 and the following paragraph. We show that an orthogonal map preserves the relative importance of features, since it is an isometry. We encourage $f_i$ to be orthogonal during training.\n\n>  the regularization component in Theorem 1 has no effect in preserving the component role ....\n\nThe cosine distance (the first term in the loss function) measures how well  $\\sum f_i(c_i)$ match z_CLIP. The orthogonality regularizer (the second term) encourages the mapping to be an isometry so as to not distort the relative features (like you mentioned). Since these are two distinct objectives, the regularization component is not expected to have any effect on the cosine similarity - in fact it is a bit surprising that the cosine distance marginally decreases. The relevant metric for faithfulness of the mapping is the match rate which measures how well the TextSpan descriptions of a component (using the ImageNet trained linear head as the \"text bank\") match the corresponding TextSpan description of the *same* component after CLIP mapping and using the CLIP text embeddings of ImageNet classes as text bank. We observe a significant improvement in the match rate when the regularizer is introduced (in Table 1).\n\nHowever, you indeed have a point that the improvements are not very large. This is because any randomly sampled high dimensional matrix is approximately orthogonal already, as for any two independent zero-mean Gaussian/uniform random variables $x, y$,  $E[xy] = 0$ . In our experiments, training this matrix via gradient descent does not significantly change this property. However, since the orthogonality property is crucial to preserve the faithfulness of the mapping, we explicitly add it in the loss term. With this, the norm $\\|| f_i f_i^T - I \\||_F$ decreases by around 50% and match rate increases by around 0.03.\n\n> [W2] The authors did not show the validity of RepDecompose algorithm. ...  It is shown that the match rate is extremely low (0.18), which disproves the validity of the algorithm.\n\n> [W5]  The pseudo-algorithm in Page 4 is not clear and there are a lot of unexplained steps.\n\nWe address W2 and W5 together. RepDecompose is a generalization of the manual process by which Gandelsman et al computed component contributions. We have added a step-by-step explanation of the execution of RepDecompose on an attention-mlp block in the global rebuttal. This is still a simplified graph for illustrative purposes, the actual computational graph has many more nodes and is somewhat more complicated. We provide the code we use in the supplementary, the RepDecompose algorithm can be found in the file `helpers/linear_decompose.py`. \n\nThe match rate in Table 1 (as mentioned previously) is only meant to validate the faithfulness of the mapping functions $f_i$, and it is intended as a metric for CompAlign. It does not validate the RepDecompose algorithm in itself. The contributions returned by RepDecompose match those computed by Gandelsman et al exactly in the case of vanilla ViTs.\n\nWe acknowledge that the pseudocode in Algorithm 1 is unclear. While the intuition is very simple ( traverse through linear nodes by applying the distributive property and stop at non-linear nodes), the actual implementation is lengthy and requires extensive handling of edge cases which might have obscured the intuition. We will present the pseudocode at a higher level of abstraction in the camera ready.\n\n> [W3]\n\nOur experiments on text-based image retrieval using components is only meant to show that (a) identifying certain image properties can be localised to certain individual components, and (b) our scoring function provides a useful signal for identifying these components. We do not claim that the individual components can retrieve images better than the overall representation or the CLIP representation.  We show this quantitatively in Table 2 and 4. Please let us know If there are any concerns with the experimental setup for this section.\n\n> [W4]  ... quantitative experiments on visualizing token contributions ... \n\nThank you for your suggestion. We have now added results of our experiments on image segmentation with competitive zero-shot segmentation baselines like GradCam and Chefer et al's method in the global rebuttal. Some of the methods are not equipped to work with certain models, in which case the cells have been left blank. \n\nWe hope that we were able to sufficiently address your concerns in the rebuttal. Please let us know about any remaining issues or concerns, and do consider increasing your score if you are satisfied with our answers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their extensive and insightful comments. We are encouraged that the reviewers found our work novel, clear, and impactful. We address some common concerns in this global rebuttal:

## How does RepDecompose work?

To illustrate the workings of our algorithm, we describe the steps that the RepDecompose algorithm takes on a simple attention-mlp block of a vanilla ViT transformer. We include a figure of a simplified computational graph in the pdf, please refer to it for the variable names in the following steps.

1. First, we mark (with green borders in the figure in the pdf) the computational nodes in which the contributions of the components get reduced. For the tokenwise contributions, this is the 'matmul' operation, while for the attention head contributions, it is the 'concat' operation. We also detach the graph at the input of each component to stop the algorithm from gathering only the direct contributions and not any higher-order contribution terms arising from the interaction between multiple components.

2. Let the RepDecompose function be denoted by $d(.)$ which takes in a representation and returns an array of contributions. Here, $n$, wherever it appears, is the number of contributions in the decomposition of the input. The $\text{map}(f, d(z))$ operation applies $f$ to every contribution vector in $d(z)$. At each step, it is ensured that the sum of all contribution vectors/tensors in the RHS is equal to the vector/tensor that is being decomposed in the LHS via the distributive property for linear transformations. Then:

   a. $d(z) =\text{map}( \lambda x. \frac{1}{\sigma}(x - \frac{\mu}{n}), d(z_1)) $ (LayerNorm linearized as in Gandelsman et al [1],  $n$ here is the  number of contributions in $d(z_1)$)

   b. $d(z_1) = (d(z_2), d(z_3))$

   c. $d(z_2) = \text{map}(\lambda x. xW_1 + \frac{b_1}{n}, d(z_4))$ ($n$ here is the  number of contributions in $d(z_4)$ )

   d. $d(z_4) = [z_4]$ (stops when it hits a non-linear node)

   e. $d(z_3) = (d(z_5), d(z_6))$

   f. $d(z_6) = \text{map}(\lambda x. xW_o + \frac{b_o}{n}, d(z_7))$ ($n$ here is the  number of contributions in $d(z_7)$ )

   g. $d(z_7) = [[\text{zeropad}(v) \text{ for } v \in u] \text{ for } u \in d(z_8)]$ (Concatenation of a tensor along a dimension can be expressed as a sum of zero-padded tensors)

   h. $d(z_8) = [ [uv \text{ for } (u, v) \in \text{zip}(U.\text{cols}, V.\text{rows}) ]  \text{ for } U \in  d(z_9) \text{ for }  V \in d(z_{10}) ]$ (via the distributive property for matrix multiplication)

   i. $d(z_9) = [z_9]$  (stops when it hits a non-linear node)

   j. $d(z_{10}) = \text{map}(\lambda x. xW_v + \frac{b_v}{n}, d(z_{11}))$ ($n$ here is the  number of contributions in $d(z_{11})$ )

   k. $d(z_{11}) = \text{map}( \lambda x. \frac{1}{\sigma}(x - \frac{\mu}{n}), d(z_{12})) $  ($n$ here is the  number of contributions in $d(z_{12})$ )

   l. $d(z_{12}) = [z_{12}] = [z_5]$ (Stopped since the comp graph is detached, if not the algorithm would return higher-order terms.)

Thus, the final decomposition contains contributions from the MLP, contributions from each attention head from each token via the OV circuit, and $z_5$, all appropriately transformed by the residual transformations like LayerNorm. Note that this is the same process by which the contributions were derived in Gandelsman et al. [1]

## What about real-world applications (like zero-shot segmentation)?

We also present the results of zero-shot segmentation on the Pixel-ImageNet dataset [4], which contains segmentation masks for ImageNet. We compare our method with the two other best performing zeroshot segmentation methods in Gandelsman et al. [1], that is Chefer at al [2] and Gradcam [3]. The implementation of Gradcam is from the `pytorch-grad-cam` library [5]. Our method outperforms the other two methods on all metrics. Note that since both SWIN and MaxVit operate on 32 x 32 patches, their segmentation metrics are worse than the DeiT and DINO models which operate on 16 x 16 patches.

### References:

1. Y. Gandelsman, A. A. Efros, and J. Steinhardt. Interpreting CLIP’s image representation via text- based decomposition. In The Twelfth International Conference on Learning Representations, 2024.

2. Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

3. R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh and D. Batra, "Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization," 2017 IEEE International Conference on Computer Vision (ICCV)

4. S. Zhang, J. H. Liew, Y. Wei, S. Wei and Y. Zhao, "Interactive Object Segmentation With Inside-Outside Guidance," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (https://github.com/shiyinzhang/Pixel-ImageNet)

5. Jacob Gildenblat and contributors, "PyTorch library for CAM methods" (https://github.com/jacobgil/pytorch-grad-cam)

Meta-review: This work introduces a general framework to identify and interpret the roles of various components in vision transformers beyond CLIP by decomposing final representations and mapping them to CLIP space for textual interpretation. The framework also includes a novel scoring function to rank component importance, offering insights that enhance tasks like image retrieval, visualizing token importance, and mitigating spurious correlations.

Summary Of Reasons To Publish:

1) Novel method for interpreting arbitrary vision transformers.

2) Extensive and solid experiments validating the effectiveness of the proposed method.

Summary Of Suggested Revisions:

All the major concerns have been addressed by the authors in their rebuttal. I recommend incorporating some of the key clarifications made in the rebuttal into the final version.
Decision: Accept (poster)