Interpreting CLIP's Image Representation via Text-Based Decomposition
Authors: Yossi Gandelsman
Alexei A Efros
Jacob Steinhardt
Venue: ICLR 2024 oral (2024)
Keywords: CLIP
interpretability
explainability
Primary area: visualization or interpretation of learned representations
TL;DR: We investigate the CLIP image encoder by analyzing how individual model components affect the final representation

Abstract: We investigate the CLIP image encoder by analyzing how individual model components affect the final representation. We decompose the image representation as a sum across individual image patches, model layers, and attention heads, and use CLIP's text representation to interpret the summands. Interpreting the attention heads, we characterize each head's role by automatically finding text representations that span its output space, which reveals property-specific roles for many heads (e.g. location or shape). Next, interpreting the image patches, we uncover an emergent spatial localization within CLIP. Finally, we use this understanding to remove spurious features from CLIP and to create a strong zero-shot image segmenter. Our results indicate that scalable understanding of transformer models is attainable and can be used to repair and improve models.

Published as a conference paper at ICLR 2024
INTERPRETING CLIP'S IMAGE REPRESENTATION VIA TEXT-BASEE DECOMPOSITION
Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt
UC Berkeley
(yossi gandel sman, aae caefros, j edd
edu
ABSTRACT
We investigate the CLIP image encoder by analyzing how individual model com- ponents affect the final representation. We decompose the image representation as a sum across individual image patches, model layers, and attention heads, and use CLIP's text representation to interpret the summands. Interpreting the attention heads, we characterize each head's role by automatically finding text represen- tations that span its output space, which reveals property-specific roles for many heads (e.g. location or shape). Next, interpreting the image patches, we uncover emergent spatial localization within CLIP. Finally, remove spurious features from CLIP and to create a strong zero-shot image seg- menter. Our results indicate that scalable understanding of transformer models is attainable and can be used to repair and improve models.!
this understanding
INTRODUCTION
Recently, Radford al. (2021) introduced CLIP, class of neural networks that produce image representations from natural language supervision. As language is more expressive than previously used supervision signals (e.g. object categories) and CLIP is trained on lot more data, its rep- resentations have proved useful on downstream tasks including classification (Zhou et al., 2022), segmentation (Lüddecke & Ecker, 2022), and generation (Rombach et al., 2022). However, we have limited understanding of what information is actually encoded in these representations.
To better understand CLIP, we design methods to study its internal structure, focusing on CLIP- VIT (Dosovitskiy et al., 2021). Our methods leverage several aspects of CLIP-ViT's architecture: First, the architecture uses residual connections, so the output sum of individual layer outputs. Moreover, it uses attention, so the output is also sum across individual locations in the image. Finally, the representation lives in a joint vision- lisiunnlanguage space, so we can label its directions with text. We use these properties decompose the representation into text-cxplainable directions that are attributed to specific attention heads and image locations.
As a preliminary step, direct effect on the output. We find that ablating all layers but the last attention layers has only small effect CLIP's zero-shot classification accuracy (Section 3). We conclude that the CLIP image representation is primarily constructed by these late attention layers.
the residual structure to investigate which layers have significant
We next investigate the late attention layers in detail, leveraging the language space uncover interpretable structure. We propose an algorithm, TEXTSPAN, that finds basis for each attention head where each basis vector is labeled by text description. The resulting bases reveal specialized roles for each head: for example, one head's top 3 basis directions are A semicircular arch, A isosceles triangle and oval, suggesting that specializes in shapes (Figure ((a)).
We present two applications of these identified head roles. First, we can reduce spurious correlations by removing heads associated with the spurious cue: we apply this on the Waterbirds dataset (Sagawa et al., 2019) to improve worst-group accuracy from 48% to 73%. Second, the representations of heads with a property-specific role can be used retrieve images according that property; we use it to perform retrieval based discovered senses of similarity, such as color, location, and texture.
We next exploit the spatial structure provided by attention layers. Each attention head's output is weighted sum across image locations, allowing us to decompose the output across these locations.
Project page and code:
sman . thub. io/elig pe__ppddddopsseceeeccccccppppppe ai

---

Published as a conference paper at ICLR 2024
CLIP-VIT
(b)
Image Tokens Decomposition
Represantation
Image
"A photo of a pyramid"
"A photo of a camel
proj layer
Input Image
Layers Heads
Attention Heads Decomposition
(c)
Joint Decomposition
Laver Head
Laver Head
Layer 22, Head
Layer 22, Head
Layer 22, Head
(a "number" head)
la "shape" head)
(a "color" head)
"Image with an orange color"
Image with subjects
1 semicrouter arch
Image with a yellow color
2mege with a four people
2. iocceels triangle
2.meae with orange color
isosceles triangle"
3.An image of the number 3
3.An oval
3. with cold green tones
Figure I: CLIP-VIT image representation decomposition. By decomposing CLIP's image rep- resentation as sum across individual image patches, model layers, and attention heads, we can (a) characterize each head's role by automatically finding text-interpretable directions that span its output space, (b) highlight the image regions that contribute to the similarity score between image and text, and (c) present what regions contribute towards a found text direction at specific head.
We use this to visualize how much each location writes along given text direction (Figure 1(b)). This yields a zero-shot image segmenter that outperforms existing CLIP-based zero-shot methods.
Finally, we consider the spatial structure jointly with the text basis obtained from TEXTSPAN. For each direction in the basis, the spatial decomposition highlights which image regions affect that basis direction. We visualize this in Figure (c)), and find that it validates our text labels: for instance, the regions with triangles are the primary contributors to a direction that is labeled as isosceles triangle.
In summary, interpret CLIP's image representation by decomposing into text-interpretable el- ements that are attributed to individual attention heads and image locations. We discover property- specific heads and emergent localization, and use our discoveries to reduce spurious cues and im- prove zero-shot segmentation, showing that understanding can improve downstream performance.
2 RELATED WORK
Vision model explainability. A widely used class of explainability methods produces heatmaps to highlight parts in the input image that are most significant to the model output (Selvaraju et al., 2017; Sundararajan et al., 2017; Binder et al., 2016; Voita et al., 2019; Lundberg & Lee, 2017; Chefer et al., 2021). While these heatmaps are useful for explaining the relevance of specific image regions to the output, they do not show how attributes that lack spatial localization (e.g. object size or shape) affect the output. To address this, a few methods interpret models by finding counterfactual edits using generative models (Goetschalekx et al., 2019; Lang et al., 2021; Aberman al., 2021). All these methods aim to explain the output of the model without interpreting its intermediate computation.
Intermediate representations interpretability. An alternate way explain vision models is to study their inner workings. One approach is to invert intermediate features into the input image space (Dosovitskiy & Brox, 2015; Mahendran & Vedaldi, 2014: Goh et al., 2021). Another approach is to interpret individual neurons (Bau et al., 2020; 2019; Dravid et al., 2023) and connections between neurons (Olah et al., 2020). These approaches interpret models by relying only on visual outputs.
Few methods use text to interpret intermediate representations in vision models. Hernandez et al. (2022) provide text deseriptions for image regions in which a neuron active. Yuksekgonul et al. (2023) project model features into bank of text-based concepts. More closely to us, few methods analyze CLIP's intermediate representations via text-Goh al. (2021) find multimodal neurons in CLIP that respond to different renditions of the same subject in images. Materzynska et al. (2022) study entanglement in CLIP between images of words and natural images. We differ from these works by using CLIP's intrinsic language-ing imagg space and by exploiting decompositions in CLIP's architecture for interpreting intermediate representations.
Contrastive vision-language models. Contrastive vision-ande language models like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) showed promising zero-shot transfer capabilities for down- stream tasks, including OCR, loaaliaziz and classification (Wortsman, 2023). Moreover, CL.IP
2

---

Published as a conference paper at ICLR 2024
representations are used for segmentation (Lüddecke & Ecker, 2022), querying 3D scenes (Kerr et al., 2023), and text- image generation (Ramesh et al., 2021; Rombach et al., 2022). We aim to interpret what information is encoded in these representations.
DECOMPOSING CLIP IMAGE REPRESENTATION INTO LAYERS
We start by presenting the CLIP model (Radford et al., 2021) and describe how the image represen- tation of CLIP-VIL VIT is computed. We show that this representation can be decomposed into direct contributions of individual layers of the image encoder ViT architecture. Through this decomposi- tion, we find that the last few attention layers have most of the direct effects on this representation.
3.1 CLIP-VIT PRELIMINARIES
Contrastive pre-training, CLIP is trained to produce visual representations from images I coupled with text descriptions I. It uses two encoders-a transformer-based text encoder Maet and an image encoder Mmage Both Mex and Maage map shared rision-and-le and-language latent space, allowing us to measure similarity between images and text via cosine similarity:
sim(l. = (Minage (1), (te (() (2)
(I)
Given a batch of images and corresponding text descriptions {(Te, (.)eett...))) CLIP is trained to maximize the similarity of the image representation Minge (I;) to its corresponding text representa- tion (ta), while minimizing sim tÿ)) for every + j in the batch.
Zero-shot classification. CLIP can be used for shot image classification. To classify an image given a fixed set of classes, each name of a class (e.g. "Chihuahua") is mapped to fixed template (e.g. "An image of {class]"") and encoded by the CLIIP text encoder. The prediction for given image is the class whose text description has the highest similarity to the image representation.
CLIP image representation. Several architectures have been proposed for computing CLIP's im- age representation. We focus on the variant that incorporates VIT (Dosovitskiy et al., 2021) backbone. Here a vision transformer (ViT) is applied to the input image E R x3 to obtain d-dimensional representation (I) The CLIP image representation Munge (I) is a linear pro- jection of this output to a d' ''-dimensional representation in the joint vision-and -ad---an-d----gee space?. Formally, denoting the projection matrix by P E Rd xd;
Minage (1) PVIT(I)
Both the parameters of the VIT and the projection matrix P are learned during training.
VIT architecture. VIT a residual network built from L layers, each of which contains a melti-head self-attention (MSA) followed by MLP block. The input I is first split into N non-overlapping image patches. The patches are projected linearly into N d-dimensional vectors, and positional embeddings are added to them create the image tokens (-] "+(((.....))) An additional learned
token 2 Rd named the class token, is also included and later used the output token.
Formally, the matrix 2 E Ixx((+1), with the tokens z0 =f:..., z as columns, constitutes the initial state of the residual stream. It is updated for L iterations via these two residual steps:
2'=
z' z'= MLLL' (2') 2'.
(3)
We denote the first column in the residual stream Z!, corresponding the class token, by [ats. The output of the ViT is therefore [24]ds
3.2 DECOMPOSITION INTO LAYERS
The residual structure of ViT allows us to express its output as a sum of the direct contributions of individual layers of the model. Recall that the image representation Minge (7) is linear projection
Both here and in Fq. 3, we ignore layer-normalization term to simplify derivations. We address layer- normalization in detail Section A.I.
3

---

Published as a conference paper at ICLR 2024
Accuracy
s0-
Base accuracy
+ MLPs ablation
VIT-B-16
ViT-B-16
70.22
67.04
20
VIT-L-14
ViT-L-14
75,25
74.12
VIT-H-14
ViT-H-14
77.95
76.30
32 Accumulated mean-ablated layers
Table MLPs mean-ablation. We simultane- ously replace all the direct effects of the MLPs with their average taken across ImageNet's valida- tion set. This results in only small reduction in zero-shot classification performance.
Figure MSAs accumulated mean-ablation. We replace all the direct effects of the MSAs up to given layer with their average taken across the ImageNet validation set. Only the replacement of the last few layers causes large decrease in accuracy.
of the ViT output. By unrolling Eq. across layers, the image representation can be written as:
Minage ()) = (T) = [ dts
(2-1 ols
(MLP((2))
(4)
MSA terms
terms
Eq. 4 decomposes the image representation into direct contributions of MLPs, MSAs, and the in- put class token, allowing us to analyze each term separately. We ignore here the indirect effects of the output of layer another downstream layer. We this decomposition (and further decompositions) to analyze CLIP's representations in the next sections.
Evaluating the direct contribution of layers. As preliminary investigation, we study which of the components in Eq. 4 significantly affect the final image representation, and find that the large majority of the direct effects come from the late attention layers.
To study the direct effect of component (or set of components). we use mean-ablation (Nanda al., 2023), which replaces the component with its mean value across dataset of images. Specif- ically, we measure the drop in zero-shot accuracy on classification task before and after ablation. Components with larger direct effects should result in larger accuracy drops.
In our experiments, we compute means for each component over the ImageNet (IN) validation set and evaluate the drop in IN classification accuracy. We analyze the OpenCLIP ViT-H-14 14, L- and B-16 models (Ilharco et al., 2021), which were trained on LAION-2B (Schuhmann et al., 2022).
MLPss have negligible direct effect. Table presents the results of simultaneously mean-ablating all the MIPs. The MIPs do not have significant direct effect on the image representation, as ablating all of them leads to only small drop accuracy (1%-3%).
Only the last MSAs have significant direct effect. We next evaluate the direct effect of different MSA layers. sults: removing all the early MSA layers (up to the last 4) does not change the accuracy significantly. Mcan-ablating these final MSAs, on the other hand, reduces the performance drastically.
so, we mean-ablate all MSA layers up to some layer l. Figure 2 presents the re-
In summary, the direct effect the output is concentrated in the last 4 MSA layers. We therefore focus only on these layers in our subsequent analysis, ignoring the MLPs and the early MSA layers.
3.3 FINE-GRAINED DECOMPOSITION INTO HEADS AND POSITIONS
We present more fine- fine-grained decomposition of the MSA blocks that will be used in the next two sections. We focus on the output at the class token, as that is the only term appearing in Eq. 4. Following Elhage et al. (2021), write the MSA output sum H independent attention heads and the N input tokens:
H N =Σ>
2. = -. W.
(5)
h=1i=0 ==0
where W--h Raxd transition matrices and a. E IR are the attention weights from the class token to the i-th token (>i/ a; = 1). Therefore, the MSA output can be decomposed into direct effects of individual heads and tokens.
4

---

Published as a conference paper at ICLR 2024
L21.HI1 ("Gec-locatioos")
L23.HI0 ("Counting")
L22.H8 ("Letters")
Photo captured in the Arizona desert
Image with six subjects
A photo with the letter V
Picture taken in Alberta, Canada
Image with four people
A photo with the letter F
Photo taken in Rio de Janeiro, Brazil
An image of the number
A photo with the letter D
Picture taken in Cyprus
image of the number
A photo with letter
Photo taken in Seoul, South Korea
The number fifteen
A photo with the letter X
122.H11 ("Colors")
L22.H6 ("Animals")
L22.H3 ("Objects")
A charcoal gray color
Curious wildlife
An image of legs
Sepia-tone photograph
Majestic soaring birds An image with dogs
A jacket
Minimalist white backdrop
A helmet
High-contrast black and white
Image with dragonfly
A scarf
Image with red color
An image with cats
A table
L23.H12 ("Textures")
L22.H1 ("Shapes")
L22.H2 ("Locations")
Artwork with pointillism technique
A semicircular arch
Urban park greenery
Artwork with woven basket design
An isosceles triangle
Cozy home interior
Artwork featuring barcode arrangement
An oval
Urban subway station
Image with houndstooth patterns
Rectangular object
Energetic street scene
Image with quilted fabric patterns
A sphere
Tranquil boating on a lake
Table 2: Top-5 text descriptions extracted per head by our algorithm. Top 5 components returned by TEXTSPAN applied to ViT-L,, for several selected heads. See Section A.5 for results all the heads.
Plugging the MSA output definition in Eq. 5 into the MSA term in Eq. 4, we obtain:
Ep [M(A'A((Z dis
CIILA Pxr 1.b
(6)
In other words, the total direct effect of all attention blocks is the result of contracting the tensor e across all of its dimensions. By contracting along only some dimensions, we can decompose effects in a variety of useful ways. For instance, we can contract along the spatial dimension i to get
I.h Σ Zimo Ci Alternatively, we can contract along layers and contribution for each head: Chend
heads to get contribution from each image token: Cooken Ete ΣH Ci,.h.
The quantities G, head and Guken all live in the d'-dimensional joint text-image representation space, which allows to interpret them via text. For instance, given text description f, the quantity (Muaatt) heea)) intuitively measures the similarity of that head's output to description
DECOMPOSITION INTO ATTENTION HEADS
Motivated by the findings in Section 3.2, we turn to understanding the late MSA layers in CLIP. We use the decomposition into individual attention heads (Section 3.3), and present an algorithm for labeling the latent directions of each head with text descriptions. Examples of this labeling are depicted in Table 2 and Figure 4, with the labeling for all 64 late attention heads given in Section A.5.
Our labeling reveals that some heads exhibit specific semantic roles, e.g. "counting" or "location", in which many latent directions in the head track different aspects of that role. We show how to exploit these labeled roles both for croperty-speciffe image retrieval and for reducing spurious correlations.
4.1 TEXT-INTEPPRETARL.. DECOMPOSITION INTO HEADS
We decompose an MSA's output into text-related directions in the joint representation space. We rely on two key propertics: First, the output of each MSA block is a sum of contributions of individual attention heads, demonstrated in Section 3.3. Second, these contributions lie in the joint text- image representation space and so can be associated with text.
Recall from Section 3.3 that the MSA terms of the image representation (Eq. 4) can be written as I.h sum over heads, Σt,, head' To interpret head's contribution d.h we will find a of text de- scriptions that explain most of the variation in the head's output (the head "principal components").
To formalize this, we take input images I,.... Ix with associated head outputs C1, ..., Cn (for sim- plicity, we fix the layer and head h and omit from the notation). As C1, -ng CK vectors in the
5

---

Published as a conference paper at ICLR 2024
Algorithm 1: TEXTSPAN
Input: Head (1, k) contribution did for K images stacked in matrix C Rxxxd pool of M text descriptions (44 / their corresponding CLIP text representations RMxx (projected to the head output space), and basis size m
Output: of text descriptions T and projected representations C' E RRx
Initialization: C' < Ok xa',
for in b..., m do
D RC"
j + arg max Var(D(j))
T +- TU {4y-)
for k in [1,-..; K] do
[*( [*] + : Rb""]
||
C[k] 4- [(k] [C
R[J*]
for in [1,..., .) do
RMM <- R[[A]
(RIA)RG RO") RG-
joint text-image space, each text input defines direction (t) in that space. Given collection of text directions T, let Proir denote the projection onto the span of (Mea(!) | TJ. We define the variance explained by as the variance under this projection:
1
1 K
Vexplained (T)
-===
(rojy(a: -cavg)l where Cavg
Ck
k=1
k=
We aim to find a set of m descriptions T for each head that maximizes Vexplained (T). Unlike regular
PCA, there no closed-form solution to this optimization problem, so we take greedy approach.
Greedy algorithm for descriptive set mining. To approximately maximize the explained variance in Eq. we start with a large pool of candidate descriptions } and greedily select from it to obtain the set T.
Our algorithm, TEXTSPAN, is presented in Alg. 1. It starts by forming the matrix C' R of outputs for head (1, h). as well as the matrix R E Mxd of repre- sentations for the candidate descriptions, projected onto the span of C. In each round, TEXTSPAN computes the dot product between each row of R and the head outputs C, and finds the row with the highest variance R[j"] (the first "principle component").
Accuracy
general descriptions most common words vactors
then projects that compo- nent away from all rows and repeats the process to find the next components. The projection step ensures that each new component adds variance that is orthogonal to the earlier components.
Descriptions per head
Figure 3: ImageNet classification accu- racy for the image representation pro- jected to TEXTSPAN bases. We evalu- ate our algorithm for different initial de- scription pools, and with different out- put sizes.
TEXTSPAN requires an initial set of descriptions (t: li=1 that is diverse enough to capture the output space of each head. We use set of sentences that were generated by prompting ChatGPT-3.5 to produce general image de- scriptions. After obtaining an initial set, we manually prompt ChatGPT to generate more examples of specific patterns we found (e.g. texts that describe more colors). This results in 3498 sentences. In our experiments, most common words English, and a set random d'-dimensional vectors that do not correspond text (see Section A.3 for the ChatGPT prompt and more details about the baselines).
also consider two simpler baselines wordwwrr descriptions comprising the
4.2 EXPERIMENTS
We apply TEXTSPAN to find a basis of text descriptions for all heads in the last MSA layers. We first verify that this set captures most of the model's behavior and that text descriptions track

---

Published as a conference paper at ICLR 2024
Layer 22, Head B: "A photo with the letter V
Layer 22, Head 2: "Urban park preenery"
Layer 23, Head 12: "Image with polka dot patterms"
Layer 22, Head 7: 'Serene winter wonderiand
Figure 4: Top-4 images for the top head description found by TEXTSPAN. We retrieve images with the JAA highest similarity score between hhead and the top text representation found by TEXTSPAN. They correspond to the provided text descriptions. See Figure 11 in the appendix for randomly selected heads.
image properties. We then show that some heads are responsible for capturing specific image prop- erties (see Figure I(1)). We use this finding for two applications ceaautinnn-rruuiinngg known spurious cues in downstream classification and property -speciiic image retrieval.
Experimental setting. We apply TEXTSPAN to all the heads in the last 4 layers of CLIP ViT-L, which are responsible for most of the direct effects on the image representation (see Section 3.2). We consider a variety of output sizes m {10, 20, 30, 40, 50, 60).
We first verify that the resulting text representations capture the important variation in the image representation, as measured by zero-shot accuracy on ImageNet. We simultaneously replace each head's direct contribution head with its projection to the text representations ProjT(..)) Gead (where T(I, h) is the obtained text set for head (1, h)). We also mean-ablate all other terms in the represen- tation (MLPs and the rest of the MSA layers).
1.h
J.h
The results are shown in Fig. 3: 60 descriptions per head suffice reach 72.77% accuracy (compared to 75.25% base accuracy). Moreover, using our CharGPT-generated descriptions as the candidate pool yields higher zero-shot accuracy than either common words or random directions, for all the different sizes m. In summary, we can approximate CLIP's representation by projecting each head output, a -o8-dimensional vector, to (head-speeificc 60-dimensional rext-interpretable subspace.
Some attention heads capture specific image properties. We report selected head descriptions from TEXTSPAN (m = 60) Table 2, with full results in Appendix A.5. For some heads, the top descriptions center around a single image property like texture (I.23H12), shape (I.22H1), object count (L23H10), and color (L22HH1),. 1). This suggests that these heads capture specific image proper- ties. We qualitatively verify that the text tracks these image properties by retrieving the images with the largest similarity (t,) d.be)) for the top extracted text descriptions t,. The results in Fig. and 11 show that the returned images indeed match the text.
Reducing known spurious cues. We remove spurious correlations. For instance, if location being used as a spurious feature, we can ablate heads that specialize in geolocation hopefully reduce reliance on the incorrect feature.
knowledge of head-specific roles to manually
We validate this idea on the Waterbirds dataset (Sagawa et al., 2019), which combines waterbird and landbird photographs from the CUB dataset Welinder al. (2010) with image backgrounds (water/land background) from the Places dataset (Zhou et al., 2016). Here image background is spurious cue, and models tend misclassify waterbirds on land backgrounds (and vice versa).
To reduce spurious cues, we manually annotated the role of each head using the text descriptions from TEXTSPAN, mean-ablated the direct contributions of all "geolocation" and Vimage-location heads, and then evaluated the zero-shot accuracy on Waterbirds, computing the worst accuracy across subgroups as Sagawa et (2019). As a baseline, we also ablated 10 random heads and reported the top accuracy out of 5 trials. As shown in Table 3, the worst-group accuracy increases by a large margin-by 25.2% for ViT-L. This exemplifies that the head roles we found with TEXTSPAN help us to design representations with less spurious cues, without any additional training.
Property-basee image retrieval. Since some heads specialize to image properties, we can use their representations to obtain property-specific similarity metric. To illustrate this, for given head (h, !), we compute the inner product (ch (T), dilla(")) between a base image / and all other
7

---

Published as a conference paper at ICLR 2024
Layer 20, Head
"cconningg" head!
Layer 21, Head 15 la "locstoon head!
Inout Image
Layer Head
(an "objeor" head!
Input Image
Layer Head "bolor" head!
Layer 23, Head 8 ""eolor"head!
Layer 23, Head 12 la "texture"had
Figure 5: Top-4 nearest neighbors per head and image. We retrieve the most similar images to an input image by computing the similarity of the direct contributions of individual heads. As some heads capture specific aspects of the image (e.g. colors/objects), retrieval according to this metric results in images that are most similar regarding these aspects. See additional results in the project page and appendix (Fig. 12).
images in the dataset, retrieving the images with the highest similarity. Figure 5 shows the resulting nearest neighbors for heads that capture different properties. The retrieved images are different for each head and match the head-specific properties. In the left example, if we use a head that captures color for retrieval, the nearest neighbors are images with back-and-whing objects. we use a head that counts objects, the nearest neighbors are images with two objects.
DECOMPOSITION INTO IMAGE TOKENS
Decomposing the image representation across heads enabled us to answer what each head con- tributes to the output representation. We can alternately decompose the representation across image tokens to tell us which image regions contribute to the output for given text direction Mtx We find that these regions match the image parts that é describes, thereby yielding a zero-shot semantic image segmenter. We compare this segmenter to existing CLIP-based zero-shot methods and find that it is state-of-the-ar Finally, we decompose each head's 's direct contributions into per-head image tokens and this to obtain fine-grained visualizations of the information flow from input images to output semantic representations.
Decomposing MSA outputs into image tokens. Applying the decomposition from Section 3.3, if we group the terms C.I.A by position instead of head (1, h), we obtain the identity Minage (T) = Ei= Stoken (T), where ctoken (T) is the sum of the output location i across all heads (1, We empirically find that the contribution of the class token Goken has negligible direct effect on zero- shot accuracy (see mean-ablation in A.2). Therefore, we focus on the N image tokens.
We use the decomposition into image tokens to generate heatmap that measures how much the out- put from each image position contributes to writing in a given text direction. Given a text description we obtain this heatmap by computing the score (Cooeen (T), M (t)) for each position é.
Quantitative segmentation results. We follow a standard protocol for evaluating heatmap- based explainability methods (Chefer et al., 2021). We first compute image heatmaps given deseriptions of the image class (e.g. "An image {class}"")? We then binarize them (by applying a threshold) to obtain a foreground/bcckroound segmentation. We compare the segmentation quality to zero-shot segmentations produced by other explainability methods in the same manner.
We evaluate the methods ImageNet-segmenaaee subset of 4,276 images from the ImageNet validation set with annotated segmentations. Table 4 dis- plays the results: our decomposition is more accurate than existing methods across all metrics, See Chefer et al. (2021) for details about the compared methods and metrics, and additional qualitative comparisons in Section A.6.
(Guillaumin et al., 2014), which contains
Joint decompasition into her-head image tokens. Finally, we can jointly decompose the output of CLIP across both heads and locations. We use this decomposition affect each of the basis directions found by TEXTSPAN. Recall that CrJ,A from Eq. 6 is the direct contribution of token at head (h, l) to the representation. For each image token i, we take the inner
visualize what regions
We normalize out bias terms by subtracting from the heatmap averaged heatmap computed across all class descriptions in ImageNet.
8

---

Published as a conference paper at ICLR 2024
Layer Head 13 "peolocation" head
Layer 22, Head "season" head
Layer 23, A "color" head
Input Image
Photo taken Paris, France"
"Image captured in the Australian
Blossooing springlime blcoms"
"Serene winter wonderland"
"Photograph with red color palette"
"Soft pestel hues"
'Image taken Brazil
"Picture taken in Pakistan"
photo the summer"
"Cisp leeves'
image with cold green tones"
"A charccal gray color
Figure 6: Joint decomposition examples. the description that is most similar to Tuu among the TEXTSPAN output set. The right heatmap (red border) corresponds the least similar
each head (1, the heatmap (green border) corresponds
this set (for 60). Figure more results.
products between C...A and a basis direction Mex and obtain per-head similarity heatmap. This visualizes the flow of information from input images to the text-labeled basis directions.
In Figure 6, we compute heatmaps for the two TEXTSPAN basis elements that have the largest and smallest (most negative) coefficients when producing each head's output. The highlighted regions match the text description for that basis direction-for for instance, L22HI3 is geolocation head, its aighest-activating direction for the top image "Photo taken in Paris, France", and the image tokens that contribute to this direction are those matching the Eiffel Tower.
Pixel Acc.
mloU
mAP f
LRP (Binder et al., 2016)
52.81
33.57
54.37
base
random
partial-LRP (Voita al., 2019)
61.49
40.71
72.29
VIT-B-16
57.5
rollout (Abnar & Zuidema, 2020)
60.63
40.64
74.47
VIT-L-14
47.7
57.7
72.9
raw attention
65.67
43.83
VIT-H-14
37.2
37.0
43,3
GradCAM Selvaraju et al. (2017)
70.27
44.50
70.30
Chefer et al. (2021) Ours
69.21
47.47
78.29
Table 3: Worst-group accuracy on Waterbirds. We reduce spu- rious by ablating property- specific heads. Tables 11-14 for fine-grained results.
75.21
54.50
81.61
Table 4: Segmentation performance ImageeNdtssgmmentttio
image tokens decomposition results significantly more accurate zero-shot segmentation than previous methods.
LIMITATIONS AND DISCUSSION
We studied CLIP's image representation by analyzing how individual model components affect it. Our findings allowed to reduce spurious cues in downstream classification and improve zero-shot segmentation. We present two limitations of our investigation and conclude with future directions.
Indirect effects. We analyzed only the direct effects of model components on the representation. Studying indirect effects (e.g. information flow from early layers to deeper ones) can provide addi- tional insights into the internal structure of CLIP and unlock more downstream applications.
Not all attention heads have clear roles. The outputs TEXTSPAN show that not every head cap- single image property (see results in Section A.5). We consider three possible explanations for this: First, some heads may not correspond to coherent properties. Second, the initial descrip- tions pool does not include descriptions of any image property. Third, some heads may collaborate and have a coherent role only when their outputs are addressed together. Uncovering the roles of more complex structures in CLIP can improve the performance of the described applications.
Future work. We believe that similar analysis for other CLIP architectures (e.g. ResNet) can shed light on the differences between the output representations of different networks. Moreover, insights may help to design better CLIP image encoder architectures and feature extractors for downstream tasks. We plan to explore these directions future work.

---

Published as a conference paper at ICLR 2024
Acknowledgements. We would like to thank StannStansslas Denain for the insightful discussions and comments. We thank Jixahai Feng, Fred Zhang, and Erik Jones for helpful feedback on the manuscript. YG is supported by the Google Fellowship. AE is supported in part by DoD, including DARPA's MCS and ONR MURI, as well as funding from SAP. JS is supported by the NSF Awards No. 1804794 & 2031899.
REFERENCES
Kfir Aberman, Junfeng Hc, Yossi Gandelsman, Inbar Mosseri, David Jacobs, Kai Kohlhoff, Yael Pritch, and Michael Rubinstein. Deep saliency prior for reducing visual distraction. CoRR, bbss/209.01180) 2021. URL https: /ttpiv.xvvvv..///////../.a.aa org/ aba/bbs//100 01980.
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190-4197, Online, July 2020. Association for Computational Linguisties. doi: 10. (0.18653//1/2020.acl aciiminn- 385. URL ht https///aaaaaaahhaccoeco
owtooo/g//////. 2000 acl act.m/0mm.....m......m.mm
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial net- works. In Proceedings of the International Conference Learning Representations (ICLR), 2019.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role individual units deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10. 1073/pnas 190.3/77111997771111 URL https: /111////1/1117111111
//www pnw..pnssss.. org/ htt////otteeteeeeee cotr///oeee///erllr..er.
www.pnas...
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin. -laur-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. volume 9887 of Lecture Notes in Computer Science, pp. 63-71. Springer Berlin / Heidelherg, 2016. doi: 10. 000007977-33319-44777
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 182-791, June 2021.
Alexey Dasovitskiy and Thomas Brox. Inverting convolutional networks with convolutional net- /1566 02753.
works. CoRR, abs/l 2015. URL http:
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2021.
Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in model zoo. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 1934-1943, October 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfiel-Doodde Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defini- tions of cognitive image properties, 2019.
Gabriel Goh, Nick Cammarata Y, Chelsea Voss t. Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23915/4 5/distil 1/000.. http://iillllpb//22/mmmmmmaaaala
10

---

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction circuits. Distill, 2020. doi: 1..39915/disilll.0004..00 https//dbsilllloo/2222ecccctttts Published as a conference paper at ICLR 2024
Matthieu Guillaumin, Daniel Küttel, and Vittorio Ferrari. Imagenet auto anto-annotation with segmenta- tion propagation. Int. Comput. Vision, 110(3):328 348, dec 2014. ISSN 0920-5691. doi: 10. 1007//11223--11-077133 URL https: : 04
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. CoRR, abs/2201.11114, 2022.
URL https: /tttpp://arxiii. org/ abss/201.... 11114.
Gabriel Uharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL https: : /tt///diii/// org/ 12811 zenodo. 5143773. If you this software, please cite it below.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and liionnlanguaage representation learning with noisy text supervision. In International Conference on Machine Learning. 2021. URL attpss:/aptt www.nticscholeeeeeeeeee
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Lan- guage embedded radiance fields, 2023.
Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T. Freeman, Phillip Isola, Amir Globerson, Michal Trani, and Inbar Mosseri. Explaining in style: Training gan explain classifier in stylespace. arXiv preprint 20X 2021.
Timo Lüddeeke and Alexander Ecker. Image segmentation using text and image prompts. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 7086-7096, June 2022.
Scott M. Lundberg and Su-In Lee. A unified approach interpreting model predictions. CoRR, abs/1705.07874, 2017. URL http: //apivva//rivvv/// org/ hbsp//bs/11000 /1705 07874.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. CoRR, 411.00035 2014. URL http: www.aaiivvvor...a.aa www.oogg///oo...oooo abs//b/////// 42. 0035.
Joanna Materzynska, Antonio Torralba, and David Bau. Disentangling visual and written concepts in clip, 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8766 PMLR, 18-24 Jul 2021. URL https: https::/pproceeinng. mlr www...prre//vlll.. 1 radford21a. html.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102. 2021. URL https: http:://xxvvvvvoo https://arxiv.orrr/aass
wrt/ /twps////2..... 2102. 12092.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.

---

Published as a conference paper at ICLR 2024
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. CoRR, 0bs/1911.0873.. 2019. URL http:/ http://gggbbbb.la/oaa/
abs/ 19t.//11111..11.11
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In siitt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. https: /openrevies net. meet/forrttt
vmsMcY.
Toby Segaran and Jeff Hammerbacher (eds.). Beautiful Data: The Stories Behind Elegant Data Solutions. O' O'Reilly, Beijing, 2009. ISBN 978-0-396-15711-1. URL https://ww/
w.faribbcksooo w wtt.//oom///ll/ee i ful /
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In ICCV, pp. 618-626. IEEE Computer Society, 2017. ISBN 978-
1-5386-1032-9. URL http:/ /ttp::///bbppppp uni mimittrier.. de/ /tl.d///////////eee html #Sel#Sellaarrr juCDVVPB1..
Mukund Sundararajan. Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. CoRR,
abs/1703.01365, 2017. URL http: wtppaaxxvvvvv.......... .rg. wwwp/bs/////oooooo.ttt....t...tt /bs//703- 1703. 013
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Anmual Meeting the Association for Computationnll Linguistics, pp. 5797-5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1111665/11/1191115511 URL https: httpp:://clanthology: org/ 1580.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Be- longie, and Pietro Perona. Caltech-uesd birds 200. Technical Report CNS-TR-201, Caltech, 2010. URL /te3////////// pdf, , http: httt://wwerr /twww/wwww///mm.m/ vision. cawtech.tch...... edu/ wwtpe/iiivivaeaeeeeeeeeeeeeeeeee 200. 2ttttm10m............ html.
CUB 200.
Mitchell Wortsman. Reaching 80% zero-shot accuracy with openclip: Vit-g/14 trained laion-2b. 2023. URL httpt: //taios://///o../o////// a /t.ii///// ho//llg//// giant-openclllg wwp.ooicnll..............m..ppp.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models, 2023.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. 2016.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 LAYER NORMALIZATION
We describe here the modifications that needed to be incorporated in method take into account layer-normalizations There are two places where layer-normalizations are used - before the projection layer (to the output of the ViT), and before each layer the ViT (to the layer input). We present how the individual contributions of Ci,,Ah Chead
I.h and clokon should be changed.
Pre-projection layer normalization. As mentioned in the Section 3.1, in many implementations of CLIP, a layer-normalization LN applied to the output of the ViT before the projection layer. Formally, the image representation of image I is then:
Minage (T) PLN(ViT
(8)
The normalization layer can be rewritten
LN(x) = 7 I 14 Vo +
7 of
PLY LVo +
x
(9)
where x E Rd is the input token, (4, 01 I are the mean and standard deviation, and 7, E R are learned vectors. To incorporate the layer normalization in our decomposition, we compute the mean and the standard deviation during the forward pass of the model. The multiplicative term, Vor-e is
absorbed into the projection matrix P. The contribution of Mry 3 is split equally between all Voî+e
the C terms in the Eq. 6. We apply these modifications when we decompose OpenCLIP-based models.
MLPs and MSAs input layer normalizations. In the main paper, we do not describe the normal- ization layers that are applied each input MLP and MSA in the model. More accurately, the residual updates of the ViT are:
Z' MSA (LN' (Z'-1))+ z' = MLL (2'))
(10)
Where LiV' and LN are the layer normalizations applied to each token in the input matrix of the MLP layers and MSA layers. This modification does not affect our corollaries about the direct contributions of the MI.P layers and MSA layers, as we only address the outputs of these layers. The only other equation in which this modification takes place is in Eq. 5:
#
MSA MSA((Z-1)) (7-1))
Lh = of (f tW I
(11)
4=1 1=0
A.2 MEAN-APE ABLATION OF THE CLASS-TOKEN ATTENDED FROM ITSELF
We show that we can ignore the direct effect of the class token in the MSAs term when we decom- pose it into tokens (see section 5). We mean-ablate the direct contribution of the class token the MSAs term in Eq. 6. We simultancously ablate both the class token and the MLPs. The ImageNet zero-shot classification performances of the three ViT models are shown in Table 5. As shown, the direct contributions of all the MLP layers and the direct contributions of the class token in the decomposed MSAS term results negligible drop performance for all the models.
A.3 TEXT DESCRIPTIONS
General text descriptions. To generate the set of text descriptions that are used by our algorithm, we prompted ChatGPT (GPT-3.5) to produce image descriptions. We used the prompt provided in Table 6, and manually prompted the language model generate more examples for specific patterns we found in the initial result (e.g. more colors, more letters). This process resulted in 3498 sentences.
Most common words. For the set of most common words, used the same number of examples, and took the 3498 most common English words, as determined by n-gram frequency analysis of Google's Trillion Word Corpus (Segaran & Hammerbacher (2009)).
13

---

Published as a conference paper at ICLR 2024
Base
class token
MM.Ps
accuracy
ablation
ablation
ViT-B-16
70.22
69.37
67.32
ViT-L-14
75.25
74.38
73.87
ViT-H-14
77.95
76.89
76.29
Table 5: Mean-ablation of the class token contribution to the MSAs term. The overall drop
in accuracy is relatively small, when the ML.s replaced by their mean across ImageNet validation set.
Class-specific text descriptions. We generate additional class-specific text descriptions, by prompt- ing ChatGPT with the prompt template provided in table 6. We queried to model for each of the ImageNet class names. This process resulted in 28767 unique sentences.
Random vectors. As a baseline we created random set of 3498 vectors sampled from unit Gaussian.
General text descriptions initial prompt
Imagine you are trying explain photograph by providing a complete set of image characteristics. Provide generic image characteristics. Be as general possible and give short descriptions present- ing one characteristic at a time that can describe almost the possible images of a wide range of categories. Try cover many categories possible, and don't repeat yourself. Here some possible phrases: "An image capturing an interaction between subjects", "Wildlife their natural habitat", "A photo with texture of mammals", "An image with cold green tones", "Warm indoor scene", "A photo that presents anger". Just give the short titles, don't I explain why, and don't com- bine different concepts (with "or" "and"). Make each item in the list short but descriptive. Don't be too specific.
Class-specific text descriptions prompt
Provide 40 image characteristics that are true for almost all the images of (class). Be as general as possible and give short descriptions presenting one characteristic a time that can describe almost all the possible images of this category. Don't mention the category name itself (which is "[class) "") Here are some possible phrases: "Image with texture ". "Picture taken in the geographical location of..." "Photo that is taken outdoors". "Caricature with text", "Image with the artistic style of...", "Image with one/two/three objects", "Illustration with the color palette ...", "Photo taken from above/below". "Photograph taken during ... season". Just give the short titles, don't explain why, and don't 't combine two different concepts (with "or" or "and"".
Table 6: ChatGPT prompts for image descriptions generation.
A.4 ADDITIONAL INITIAL DESCRIPTION POOL ABLATION
We present additional ablation of the initial set of text descriptions provided to TEXTSPAN. The text description generation processes for each of the pools are described Section A.3.
As shown in Figure 7, using the class-specific descriptions pool that includes around x8 more exam- ples than the general descriptions pool, allows us to obtain higher accuracy with fewer descriptions per head (smaller m). Nevertheless, using each of the pools results in relatively similar accuracy with m = 60.
A.5 TEXTSPAN OUTPUTS FOR CLIP-VIT-L
We apply TEXTSPAN the attention heads of the last layers of CLIP VIT-I. Tables 7-10 present the first 5 descriptions per head.
A.6 QUALITATIVE RESULTS FOR IMAGE TOKEN DECOMPOSITION
Figure 8(a) shows the similarity heatmaps for text descriptions. As presented our heatmaps high- light the objects that described in the text. Figure 8(b) presents the relative similarity heatmaps
14

---

Published as a conference paper at ICLR 2024
Accuracy
general descriptions
20
words
random vectors
class-specific descriptions
Descriptions per head
Figure 7: ImageNet classification accuracy for the image representation projected to TEXTSPAN bases (additional results). We evaluate our algorithm for different initial description pools, and with different output sizes.
"The player with
"The grown-up elephant"
"The dog with ginger
"The player with the ball"
"The baby elephant"
"The dog with the black and white fur"
"An image ofla liger"
"An image of ram (adult male sheap)"
Figure 8: Heatmaps produced by the image token decomposition. We visualize (a) what areas in the image directly contribute to the similarity score between the image representation and a text representation and (b) what areas make image representation more similar to text represen- tation rather than another.
given two descriptions (by subtracting between the two heatmaps). The areas in the images that make the image representations more similar to one of the text representations rather than the other, correspond to the areas that are mentioned by and ignored by the other text.
A.7 MOST SIMILAR IMAGES TO TEXTSPAN RESULTS
We randomly choose attention heads from the last 4 layers of CLIP VIT-L. For each head (1, h), I.h we retrieve the 3 images with the highest similarity score between their L and the top 10 text representations found by our algorithm. The retrieval is done from ImageNet validation set. The results are presented in Figure II. As shown, in most cases, the top text representation corresponds to the attributes of the images.
15

---

Published as a conference paper at ICLR 2024
Layer 20, Head
Layer 20, Head
Picture taken in Hungary
Picture taken in Seychelles
Image taken in New England
Picture taken in Saudi Arabia
Futuristic technological concept
Muted urban tones
Playful siblings
Man-made pattern
Picture taken in the English countryside
an image of glasgow
Layer 20, Head
Layer 20, Head 3
Image of police
Intrica wood carvingte
Picture taken in Laos
Image snapped Spain
Remote alpine chalet
Photo taken in Bora Bora, French Polynesia An image of a Preschool Teacher A breeze
A photograph of small object
Desert sandstorm
Layer 20, Head
Layer 20, Head 5
Image with pair of subjects
an image of samoa Urban nostalgia A photo with the letter K Image snapped in the Colorado Rockies Serendipitous discovery
Image with five subjects
Image with trio of friends A photo of adult
Image with seven people
Layer 20, Head
Layer 20, Head 7
Bustling city square
Energetic children
Peaceful village alleyway
Grumpy facial expression
ornate cathedral
Intricate ceramic patterns
Image taken in the Alaskan wildemess
Photo taken in Bangkok, Thailand
Modern airport terminal
Subdued moments
Layer 20, Head 8
Layer 20, Head 9
Photo taken Rioja, Spain
Tranquil Asian temple
Photo taken Boreeo
Vibrant city nightlife
Vibrant urban energy
A photo with the letter R intricate mosaic artwork
Picture captured in the Icelandic glaciers oceanside
Photo taken in the Rub' Khali (Empty Quarter)
Layer 20, Head 10
Layer 20, Head 11
A bowl
Photo taken in Beijing, China
A bottle
Photo with retro color filters
Nostalgic pathways
Image with holographic cyberpunk aesthetics
A laptop
Urban fashion
Reflective ocean view
Photograph with the artistic style tilt-shift
Layer 20, Head 12
Layer 20, Head
Photo with grainy, old film effect Detailed illustration
Image taken from distance
Photograph with the artistic style of split toning Photo taken Beijing, China
Serene beach sunset
An image of the number 10
close-up shot An image of a Novelist
An image of the number 5
Layer 20, Head 14
Layer 20, Head 15
Quirky street performer
Remote hilltop hut
Antique sculptural element
Photo taken in Barcelona, Spain
Celebratory atmosphere
Dynamic movement
Overwhelmed facial expression
Caricature of influential leader
Serene winter wonderland
A picture of Samoa
Table 7: Top-5 results of TEXTSPAN. Applied to the heads at layer 20 of CLIP-VIT-L.
16

---

Published as a conference paper at ICLR 2024
Layer 21, Head 0
Layer 21, Head
Timeless black and white
Picture taken in the southeastern United States
Vintage sepia tones
Picture taken in the Netherlands
Image with a red color
Image taken Brazil
charcoal gray color
Image captured in the Australian bushlands
Soft pastel hues
Picture taken in the English countryside
Layer 21, Head 2
Layer 21, Head 3
A photo of woman
Precise timekeeping mechanism
photo of man Energetic children
Image snapped the Canadian lakes
An image of Andorra
An image with dogs
thrilling sports challenge
A picture of a baby
Photo taken in Namib Desert
Layer 21, Head 4
Layer 21, Head 5
An image with dogs
Inquisitive facial expression
A picture of a bridge
Artwork featuring typographic patterns
A photo with the letter R
A photograph of big object
Dramatic skies
Reflective landscape
Ancient castle walls
Burst motion
Layer 21, Head 6
Layer 21, Head
Photo taken in the Italian pizzerias
A pin
thrilling motorsport
A thimble
Urban street fashion An image of Animal Trainer
bookmark
Picture taken in Rwanda
Screne countryside sunrise
A pen
Layer 21, Head 8
Layer 21, Head 9
Inviting coffee shop
Photograph with blue color palette
Photograph taken in music store An image of News Anchor
Image with a purple color
Image with a pink color
Joyful family picnic scene cozy home library
Image with orange color Timeless black and white
Layer 21, Head 10
Layer 21, Head 11
Playful winking facial expression
Photo captured in the Arizona desert Picture taken in A Alberta, Canada
Joyful toddlers
Closc-up of textured plastic An image of Teacher
Photo taken in Rio de Janeiro, Brazil
Picture taken in Cyprus
Image with a seven people
Photo taken in Seoul, South Korea
Layer 21, Head 12
Layer 21, Head 13
Photo with grainy, old film effect Macro botanical photography
Quiet rural farmhouse Lively coastal fishing port an image of liechtenstein Image taken in the Florida Everglades thrilling motorsport race
laptop
Vintage nostalgia
serene mountain retreat
Layer 21, Head 14
Layer 21, Head 15
Photo taken in Beijing, China
Submerged underwater scene
Cheerful adolescents
Artwork featuring overlapping scribbles
Picture taken in Ecuador
Surrealist artwork with dreamlike elements Serene winter wonderland Wildlife in their natural habitat
Dreamy haze
Image captured in the Greek islands
Table 8: Top-5 results of TEXTSPAN. Applied to the heads at layer 21 of CLIP-ViT-L.
17

---

Published as a conference paper at ICLR 2024
Layer 22, O
Layer 22, Head 1
Artwork with pointillism technique
A semicircular arch An isosceles triangle
Artwork with woven basket design
Artwork featuring barcode arrangement
An oval
Image with houndstooth patterns
Rectangular object
Image with quilted fabric patterns
A sphere
Layer 22, Head
Layer 22, Head
Urban park greenery cozy home interior
An image of legs
jacket
Urban subway station
A helmet
Energetic street scene
A scarf
Tranquil boating lake
A table
Layer 22, Head
Layer 22, Head
An image with dogs
Harmonious color scheme An image of checks
Joyful toddlers
Serene waterfront scene thrilling sports action
Vibrant vitality
Captivating scenes
A picture of baby
Dramatic chiaroscuro photography
Layer 22, Head
Layer 22, Head
Curious wildlife
Serene winter wonderland
Majestic soaring birds
Blossoming springtime blooms
An image with dogs Image with dragonfly
Crisp autumn leaves
A photo taken the summer
An image with cats
Posed shot
Layer 22, Head 8
Layer 22, Head 9
A photo with the letter V
A photo food
A photo with the letter F
delicate soap bubble play
A photo with the letter D
Dynamic and high-energy music performance
A photo with the letter T
Hands in embrace
A photo with the letter X
Futuristic technology display
Layer 22, Head 10
Layer 22, Head 11
Image with yellow color
A charcoal gray color
Image with orange color
Sepia-toned photograph
An image with cold green tones
Minimalist white hackdrop High-contrast black and white Image with red color
Image with pink color
Sepia-tone toned photograph
Layer 22, Head 12
Layer 22, Head 13
Photo taken in Namib Desert Ocean sunset silhouette
Image taken in Thailand
Picture taken in the Netherlands
Photo taken
Brazilian rainforest
Picture taken in the southeastern United States
Serene countryside sunrise
Image captured in the Australian bushlands Picture taken in the gengraphical location of Spain
Bustling cityscape at night
Layer 22, Head 14
Layer 22, Head 15
A silver color
contemplative urban view
Play of light and shadow Image with white color
Photograph revealing frustration
Celebratory atmosphere
A charcoal gray color Cloudy sky
Captivating authenticity
Intense athletic competition
Table 9: Top-5 results of TEXTSPAN. Applied to the heads at layer 22 of CLIP-ViT-L
18

---

Published as a conference paper at ICLR 2024
Layer 23, Head
Layer 23, Head
Intrica wood carvingte
Photograph taken in retro diner
Nighttime illumination
Intense athlete
Image with woven fabric design
Detailed illustration of futuristic bioreactor
Image with shattered glass reflections photo of food
Image with holographic retro gaming aesthetics
Antique historical artifact
Layer 23, Head 2
Layer 23, Head
Image showing prairie grouse
Bustling city square
Image with a penguin magnolia
Serene park setting
Warm and cozy indoor scene
An image with dogs
Modern airport terminal
An image with cats
Remote hilltop hut
Layer 23, Head 4
Layer 23, Head
Playful siblings
Intertwined tree branches
A photo of young person
Flowing water bodics
Image with three people
A meadow
photo of
A smoky plume
photo of man
Blossoming springtime blooms
Layer 23, Head 6
Layer 23, Head 7
Picture taken in Sumatra
A paddle
Picture taken in Alberta, Canada
A ladder
Picture taken the geographical location of Spain
Intriguing and enigmatic passageway
Image taken New England
A bowl
Photo captured in the Arizona desert
A table
Layer 23, Head 8
Layer 23, Head 9
Photograph with red color palette
ornate cathedral
An image with cold green tones
detailed reptile close-up
Timeless black and white Image with a yellow color
Image with seagull A clover
Photograph with blue color palette
Futuristic space exploration
Layer 23, Head 10
Layer 23, Head 11
Image with six subjects
A photo with the letter N
Image with a four people
A photo with the letter
An image of the number 3 An image of the number 10 number fifteen
Serendipitous discovery
A fin
Unusual angle
Layer 23, Head 12
Layer 23, Head 13
Image with polka dot patterns Striped design
Photo taken in museum
Surreal digital collage
Checkered design
Cinematic portrait with dramatic lighting
Artwork with pointillism technique
Collage of vintage magazine clippings
Photo taken in Galápagos Islands
Candid documentary photography
Layer 23, Head 14
Layer 23, Head 15
image with dogs
Resonant harmony
Majestic soaring birds
Subtle nuance
Graceful swimming fish
An image of cheeks emotional candid gaze
An image with bikes
Picture with boats
Whimsicachildren'ss scenel
Table 10: Top-5 results of TEXTSPAN. Applied to the heads at layer 23 of CLIL-VIT-L
19

---

Published as a conference paper at ICLR 2024
Layer 22, Head A "location" head
Layer 23, Head 12 A "texture" head
Layer 23, Head 4 "people" head
PPoto takee in museum"
"Serene countryside sunrise"
"Checkered design"
Artwork with pointilliam technique'
"An image friends hanging out"
"A photo of woman"
"Quaint villages'
"Image polka dot patterns"
"Striped design"
"Joyful toddlers'
museum'
woman"
Figure 9: Additional joint decomposition examples.
Input Image
rollout
attention
GradCAM
partial LRP
Chefer et al.
Ours
"An image
echidna"
"An image of tiger"
"An image ram (adult male sheep)"
Figure 10: Comparison to other explainability methods. The highlighted regions produced by our decomposition are more aligned with the areas of the image that are mentioned text.
base
ours
ViT-B-16
76.7
83,8
VIT-L-14
73.1
84.2
ViT-H-14
77.0
84.1
Overall classification accuracy Waterbirds dataset. We reduce spurious cues by zeroing the direct effects property-specific heads.
Table 11:
water background
land background
waterbird class landbird class
92.1 (93.1)
77.8 (66.2)
72.9 (47.7)
94.9 (94.8)
Table 12: Zero-shot classification accuracy on Waterbirds dataset, per class and background (ViT-L). The accuracy for the baseline CLIP model is in parentheses. As shown, reduce the spurious correlation between the background and the object class.
water background land background 62.3 (69.8)
waterbird class landbird class
43.3 (37.2)
87.9 (71.0)
98.0 (96.4)
Table 13: Zero-shot classification accuracy on Waterbirds dataset, per class and background (VTT-H). The accuracy for the baseline CLIP model is in parentheses.
20

---

Published as a conference paper at ICLR 2024
water background
land background
waterbird class
80.5 (86.1)
81.6 (63.5)
landbird class
57.5 (45.6)
94.3 (96.1)
Table 14: Zero-shot classification accuracy Waterbirds dataset, per class and background (ViT-B). The accuracy for the baseline CLIP model is in parentheses.
Layer 23, Head 7
Layer 23, Head
Layer 21, Head 10
paddle
Image showing prairie grouse
Timeless black and white
Image penguin
Vintage sepia tones
Intriguing and enigmatic passageway
magnolii
Image with red color
A bowl
An image with dogs
A charcoal gray color
table
An image with cals
Soft pastel hues
shoelace
irregular hexagon
Picture taken
Hands embrace
parabola
Peaceful lakeside retreat
confemplative mountain view
Image with zebra
Image with traditional African motifs
roof
A spiky texture
Photograph with blue color palette
semiciroular
Picture mammels
Galápagos
Figure 11: Top 3 images with highest similarities to TEXTSPAN outputs. For 3 randomly se- lected attention heads, retrieve the images with the highest similarity score between their head contributions hed and the top 10 text representations found by our algorithm.
21

---

Published as a conference paper at ICLR 2024
Layer Head
(a "people" head)
TOULS
Input Image
Layer 22, Head 2 la "location" head)
Layer Head "oolor" head)
Layer 22, Head 6 (an "animal"
Input Image
Layer 22, Head
(an "object" head)
Layer 23, Head 12 (a "texture" head)
Layer 22, Head 7 La "season" head)
Input Image
Layer 23, Head
la "people" head)
Layer 23, Head 6 la "geo-locsteoe pecoloocation" head]
KIRZLLA
Layer Head
"shape" head)
Input Image
Layer 22, Head 8 (a "letter" head)
AAA
354
Layer Head "color" head]
Figure 12: Additional results for image retrieval based on head-specific similarity.
22


Reviews:
[{"review_id": "MFNLsis9r2", "reviewer": "Reviewer_ETim", "summary": "This paper delves deep into the famous CLIP vision-langauge model for understanding its image-encoder components with the help of text representation. This includes understanding the role of attention heads, self-attention layers, MLP layers as well as the effect of each layer on the final representation in terms of downstream task performance. As the image backbone is completely aligned with text embedding representations, this work makes advantage of this property to interpret the image encoder components. \nInterestingly, each head role is associated with capturing specific attributes and properties, which leads to several downstream applications including, image retrieval with specific properties defined by head and removing spurious features by neglecting the heads which focuses such features.\nThe analysis is performed on CLIP models with various scales, which shows the generality of this study.", "questions": "Please refer to the weaknesses section! Thank you.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) This paper presents important timely study to understand the inner working of image backbones of the large scale vision-language models like CLIP which has become a very popular paradigm. This will help to improve the next generation of such models in terms of architecture and training.\n2) The use of text to interpret the image backbone components is motivating by the fact that CLIP backbone understands text representations. \n3) Extensive analysis shows the main performance drivers of the image backbone of CLIP, and could help in rejecting the redundant modules present in such vision-language architectures.\n4) The proposed TextBase algorithm to associate specific roles per head using text is fairly motivated and its effectiveness is shown via downstream retrieval experiments.\n5) This analysis unlocks several improvements for downstream tasks including reducing known spurious cues and zero-shot segmentation.\n5) Finally the paper is well written and easy to understand.", "weaknesses": "I could not think much about any significant weaknesses in this work. I have some questions as follows:\n\n1) How the zero-shot results compare against from methods like MaskCLIP [1]?\n2) It has been shown that the zero-shot accuracy of the embeddings from only late attention layers is very competitive to the original performance. Will this also hold true where the representations are used for downstream tasks which require adaption? For example, it will be good to see the linear probe performance of the filtered embeddings on imagenet or other datasets like CIFAR100, Caltech101. \n\n\n\n\n\n\n\n[1] Extract Free Dense Labels from CLIP, ECCV 2022, Oral", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6b53O1BQRR", "reviewer": "Reviewer_uT2r", "summary": "The paper delves into the analysis of the CLIP image encoder, breaking down the image representation by considering individual image patches, model layers, and attention heads. Using CLIP's text representation, the authors interpret these components, discovering specific roles for many attention heads, such as identifying location or shape. They also identify an emergent spatial localization within CLIP by interpreting the image patches. Leveraging this understanding, they enhance CLIP by eliminating unnecessary features and developing a potent zero-shot image segmenter. The research showcases that a comprehensive understanding of transformer models can lead to their improvement and rectification. Furthermore, the authors demonstrate two applications: reducing spurious correlations in datasets and using head representations for image retrieval based on properties like color and texture.", "questions": "Questions:\n1. “all layers but the last 4 attention layers has only a small effect on CLIP’s zero-shot classification accuracy” maybe just because the early layers’ feature are not semantically distinctive? But they should be still important to extract low-level features.\n2. Is it possible to achieve the “dual” aspect of the text encoder of the CLIP: 1) find layer-wise importance of the text encoder; 2) find and remove redundant heads to reduce spurious correlations; 3) perform head-based text retrieval based on query images.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength:\n1. Paper is well-organized\n2. The proposed analysis (importance of different attention layers) and two use cases (removal of spurious correlations and head-based image retrieval) are interesting.", "weaknesses": "Weakness:\nNo ablation studies on the impact of the pool size (M) and the basis size (m) on the performance of the decomposition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EwGCcxaY2g", "reviewer": "Reviewer_eYDd", "summary": "The paper ’Interpreting CLIPs Image Representation via Text-based Decomposition’ studies the internal representation of CLIP by decomposing the final [CLS] image representation as a sum across different layers, heads and tokens. Using direct effect to ablate certain layers, the authors first understand that the late attention layers are important for the image representation. Furthermore, the authors further study the attention heads and token decompositions in the later layers to come up with two applications: (i) Annotating attention heads with image-specific properties which was leveraged to primarily reduce spurious cues; (ii) Decomposition into image tokens enabled zero-shot segmentation.", "questions": "Check Cons / Questions;\n\nOverall, the paper is an interesting take on understanding the internal representations of CLIP with the additional benefit of showing applications on image retrieval and reducing spurious correlations.  I am leaning towards acceptance and happy to increase my score if the authors adequately respond to the questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is one of the well executed papers in understanding the internal representations of CLIP. The writing is excellent and clear! \n- The text-decomposition based TextBase is technically simple, sound and a good tool to annotate internal components of CLIP with text. - This tool can infact be extended to study other non-CLIP vision-language models too.\n- Strong (potential) applications in removing spurious correlation, image retrieval or zero-shot segmentation.", "weaknesses": "Cons / Questions:\n- While the authors perform the direct effect in the paper, can the authors comment on how indirect effects can be leveraged to understand the internal representations? I think this is an important distinction to understand if understanding the internal representations in more detail can unlock further downstream capabilities. If affirmative, what downstream capabilities will be feasible?\n- I am not sure if the current way to create the initial set of descriptions is diverse enough to capture “general” image properties or attributes. I believe the corpus of 3.4k sentences is too small for this analysis. While this set is a good starting point, can the authors comment on how this set can be extended to make it more diverse?\n- Did the authors analyse the OpenAI CLIP variants using this framework? The OpenCLIP and OpenAI variants are trained on different pre-training corpus, so a good ablation is to understand if these properties are somehow dependent on the pre-training data distribution. \n- Can the authors comment on how the images set in Sec. 4.1 is chosen? This is not very clear from the text. Is this set a generic image set that you use to obtain m text-descriptions per head from a bigger set of text-descriptions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0GI6TMqM9G", "reviewer": "Reviewer_bDCR", "summary": "The research dissects the CLIP image encoder, identifying specialized roles of its internal components, and reveals an inherent spatial awareness in its processing by using the CLIP’s text representation. Insights gained from this analysis facilitate the removal of extraneous data and the enhancement of the model, notably demonstrated through the creation of an effective zero-shot image segmenter. This underscores the potential for in-depth, scalable understanding and improvement of transformer models.", "questions": "1. Human AI teaming. \n- I am thinking about the task from a human-AI taming point of view. How hard is it for the human to identify the role of each head? Is it possible to introduce some humans during inference time to prune/select which head to use for final prediction?\n2. “Note everything has a direct role” \n- Does it mean there exists complicated features that we need to leave it as a black box, or we can ignore them as redundant feature points?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Well written text\n2. Excellent figure to explain the pipeline. (Fig 1)\n3. Tested on various backbone and datasets\n4. Carefully designed ablation study to show why we should focus on MSA blocks instead MLP blocks. This also provides nice reasoning for the decomposition algorithm design. \n\n5. An excellent way to provide explanations to researchers to understand the trained models. Instead of providing an enormous amount of random feature number to explain the model, the proposed method is able to align the reasoning to text. This could serve as a nice tool to collaborate with human researchers. \n6. Smart way to utilize ChatGPT 3.5 to provide text prompts", "weaknesses": "1. Seems like human users are still required to provide some sort of heuristic to decide the role of a head. I would like to know how hard it is from the user point of view. \n2. Seems like most of the experiments have been done on general image datasets. I am curious about the results on some fine grained tasks or datasets for example human face recognition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)