Interpreting the Second-Order Effects of Neurons in CLIP
Authors: Yossi Gandelsman
Alexei A Efros
Jacob Steinhardt
Venue: ICLR 2025 Poster (2025)
Keywords: CLIP
neurons
interpretability
Primary area: interpretability and explainable AI
TL;DR: We present the second-order lens to interpret neurons in CLIP with text, and use it for segmentation, properties discovery in images, and mass production of semantic adversarial attacks.

Abstract: We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation, outperforming previous methods. Our results indicate that a automated interpretation of neurons can be used for model deception and for introducing new model capabilities

Published as a conference paper at ICLR 2025
INTERPRETING THE SECOND-ORDER EFFECTS OF NEURONS IN CLIP
Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt UC Berkeley
(yossi _aande 1 sman, aaef ros, @bberkkeeyy edu
ABSTRACT
We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) the indirect effeets (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for < 2% of the images. Moreover, each effect be approximated by single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior-eace neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass- passs-prouuce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation, outperforming previous methods. Our results indicate that an automated interpretation of neurons can used for model deception and for introducing new model capabilities'.
INTRODUCTION
Automated interpretability of the roles of components in neural networks enables the discovery of model limitations and interventions to overcome them. Recently, such a technique was applied for interpreting the attention heads in CLIP (Gandelsman al., 2024), widely used class of image representation models (Radford et al., 2021). However, this approach has only scratched the surface, failing to explain major set of CLIP's components- neuronss. Here we will introduce new interpretability lens for studying the neurons and use the gained understanding for zero-shot segmentation and mass-pro mass--rrouuction semantic adversarial examples.
Interpreting the neurons in CLIP is harder task than interpreting the attention heads. First, there are more neurons than individual heads, which requires more automated approach. Second, their direct effect on the output-the flow from the neuron, through the residual stream directly to the output-is negligible (Gandelsman et al., 2024). Third, most information stored redundantly-mann neurons encode the same concept, so just ablating neuron (i.e. examining indirect effects) does not reveal much since other neurons make up for it.
The limitations presented above mean that can neither look at the direct effect the indirect effect to analyze single neuron. To address this, we introduce a "second-order lens" for investigating the second-order effect of a neuron its total contribution the output, flowing via all the consecutive attention heads (see Figure 1).
We start by analyzing the empirical behavior of second-order effects of neurons. We find that these effects have high significance in the late layers. Additionally, each neuron is highly selective: its second-order effect significant for only a small set (about 2%) of the images. Finally, this effect can be approximated by single direction in the joint text-image representation space of CLIP (Section 3.3).
Project page: https://////...... /ots::///osss yossiisssssssooooo gandel sman, www..mmitub......

---

Published as a conference paper at ICLR 2025
Second order effects of CLIP's neurons
MLP 0
MLP L-1
MLP
MSA O
MSAL
P
Text extt-based sparse neuron decomposition
Automatic generation of adversarial examples
dog elephant
"A cat lounging in the sun, with group of elephants
CLIP's prediction:
3. value cabbage
Language Model
Text-to-lmage Model
value sign in the foreground"
background and
Value
dog: 65%
cal: 35%
32. sun
Figure Second order effects of CLIP's neurons. Top: We analyze the second-ordes effects of neurons in CLT (flow in pink). Bottom-left: Each second-order effect of neuron can be decomposed to a sparse set word directions the joint text-image space. Bottom-right, co- appearing words in these sets can be used for mass-generation of semantic adversarial images.
As each direction that corresponds to neuron lives in a joint representation space, it can be decomposed as sparse of text representations that describes the neurons' functionality (see Figure 1). These text representations show that neurons are polysemantic (Elhage et al., 2022) each neuron corresponds to multiple semantic concepts. To verify that the neuron decompositions are meaningful, we show that these concepts correctly track which inputs activate given neuron (Section 4).
The polysemantic behavior of neurons allows us to find concepts that inadvertently overlap in the network, due to being represented by the same neuron. We use these spurious cues for mass production of "semantic" adversarial examples that will fool CLIP (see bottom of Figure 1). We apply this technique automatically produce adversarial images for variety of classification tasks. Our qualitative and quantitative analysis shows that incorporating spuriously overlapping concepts in an image deceives CLIP with a significant success rate (Section 5.
The text representations that describes the neurons' functionality enable additional application- zero-shot segmentation. Mining for text representations of class names, we can identify class-relevant neurons with the second-order lens. Averaging the activation patterns of such neurons, we generate attribution heatmaps. Binarizing them yields a strong zero-shot image segmenter that outperforms recent work (Chefer et al., 2021; Gandelsman et al., 2024).
In summary, we present an automated interpretability approach for CLIP's neurons by modeling their second-order effects and spanning them with text descriptions. We these descriptions to automatically understand neuron roles and apply this to two applications. This shows that scalable understanding of internal mechanisms both uncovers emror and elicits new capabilities from models.
RELATED WORK
Contrastive vision-language models. Models like ALIGN (Jia al., 2021). CLIP (Radford et al., 2021), and its variants (Zhai et al., 2023; Li et al., 2023) produce image representations from pre- training on images and their captions. They demonstrated impressive zero-shot capabilities for various downstream tasks, including OCR, localizzzz,,,,,, and classification (Wortsman, 2023). These models' representations are also used for segmentation (Lüddecke & Ecker, 2022), image generation (Ramesh et al., 2021; Rombach et al., 2022) and 3D understanding (Kerr et al., 2023). We aim to reveal the roles of neurons such models.
Mechanistic interpretability of vision models. Mechanistic interpretability aims to reverse engineer the computation process neural networks. In computer vision, this approach was applied to model individual network components (Shah et al., 2024) and to extract intermediate mechanisms like curve detectors (Olah et al., 2020), object segmenters (Bau et al., 2019; 2020), high-frequency boundary detectors (Schubert et al., 2021), and multimodal concepts detectors (Goh et al., 2021). More closely to us, few works made use the intrinsic language-imsn -mage space of CLIP to interpret the direct effect of attention heads and the output representation CL.PP with automatic text descriptions (Gandelsman
2

---

Published as a conference paper at ICLR 2025
al., 2024; Bhalla et al., 2024). We go beyond the output and direct effects of individual layers to interpret intermediate neurons in CLIP.
Neurons interpretability. The role of individual neurons (post-non-linearity single channel acti- vations) is broadly studied in computer vision models (Bau et al., 2019; 2020; Goh et al., 2021) and language models (Radford al., 2017; Geva et al., 2021; Meng al., 2022). (Dravid et al., 2023; Gurnee et al., 2024) demonstrate that neurons can learn universal mechanisms across different models in both domains. Elhage et al. (2022) show that neurons can be polysemantic (i.e. activated on multiple concepts) and exploit this property for generation of L2 adversarial examples. Some work seeks to extract neurons' concepts by learning sparse dictionaries (Bricken et al., 2023; Raja- manoharan et al., 2024). Other methods use large language models to automatically describe neurons based on which examples they activate on (Bills et al., 2023; Oikarinen & Weng, 2023; Hernandez et al., 2022; Shaham et al., 2024). In contrast, we focus on the contribution of neurons to the output representation.
SECOND-ORDER EFFECTS OF NEURONS
We start by presenting the CLIP-ViT architecture. Then, we derive the second-order effect of neurons and present their benefits over first-order and the indirect effects. Finally, we empirically characterize the second-order effects, setting the stage for automatically interpreting them via text in Section 4.
3.1 CLIP-VIT PRELIMINARIES
Contrastive pre-training. CLIP is trained via contrastive loss to produce image representations from weak text supervision. The model includes an image encoder Mimage and text encoder Mest that map images and text descriptions to a shared latent space RRd, The two encoders are trained together to maximize the cosine similarity between the output representations Mingo (T) and Mea(() for matching input text-image pairs (t, I):
ti, (Minage (f), Mgxa(t)) Mimage (aammn:(())))
(1)
Using CLIP for zero-shot classification. Given set of classes, each name of a class (e.g. the class "'og"") is mapped to fixed template template(c)) (e.g. "A photo of (class]"), and encoded via the text encoder Men(terppaaee(c;) The classification prediction for a given image is the class e; whose text representation is most similar to the image representation: arg max, sim(I, template(c))).
CLIP-ViT architecture. The CLIP-ViT image encoder consists of a Vision Transformer followed by a linear projection The vision transformer (ViT) is applied to the input image E H x3 to obtain a d' dimeeniioonll representation VIT Denoting the projection matrix by P E Raxd'
Minge (I) (I))T
The input to VIT is first split into K non-overlapping image patches that are encoded into K d' d'-dimensional image tokens. An additional learned token, named the class token, is included and used later as the output token. As shown in Figure 1, the tokens are processed simultancously by applying L alternating residual layers of multi-head self-attention (MSA) and MLP blocks.
MLP neurons in CLIP. The MLP layers are applied separately on each image token and the class token. They consist of an input linear layer, parametrized by W E Ng.x xx'" followed by a GELU in
lin-liiyyy and an output linear layer, parametrized by Wi out E Rd' X Here / is the layer number and N is the width (number of neurons) of the MLP. We next analyze the contributions of each individual neuron E [1,..., N) for each layer.
3.2 ANALYZING THE NEURON EFFECTS ON THE OUTPUT
Individual neurons have different types of contributions to the output-the first-order (direct) effects, second-order effects, and (higher-order) indirect effects. We introduce them and explain the limita-
Throouhoou the paper, ignore layer-normalization terms simplify derivations. We address layers- normalization in detail in Appendix A.6.
3

---

Published as a conference paper at ICLR 2025
First order
MSA I
60
Second order
W Head
Accuracy (%)
MLP
W
Head 2
50
baseline
w/o all neurons
40
w/o large norms
Wh
Head
Projection
w/o small norms
rec. #1
p
8 9 10
Layer index
Figure 2: First/Second-order effects. The first order is the flow coming from neuron to the projection layer and the output (blue). The sec- ond order goes from single neuron through all the consecutive attention heads, to the projection layer, and to the output (pink).
Figure 3: Mean-ablation of second order ef- fects (ViT-B-32). We evaluate the performance on ImageNet validation set. Second-order effects concentrate in late layers, significant for only part of the images, and can be approximated by one direction in the output space.
tions of the direct and indirect effects before continuing to characterize the second-order effects in Section 3.3.
First-order effects (logit lens (nostalgebraist, 2020)). The first-order effect is the direct contribution of component to the residual stream, multiplied by the projection layer (see blue flow in Figure 2). For an individual neuron n in layer let p, (I) E R denote its post-GELU activation on the i-th token of the input image T. Then the contribution ef of the n-th neuron to the i-th token in the residual stream is:
c =pppp (I)oolin
where w E Rdd is the the n-th column of W out As the output representation is the class token (indexed 0) multiplied by P, the first-order effect for
the output is Peo
As observed by Gandelsman et al. (2024), the first-order effects MLP layers are close to constants in CLIP and most of the first-order contributions are from the late attention layers. We therefore focus on the second-order effects: the flow of information from the neurons through the attention layers.
Second-ordes effects. The contribution e to the residual stream directly affects the input to later layers. We focus the flow of e/.T through subsequent MSAs and then to the output (pink flow in Figure 2). We call this interpretability lens the "second-order lens", in analogy to the "logit lens".
Following Elhage et al. (2021), the output of an MSA layer MSA' that corresponds to the class token is weighted sum of its K input tokens [20. 2x|:
K
MMA ([20 =k])
ΣΣ
Σg
d
where Who Rdd xdd' transition matrices (the OV matrices) and a ( (T) E R are the attention weights from the class token to the i-th token (> a, 1).
To obtain the second-order effect of neuron n at layer 1, di, (T), we compute the additive contribution of the neuron through all the later MSAs and project it to the output space via P. Plugging in Equation (3) as the contribution to 24 in Equation (4) and summing over layers, the second order
effect of neuron is then:
{,(1)
ΣΣ
p (a
(PWtouu
(5)
1'=41/==
atlention weighted activations
ingut indepennentt
Indirect effects. An alternative approach is to analyze the indirect effect of a neuron by measuring the change in output representation when intervening is done by replacing the activation p.."' of the neuron for each token with pre-computed per-token mean. However. as was shown by McGrath et al. (2023), models often learn "self-repair" mechanisms that can obscure the individual roles of neurons. We illustrate these issues in the next section.
neuron's output. Specifically, the intervention
4

---

Published as a conference paper at ICLR 2025
60
Accuracy (55)
55
50
variance explained by first PC
Baseline (rec. from PC #11
effect type
accuracy after mean-ablation
45
common (10k)
common words (30k)
40
IN Class Descriptions
indirect second-order
52.3
11.0 48.2
2
29.6
Descriptions per Neuron
Table 1: Comparison to indirect effect. We compare the second-order effects and the indi- rect effects by mean-ablating layer 9 in ViT-B-32 on ImageNet validation set.
Figure 4: Accuracy for neuron reconstructed from sparse text representations (ViT-B-32, layer 9). We evaluate the sparse text decompo- sitions for different initial description pools and description set sizes.
3.3 CHARACTERIZING THE SECOND-ORDER EFFECTS
We analyze the empirical behavior of the second-order effects of neurons o', derived in the previous section. We find that only neurons from the late MLP layers have significant second-order effect and that each individual neuron has a significant effect for less than 2% of the images. Finally, we show that di can be approximated by one linear direction in the output space. These findings will help motivate our algorithm for describing output spaces of neurons with text in Section 4.
Experimental setting. To evaluate the second-order effects and their contributions to the output representation, we measure the downstream performance on the ImageNet classification task (Deng al., 2009) after ablating these effects for each neuron. Specifically, we apply mean-ablation (Nanda et al., 2023), replacing the additive contributions of individual &¹, (T)'s 's to the representation with the mean computed across a dataset D. our experiments, we mean-ablate all the neurons layer simultaneously and evaluate the downstream classification performance before and after ablation. Components with larger effects should result in larger accuracy drops.
We take D to be ~ 5000 images from the ImageNet training set. We report zero-shot classification accuracy on the ImageNet validation set. Our model is OpenAl's ViT-B-32 CLIP, which has 12 layers. We present additional results for ViT-L-14 and for ImageNet-R (Hendrycks et al., 2021) in Appendix A. and Figure 10.
Second-order effeets concentrate in moderately late layers. We evaluate the contributions of all the o across different layers and observe that the neurons with the most significant second-order effects appear relatively late in the model. The results for different layers in ViT-B-32 CLIP model presented in Figure 3 ("w/o all neurons")). As shown, mean-ablating layers 8-10 leads to the largest drop in performance. These layers appear right before the MSA layers with the most significant direct effects, as shown in Gandelsman et al. (2024) (layers 9-11; 1-11; see Appendix A.2). The same trend is preserved for a larger model size as well (see Appendix A.
The second-order effect is sparse. We find that the second-order effect of each individual neuron significant only for less than 2% of the images across the validation set. We repeat the same experiment as before, but this time we only mean-ablate o', (T) for subset of images, while keeping the original effects for other images. For most of the images, except the subset of images in which di, (T) has a large norm, we can mean-ablate of (T) without changing the accuracy significantly, as shown in Figure 3 ("w/o small norm"). Differently, mean-ablating the contributions for the 100 images with the largest o', (I) norms results in significant drop in performance ("w/o large norm"). The same trend is shown for images from ImageNet-R Figure 10.
The recond-ordes effect is approximately rank 1. While the second-order effect for given neuron can write to different directions in the joint representation space for each image, we find that ø, (I) can be approximated by one direction rf. d in this space, multiplied by coefficient & (I) that depends on the image. We use the set S which contains the largest second-order effects in
norm from D. and set ri, to be the first principle component computed from St We approximate
5

---

Published as a conference paper at ICLR 2025
Neuron
ImageNet class descriptions
Common words (30k)
"Picture with falling snowflakes"
+"snowy"
"Picture portraying a person [..]] extreme weather conditions"
"frost"
#4
"Picture with a bucket in construction site"
"closings"
+ *Photograph taken during holiday service"
+ "advent""
+"Image with a traditional wooden sled"
*"woodworking"
+"Image with wooden cutting board"
"swelling"
#391
*Picture showcasing beach accessories"
+"cedar"
"Photograph with syringe and surgical mask"
""heirloom"
+"Photo with lime garnish"
*"refreshments"
+ +IImage with candies in glass containers"
+ "gelatin"
#2137
"Picture fcaturing lifehoat equipment"
+"sour"
+ *Close-up photo of melting popsicle"
+ "cosmopolitan"
"Photo that features stretch limousine"
"motorhome"
"Image capturing suit with pinstripes"
"yacht"
#2914
"Caricature with celebrity endorsing the brand"
"cirrus"
"Image showcasing a Bullmastiff's s prominent neck folds"
"cabriolet"
Table 2: Examples of sparse decompositions (ViT-B-32, layer 9). We present the top-4 texts corresponding to the sparse decomposition of each neuron and the signs of the decomposition coefficients, for two initial pools (m = 128). See Table 5 for more neurons.
o. (I) with x., (T)rk, + by, where b/, E Rd is the bias computed by averaging (T) across D, and F. (T) E R is the norm of the projection of øi onto r/,
To verify that this approximation recovers o', (T) we replace each &% (I) for each neuron and image in the validation with the approximation. We then evaluate the downstream classification per- formance. As shown in Figure 3 ("reconstruction from PC #1"), for each layer I, this replacement results in negligible drop in performance from the baseline, that uses the full representation. The same behavior is observed for ViT-L model and for a different initial set of images in the Appendix.
Comparison indirect effeet. We compare the second-order effect the indirect effect and present the variance explained by the first principle component for each of them and the drop in performance when simultaneously mean-all aeen-aalatinng all the effects from one layer. As shown Table Mean-ablating the indirect effects results in a smaller drop in performance due to self-repair behavior. Moreover, the first principle component explains significantly less of the variance in the indirect effect, than in the second-order effect. This demonstrates two advantages of the second-order effects- uncovering neuron functionality that is obfuscated by self-repair, and one-dimensional behavior that can be easily modeled and decomposed, as will show in the next section.
4 SPARSE DECOMPOSITION OF NEURONS
We aim to interpret each neuron by associating its second-order effect with text. We build on the previous observation that each second-order effect of neuron d', is associated with vector direction r Since r lies in a shared image-text space, we can decompose it to a sparse set of text directions. We sparse coding method (Pati et al., 1993) mine for small set of texts for each neuron, of large pool of descriptions. We evaluate the found texts across different initial pools with different set sizes.
Decomposing a neuron into a sparse set of descriptions. Given the first principal component the second-order effect of each neuron, r/. will decompose
sparse sum of text directions ty: 7 = P/, Σラ 2 (ty) To do this, we start from large pool T of M text descriptions (c.g. the most common words in English). We apply sparse coding algorithm to approximate r', the sum above, where only m of the
7 1.n are non-zero, for some M.
Experimental settings. We verify that the reconstructed Pl, from the text representations captures the variation in the image representation, measured by zero-shot accuracy ImageNet. We simul-

---

Published as a conference paper at ICLR 2025
Neuron #4
Neuron #391
Neuron #2137
Neuron #2914 14
Figure 5: Images with largest second-order effect per neuron. We present the top images from 10% of ImageNet validation set for the neurons in Table 2. Note that neurons are polysemantic they have large second-order effects on images that show multiple concepts (e.g. cars and boats). See top-50 images in Figure 13.
tancously replace the neurons' second-order contributions in single layer with the approximation x", (1) + B,
To obtain sparse decomposition for each neuron, we use scikit-learn's implementation of orthogonal matching pursuit (Pati et al., 1993). We consider two strategies for constructing the pool of text descriptions T. The first type single words the 10k and 30k most common words in English. The second type is image descriptions we prompt ChatGPT-3.5 to produce descriptions of images that include object of specific class. Repeating this process for all the ImageNet (IN) classes results in 288 unique image descriptions. We then evaluate the reconstruction of r for different m 'ss and pools.
Effect of sparse set size m and different pools. We experiment with m E [4, 8, 16, 32, 64, 128} and the three text pools, and present the accuracy on 10% of ImageNet validation set in Figure 4. We approach the original classification accuracy with 128 text descriptions per neuron reconstruction P,, Using full descriptions outperforms using single words for the text pool, but the gap vanishes for larger
Qualitative results. We present the images with the largest second-order norms in Figure and the corresponding top-4 text descriptions in Table 2. As shown, the found descriptions match the objects in the top 10 images. Moreover, some individual neurons correspond to multiple concepts (e.g. writing both toward "yacht" and type of car "cabriolet").. This property is even more apparent if more nearest neighbors are presented (see Figure 13 for the top 50 nearest neighbors). This corroborates with previous literature neurons' polysemantic behavior (Elhage et al., 2022) single neurons behave as a superposition of multiple interpretable features. This property will allow generate adversarial images in Section 5.1.
5
APPLICATIONS
5.1
AUTOMATIC GENERATION ADVERSARIAL EXAMPLES
The sparse decomposition of r allows us to find overlapping concepts that neurons are writing to. We use these spurious cues to generate semantic adversarial images. Our pipeline, shown in Figure 1, mines for spurious words that correlate with the incorrect class in CLIP (e.g. "clephant", that correlates with "dog"), combines them into image descriptions that include the correct class name ("cat"), and generates adversarial images by providing these deseriptions model. We explain the steps in the pipeline and provide quantitative and qualitative results.
text-to-image
Finding relevant spurious cues in neurons. Given two classes and c2 first select neurons that contribute the most toward the classification direction = Mcxx(c2) Mces(C))), then mine their sparse decompositions for spurious cues. Specifically, we extract the set of neurons A whose
7

---

Published as a conference paper at ICLR 2025
horse auttooblllo
dog deer
frog bird
truck
ship
ship automobile
Thank
Emission
HAPPY THANKS
horse is driven by trucker on road sign saying "Emissions" the background.
A dog is running through tishery, chasing atter squirrel and rabbit, with sconic view the surrunnding forests.
poodle weasing ring and holding frog.
duo of trainers are run- ning alongside truck, with pirate flag waving in the background, and sea people chearing
shi sailigg thooug a quartet cyclists Brooklyn med, with a hus- band and wife driving mo- toreycle in the background.
standing next to sign saying "Thakksgiving" festive atmosphare.
Thankkn
HAPPY
THANKSGMIN
Figure 6: Adversarial images generated by our method. For each binary classification task, we present the generated images, the input text to the text-to-image model (words from W are bold), and an attribution map (Gandelsman et al., 2024) for the classification (areas that contribute the incorrect class score are red). See additional results in Figure 15.
directions are most similar to v: N top-k, (v, r. Utilizing the sparse decomposition from before, we compute contribution score w for each phrase j in the pool T:
w
Σ BEN
(v.r1))
This looks at the weight that each neuron in N assigns to j in its sparse decomposition, weighted by how important that neuron for classification. A phrase with a high contribution score has significant weight in one or more important neurons, and so is potential spurious cue. The top phrases, sorted by the contribution score are collected into a set of phrase candidates W..
Generating "semantic" adversarial examples. We use text and image generative models to create examples with the object 02 that are classified as 61. First, we generate image descriptions with large language model (LLM) by providing it phrases from the set W and the class name с₁ and prompting to generate image descriptions that include elements from both. We prompt the model to exclude anything related 02 from the descriptions and use visually distinctive words from W..
The resulting descriptions are fed into a -o-tago- model to generate the adversarial images. Note that the adversarial images lie the manifold of generated images, differently from non-semantic adversarial attacks that modify individual pixels.
Experimental settings. We generate adversarial images for classifying between pairs of classes from CIFAR-10 10 (Krizhevsky, 2009). We use the 30k most common words as our pool T. We choose the top 100 neurons from layers 8-10 for N,, and the top 25 words according their contribution scores for prompting the LLM. We prompt LLaMA3 (Touvron et al., 2023) to generate 50 descriptions for each classification task (see prompt in Appendix A.7). We then filter out descriptions that include the class name and choose 10 random descriptions. We generate 10 images for each description with DeepFloyd IF text-to- -o-tgee model (StabilityAI, 2023). This results 100 images per experiment. We repeat the experiment 3 times and manually remove images that include objects or do not include ej objects.
We report three additional baselines. First, we repeat the same process with 100 random neurons instead the set Second, we repeat the same generation process with sparse text decompositions computed from the first principle components of the indirect effects instead of the second-order effect. Third, we do not rely on the neuron decompositions, and instead prompt the language model with the words from M for which their text representations the baselines, we automatically filter out synonyms of c2 from the phrases provided to the language model according to their sentence similarity to c2 (Reimers & Gurevych, 2019).
the most similar to Both for pipeline and
8

---

Published as a conference paper at ICLR 2025
Indirect effect
Similar words
Second order
Task
Random
horse + automobile
1.0 (E:14)
2.8 (4:3.7)
1.0 (2:1.4)
5.3 (::1.9)
dog + deer
0,3 (1005)
6,3 (2:48)
3.3 (2:0.)
22.7 (1:0.)
bird frog
0.3 (20.5)
1.0 (#114)
5.0 (#22.9)
8.0 (b455)
ship - truck
0.0 (2000)
0.0 (#:00)
0.0 (=:0.0)
5.7 (b0.9)
ship - automobile
1.3 (2119)
0.0 (2000)
1.3 (220,))
7.0 (2:4.5)
Table Accuracy adversarial images. We report how many generated images out of 100, fooled
the binary classifier (standard deviation in parentheses).
Pix. Acc.
mloU
mAP
Partial-LRP (Voita al., 2019)
55.0
35.5
66.9
Rollout (Abnar & Zuidema, 2020)
61.8
42.6
74.0
LRP (Binder et al., 2016)
62.9
35.8
68.5
GradCAM (Selvaraju et al., 2017)
67.3
39.3
61.9
Chefer al. (2021)
68.9
49.1
79.7
Raw-attention
69.6
49.8
80.0
TextSpan (Gandelsman al., 2024)
76.5
58.1
84.1
Ours
78.1
59.0
84.9
Table 4: Segmentation performance on ImageeNet-segmentaatee Our zero-shot segmentation is more accurate than previous methods across all metrics.
Quantitative results. The classification accuracy results for the adversarial images are presented in Table 3. The success rate of our adversarial images is significantly higher than the indirect effect baseline, the similar words baseline, and the random baseline, which succeeds only accidentally. For the task of generating "ship" images the will be missclassified "truck", no other baseline manged to generate any adversarial images, while ours generated 5.7 images on average.
Qualitative results. Figure 6 includes generated adversarial examples and the descriptions that were used in their generation. The presented attribution heatmaps (Gandelsman et al., 2024) show that the found spurious objects from W, contribute the most the misclassification, while the object from the correct class (e.g. horse in the left-most image) contributes the least. We provide more results for additional classification tasks (e.g. "stop-sign v.s. yield") in Figure 15.
We show that understanding internal components models can be grounded by exploiting them for adversarial attacks. Our attack is optimization-free and compute-intensin Hence, it can be used for measuring interpretability techniques, with better understanding leading to improved attacks.
5.2 ZERO-SHOT SEGMENTATION
Finally, we use our understanding of neurons for zero-shot segmentation. Each neuron corresponds to an attribution map, by looking at its activations p, (I) on each image patch. Ensembling all the neurons that contribute towards concept results in aggregated attribution map that can be binarized to generate reliable segmentations.
Specifically, to generate segmentation map for image I, find set of neurons with the largest absolute value the dot product with the encoded class name c; we aim to segment: | Misa (c)))l- We then average their spatial activation maps Pt I.n (T), standardize the average activations into [0, 1], and binarize the values into foreground/backgrooudd segments by applying threshold of 0.5.
Segmentation results. We provide results on mageeNet-Segmennttion (Guillaumin et al., 2014), which includes foreground/backgromn segmentation maps of ImageNet objects. We use activation maps from the top 200 neurons of layers 8-10. Table 4 presents quantitative comparison to previous explainability methods. Our method outperforms other zero-shot segmentation methods across all standard evaluation metrics. We provide qualitative results before thresholding in Figure 7. While the first-order effects ("TextSpan")) highlight individual discriminative object parts, our heatmaps capture more parts of the full object.

---

Published as a conference paper at ICLR 2025
Input image
TextSpan
Ours
Figure 7: Qualitative results on ImageNet--Sgmeentttin (ViT-B-32). Our heatmaps capture more object parts than the first-order token decomposition of Gandelsman et al. (2024).
LIMITATIONS AND DISCUSSION
We analyzed the second-order effects of neurons on the CLIP representation and used our understand- ing to perform zero-shot segmentation and generate adversarial images. We present mechanisms that did not analyze in our investigation and conclude with a discussion of broader impact and future directions.
-attonnn maps mechanisms. We investigated how the neurons flow through individual consecutive attention values, and ignored the effect of neurons on consecutive queries and keys in the attention mechanism. Investigating these effects will allow to find neurons that modify the attention map patterns. We leave it for future work.
Neuron-neuron mechanisms. We did analyze the mutual effects between neurons in the same layer or across different layers. Returning to our adversarial "frog/bird" attack example, a neuron that writes toward "dog" may not be activated if different neuron writes simultaneously toward "frog", thus reducing our attack efficiency. While we can still generate multiple adversarial images, we believe that understanding dependencies between neurons can improve it further.
Future work and broader impact. The mass production of adversarial images can be harmful to systems that rely on neural networks (e.g., the adversarial attack that causes misclassification between "yield" and "stop sign" in Figure 15). Automatic extraction of such cases allows the defender to be prepared for them and, possibly, fine-tune the model on the generated images avoid such attacks. We plan to investigate this approach to improve CL.IP's robustness in future work.
Currently, our attack pipeline relies on a few independent components, each of which has failure modes. For example, the language model can fail to generate a coherent sentence that includes many phrases from W., and can omit the class name 02 or accidentally include the class name e1- Additionally, the text-to-image model can fail to generate image that follows the exact description and can drop crucial elements from the description. We believe that future improvements in the language and vision models will increase the success rate of our attack, and plan to continue to develop and improve it in the future.
Acknowledgments. We would like to thank Alexander Pan for helpful feedback the manuscript. YG is supported by the Google Fellowship. AE is supported in part by DoD, including DARPA's MCS and ONR MURI, well funding from SAP. JS is supported by the NSF Awards No. 1804794 & 2031899.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190 4197, Online, July 2020. Association for Computational Linguistics. doi: aa0.://///22022.lmmaa..... URL https: httpe///aaaaaahhaaaeooee htwpelooyyo.rg///ee wwtp/otg/..o.oo
main.385.
David Bau, Jun- Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
David Bau, Jun Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10. 1t1773//pnas..90777111111 URL https: /ttp//////1111/1111111
//www. pnas. org/ wowtent/early/
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. Interpret- ing clip with sparse linear concept embeddings (splice), 2024.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language mod- els. https: / poppenaipubliccoom bic.blbbbbb......c core. windows net/neuron http://euroooeepllaieeeeeeeeeeee explainer/ paper/index: html, 2023.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Rober Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. Artificial Neural Networks and Machine Learning ICANN 2016, 9887:63-71, 2016. doi: 10. 1007/
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph. Zac Hatfield-Dodds. Alex Tamkin, Karina Nguyen, Brayden Mcleen, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https: pub/ 2023 htt////ooooaatiiiemoo
features/ index. html.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782-791, June 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher. Rosetta neurons: the common units in model zoo. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 1934-1943, October 2023.
Mining
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. hutp::/traassormerrcicciits hbtp://maeeeteeteecciiiiaaiiaala 1fwp:2//22//aaaeoooooo/eeeee htmml.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Suifield-Doddse Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi- tion. Transformer Circuits Thread, 2022. URL https: / ttrans circuite pub/ 1 ndex. html.
11

---

Published as a conference paper at ICLR 2025
Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting CLIP's image representation via text-based decomposition. In The Twelfth International Conference on Learning Representa- tions, 2024. URL htttps://cpenrevieet
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021.
Gabriel Goh, Nick Cammarata +, Chelsea Voss f, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.2391 1 00030. https://dssiillpubb222
nnurons.
Matthieu Guillaumin, Daniel Küttel, and Vittorio Ferrari. Imagenet auto-annotation with segmentation propagation. Int. J. Comput. Vision, 1 110(3):22--33 dec 2014. ISSN 0920-5691. doi: 10.1007/ 11263-001400713-9 URL https: doi. org 10. 111.1011001100 s11//311263331 -14--666--1--113
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaci Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models, 2024.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness; A critical analysis of out-of-distribution generalization. ICCV, 2021.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions deep visual features. CoRR, abs/2201.11114 2022. URL htttt::ttaaaiiitii org/ abs/ 2200 11114.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, 2021. URL
https://aphs http://app.seaaiiiseessee semanticscho arr wttg/oorgueeeeeoooeeee Corpus 8795566.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields, 2023.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language- image pre- treiiriinigg via masking, 2023.
Timo Lüddecke and Alexander Beker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2022.
pp.
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair language model computations, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations GPT. Advances Neural Information Processing Systems, 36, 2022. arXiv:2202.0522 02262262.........
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. 2023.
nostalgebraist, interpreting gpt: the logit lens, 2020. URL /teatt/cccckssoooaaaaa/occtettee
com/
Tuomas Oikarinen and Tsui-Wer Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks, 2023.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/da http://dhhilllp////////eeeiiinoo
12

---

Published as a conference paper at ICLR 2025
Yagyensh C. Pati, Ramin Rezaïfar, and Perinkulam S. Krishnaprasad. Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. Proceedings of 27th Asilomar Conference Signals, Systems and Computers, pp. 40-44 44 vol.1, 1993. URL https:: / /ppp semant inschol ar. .r.arroo .o..la...oroogreeee..ee...eeee.. Corg/orp : 3805.
Alec Radford, Rafal Jozefowicz. and Ilya Sutskever. Learning to generate reviews and discovering sentiment, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 18 24 Jul 2021. URL https: / http/procceee ngs ml p.ressp..sss...... / v1 39/ rad ord2 a.mmml.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ily Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092, 2021. URL https: /ttpii//aaivvv//ia
https::/arxiv..rrgt http:/aaxvv....g/aaa/// batgg//bs//1000 12092.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL http:::://wwwww/rr//eee/o 1908. 10084.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. High-low frequency detectors. Distill, 2021. doi:
edges.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: opw.review.eeeepppepppe... net/ forum -1dM3377 4vmaMcY.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. /CCV, pp. 618-626. IEEE Computer Society, 2017. ISBN 978- 1--388--10322 URL http:
//dblp. wwwuuuiiierrr... .eleerrde////ee coof/oof/////
html 1 #Se va juCDVPB17
Harshay Shah, Andrew Ilyas, and Aleksander Madry. Decomposing and editing predictions by modeling model computation. arXiv preprint arXiv:2404.11554 2024.
Tamar Rott Shaham, Sarah Schwettmann. Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. A multimodal automated interpretability agent, 2024.
StabilityAI. DeepFloyd-IF.
http://///ggttbb.cm///eeeeeeeeee 2f 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ama Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
13

---

Published as a conference paper at ICLR 2025
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 5797-5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18655/11/11191155 URL https: nttps://aclaaae antho wtw.olgyy..ooo1111111
Mitchell Wortsman. Reaching 80% zero-shot accuracy with openelip: Vit-g/14 trained on laion-2b, 2023. URL https:
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 SECOND ORDER ABLATIONS FOR VIT-L
We repeat the same experiments from Section 3.3 for ViT-L-14, trained on LAION dataset (Schuh- mann et al., 2022). For this model, we only use 10% of ImageNet validation set. Here, the maximal drop in performance when ablating the second order is relatively smaller and spread across more layers. Nevertheless, the same properties presented and discussed in Section 3.3 hold for this model.
A.2 FIRST ORDER ABLATIONS
For the two models discussed above, ViT-B-32 and ViT-L-14, we provide the mean-ablation results for the first-order effects of MSA layers, computed in Gandelsman et al. (2024). For each model, we present the performance before and after accumulative mean-ablation of all the first-order effects of MSA layers. As shown in Figure 11 and Figure 12, the neurons with the significant second-order effects appear right before the layers with the significant first-order effects.
A.3 ADDITIONAL ADVERSARIAL IMAGES
We present additional semantic adversarial results, generated by our method for ViT-B-32, in Fig- ure 15. We demonstrate a wide variety of tasks, including additional pairs from CIFAR-I6 10 dataset, and adversarial attacks related to traffic signs (e.g. misclassification between stop sign and yield sign or a crossroad). For each image, we provide the text used for generating it, and highlight the spurious cues words from the sparse decompositions.
A.4 ADDITIONAL SPARSE DECOMPOSITION RESULTS
We provide additional examples of sparse decompositions of neurons in Table 5 and the images with the top norms for the second-order effects of the same neurons in Figure 14, As shown, the found descriptions match the objects in the top 10 images.
A.5 CONCEPT DISCOVERY IN IMAGES
We present additional application concept discovery in images. We aim discover concepts in image I, by aggregating phrases that correspond to the neurons that are activated on /. Here, we start from the of activated neurons N (for which ||% (f)ll2 is above the 98th percentile of norms computed across ImageNet images). Similarly to the contribution score described in Section 5.1, we compute an image-contribution score wi for each phrase j according to its combined weight in the decompositions of neurons in N. Formally, w is the overall sum of weights that each neuron in N assigns in its decomposition, weighted by the neuron second-order norms: w/ = 7 2 1 [2) The phrases with the highest imtte-cont score are picked to describe the image concepts.
0.39 ducks chickans eaple dlover cormel park
0.89 vioiin quitar
sweetwater
golting
1.29 primates
0.50 bridge
0.66 chimp
0.36 log
0.63 alejandro
0 statton
0.50 zoology
0.23 tektronix
0.20 kong
0:23 mel
0.47 bolivts
0:21 postmaster
0.4 inverter
0.21 bridges
0.50 ears
0.20 yagich
0.57 malf
0.20 continually
0.56 chosts
0.20 laaoon
Figure Concept discovery in images (VIT-B-32), We include top-10 words discovered by aggregating words in sparse decompositions of activated neurons.
15

---

Published as a conference paper at ICLR 2025
Qualitative results. We present qualitative results for neurons and the top-10 discovered concepts from layer ViT-B-32 in Figure 8, when using the most common words as the pool. The number of neurons activated on these images, [W] is between 29 and 59, less than 2% of the neurons in the layer. Nevertheless, the top words extracted from these neurons relate semantically to the objects in the image and their locations. Surprisingly, the top word for each of the images appears only in one or two of the neuron sparse decompositions and is not spread across many activated neurons.
We acknowledge that while this application discovers meaningful concepts that correspond the input images, there are other approaches for extract these concepts (e.g. sparsely decomposing the image representation, as shown in Bhalla al. (2024)).
A.6 DERIVATIONS WITH LAYER NORMALIZATION
In many implementations of CLIP, there is layer normalization between the Vision Transformer and projection layer P. In this case, the representation is:
Minage (T) = P(L.N(VT(()))
where the LN is the layer normalization. Specifically, L.N can written as:
I + + LN(x) = = vo +
Mr
B
(8)
+
=B
where E Rd is the imput token, At, 01 R are the mean and standard deviation, and y, R are learned vectors. To include A and B in the second-order effect of a neuron flow, we replace the input-independent component in Equation (5), PW h wn with:
P(A W Wvow
Where is normalization constant that splits B equally across all the neurons that can additively contribute to it.
Except for the layer normalization before the projection layer, the input to the MSA layers that comes from the residual stream also flows through layer normalization. Thus, if the input to the MSA layer layer is the of tokens [.=],. the output that corresponds to the class token is:
H
K
MSA ((=0) ([=0)..,,<I]]]
ΣΣ h=1 1=0
d ((T))
(10)
where LN is the normalization layer at layer I, that can be parameterized similarly to Equation (8)
by A', ; B' E Rdd We modify the definition of the second-order effect accordingly:
d, = ΣE
L
H
K
(
((a)()a((()) PP
(p
(p(A~ A
who + B" +z))+ +
B"
2 (1I)
B
"=++1/=1 i=
where d' is is normalization coefficient that splits 8" equally across all the neurons before layer l'.
In all of our experiments, we use this modification. Most of the elements in the modification add constant biases. Therefore, they can be ignored in our experiments as many the experiments constant biases do not change the results. For example, in our mean-ablation experiment, we subtract mean, computed across a dataset.
A.7 PROMPTS
We provide the prompt that was used for generating sentences given the set of words W. presented in Section 5.1, in Table 6. This prompt is given to LLAMA3 model (Touvron et al., 2023).
Additionally, provide the prompt that was used for generating the pool of ImageNet class descriptions, presented in Section We prompt ChatGPT (GPT 3.5) with the prompt template provided in Table 7.
16

---

Published as a conference paper at ICLR 2025
75.0
50
baseline
72.5
Accuracy (%)
all neurons
Accuracy (86)
70.0
40
large
67.5
baseline
w/o all
30
rec.
wi large norms
62.5
rec. from PC #1
20
9
10
Layerindex
Layer index
Figure VIT-L-14 second-order ablations.
Figure 10: Mean-ablation of second order ef- fects on ImageNet-R (ViT-B-32, layers 8-10), We repeat the evaluation Figure 3 on ImageNet- R. The performance of different ablations follows the same trends as that of ImageNet.
60
50
Accuracy (%)
Accuracy (96
baseline attentions mean-ablation
baseline
attentions mean-ablation
2
3
6
7
8
10
11
2
4
6
8
16 18 20 22 Layer index
10
12
14
Layer index
Figure 11: VIT-B-32 first-order MSAs ablation.
Figure 12: ViT-L-14 first-order MSAs ablation.
Neuron
ImageNet class descriptions
Common words (30k)
"Image with wiry, weather-resistant coat"
+"tents"
"Image showcasing a compact and lightweight sleeping bag"
+ +"svalbard"
#600
"Picture of camper towing bicycles"
+* "miless
"Image with a Border Terrier jumping"
"mountainous"
"Photograph taken during race"
"runners"
"Silhouctte of a running dog"
"races"
#974
"Picture taken in fishing competition"
"dolphin"
"Silhhuuttte of hammerhead shark with other ocean creatures"
"expiration"
+"Chiir with a foot pedal control"
+ +"bus"
"Picture that captures the breed's intelligence"
"filings"
#1517
"Image with snow-capped mountains scenery"
"percussion"
"Image with graffiti on train"
"whhelchairrs"
"Image depicting a sustainable living option"
"genres"
"Photo taken in a train yard"
"governance"
#2002
"Image featuring snow-covered rooftops"
"gravel"
"Rescue equipment"
"conserve"
Table 5: Additional examples of sparse decomposition results. For each neuron, present the top-4 texts corresponding to the sparse decomposition with m = 128 and the signs of the coefficients in the decomposition.
17

---

Published as a conference paper at ICLR 2025
Top
Top 10
Top 15
Top 20
Top 25
Top 30
Top 35
Top 40
Top 45
Top 50
Neuron #4
Neuron #391
Neuron #2137
Neuron #2914
Figure 13: Images with largest second-order effect norm per neuron. We present the top images from 10% ImageNet validation set, for the neurons in Table 2. Notice that additional concepts that not captured by the top-4 descriptions in Table 2 are starting to appear.
A.8 COMPUTE
As our method does not require additional training, the time of our experiments depends linearly on the inference time of CLIP (and other generative models that were used for the adversarial images generation), and on the number of images we use for the experiments (~5000 our case). All our experiments
A100 GPU. The most time-consuming experimentt-computing the per-layer mean-alt ablation results for ViT-L-14 -ookk days.
Neuron #600
Neuron #974
Neuron #1517
Neuron #2002
Figure 14: Images with largest second-order effect norm per neuron. We present the top images from 10% of ImageNet validation set, for the neurons in Table 5.
18

---

Published as a conference paper at ICLR 2025
stop sign
stop sign yleld
stop sign yleld
dog deer
dog deer
yield
STOP
YU'S BANKING
STOP
STOP
dog walking with pat- terned leash through forest with rabbits and squirrets, with symmatri- call patterned free the background.
dog running through forest, chesing after squirrel, with helicopter flying avarhead and pat- terned stream the dis- tance.
group people wandered through marset filled with cans, and perfumes, with
A slop sign slands in front of building with sign that says "Yu's Banking Sar- vicees
A stop sign stands in front of building with sign that seys "Yu's Banking Sar- vicees
the dis-
cat + vacuum cleaner
cat + vacuum
dog cat
frog bird
frog - bird
cleaner
frog back elephant, purple orange a rounding
cal is playing with hockey stick near shovel and injection
A cal brushing ils for with a blunt comb, surrounded drops ethanol and dvr
looking group
pony, wearing a yuri-themed
perched neerby
dog + cat
bird + frog
bird - frog
bird + frog
bird - cat
A TUER
dog sitting moon- light. looking a group owis perched on nearby branch.
bird as awims pool filld with reptiles and butter- flies.
pug's green and flag wavvig in the wind.
through Jungle, searing above
with swimming nearby pond fisherman in distance.
stop sign crossroad
stop sign crossroad
stop sign crossroad
stop sign crossroad
bird-
STOP
STOP
OUSE
GROSSD
UUMZLE
bird foading from hand whil elephants bothh
A stop stands tai gorge. surrounded blocks colorful
A stop sign marks the and of journey. with grandson and his grandlather surounded pertumes blocks.
A stop sign is placed block wood, with chick- siting crossword puzzl
stop sign painted on rock, with chicken perched way leading joumery.
path-
chicken
pathway leading
bant marathon
Figure 15: Additional adversarial examples generated by our method. We provide the sentence that was given to the text-to-image model to generate it. Words from W are highlighted in bold.
19

---

Published as a conference paper at ICLR 2025
You are capable instruction-followingg agent.
I want generate image by providing image descriptions imput text-to-image model.
The image descriptions should be short. Each of them must include the word "[class_]]
They must not include the word "(class_2) __" any synonym of it, plural version! The image descriptions should include as many words as possible from the next list and almost no other words: {list}
Do names of people places from the list unless they are famous and there is something visually distinctive about them. In each of the image descriptions mention many objects and animals as possible from the list above. you want to mention the place which the image taken or name of person, describe it with visually distinctive words. For example, if "Paris" is in the list, instead of saying "... in Paris", say "... with the Eiffel Tower in the background" saying Pariss". Don't mention words that too similar to "(class_2) _ even if they are in the list above. For example, the word was "tree" you should not mention "trees", "bush" or "eucalyptus". Only words that you know what they mean. Generate list of 50 image descriptions.
"... next to sign
Table The language model prompt for generating image descriptions.
Provide 40 image characteristics that are true for almost all the images of [class). Be as general as possible and give short descriptions presenting one characteristic a time that can describe almost all the possible images of this category. Don't mention the category name itself (which is "(class) "") Here are some possible phrases: "Image with texture of ". "Picture taken in the geographical location of..." "Photo that is taken outdoors". "Caricature with text", "Image with the artistic style of...", " "Image with onc/two/three objects", "Illustration with the color palette ...", "Photo taken from above/below". "Photograph taken during ... season". Just give the short titles, don't 't explain why, and don't combine two different concepts (with "or" or "and").
Table The prompt for generating the pool of class deseriptions. We prompt the model with the ImageNet classes.
20


Reviews:
[{"review_id": "O6demcC0Sc", "reviewer": "Reviewer_g9ZC", "summary": "This work presents a novel approach for examining potential second-order effects of neurons of CLIP representations and how these can be used in the context of zero-shot segmentation and generation of adversarial images. \n\nTo this end, the authors focus on the contribution of each individual neuron to the output in terms of second-order effects via the computation of the additive contribution of each neuron through the MSAs and projection to the input space. \n\nThe experimental evaluation focuses on the empirical analysis of the obtained effects and how these insights can be used in the context of generating \"semantic\" adversarial examples and using said effects for zero-shot segmentation.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper draws inspiration from recent approaches that aim to examine and evaluate the functionality of each neuron in a given architecture. Automated interpretability constitutes an important challenge for modern architectures and this work aims to approach this in a different way via the contribution of neurons to the output representation and the information flow through the MSA blocks.", "weaknesses": "The connection of the proposed approach to highly relevant work is a bit lacking. Can the authors provide a discussion on [1], highlighting the differences in the decomposition and analysis of the direct effects of the neurons?\n\nI find the focus on a single dataset, i.e., ImageNet, to be a bit restrictive in terms of analysing the behavior of the proposed approach. Indeed, most approaches in this line of work considered additional datasets, e.g., Waterbirds, CUB and Places365. The same applies for the adversarial examples setting, where the authors only consider CIFAR-10. What happens when trying to generate adversarial examples in a more complicated dataset, e.g., CUB or ImageNet?\n\nDid the authors reproduce the results for all the method in Table 4? Since they are different than the ones reported [1], I would expect that to be the case. \n\nCan the authors provide the weights of the texts corresponding to the sparse decomposition of each neuron? A full list for some neurons would also be helpful. \nQualitatively, what are the differences when choosing a different sparse set size m. \n\nWhat is the motivation behind the consideration a binary classification task instead of a classical setting? What classifier is used and how is it trained?\n\nThe authors mention that the generated adversarial images lie on the manifold of generated images differently from non-semantic adversarial attacks. Can the authors elaborate on this claim?\n\nWhat is the complexity of the proposed approach compared to other interpretability focused methods?\n\n[1] Gandelsman et al.,  Interpreting CLIP’s image representation via text-based decomposition, ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ddySXMfJM", "reviewer": "Reviewer_M6ey", "summary": "The paper presents an interpretability study focused on understanding the second-order effects of neurons in CLIP. The authors propose a novel \"second-order lens\" to analyze neuron contributions that flow through attention heads to the model output.", "questions": "1. What happens if you apply the same method to the text encoder in CLIP?\n2. Have the authors tried it on other variants of CLIP like MaskCLIP?\n3. Do the findings of this paper apply to other domains like medical imaging (MedCLIP)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The technical contributions are sound  and interesting.\n2. The paper is well written. \n3. The paper included thorough evaluations.", "weaknesses": "Generally good paper so please see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ykR7cSVYfQ", "reviewer": "Reviewer_XYxA", "summary": "This paper introduces an approach to generalize the first order effects of a neuron in attention networks into second order effects through successive layers. This methodology helps uncover the sparseness of the neuron second order effect as well as the decomposition of each neurons second order effect into a single direction vector.", "questions": "- Are there any gradient based approaches to finding textual descriptions of each neuron's second order effect?\n- Were any additional text to image models evaluated other than DeepFloyd IF, if so was there similar success to images generated with DeepFloyd IF?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Extensive empirical validation of second order effects (e.g. second order effect neuron sparseness)\n- Intuitive and interesting applications of second order effect control in the semantic adversarial example generation\n- Increased understanding of internal attention model mechanism through semantic adversarial examples\n- Improved segmentation results over TextSpan", "weaknesses": "- Sparse coding to find textual descriptions of neurons may be very computationally expensive\n- Not considering nonlinearities in second order effects (Eqn 5)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6RY8ax2pKc", "reviewer": "Reviewer_XFDa", "summary": "This paper proposes to interpret the functional of individual neurons in CLIP by investigating their \"second order\" effects at the joint text-image output space. Specifically, a neuron's contribution to the output feature space is defined as the summing of the effects flowing from the neuron through the later attention heads, to the output. Experimental results show several observations: (1) the effects of an individual neuron are highly selective. (2) Each neuron's contribution in the output feature space can be approximated by a single direction, which can be decomposed into representations of a sparse set of texts. (3) By exploiting the polysemantic property of neuron, it is possible to generate adversarial images that fool the CLIP classifier. (4) The semantic-awareness of neurons can be utilized for semantic segmentation.", "questions": "1. In the related work section, the paper lists several other neurons interpretability works that explore what concepts the neuron activated on, and saying that in contrast their work focuses on the neuron's contribution to the output space. My question is: what is the relation between the two approaches (exploring what examples the neurons activate on vs. exploring their contribution in the output space), what are the advantages / limitations of one over the other, does one show complementary aspects that the other cannot  observe?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well motivated, as exploring the functions of neurons in CLIP model facilitates better understanding of its representation learning mechanism, and guide further improvements\n2. The paper proposes to study the \"second order\" effect of neurons, which is demonstrated to show a clearer signal of their contribution comparing to first order and indirect effects\n3. The paper shows an interesting finding that each neuron's contribution to the joint text-image output space is approximately a single direction vector, and implies a ploysemy property.\n4. The paper further exploits the finding of 3 for generating adversarial images and application of semantic segmentation, showing that by taking advantage of polysemy neuron's output representations, it's possible to fool the CLIP classifier by using words that's spuriously correlated with a wrong class; and that by aggregating the activation maps of neurons that highly correlated with a query concept, its semantic segmentation mask can be acquired.", "weaknesses": "Please see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)