Interpreting the Second-Order Effects of Neurons in CLIP

## Abstract

Abstract We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons’ function in CLIP. Therefore, we present the “second-order lens”, analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for < 2 % absent percent 2 <2\% < 2 % of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior—each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce “semantic” adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation and attribute discovery in images. Our results indicate that a scalable understanding of neurons can be used for model deception and for introducing new model capabilities. 1 1 1 Project page and code: https://yossigandelsman.github.io/clip_neurons/

## 1 Introduction

Automated interpretability of the roles of components in neural networks enables the discovery of model limitations and interventions to overcome them. Recently, such a technique was applied for interpreting the attention heads in CLIP [], a widely used class of image
representation models []. However, this approach has only scratched the surface, failing to explain a major set of CLIP’s components—neurons. Here we will introduce a new interpretability lens for studying the neurons and use the gained understanding for segmentation, property discovery in images, and mass-production of semantic adversarial examples.

Interpreting the neurons in CLIP is a harder task than interpreting the attention heads. First, there are more neurons than individual heads, which requires a more automated approach. Second, their direct effect on the output—the flow from the neuron, through the residual stream directly to the output—is negligible []. Third, most information is stored redundantly—many neurons encode the same concept, so just ablating a neuron (i.e. examining indirect effects) does not reveal much since other neurons make up for it.

The limitations presented above mean that we can neither look at the direct effect nor the indirect effect to analyze a single neuron. To address this, we introduce a “second-order lens” for investigating the second-order effect of a neuron—its total contribution to the output, flowing via all the consecutive attention heads (see Fig. 1).

We start by analyzing the empirical behavior of second-order effects of neurons. We find that these effects have high significance in the late layers. Additionally, each neuron is highly selective: its second-order effect is significant for only a small set (about 2%) of the images. Finally, this effect can be approximated by a single direction in the joint text-image representation space of CLIP (Sec. 3.3).

As each direction that corresponds to a neuron lives in a joint representation space, it can be decomposed as a sparse sum of text representations that describes the neurons’ functionality (see Fig. 1). These text representations show that neurons are polysemantic []—each neuron corresponds to multiple semantic concepts. To verify that the neuron decompositions are meaningful, we show that these concepts correctly track which inputs activate a given neuron (Sec. 4).

The polysemantic behavior of neurons allows us to find concepts that inadvertently overlap in the network, due to being represented by the same neuron. We use these spurious cues for mass production of “semantic” adversarial examples that will fool CLIP (see bottom of Fig. 1). We apply this technique to automatically produce adversarial images for a variety of classification tasks. Our qualitative and quantitative analysis shows that incorporating spuriously overlapping concepts in an image deceives CLIP with a significant success rate (Sec. 5.1).

We present two additional applications – discovery of concepts in images, and zero-shot segmentation.
For concept discovery, we interpret the concepts that CLIP associates with a given image by aggregating the text descriptions of the sparsely-activated neurons for that image. For segmentation, we can generate attribution heatmaps from the activation patterns of neurons. We do this by identifying class-relevant neurons with the second-order lens and averaging their heatmaps. This yields a strong zero-shot image segmenter that outperforms recent work [, ].

In summary, we present an automated interpretability approach for CLIP’s neurons by modeling their second-order effects and spanning them with text descriptions. We use these descriptions to automatically understand neuron roles and apply this to three applications. This shows that a scalable understanding of internal mechanisms both uncovers errors and elicits new capabilities from models.

Figure: Figure 1: Second order effects of CLIP’s neurons. Top: We analyze the second-order effects of neurons in CLIP-ViT (flow in pink). Bottom-left: Each second-order effect of a neuron can be decomposed to a sparse set of word directions in the joint text-image space. Bottom-right: co-appearing words in these sets can be used for mass-generation of semantic adversarial images.
Refer to caption: x1.png

## 2 Related work

Contrastive vision-language models. Models like ALIGN [], CLIP [], and its variants [, ] produce image representations from pre-training on images and their captions. They demonstrated impressive zero-shot capabilities for various downstream tasks, including OCR, geo-localization, and classification []. These models’ representations are also used for segmentation [], image generation [, ] and 3D understanding []. We aim to reveal the roles of neurons in such models.

Mechanistic interpretability of vision models.
Mechanistic interpretability aims to reverse engineer the computation process in neural networks. In computer vision, this approach was applied to model individual network components [] and to extract intermediate mechanisms like curve detectors [], object segmenters [, ], high-frequency boundary detectors [], and multimodal concepts detectors []. More closely to us, a few works made use of the intrinsic language-image space of CLIP to interpret the direct effect of attention heads and the output representation in CLIP with automatic text descriptions [, ]. We go beyond the output and direct effects of individual layers to interpret intermediate neurons in CLIP.

Neurons interpretability.
The role of individual neurons (post-non-linearity single channel activations) is broadly studied in computer vision models [, , ] and language models [, , ]. [, ] demonstrate that neurons can learn universal mechanisms across different models in both domains. [] show that neurons can be polysemantic (i.e. activated on multiple concepts) and exploit this property for generation of L2 adversarial examples. Some work seeks to extract neurons’ concepts by learning sparse dictionaries [, ]. Other methods use large language models to automatically describe neurons based on which examples they activate on [, , , ]. In contrast, we focus on the contribution of neurons to the output representation.

## 3 Second-order effects of neurons

We start by presenting the CLIP-ViT architecture. Then, we derive the second-order effect of neurons and present their benefits over first-order and the indirect effects. Finally, we empirically characterize the second-order effects, setting the stage for automatically interpreting them via text in Sec. 4.

### 3.1 CLIP-ViT Preliminaries

Contrastive pre-training. CLIP is trained via a contrastive loss to produce image representations from weak text supervision. The model includes an image encoder $M_{\text{image}}$ and a text encoder $M_{\text{text}}$ that map images and text descriptions to a shared latent space $\mathbb{R}^{d}$. The two encoders are trained together to maximize the cosine similarity between the output representations $M_{\text{image}}(I)$ and $M_{\text{text}}(t)$ for matching input text-image pairs $(t,I)$:

$$ $\operatorname{sim}(I,t)=\langle M_{\text{image}}(I),M_{\text{text}}(t)\rangle/ (||M_{\text{image}}(I)||_{2}||M_{\text{text}}(t)||_{2}).$ (1) $$

Using CLIP for zero-shot classification.
Given a set of classes, each name of a class $c_{i}$ (e.g. the class “dog”) is mapped to a fixed template $template(c_{i})$
(e.g. “A photo of a {class}”), and encoded via the text encoder $M_{\text{text}}(template(c_{i}))$. The classification prediction for a given image $I$ is the class $c_{i}$ whose text representation is most similar to the image representation: $\operatorname*{arg\,max}_{c_{i}}\operatorname{sim}(I,template(c_{i}))$.

CLIP-ViT architecture. The CLIP-ViT image encoder consists of a Vision Transformer followed by a linear projection(^2^22Throughout the paper, we ignore layer-normalization terms to simplify derivations. We address layers-normalization in detail in A.5.). The vision transformer (ViT) is applied to the input image $I\in\mathbb{R}^{H\times W\times 3}$ to obtain a $d^{\prime}$-dimensional representation $\mathsf{ViT}(I)$. Denoting the projection matrix by $P\in\mathbb{R}^{d\times d^{\prime}}$:

$$ $M_{\text{image}}(I)=P(\mathsf{ViT}(I)).$ (2) $$

The input $I$ to ViT is first split into $K$ non-overlapping image patches that are encoded into $K$ $d^{\prime}$-dimensional image tokens. An additional learned token, named the class token, is included and used later as the output token. As shown in Fig. 1, the tokens are processed simultaneously by applying $L$ alternating residual layers of multi-head self-attention (MSA) and MLP blocks.

MLP neurons in CLIP. The MLP layers are applied separately on each image token and the class token. They consist of an input linear layer, parametrized by $W^{l}_{\text{in}}\in\mathbb{R}^{N\times d^{\prime}}$, followed by a GELU non-linearity $\sigma$ and an output linear layer, parametrized by $W^{l}_{\text{out}}\in\mathbb{R}^{d^{\prime}\times N}$. Here $l$ is the layer number and $N$ is the width (number of neurons) of the MLP. We next analyze the contributions of each individual neuron $n\in\{1,...,N\}$ for each layer.

Figure: Figure 2: First/Second-order effects. The first order is the flow coming from a neuron to the projection layer and the output (blue). The second order goes from a single neuron through all the consecutive attention heads, to the projection layer, and to the output (pink).
Refer to caption: x2.png

### 3.2 Analyzing the neuron effects on the output

Individual neurons have different types of contributions to the output—the first-order (direct) effects, second-order effects, and (higher-order) indirect effects. We introduce them and explain the limitations of the direct and indirect effects before continuing to characterize the second-order effects in Sec. 3.3.

First-order effects (logit lens []). The first-order effect is the direct contribution of a component to the residual stream, multiplied by the projection layer (see blue flow in Fig. 3). For an individual neuron $n$ in layer $l$, let $p_{i}^{l,n}(I)\in\mathbb{R}$ denote its post-GELU activation on the $i$-th token of the input image $I$.
Then the contribution $e_{i}^{l,n}$ of the $n$-th neuron to the $i$-th token in the residual stream is:

$$ $e_{i}^{l,n}=p_{i}^{l,n}(I)w^{l,n}$ (3) $$

where $w^{l,n}\in\mathbb{R}^{d^{\prime}}$ is the the $n$-th column of $W^{l}_{\text{out}}$. As the output representation is the class token (indexed 0) multiplied by $P$, the first-order effect for neuron $n$ on the output is $Pe_{0}^{l,n}$.

As observed by Gandelsman et al. [], the first-order effects of MLP layers are close to constants in CLIP and most of the first-order contributions are from the late attention layers. We therefore focus on the second-order effects: the flow of information from the neurons through the attention layers.

Second-order effects. The contribution $e^{l,n}_{i}$ to the residual stream directly affects the input to later layers. We focus on the flow of $e^{l,n}_{i}$ through subsequent MSAs and then to the output
(pink flow in Fig. 3). We call this interpretability lens the “second-order lens”, in analogy to the “logit lens”.

Following [], the output of an MSA layer $\mathsf{MSA}^{l}$ that corresponds to the class token is a weighted sum of its $K+1$ input tokens $[z_{0},...,z_{K}]$:

$$ $\left[\mathsf{MSA}^{l}([z_{0},...,z_{K}])\right]_{0}=\sum_{h=1}^{H}\sum_{i=0}^ {K}a_{i}^{l,h}(I)W_{VO}^{l,h}z_{i}$ (4) $$

where $W_{VO}^{l,h}\in\mathbb{R}^{d^{\prime}\times d^{\prime}}$ are transition matrices (the OV matrices) and $a_{i}^{l,h}(I)\in\mathbb{R}$ are the attention weights from the class token to the $i$-th token ($\sum_{i=0}^{K}{a_{i}^{l,h}}=1$).

To obtain the second-order effect of a neuron $n$ at layer $l$, $\phi_{n}^{l}(I)$, we compute the additive contribution of the neuron through all the later MSAs and project it to the output space via $P$. Plugging in Eq. 3 as the contribution to $z_{i}$ in Eq. 4 and summing over layers, the second order effect of neuron $n$ is then:

$$ $\phi_{n}^{l}(I)=\sum_{l^{\prime}=l+1}^{L}\sum_{h=1}^{H}\sum_{i=0}^{K} \underbrace{\left(p_{i}^{l,n}(I)a_{i}^{l^{\prime},h}(I)\right)}_{\text{ attention-weighted activations}}\underbrace{\left(PW_{VO}^{l^{\prime},h}w^{l,n }\right)}_{\text{input-independent}}$ (5) $$

Indirect effects. An alternative approach is to analyze the indirect effect of a neuron by measuring the change in output representation when intervening on a neuron’s output. Specifically, the intervention is done by replacing the activation $p_{i}^{l,n}$ of the neuron for each token with a pre-computed per-token mean. However, as was shown by McGrath et al. [], models often learn “self-repair” mechanisms that can obscure the individual roles of neurons. We illustrate these issues in the next section.

Figure: Figure 4: Comparison to indirect effect. We compare the second-order effects and the indirect effects by mean-ablating layer 9 in ViT-B-32 on ImageNet test set.
Refer to caption: x4.png

### 3.3 Characterizing the second-order effects

We analyze the empirical behavior of the second-order effects of neurons $\phi^{l}_{n}$ derived in the previous section. We find that only neurons from the late MLP layers have a significant second-order effect and that each individual neuron has a significant effect for less than 2% of the images. Finally, we show that $\phi^{l}_{n}$ can be approximated by one linear direction in the output space. These findings will help motivate our algorithm for describing output spaces of neurons with text in Sec. 4.

Experimental setting. To evaluate the second-order effects and their contributions to the output representation, we measure the downstream performance on the ImageNet classification task [] after ablating these effects for each neuron. Specifically, we apply mean-ablation [], replacing the additive contributions of individual $\phi_{n}^{l}(I)$’s to the representation with the mean computed across a dataset $D$. In our experiments, we mean-ablate all the neurons in a layer simultaneously and evaluate the downstream classification performance before and after ablation. Components with larger effects should result in larger accuracy drops.

We take $D$ to be $\sim\!5000$ images from the ImageNet (IN) training set. We report zero-shot classification accuracy on the IN test set. Our model is OpenAI’s ViT-B-32 CLIP, which has 12 layers (additional results for ViT-L-14 are in App. A.1)

Second-order effects concentrate in moderately late layers. We evaluate the contributions of all the $\phi^{l}_{n}$ across different layers and observe that the neurons with the most significant second-order effects appear relatively late in the model. The results for different layers in ViT-B-32 CLIP model are presented in Fig. 3 (“w/o all neurons”). As shown, mean-ablating layers 8-10 leads to the largest drop in performance. These layers appear right before the MSA layers with the most significant direct effects, as shown in [] (layers 9-11; see App. A.2).

The second-order effect is sparse. We find that the second-order effect of each individual neuron is significant only for less than 2% of the images across the test set. We repeat the same experiment as before, but this time we only mean-ablate $\phi^{l}_{n}(I)$ for a subset of images, while keeping the original effects for other images. We find that for most of the images, except the subset of images in which $\phi^{l}_{n}(I)$ has a large norm, we can mean-ablate $\phi^{l}_{n}(I)$ without changing the accuracy significantly, as shown in Fig. 3 (“w/o small norm”). Differently, mean-ablating the contributions for the 100 images with the largest $\phi^{l}_{n}(I)$ norms results in a significant drop in performance (“w/o large norm”).

The second-order effect is approximately rank 1. While the second-order effect for a given neuron can write to different directions in the joint representation space for each image, we find that $\phi^{l}_{n}(I)$ can be approximated by one direction $r_{n}^{l}\in\mathbb{R}^{d}$ in this space, multiplied by a coefficient $x^{l}_{n}(I)$ that depends on the image. We use the set $S^{l}_{n}$, which contains the largest second-order effects in norm from $D$, and set $r_{n}^{l}$ to be the first principle component computed from $S^{l}_{n}$. We approximate $\phi^{l}_{n}(I)$ with $x^{l}_{n}(I)r^{l}_{n}+b^{l}_{n}$, where $b^{l}_{n}\in\mathbb{R}^{d}$ is the bias computed by averaging $\phi^{l}_{n}(I)$ across $D$, and $x^{l}_{n}(I)\in\mathbb{R}$ is the norm of the projection of $\phi^{l}_{n}(I)$ onto $r^{l}_{n}$.

To verify that this approximation recovers $\phi^{l}_{n}(I)$ we replace each $\phi^{l}_{n}(I)$ for each neuron and image in the test set with the approximation. We then evaluate the downstream classification performance. As shown in Fig. 3 (“reconstruction from PC #1”), for each layer $l$, this replacement results in a negligible drop in performance from the baseline, that uses the full representation.

Comparison to indirect effect. We compare the second-order effect to the indirect effect and present the variance explained by the first principle component for each of them and the drop in performance when simultaneously mean-ablating all the effects from one layer. As shown in Tab. 5, Mean-ablating the indirect effects results in a smaller drop in performance due to self-repair behavior. Moreover, the first principle component explains significantly less of the variance in the indirect effect, than in the second-order effect. This demonstrates two advantages of the second-order effects—uncovering neuron functionality that is obfuscated by self-repair, and one-dimensional behavior that can be easily modeled and decomposed, as we will show in the next section.

## 4 Sparse decomposition of neurons

We aim to interpret each neuron by associating its second-order effect with text. We build on the previous observation that each second-order effect of a neuron $\phi_{n}^{l}$ is associated with a vector direction $r_{n}^{l}$. Since $r_{n}^{l}$ lies in a shared image-text space, we can decompose it to a sparse set of text directions. We use a sparse coding method [] to mine for a small set of texts for each neuron, out of a large pool of descriptions. We evaluate the found texts across different initial pools with different set sizes.

Decomposing a neuron into a sparse set of descriptions. Given the first principal component of the second-order effect of each neuron, $r_{n}^{l}$, we will decompose it as a sparse sum of text directions $t_{j}$: $r^{l}_{n}\approx\hat{r}^{l}_{n}=\sum_{j=1}^{M}\gamma^{l,n}_{j}M_{\text{text}}(
t_{j})$. To do this, we start from a large pool $T$ of $M$ text descriptions (e.g. the most common words in English). We apply a sparse coding algorithm to approximate $r_{n}^{l}$ as the sum above, where only $m$ of the $\gamma^{l,n}_{j}$’s are non-zero, for some $m\ll M$.

**Table 1: Examples of sparse decompositions. For each neuron, we present the top-4 texts corresponding to the sparse decomposition with $m=128$ and the signs of the coefficients in the decomposition for two initial pools.**
| Neuron | ImageNet class descriptions | Common words (30k) |
| --- | --- | --- |
| #4 | +“Picture with falling snowflakes” | +“snowy” |
| +“Picture portraying a person […] in extreme weather conditions” | +“frost” |  |
| -“Picture with a bucket in a construction site” | +“closings” |  |
| +“Photograph taken during a holiday service” | +“advent” |  |
| #391 | +“Image with a traditional wooden sled” | +“woodworking” |
| +“Image with a wooden cutting board” | -“swelling” |  |
| +“Picture showcasing beach accessories” | +“cedar” |  |
| -“Photograph with a syringe and a surgical mask” | +“heirloom” |  |
| #2137 | +“Photo with a lime garnish” | +“refreshments” |
| +“Image with candies in glass containers” | +“gelatin” |  |
| -“Picture featuring lifeboat equipment” | +“sour” |  |
| +“Close-up photo of a melting popsicle” | +“cosmopolitan” |  |
| #2914 | +“Photo that features a stretch limousine” | +“motorhome” |
| +“Image capturing a suit with pinstripes” | +“yacht” |  |
| +“Caricature with a celebrity endorsing the brand” | +“cirrus” |  |
| +“Image showcasing a Bullmastiff’s prominent neck folds” | +“cabriolet” |  |

Figure: Figure 6: Images with largest second-order effect norm per neuron. We present the top images from 10% of ImageNet test set for the neurons in Tab. 1. Note that neurons are polysemantic - they have large second-order effects on images that show multiple concepts (e.g. cars and boats).
Refer to caption: x5.png

Experimental settings. We verify that the reconstructed $\hat{r}^{l}_{n}$ from the text representations captures the variation in the image representation, as measured by zero-shot accuracy on IN. We simultaneously replace the neurons’ second-order contributions in a single layer with the approximation $x^{l}_{n}(I)\hat{r}^{l}_{n}+b^{l}_{n}$.

To obtain sparse decomposition for each neuron, we use [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.OrthogonalMatchingPursuit.html)’s implementation of orthogonal matching pursuit []. We consider two strategies for constructing the pool of text descriptions $T$. The first type is single words - the 10k and 30k most common words in English. The second type is image descriptions - we prompt ChatGPT-3.5 to produce descriptions of images that include an object of a specific class. Repeating this process for all the IN classes results in $\sim$28k unique image descriptions. We then evaluate the reconstruction of $r^{l}_{n}$ for different $m$’s and pools.

Effect of sparse set size $m$ and different pools. We experiment with $m\in\{4,8,16,32,64,128\}$ and the three text pools, and present the accuracy on 10% of IN test set in Fig. 5. We approach the original classification accuracy with 128 text descriptions per neuron reconstruction $\hat{r}^{l}_{n}$. Using full descriptions outperforms using single words for the text pool, but the gap vanishes for larger $m$.

Qualitative results. We present the images with the largest second-order norms in Fig. 6, and the corresponding top-4 text descriptions in Tab. 1. As shown, the found descriptions match the objects in the top 10 images. Moreover, some individual neurons correspond to multiple concepts (e.g. writing both toward “yacht” and a type of a car - “cabriolet”). This corroborates with previous literature on neurons’ polysemantic behavior [] - single neurons behave as a superposition of multiple interpretable features. This property will allow us to generate adversarial images in Sec. 5.1.

## 5 Applications

### 5.1 Automatic generation of adversarial examples

The sparse decomposition of $r_{n}^{l}$’s allows us to find overlapping concepts that neurons are writing to.
We use these spurious cues to generate semantic adversarial images.
Our pipeline, shown in Fig. 1, mines for spurious words that correlate with the incorrect class in CLIP (e.g. “elephant”, that correlates with “dog”), combines them into image descriptions that include the correct class name (“cat”), and generates adversarial images by providing these descriptions to a text-to-image model. We explain the steps in the pipeline and provide quantitative and qualitative results.

Figure: Figure 7: Adversarial images generated by our method. For each binary classification task, we present the generated images, the input text to the text-to-image model (words from $W^{v}$ are bold), and an attribution map [] for the classification (areas that contribute to the incorrect class score are red).
Refer to caption: x6.png

Finding relevant spurious cues in neurons. Given two classes $c_{1}$ and $c_{2}$, we first select neurons that contribute the most toward the classification direction $v=M_{\text{text}}(c_{2})-M_{\text{text}}(c_{1})$, then mine their sparse decompositions for spurious cues. Specifically, we extract the set of neurons $\mathcal{N}$ whose directions are most similar to $v$: $\mathcal{N}=\text{top-k}_{n\in N}|\langle v,r^{l}_{n}\rangle|$. Utilizing the sparse decomposition from before, we compute a contribution score $w_{j}$ for each phrase $j$ in the pool $T$:

$$ $w_{j}^{v}=\sum_{n\in\mathcal{N}}\gamma^{l,n}_{j}\langle v,r^{l}_{n}\rangle.$ (6) $$

This looks at the weight that each neuron in $\mathcal{N}$ assigns to $j$ in its sparse decomposition, weighted by how important that neuron is for classification. A phrase with a high contribution score has significant weight in one or more important neurons, and so is a potential spurious cue. The top phrases, sorted by the contribution score are collected into a set of phrase candidates $W_{v}$.

Generating “semantic” adversarial examples. We use text and image generative models to create examples with the object $c_{2}$ that are classified as $c_{1}$. First, we generate image descriptions with a large language model (LLM) by providing it phrases from the set $W^{v}$ and the class name $c_{1}$ and prompting it to generate image descriptions that include elements from both. We prompt the model to exclude anything related to $c_{2}$ from the descriptions and use visually distinctive words from $W_{v}$.

The resulting descriptions are fed into a text-to-image model to generate the adversarial images. Note that the adversarial images lie on the manifold of generated images, differently from non-semantic adversarial attacks that modify individual pixels.

Experimental settings. We generate adversarial images for classifying between pairs of classes from CIFAR-10 []. We use the 30k most common words as our pool $T$. We choose the top $100$ neurons from layers 8-10 for $\mathcal{N}$, and the top 25 words according to their contribution scores for prompting the LLM. We prompt LLaMA3 [] to generate 50 descriptions for each classification task (see prompt in App. A.6). We then filter out descriptions that include the class name and choose 10 random descriptions. We generate 10 images for each description with DeepFloyd IF text-to-image model []. This results in 100 images per experiment. We repeat the experiment 3 times and manually remove images that include $c_{2}$ objects or do not include $c_{1}$ objects.

We report three additional baselines. First, we repeat the same process with $100$ random neurons instead of the set $\mathcal{N}$. Second, we repeat the same generation process with sparse text decompositions computed from the first principle components of the indirect effects instead of the second-order effect. Third, we do not rely on the neuron decompositions, and instead prompt the language model with the words from $M$ for which their text representations are the most similar to $v$. Both for our pipeline and the baselines, we automatically filter out synonyms of $c_{2}$ from the phrases provided to the language model according to their sentence similarity to $c_{2}$ [].

Quantitative results. The classification accuracy results for the adversarial images are presented in Tab. 3. The success rate of our adversarial images is significantly higher than the indirect effect baseline, the similar words baseline, and the random baseline, which succeeds only accidentally.

Qualitative results. Fig. 7 includes generated adversarial examples and the descriptions that were used in their generation. The presented attribution heatmaps [] show that the found spurious objects from $W_{v}$ contribute the most to the misclassification, while the object from the correct class (e.g. a horse in the left-most image) contributes the least. We provide more results for additional classification tasks (e.g. “stop-sign v.s. yield”) in Fig. 14.

We show that understanding internal components in models can be grounded by exploiting them for adversarial attacks. Our attack is optimization-free and is not compute-intensive. Hence, it can be used for measuring interpretability techniques, with better understanding leading to improved attacks.

Figure: Figure 8: Concept discovery in images (ViT-B-32). We include top-10 words discovered by aggregating words in sparse decompositions of activated neurons.
Refer to caption: x7.png

### 5.2 Concept discovery in images

We aim to discover concepts in image $I$, by aggregating phrases that correspond to the neurons that are activated on $I$. Here, we start from the set of activated neurons $\mathcal{N}$ (for which $||\phi_{n}^{l}(I)||_{2}$ is above the 98th percentile of norms computed across IN images). Similarly to the contribution score described earlier, we compute an image-contribution score $w_{j}^{I}$ for each phrase $j$ according to its combined weight in the decompositions of neurons in $\mathcal{N}$. Formally, $w_{j}^{I}$ is the overall sum of weights that each neuron in $\mathcal{N}$ assigns to $j$ in its decomposition, weighted by the neuron second-order norms: $w_{j}^{I}=\sum_{n\in{\mathcal{N}}}{\gamma_{j}^{l,n}||\phi_{n}^{l}(I)||_{2}}$.
The phrases with the highest image-contribution score are picked to describe the image concepts.

Qualitative results. We present qualitative results for neurons and the top-10 discovered concepts from layer 9 of ViT-B-32 in Fig. 9, when using the most common words as the pool. The number of neurons activated on these images, $|\mathcal{N}|$, is between 29 and 59, less than 2% of the neurons in the layer. Nevertheless, the top words extracted from these neurons relate semantically to the objects in the image and their locations. Surprisingly, the top word for each of the images appears only in one or two of the neuron sparse decompositions and is not spread across many activated neurons.

**Table 2: Accuracy of adversarial images. We report how many generated images out of 100, fooled the binary classifier (standard deviation in parentheses).**
| Task | Random | Indirect | Similar | Second |
| --- | --- | --- | --- | --- |
| effect | words | order |  |  |
| horse $\rightarrow$ | 1.0 ($\pm$1.4) | 2.8 ($\pm$3.7) | 1.0 ($\pm$1.4) | 5.3 ($\pm$1.9) |
| automobile |  |  |  |  |
| dog $\rightarrow$ deer | 0.3 ($\pm$0.5) | 6.3 ($\pm$4.8) | 3.3 ($\pm$0.9) | 22.7 ($\pm$0.5) |
| bird $\rightarrow$ frog | 0.3 ($\pm$0.5) | 1.0 ($\pm$1.4) | 5.0 ($\pm$2.9) | 8.0 ($\pm$4.5) |
| ship $\rightarrow$ truck | 0.0 ($\pm$0.0) | 0.0 ($\pm$0.0) | 0.0 ($\pm$0.0) | 5.7 ($\pm$0.9) |
| ship $\rightarrow$ | 1.3 ($\pm$1.9) | 0.0 ($\pm$0.0) | 1.3 ($\pm$0.9) | 7.0 ($\pm$4.5) |
| automobile |  |  |  |  |

### 5.3 Zero-shot segmentation

Finally, we use our understanding of neurons for zero-shot segmentation. Each neuron corresponds to an attribution map, by looking at its activations $p_{i}^{l,n}(I)$ on each image patch. Ensembling all the neurons that contribute towards a concept results in an aggregated attribution map that can be binarized to generate reliable segmentations.

Specifically, to generate a segmentation map for an image $I$, we find a set of neurons with the largest absolute value of the dot product with the encoded class name we aim to segment: $|\langle r_{n}^{l},M_{\text{text}}(c_{i})\rangle|$. We then average their spatial activation maps $p_{i}^{l,n}(I)$, standardize the average activations into $[0,1]$, and binarize the values into foreground/background segments by applying a threshold of 0.5.

Segmentation results. We provide results on IN-Segmentation [], which includes foreground/background segmentation maps of IN objects. We use activation maps from the top 200 neurons of layers 8-10. Tab. 3 presents a quantitative comparison to previous explainability methods. Our method outperforms other zero-shot segmentation methods across all standard evaluation metrics. We provide qualitative results before thresholding in Fig. 9. While the first-order effects highlight individual discriminative object parts, our heatmaps capture more parts of the full object.

## 6 Limitations and discussion

We analyzed the second-order effects of neurons on the CLIP representation and used our understanding to perform zero-shot segmentation, discover image concepts, and generate adversarial images. We present mechanisms that we did not analyze in our investigation and conclude with a discussion of broader impact and future directions.

Neuron-attention maps mechanisms. We investigated how the neurons flow through individual consecutive attention values, and ignored the effect of neurons on consecutive queries and keys in the attention mechanism. Investigating these effects will allow us to find neurons that modify the attention map patterns. We leave it for future work.

Neuron-neuron mechanisms. We did not analyze the mutual effects between neurons in the same layer or across different layers. Returning to our adversarial “frog/bird” attack example, a neuron that writes toward “dog” may not be activated if a different neuron writes simultaneously toward “frog”, thus reducing our attack efficiency. While we can still generate multiple adversarial images, we believe that understanding dependencies between neurons can improve it further.

Future work and broader impact. The mass production of adversarial images can be harmful to systems that rely on neural networks. On the other hand, automatic extraction of such cases allows the defender to be prepared for them and, possibly, fine-tune the model on the generated images to avoid such attacks. We plan to investigate this approach to improve CLIP’s robustness in the future.

Acknowledgments. We would like to thank Alexander Pan for helpful feedback on the manuscript. YG is supported by the Google Fellowship. AE is supported in part by DoD, including DARPA’s MCS and ONR MURI, as well as funding from SAP. JS is supported by the NSF Awards No. 1804794 & 2031899.

## Appendix A Appendix

### A.1 Second order ablations for ViT-L

We repeat the same experiments from Sec. 3.3 for ViT-L-14, trained on LAION dataset []. For this model, we only use 10% of ImageNet test set. Here, the maximal drop in performance when ablating the second order is relatively smaller and is spread across more layers. Nevertheless, the same properties presented and discussed in Section 3.3 hold for this model.

### A.2 First order ablations

For the two models discussed above, ViT-B-32 and ViT-L-14, we provide the mean-ablation results for the first-order effects of MSA layers, as computed in []. For each model, we present the performance before and after accumulative mean-ablation of all the first-order effects of MSA layers. As shown in Figures 12 and 12, the neurons with the significant second-order effects appear right before the layers with the significant first-order effects.

### A.3 Additional adversarial images

We present additional semantic adversarial results, generated by our method for ViT-B-32, in Figure 14. We demonstrate a wide variety of tasks, including additional pairs from CIFAR-10 dataset, and adversarial attacks related to traffic signs (e.g. misclassification between a stop sign and a yield sign or a crossroad). For each image, we provide the text used for generating it, and highlight the spurious cues words from the sparse decompositions.

### A.4 Additional sparse decomposition results

We provide additional examples of sparse decompositions of neurons in Tab. 4 and the images with the top norms for the second-order effects of the same neurons in Fig. 13. As shown, the found descriptions match the objects in the top 10 images.

### A.5 Derivations with Layer Normalization

In many implementations of CLIP, there is a layer normalization between the Vision Transformer and the projection layer $P$. In this case, the representation is:

$$ $M_{\text{image}}(I)=P(LN(\mathsf{ViT}(I)))$ (7) $$

where the $LN$ is the layer normalization. Specifically, $LN$ can be written as:

$$ $LN(x)=\gamma*\frac{x-\mu}{\sqrt{\sigma^{2}+\epsilon}}+\beta=\underbrace{\left[ \frac{\gamma}{\sqrt{\sigma^{2}+\epsilon}}\right]}_{=A}*x-\underbrace{\left[ \frac{\mu\gamma}{\sqrt{\sigma^{2}+\epsilon}}-\beta\right]}_{=B},$ (8) $$

where $x\in\mathbb{R}^{d^{\prime}}$ is the input token, $\mu_{l},\sigma_{l}\in\mathbb{R}$ are the mean and standard deviation, and $\gamma,\beta\in\mathbb{R}^{d^{\prime}}$ are learned vectors. To include $A$ and $B$ in the second-order effect of a neuron flow, we replace the input-independent component in Eq. 5, $PW_{VO}^{l^{\prime},h}w^{l^{\prime},n}$, with:

$$ $P(A*W_{VO}^{l^{\prime},h}w^{l,n}+\frac{B}{c})$ (9) $$

Where $c$ is a normalization constant that splits $B$ equally across all the neurons that can additively contribute to it.

Except for the layer normalization before the projection layer, the input to the MSA layers that comes from the residual stream also flows through a layer normalization. Thus, if the input to the MSA layer in layer $l$ is the list of tokens $[z_{0}^{l},...z_{K}^{l}]$, the output that corresponds to the class token is:

$$ $\left[\mathsf{MSA}^{l}([z_{0},...,z_{K}])\right]_{0}=\sum_{h=1}^{H}\sum_{i=0}^ {K}a_{i}^{l,h}(I)W_{VO}^{l,h}LN^{l}(z_{i}),$ (10) $$

where $LN^{l}$ is the normalization layer at layer $l$, that can be parameterized similarly to Eq. 8 by $A^{l},B^{l}\in\mathbb{R}^{d^{\prime}}$. We modify the definition of the second-order effect accordingly:

$$ $\phi_{n}^{l}(I)=\sum_{l^{\prime}=l+1}^{L}\sum_{h=1}^{H}\sum_{i=0}^{K}\left(p_{ i}^{l,n}(I)a_{i}^{l^{\prime},h}(I)\right)\left(P\left(A*W_{VO}^{l^{\prime},h}( A^{l}*w^{l,n}+\frac{B^{l^{\prime}}}{c^{l^{\prime}}})+\frac{B}{c}\right)\right),$ (11) $$

where $c^{l^{\prime}}$ is is a normalization coefficient that splits $B^{l^{\prime}}$ equally across all the neurons before layer $l^{\prime}$.

In all of our experiments, we use this modification. Most of the elements in the modification add constant biases. Therefore, they can be ignored in our experiments as in many of the experiments constant biases do not change the results. For example, in our mean-ablation experiment, we subtract the mean, computed across a dataset.

Figure: Figure 10: ViT-L-14 second-order ablations.
Refer to caption: x9.png

Figure: Figure 11: ViT-B-32 first-order MSAs ablation.
Refer to caption: x10.png

**Table 4: Additional examples of sparse decomposition results. For each neuron, we present the top-4 texts corresponding to the sparse decomposition with $m=128$ and the signs of the coefficients in the decomposition.**
| Neuron | ImageNet class descriptions | Common words (30k) |
| --- | --- | --- |
| #600 | +“Image with a wiry, weather-resistant coat” | +“tents” |
| +“Image showcasing a compact and lightweight sleeping bag” | +“svalbard” |  |
| +“Picture of a camper towing bicycles” | +“miles” |  |
| +“Image with a Border Terrier jumping” | -“mountainous” |  |
| #974 | -“Photograph taken during a race” | +“runners” |
| -“Silhouette of a running dog” | +“races” |  |
| -“Picture taken in a fishing competition” | -“dolphin” |  |
| +“Silhouette of hammerhead shark with other ocean creatures” | +“expiration” |  |
| #1517 | +“Chair with a foot pedal control” | +“bus” |
| -“Picture that captures the breed’s intelligence” | -“filings” |  |
| -“Image with snow-capped mountains as scenery” | -“percussion” |  |
| +“Image with graffiti on a train” | +“wheelchairs” |  |
| #2002 | +“Image depicting a sustainable living option” | -“genres” |
| +“Photo taken in a train yard” | +“governance” |  |
| -“Image featuring snow-covered rooftops” | +“‘gravel” |  |
| +“Rescue equipment” | +“conserve” |  |

Figure: Figure 13: Images with largest second-order effect norm per neuron. We present the top images from 10% of ImageNet test set, for the neurons in Tab. 4.
Refer to caption: x12.png

Figure: Figure 14: Additional adversarial examples generated by our method. We provide the sentence that was given to the text-to-image model to generate it. Words from $W^{v}$ are highlighted in bold.
Refer to caption: x13.png

**Table 5: The language model prompt for generating image descriptions.**
| You are a capable instruction-following AI agent. |
| --- |
| I want to generate an image by providing image descriptions as input to a text-to-image model. |
| The image descriptions should be short. Each of them must include the word "{class_1}". |
| They must not include the word "{class_2}", any synonym of it, or a plural version! |
| The image descriptions should include as many words as possible from the next list and almost no other words: |
| {list} |
| Do not use names of people or places from the list unless they are famous and there is something visually distinctive about them.<br>In each of the image descriptions mention as many objects and animals as possible from the list above.<br>If you want to mention the place in which the image is taken or a name of a person, describe it with visually distinctive words.<br>For example, if "Paris" is in the list, instead of saying "… in Paris", say "… with the Eiffel Tower in the background" or "… next to a sign saying ’Paris’".<br>Don’t mention words that are too similar to "{class_2}", even if they are in the list above.<br>For example, if the word was "tree" you should not mention "trees", "bush" or "eucalyptus". |
| Only use words that you know what they mean. |
| Generate a list of 50 image descriptions. |

**Table 6: The prompt for generating the pool of class descriptions. We prompt the model with all the ImageNet classes.**
| Provide 40 image characteristics that are true for almost all the images of {class}. Be as general as possible and give short descriptions presenting one characteristic at a time that can describe almost all the possible images of this category. Don’t mention the category name itself (which is “{class}”). Here are some possible phrases: “Image with texture of …”, “Picture taken in the geographical location of…”, "Photo that is taken outdoors”, “Caricature with text”, “Image with the artistic style of…”, “Image with one/two/three objects”, “Illustration with the color palette …”, “Photo taken from above/below”, “Photograph taken during … season”. Just give the short titles, don’t explain why, and don’t combine two different concepts (with “or” or “and”). |
| --- |

### A.6 Prompts

We provide the prompt that was used for generating sentences given the set of words $W_{v}$, as presented in Section 5.1, in Tab. 5. This prompt is given to LLAMA3 model [].

Additionally, we provide the prompt that was used for generating the pool of ImageNet class descriptions, presented in Section 4. We prompt ChatGPT (GPT 3.5) with the prompt template provided in Tab. 6.

### A.7 Compute

As our method does not require additional training, the time of our experiments depends linearly on the inference time of CLIP (and other generative models that were used for the adversarial images generation), and on the number of images we use for the experiments ($\sim$5000 in our case). All our experiments were run on one A100 GPU. The most time-consuming experiment—computing the per-layer mean-ablation results for ViT-L-14—took 5 days.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the Weaknesses section.
soundness: 3
strengths: This paper draws inspiration from recent approaches that aim to examine and evaluate the functionality of each neuron in a given architecture. Automated interpretability constitutes an important challenge for modern architectures and this work aims to approach this in a different way via the contribution of neurons to the output representation and the information flow through the MSA blocks.
summary: This work presents a novel approach for examining potential second-order effects of neurons of CLIP representations and how these can be used in the context of zero-shot segmentation and generation of adversarial images. 

To this end, the authors focus on the contribution of each individual neuron to the output in terms of second-order effects via the computation of the additive contribution of each neuron through the MSAs and projection to the input space. 

The experimental evaluation focuses on the empirical analysis of the obtained effects and how these insights can be used in the context of generating "semantic" adversarial examples and using said effects for zero-shot segmentation.
weaknesses: The connection of the proposed approach to highly relevant work is a bit lacking. Can the authors provide a discussion on [1], highlighting the differences in the decomposition and analysis of the direct effects of the neurons?

I find the focus on a single dataset, i.e., ImageNet, to be a bit restrictive in terms of analysing the behavior of the proposed approach. Indeed, most approaches in this line of work considered additional datasets, e.g., Waterbirds, CUB and Places365. The same applies for the adversarial examples setting, where the authors only consider CIFAR-10. What happens when trying to generate adversarial examples in a more complicated dataset, e.g., CUB or ImageNet?

Did the authors reproduce the results for all the method in Table 4? Since they are different than the ones reported [1], I would expect that to be the case. 

Can the authors provide the weights of the texts corresponding to the sparse decomposition of each neuron? A full list for some neurons would also be helpful. 
Qualitatively, what are the differences when choosing a different sparse set size m. 

What is the motivation behind the consideration a binary classification task instead of a classical setting? What classifier is used and how is it trained?

The authors mention that the generated adversarial images lie on the manifold of generated images differently from non-semantic adversarial attacks. Can the authors elaborate on this claim?

What is the complexity of the proposed approach compared to other interpretability focused methods?

[1] Gandelsman et al.,  Interpreting CLIP’s image representation via text-based decomposition, ICLR 2024

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. What happens if you apply the same method to the text encoder in CLIP?
2. Have the authors tried it on other variants of CLIP like MaskCLIP?
3. Do the findings of this paper apply to other domains like medical imaging (MedCLIP)?
soundness: 2
strengths: 1. The technical contributions are sound  and interesting.
2. The paper is well written. 
3. The paper included thorough evaluations.
summary: The paper presents an interpretability study focused on understanding the second-order effects of neurons in CLIP. The authors propose a novel "second-order lens" to analyze neuron contributions that flow through attention heads to the model output.
weaknesses: Generally good paper so please see questions.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: - Are there any gradient based approaches to finding textual descriptions of each neuron's second order effect?
- Were any additional text to image models evaluated other than DeepFloyd IF, if so was there similar success to images generated with DeepFloyd IF?
soundness: 3
strengths: - Extensive empirical validation of second order effects (e.g. second order effect neuron sparseness)
- Intuitive and interesting applications of second order effect control in the semantic adversarial example generation
- Increased understanding of internal attention model mechanism through semantic adversarial examples
- Improved segmentation results over TextSpan
summary: This paper introduces an approach to generalize the first order effects of a neuron in attention networks into second order effects through successive layers. This methodology helps uncover the sparseness of the neuron second order effect as well as the decomposition of each neurons second order effect into a single direction vector.
weaknesses: - Sparse coding to find textual descriptions of neurons may be very computationally expensive
- Not considering nonlinearities in second order effects (Eqn 5)

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. In the related work section, the paper lists several other neurons interpretability works that explore what concepts the neuron activated on, and saying that in contrast their work focuses on the neuron's contribution to the output space. My question is: what is the relation between the two approaches (exploring what examples the neurons activate on vs. exploring their contribution in the output space), what are the advantages / limitations of one over the other, does one show complementary aspects that the other cannot  observe?
soundness: 3
strengths: 1. The paper is well motivated, as exploring the functions of neurons in CLIP model facilitates better understanding of its representation learning mechanism, and guide further improvements
2. The paper proposes to study the "second order" effect of neurons, which is demonstrated to show a clearer signal of their contribution comparing to first order and indirect effects
3. The paper shows an interesting finding that each neuron's contribution to the joint text-image output space is approximately a single direction vector, and implies a ploysemy property.
4. The paper further exploits the finding of 3 for generating adversarial images and application of semantic segmentation, showing that by taking advantage of polysemy neuron's output representations, it's possible to fool the CLIP classifier by using words that's spuriously correlated with a wrong class; and that by aggregating the activation maps of neurons that highly correlated with a query concept, its semantic segmentation mask can be acquired.
summary: This paper proposes to interpret the functional of individual neurons in CLIP by investigating their "second order" effects at the joint text-image output space. Specifically, a neuron's contribution to the output feature space is defined as the summing of the effects flowing from the neuron through the later attention heads, to the output. Experimental results show several observations: (1) the effects of an individual neuron are highly selective. (2) Each neuron's contribution in the output feature space can be approximated by a single direction, which can be decomposed into representations of a sparse set of texts. (3) By exploiting the polysemantic property of neuron, it is possible to generate adversarial images that fool the CLIP classifier. (4) The semantic-awareness of neurons can be utilized for semantic segmentation.
weaknesses: Please see questions

Meta-review: The paper introduces a novel method for interpreting neurons in CLIP (Contrastive Language-Image Pretraining) through second-order effects, which measure a neuron’s influence on the model’s output via later attention layers. This approach provides a deeper understanding of individual neurons by analyzing their contributions in the joint text-image embedding space. The authors demonstrate that these second-order effects are sparse and can be approximated by single-direction vectors in the embedding space, which are then decomposed into interpretable text representations.

The proposed method is applied to two key tasks:

- Adversarial Image Generation: The polysemantic behavior of neurons is exploited to generate "semantic adversarial examples," where images fool CLIP into making incorrect classifications based on spurious correlations.
- Zero-Shot Segmentation: The second-order effects are used to identify neurons aligned with specific semantic concepts. Aggregated activation maps from these neurons are used to assign labels to image regions, enabling segmentation without additional training.

Empirical results show that the method improves segmentation performance over existing baselines (e.g., TextSpan) and demonstrates the capability of adversarial attacks that lie on the natural image manifold. The study highlights the interpretability and practical utility of second-order effects in understanding and leveraging CLIP's representations.

Overall, it is recommended that this paper be accepted.

Additional comments can be found below.

Decision: Accept (Poster)