INViTE: INterpret and Control Vision-Language Models with Text Explanations
Authors: Haozhe Chen
Junfeng Yang
Carl Vondrick
Chengzhi Mao
Venue: ICLR 2024 poster (2024)
Keywords: Interpretation; Transformer
Primary area: visualization or interpretation of learned representations
TL;DR: Interpreting vision transformer's latent tokens in natural language without any data collection or training.

Abstract: Large-scale pre-trained vision foundation models, such as CLIP, have become de facto backbones for various vision tasks. However, due to their black-box nature, understanding the underlying rules behind these models’ predictions and controlling model behaviors have remained open challenges. We present INViTE: a framework for INterpreting Vision Transformer’s latent tokens with Text Explanations. Given a latent token, INViTE retains its semantic information to the final layer using transformer’s local operations and retrieves the closest text for explanation. INViTE enables understanding of model visual reasoning procedure without needing additional model training or data collection. Based on the obtained interpretations, INViTE allows for model editing that controls model reasoning behaviors and improves model robustness against biases and spurious correlations. Our code is available at https://github.com/tonychenxyz/vit-interpret.

Published as a conference paper at ICLR 2024
INVITE: INTERPRET AND CONTROL VISION- LANGUAGE MODELS WITH TEXT EXPLANATIONS
Haozhe Chen', Junfeng Yang', Carl Vondrick', Chengzhi Mao Columbia University', Milaa, McGill University hc32950 umbia. edu, {junfeng, vondrick, mez) www.bbia..cmmbbiceccooo.o
ABSTRACT
Large-scale pre-trained vision foundation models, such as CLIP, have become de facto backbones for various vision tasks. However, due to their black-box nature, understanding the underlying rules behind these models' predictions and controlling model behaviors have remained open challenges. We present INVITE: framework for INterpreting Vision Transformer's latent tokens with Text Explanations. Given latent token, INVITE retains its semantic information to the final layer using transformer's s local operations and retrieves the closest text for explanation. IN- ViTE enables understanding of model visual reasoning procedure without needing additional model training or data collection. Based on the obtained interpretations, INViTE allows for model editing that controls model reasoning behaviors and improves model robustness against biases and spurious correlations. Our code is available httpt:::////mm ottp:///ww....ooooooooooo//////a
INTRODUCTION
With the advent of large-scale foundation models, transformers have become the backbone for machine learning tasks (Zeng et al. 2022, Rombach et al. 2022 Girdhar et al. 2023) Devlin et 2018: Kirillov et al. (023). Recent advancements in large- scale vision-language transformers, such as CLIP (Radford et al. 2021 Jia et al. 2021) and FLAVA (Singh et al.) 2022). have enhanced
robustness and generalizability. These models leverage visual tokens to represent images and enable
visual reasoning through attention operation on different tokens (Dosovitskiy et al. 2020). However, since those latent tokens continuous, high-dimensional vectors (Geirhos et al. 2020 Mao et al.). is challenging to interpret and understand the reasoning procedure of transformers.
Interpretable models offer tremendous benefits, fostering better understanding and trust among users (Hendricks et al. (016), particularly in critical applications like medical diagnosis and treat- ment (Pereira al. 2018). Moreover, interpretable models enable users control and intervene, improving human-A Al interaction.
Previous attempts interpreting transformer models have primarily focused on generating saliency maps (Ancona et al. 2018 Selvaraju et al. 2016). lacking concept-level explanation for the latent embeddings. Although efforts such as NetDissect (Bau et al., 2017) connect concepts and neuron activations using labeled attribute datasets, these approaches require manual annotation, hindering generalization in open-world world settings. Other work generates language rationales for the model's decision, yet the prediction might not be made based on the provided rationales (Mao et al. 2022b Hendricks al. 2016, Kim et al. 2018b).
In this paper, we propose INVITE: an approach to INterpret Vision Transformer latent tokens using Text Explanations. the established CLIP model, a text description is retrieved from a set of provided vocabulary for entire image based on similarity between image and text representation. We extend this procedure to retrieve text descriptions for all latent tokens. Figure [ shows example text explanations and attention heat-maps associated with the interpreted tokens in ImageNet image. The interpretations are consistent with image areas that the tokens attend to and reveal CLIP's reasoning process of assembling parts into whole concept.
Retrieving natural language descriptions for latent tokens is challenging since transformers' latent space does not directly associate with word embeddings. Our key hypothesis is that latent tokens in

---

Published as a conference paper at ICLR 2024
Our Token Interpretation
Vision Transformer
Layer 13
Group Cars
Teoen
Orange
Vehicle
Tires
of Vehicles
Through
12 Tokon
Layer luhar 20
Tokoo
Laye 12 kuun
Figure I: Interpreting visual reasoning in transformer. Our approach allows text interpretation for latent tokens without any training or data collection. The diagram shows sample interpretations and image areas the tokens attend to. Our interpretations also reveal transformer's 's's hierarchical reasoning process between two layers.
transformers maintain same semantic information in the subsequent layers when they do not atlend to other tokens. Our approach capitalizes this property interpret latent tokens mapping latent embeddings to the final layer through forward propagation without self-attention operations.
pivotal advantage of framework is its ability leverage open-world vocabulary to explain latent token embeddings. By returning a language description for each latent token, our method directly elucidates the concepts learned within the transformers. Importantly, our approach does require data collection, model retraining, or modification of architectures (Kim et al. 2018a 8a Koh al. 2020).
Visualization and empirical experiments show that our method produces text explanations for latent tokens more aligned with the ground truth. Based on the provided interpretations, our framework enables users to intervene and control the model, such as fixing typographical attacks, removing spurious correlations, and replacing entities. Our approach works on open-world datasets, such as domain-specific satellite data, without additional training and data collection.
2 RELATED WORK
Transformer: The transformer architecture was first proposed in natural language processing by Vaswani et al. (017). Due to its ficxibility and versatility, it has become the standard architecture various domains (Devlin et al. 2018 Girdhar al. 2023 Dosovitskiy et al. 2020). The vision transformer (ViT) (osovviskiyy et al. (0200 was the first work to adapt the transformer for vision tasks. Consequently, there growing need to develop tools to understand and interpret transformer-bas based architectures. The success of transformers relies on the attention operation (Bahdanau et al. 2014). which computes each token in the model by attending becomes the backbone for vision foundation models (Zhu et al.) 2023 Radford et al. 2021 Li et al. 2023 Zhang et al. )022); ; however, the internal mechanism of transformer on visual reasoning still remains unknown.
few other tokens. Recently, transformer
Model Interpretation. Works such Netdisssect in (Bau et al. 2017) interpret neural networks with (016):Olaa Olah et al. (2017) find the inputs that maximize the outputs of features. Other works (Mahendran Vedaldi 2015 Zeiler feature visualizations (Lundberg Lee 2017 Ribeiro et al. 2016)) Simonyan al., 2014 (Shrikumar et al.) 2017 Zeiler & Fergus| 2014 Smilkov al. 2017] Sclvaraju et al. al., (016) find the input regions in images that are crucial to making decisions. Zou et al. (2023) establishes an Al transparency system that places representation at the center of analysis. Chen et al. 2023 builds sparse representation space to enhance explainbility model decisions.
top-activating image regions of individual neurons. Nguyen et al. (2017 2016);
Fergus 2014) invert neural feature maps to the pixel space for visualization. Gradient-
Recent works seek interpret models with natural language. MILAN (Hemandez al. 2022) finds natural language descriptions by maximizing the pointwise mutual information between ipput regions and human annotations. However, it requires additional data collection and training, thus cannot
2

---

Published as a conference paper at ICLR 2024
Image-Level Prediction
Token-Level Interpretation
Figure 2: I Latent token interpretation through disabling self-attention. La- tent tokens in transformers will re- tain the semantic information in the subsequent layers when they do not attend to other tokens.
MBA
No interactoon
Token Interpret
MSA
MBA
extend easily to open-world and domain specific tasks. Kim et al. (2018a) obtains concept activation vectors through learning on curated datasets. Other approaches (Mao et al. 2022b Hendricks et al. 2016 Hermandez et al. 2022) generate text explanations for model predictions, but these methods do not guarantee to represent actual reasons behind model predictions. Koh et al. (2020) interprets model prediction predefined set of concepts, which prevent its generalization to the open world. These methods also often develops on CNN and fail to provide token-level understanding of transformers. Attention saliency maps exploits transformers' attention based architectures by associating latent tokens with image areas they pay attention to (Caron et al., 2021 Abnar & ZZidemaa 2020). However, attention visualizations do not provide concept level interpretations that explain models' higher level reasoning processes such as abstractions. Concurrent to our work, Gaaddelanne et al. (2023) provides conceptual interpretations for transformers through decomposing contribution factors to final outputs.
Model Control. Deep models are often repurposed through techniques like transfer learning (Korn blith et al. 2019 Ying et al.] 2018) or fine-tuning. the context of vision foundation models, visual prompting has been used to adapt models (Jia et al. 22022; [Sandler et al. (022). However, these approaches typically require access to carefully tailored dataset specific to the task. For instance, to reduce bias, the dataset needs to be augmented with various text types to encourage the model to
disregard them (Shetty et al. 2019 Mao et al.) [021] Bissoto et al., 2020). However, engineering such training data can be challenging and may achieve perfection (Ponce et al. 2006 Torralba & Efros (011). Previous works studied controlling and rewriting generative models (Bau et al.) (020) and discriminative models (Santurkar al., (02TT) through modifying the weights. In contrast, our interpretation framework enables the model to directly modify the representations without altering the model weights during inference.
METHOD
3.1 PRELIMINARIES: ZERO-SHOV LANGUAGE RETRIEVAL
One widely used type of pretrained vision model includes the large-scale image-language pretrained models, such as CLIP (Rafford al. 2021). These models align visual and language representations through contrastive loss during training. They have demonstrated the ability to recognize objects in open-world setting and exhibit superior robustness and generalization. Consequently, they serve as the backbone for various tasks, including diffusion (Rombach et al. 2022). robust visual recognition (Mao al. 2022a), video perception (Xu al. 2021). and object detection (Gu et al.. 2021
Vision-language models are often trained on pairs of images and corresponding texts, denoted as (x;, tz). The image encoder is represented as Fo, which uses the final CLS token as the output. The text encoder is denoted as T'. The model is trained using contrastive loss to align the image feature Fo(x;) with the corresponding text feature Tt..). After training, the model can perform visual recognition through nearest neighbor retrieval in the embedding space. From all possible descriptions in set of provided vocabulary, it retrieves the nearest language descriptions predictions for the visual input.
3.2 INTERPRETING LATENT TOKENS THROUGH DISABLING SELF-ATTENIOON
We aim to gain insight into the latent tokens in a transformer model using natural language interpreta- tion. A straightforward approach would be to train the latent tokens alongside their corresponding natural language explanations. This would allow us to create a space where we can retrieve from set of provided vocabulary nearest-neighhor text description each latent token embedding. However,
3

---

Published as a conference paper at ICLR 2024
Original image
interpretation on Layer
Resulting in Layer K+1
L11130: faces
breck and white dogs
L11133:
looking troough Fance
grass and seem to be encorng the turm
L10T horse in the farm
L11T5C: strew the ground
L-2T40: caws Rilk
L12T19:
L12T31:
L12T42:
L12T48:
L1ST1:
L121B:
L121 L12122:
L
L12 12132:
garden
L19721: vegetated
L12760. blue jeans
L18T49: fences and 1eore
L12T42: suits and bags
L12T38: black vericles
112 1201 blue colored spots
L:311:: welking street
L11T18: straght hair
L:1T: 7: headband
L11T24: creat
L11726 carved
L11T39: Neath
L12T25:
Figure 3: Examples of text interpretations obtained with INVITE. We show text explanation for j-th token in i-th layer as LiTj. To qualitatively verify correctness of these explanations, we visualize image areas most crucial to the token interpreted via rollout attention generated saliency map. Interpretations produced by our method are consistent with image areas most crucial to the tokens.
Additional example explanations of tokens from earlier layers are provided in Appendix D
unlike the text labels that are naturally paired with the entire image, we lack supervised text targets explicitly paired with latent tokens in each layer transformers.
Ideally, if we could establish a mapping that converts the latent space to the final CLS output space, we could first find the final CLS output given the latent token query, and then use the final CLSS output to retrieve language interpretations. However, since we lack data pairs of latent tokens and their representations in final CLS output space, we cannot directly train a model for such mapping.
We propose method to map the latent tokens to the final CLS output space without any training or data collection, where we focus on repurposing the given transformer architecture. The Transformer architecture contains sequence of blocks, where each block is built from layer of attention (MSA) and a multi-layer perceptron (MLP) with LayerNorm (LN). Residual connections used inside the block.
self-
4

---

Published as a conference paper at ICLR 2024
The architecture of typical transformer can be formulated this: ho = [x xx3; 1 + Epos
(1)
h/g = MSA(LN((R-1)) + 1,,,,,
(2)
hg = MLP(LN((()) + -hg, 1,..,,
where we denote the CLS token Xote. the j-th visual token input as zz, the positional embedding as Eposs and the total number of block as L.
We divide the operations in the transformer into two types: in-token operations and inter-token operations. The mapping achieved by in-token operation can be denoted as a unary function Z = fx(x1), such as MLP, BN, and activation functions. The resulting feature Z cannot contain additional information than that already contained in the input X1; thus, Z does not have additional or different information than X1. Inter-token operations, specifically multi-headed self-attention (MSA), combine information from multiple tokens, which can be denoted as a multi-function Z = (x1, X2, X2.... Xn). The output token Z thus can contain different information from each of X1, X2, Xn-
Motivated by this observation, we propose an approach to interpret the j-th token in the i-th layer. Our key insight is that, the transformer computation, if a token does not interact with other tokens through eelf-attention, it will maintain the information local to the token in the subsequent layers.
We thus disable the attention operation from the i-th layer and above, retaining only the in-token operations. In MSA, this involves disabling the computation of keys (K) and queries (Q), and keeping only the token values (V) calculated from a linear transformation. We obtain the j-th output token in the final layer by forwarding the j-th token through the in-token operations. We can then use this output token's 's embedding to retrieve, from a set of provided vocabulary's embeddings, the corresponding language interpretation for the j-th token on the i-th layer. We will modify the computation of the features as follows:
i.....! h/g (L) + hg-1,
(4)
h MLP(LN(b(.) + hgg k. == i,..... L
(5)
The linear transformation operation for V in the MSA attention is denoted as fv, and i is the layer number we aim to interpret.
3.3 CONTROLLING MODEL BEHAVIOR THROUGH TOKEN REPLACEMENT
While previous works such as (Santurkar et al. 2021 Bau et al. 2020) control deep models by modifying their weights, we propose a method to control models by modifying their latent tokens. Guided by the text explanation of latent tokens, we can edit the model's reasoning procedure at inference time by replacing specific tokens with desired ones. Effective model editing based on INVITE text interpretations will also indicate the high quality of those interpretations. To demonstrate the effectiveness of our approach, consider three tasks:
Fixing Typographical Attacks. Typographical attacks involve overlaying words on images to cause model iis-preddiction,, which has been successful against the CLIP model. Guided by INVITE interpretations, our approach first identifies latent tokens that interpret to descriptions such as This is a text and This a word. We then replace these tokens with zero vectors. Since the dot product of the zero vector with all other tokens is zero, information from this token is not combined into other tokens in the next layer. The zero token will become a equal combination of all other tokens on the next layer since attention value goes through softmax. Our method intervenes in the transformer's s reasoning process and removes the text's influence during inference requiring no additional data, training, or assumption about the content of the overlayed words.
Intervening in the Reasoning Procedure. We investigate whether INVITE interpretation explains the causal influence between a token and final image representation through examining whether our interpretation enables us to intervene in the transformer's reasoning process. For instance, when looking at a satellite image of cars on concrete roads, the existing CLIP model can reason that it is highway. Our method first identifies the latent tokens that interpret to texts such as This is a car and This is an automobile in highway image. Then, we extract tokens that interpret to texts such as This is a plane and This is an aeroplane in an airport image. Finally, we replace each car token with an airplane token, thereby intervening in the reasoning procedure of the transformer. A visualization of
5

---

Published as a conference paper at ICLR 2024
Table I: Fixing Typographical Attacks. The original CLIP model performs well on clean forest images, yet it misled by typographical attacks of overlaying text ocean on images. Using INViTE interpretations, we can explicitly remove the latent tokens that represent text, which repairs attack in 97% of samples.
No Intervention
Random Intervention
Our Intervention
Our Intervention (RS)
original image predicting forest
94.00
97.00
97.00
97.00
original image prodicting ocean 4
5.00
3.00
0.00
0.00
attack image predicting forest 1
1.00
17.00
%6 attack image predicting occan
99.00
82.00
0.00
0.00
Table 2: Fixing Typographical Attacks. We followed experiment setup in Hernandez et al. (0022).
With random smoothing, our method improves accuracy on attack images by 35.20%. In contrast, best improvement in Hernandez et al.] (2022) is 4.9%. Editing based on INVITE interpretations also outperforms editing based various saliency map methods.
Accuracy (original image)
Accuracy (attack image)
No Intervention
2022 I Hernandez
69.90%
With Intervention
Hemandez 212
63.70%
No Intervention (Ours)
99.80%
$4.00%
Random Intervention (Ours)
99.40%
$1.80%
Raw Attention
98.20%
81.40%
Raw Attention Grad Selvaruuu al.
2016
95.40%
74.60%
Rollout Attention Abnar Zundemal
96.
Rollout Attention Gaad (hhter of Interpretation bused Intervention (Ours)
21X
98.20%
55 55.60%
99.20%
88. 80%
Interpretation-bue hased Intervention w/ RS (Ours)
99.20%
89.20%
the intervention process is shown in Appendix D If our intervention is successful, the transformer should predict airport through reasoning on concrete roads and airplanes.
Reducing Spurious Correlations. In contrast work that modifies the model weights, we explore how our framework could remove spurious correlations through modifying token representations. We identify the tokens associated with spurious features and disable them with zero vectors. We will demonstrate our method's effectiveness for removing spurious features through classfication tasks that finetune a linear layer on CLS output embedding.
EXPERIMENT
Dataset. VAW dataset (Pham et al. 2021) is large-scale visual attributes dataset with bounding box labels for the attribution annotation. We use it to study whether the annotated attribute emerges in vision transformer reasoning. We evaluate over the validation set, which contains 3297 images. UC Merced Land Use Dataset (Yang & Newsam 2010) contains remote sensing satellite images of classes, with 100 images in each class. We use to study fixing typographical attacks and intervening in the visual reasoning procedure. We also use 2088 unique nouns and adjectives extracted from remote sensing image captions of RSICD dataset (Lu et al.)) as textual vocabulary to retrieve interpretations from. CelebA Dataset. (Liu et al., 2015) We the task of classifying the hair color as gray or not gray. The label is spuriously correlated with gender. Our goal is to use our framework remove these spurious correlations.
Implementation Details. Our experiment focuses on the CLIP-B/32 model, while our method is general and can be used for any other transformer-based architecture. We single Titan RTX GPU with 24GB memory for our experiment. We found that applying small random smoothing noise output of each layer when forward propagate without attentions improves performance of model control tasks. We reason that random smoothing might help address potential embedding space distribution shift resulting from disabling attention. Details are discussed appendix We denote results obtained using random smoothing with RS in the following.
4.1 EVALUATING MODEL INTERPRETATION QUALITY
Quantifying Token Interpretation Quality through Causal Intervention. To quantitatively
evaluate INVITE interpretation's 's quality, we employed causal tracing, an established method to interpret neural networks Meng al. (002). We will study whether our text explanation interprets

---

Published as a conference paper at ICLR 2024
X of IOP (Oun) > 1:775
42-
(Random))
30-
120
20-
15-
000
12
15
11
Lager
(b) Percentage of tokens having high overlap with VAW bounding box labels (IOP 0.75) in differ- ent layers. Overlaps between bounding box labels and image areas most crucial to tokens interpreted as the labeled objects significantly exceed over- laps obtained with random masks, indicating interpretation's 's effectiveness in capturing visual concepts.
(a) Rank change under causal intervention on ob- jects. Masking out interpreted objects result in larger rank change of interpreted descriptions than applying random masks, indicating that inter- pretations are aligned with the actual object.
Figure 4: Results from quantitative evaluations of interpretation quality.
the right object by masking out that object from the input. We first interpret each token with our method (top text in cosine correlation ranking). Then, we mask all presence of the object that the token interprets to with bounding boxes provided by VAW and produce another cosine correlation ranking of texts for each token. We then measure how much the original top-ranked text changes in ranking after the masking intervention. We focus on the tokens that interpret to object names or positive attributes labeled in VAW. A large rank change implies that our method provides meaningful interpretations since masking out the object directly affects our token explanation. We also compare this result to rank changes after applying random masks (same total area as VAW mask for the image) as baseline.
Figure 4a shows the results of rank change after masking out the explained objects. For both vanilla interpretations and interpretations with random smoothing (RS), masking out the labeled objects causes interpretions' rank change larger than applying random masks by 40% to 120%. This significant gap demonstrates the validity of our text explanations through their tight links to the annotated attributes.
Quantifying Token Interpretation Quality through Saliency Map Overlap. Rollout attention flow (Abnar& & Zuidemal 2020) provides a way to associate a token and image area attends to. After min-max normalization of the rollout attention weights, produce mask with threshold of 0.99 in the input space. Suppose a token interprets to be one of the object names in the image annotation, we caleulate the Intersection over prediction (IOP) ratio between the produced mask and the union of all bounding boxes of the interpreted object. IOP is formally defined as follows: IOP mranttruthrprediction areal predittciction)
Note that the denominator differs from the commonly used Intersection over Union method, which uses the union of truth and prediction the denominator. We modify this because when a token's 's saliency mask only covers part of an object, it is still a valid interpretation. For example, if token is interpreted as a car, and its saliency map only covers part of the car, the car is still correct explanation for that highlighted part. We measure the percentage of tokens whose attended areas have high overlaps with ground truth bounding boxes (IOP 0.75). We compare this percentage with that resulted from random masks with the same area annotated bounding box area.
The result is shown Figure 4b random masks demonstrates that our text explanation is better aligned with the ground-truth objects, reinforcing the effectiveness of our interpretation in capturing visual concepts.
The large difference between percentages obtained from ours and
4.2 CONTROLLING VISION TRANSFORMER VIA TEXT EXPLANATION
Fixing Typographical Attacks. Typographical attacks Santurkar al. 1 real-world threats for vision foundation models. We tested fixing typographical attacks with satellite images of forest class (100 images). The task zero-shot classifies between classes ocean, forest, runway, parking,
7

---

Published as a conference paper at ICLR 2024
Table 3: The effectiveness of our interpretation for model editing. We show the prediction from the original model, from random replacing, from using our approach, and from our approach with random smoothing (RS). Using our approach with random smoothing can provide better intervention to the reasoning. Our method achieves 77% of the intervention efficiency. This demonstrates that our token interpretation not only explains what each token represents but also the causal relationship between a token's 's and final image's representation in ViT's s reasoning process.
Replace
Random Replace
Ours (RS)
Highway 4 Airport
100.00%
82.00% 18.00%
27.00%
5.00%
0.00%
73.00%
95.00%
Table 4: Debias CelebA result. We show the classification accuracy of original linearly probed CLIP model (baseline) and our approach, using interpretation guided intervention largely reduce the spurious correlations and improve model's performance on the worst group.
Weighted Average
Male
Male Non-Gray Hairt
Female Gray Hairt
Female Non-Gray Hairt
Gray Hairt
Baseline
58.22%
99.67%
15.85%
19.68%
99.67% %
Raw Attention
70.00%
90.71%
42.00%
59.500 59.50%
92%
Raw Attention * Grad (Selvaraju al. 2016
66.17%
99.10%
19.01%
49.52%
98, 86%
Rollout Atlention [Abnar & Zandenil 2020
69.95%
91.61%
38.27%
39.98%%
91.12%
Rollout Attention + Grad (Cheler
202
62.48%
99.43%
25.20%
28.53%
98.781%
Our Intervention
81.66%
97.23%
74.01%
58 00.00%
98. 29%
Intervention
83.91
97.23%
74.80%
66.56
97. 77.80%
and residential by selecting the top cosine correlation pair between CLS embedding and label text embedding. We conduct typographical attacks by overlaying random white boxes containing black text ocean on images. Example attack and how our interpretation reveals attack's 's effects on model reasoning are shown Figure 5 To fix typographical attack, replace tokens that interpret to the following descriptions with zero vectors: word, text, a word, a line of word, etc. Note that our method does not require any assumption about the actual content of the attack word. To demonstrate the effectiveness of fixing attacks with our natural language interpretations, we also measure results obtained by randomly removing tokens (the number of randomly removed tokens equals the number of tokens interpreted to word-related descriptions).
Table shows the result on original and typographically attacked images. Inferences with and without intervention all perform well on original images. Typographical attack subverts 95% of the images to the wrong prediction. If we remove random latent tokens by the same number as our intervention, it only mitigates 17% of the examples. Using our text explanation as guidance and removing the tokens related to words, we can correct 97% of the samples, demonstrating the accuracy of our text explanation and how it can help fix the typographical attacks. On average, our intervention (with RS) replaces 0.533 tokens per layer.
To compare our method's 's effectiveness of fixing typographical attacks with previous works, we followed the experiment setup in Hernandez et al. (2022), which uses images from 10 categories (50 images per category) in ImageNet validation set. Typographical attack is conducted by overlaying the text of another randomly selected category's 's label on the image. The procedure of identifying and removing word related tokens resembles the setup above. While best improvements sample accuracy in Hernandez et al. (2022) is 4.9%, we improved accuracy on attacked images by 35. 55.20% demonstrating the effectiveness of our text explanation in finding the right token. We also compare editing based on our method and editing based on following saliency map based methods: Raw attention values from penultimate layer final layer CLS token; GradCAM [Selvaraju al. (016)) with the same implementation for VIT in Chefer et al. (202I) (raw attention values from penultimate layer to final layer CLS tokens * their gradient respect to similarity with prompt this can be described as text); Rollout attention from CLS token to input layer Abnar & Zuidema (2020); Rollout attention from CLS token to input layer where each attention matrix multiplied by its gradient respect similarity with prompt this can be described remove typographical attack, we mask the parts of the image with map > threshold with 0 and report results with best threshold. We found that method outperforms all saliency map based methods.
attacked
text Chefer et al. (202 1). To
Object Entity Intervention. On the UC Merced land use dataset, we pick the images that are classified as Highway, and test if our approach can modify the prediction to be Airport by intervening on the latent reasoning procedure. For our random replacing, we replace the same number of tokens as the number of tokens that interpret to associate with cars. Since features in different transformer
8

---

Published as a conference paper at ICLR 2024
layers lie in different representation spaces, replace tokens with those from the same layer. Results are shown in Table demonstrating that our interpretations allow for effective model intervention. On average, intervention (with RS) replaces 1.972 tokens per layer. Details on the layerwise number of replacements are shown in Appendix F
Remove Gender Bias. Machine learning mod- els often fail due to attending to spurious cor- relations. In the CelebA dataset, gender can be a spurious correlation for predicting hair color. We conduct a binary classification of grayy hair vs. non-gray hair on the CelebA dataset by fine- tuning linear layer on CL.S token embedding. We fine-tune the linear layer for one epoch with Adam optimizer (Ir = 10-3). We first fine-tune model without intervention and measure accu- racy for each class, Then, we fine-tune a model while replacing all tokens on layer 12 that inter- pret to descriptions outside a list of hair-related words (hair, gray hair, gray, not gray hair, hairstyle, curl hair, straight hair) with zero vector. At inference, we conduct the same replacement. In addition to the hair words, we use 5000 randomly sampled unique nouns and adjectives extracted from GPT description for ImageNet (Mao al.. 2022b) and words related to gender such as gender, male, and female (a full list is shown in Appendix C as the vocabulary for interpretation. We also manually avoided the replacement of the CLS token shows the accuracy before and after our intervention to remove the gender bias. Our text explanation can guide the intervention and achieve up to 59% performance gain in the worst group, demonstrating the accuracy of our text explanation. We compare editing performance based on our interpretation to that based on the same saliency map based methods above. We mask out image parts with non- non-gradient based map >> threshold or with gradient- based map < threshold with O, where gradient is taken respect to respect to similarity with this can be described as hair and best threshold is reported. We found our method outperform all saliency maps. In addition, random smoothing can provide a better explanation, which leads to even higher overall performance. On average, replaced 43.58 tokens layer 12 (with RS). Note that this number is larger than the previous two experiments since our intervention aims to retain a minor detail and remove most other features.
LI2TO
LEITAS Mariko
Figure 5: An example token interpretations in the presence of typographical attack. The model derives lake-blue from tokens that attend to text ocean and eventually produces false CLS token interpretation ocean instead of forest.
to avoid excessive information loss. Table 4
4.3 ANALYSIS
Our Interpretations Reveal Visual Reasoning. In example latent token interpretations under our framework in 3 we demonstrate our framework's s capability to explain complex interactions in images, such as the farm animals are eating grass and seem to be enjoying the furm. In addition, our framework reveals the emerging visual reasoning procedure in the transformer. In the middle section, we show interpretations of tokens from layer that the tokens from K + 1 shown in right section attend to most and thus demonstrate vision transformer's 's hierarchical reasoning process. For example, the model combines latent tokens representing objects (motor vehicle) and context (valley) to predict an overpass. To recognize cows in field, the model combines black and white, legs, and grass. We interpreted the first two images from the VAW dataset with vocabulary extracted from MILAN annotations (Hernandez al., 2022). The second to fifth images from UC Merced Land Use Dataset are interpreted with vocabulary extracted from RSICD captions. The last image from the CelebA dataset is interpreted with vocabulary extracted from the GPT description ImageNet.
5 CONCLUSION
Our work proposes INVITE, a novel methodology for interpreting the latent tokens of visual founda- tion models with natural language descriptions by mapping token representations to the semantic space of the final layer with local transformer operations. Our approach eliminates the need for additional model training or data collection and extends to open-world and domain-specific datasets well. Qualitative and quantitative experiments demonstrate the effectiveness of our text interpretation. Our methodology gives insights on how visual foundation models reason to provide their predictions. Model editing based on our interpretations allow users to control model reasoning behaviors and enhance model robustness against spurious features and biases. Our method facilitates increased transparency and usability, fostering the development of accountable and responsible AI.

---

Published as a conference paper at ICLR 2024
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190-4197, Online, July 2020. Association for Computational Linguistics. doi:
URL
htt.es//beeaanthoooo..oo///////2 aaaamaaaaamaannnnn.aaaa 385
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- attribution methods for deep neural networks. In International Conference on Learn- ing Representations, 2018.
wttp://wwwweeeettttteeeeeeaeee htt.:/ee///ooae forum wtt.:e/eoroommmmddd2222333... 21d=5y21R9JAN
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv: 409.0444 2014.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Putlern Recognition, 2017.
David Bau, Steven Liu, Tongzhou Wang, Jun- Yan Zhu, and Antonio Torralba. Rewriting deep 2020: 16th European Conference, Glasgow, UK,
generative model. In Computer August 23-28. 2020, Proceedings, Part 16, pp- 351-369. Springer, 2020.
Alceu Bissoto, Eduardo Valle, and Sandra Avila. Debiasing skin lesion datasets and models? not so fast. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 740-741, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 182-791, June 2021.
Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. Stair: Learning sparse text and image representation in grounded tokens, 2023.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pp. 1310-1320. PMLR, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11999, 2020.
Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition, 2023.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 (1::665667777 2020.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. arXiv preprint 2006003300 2023.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulan object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.139 2021.
10

---

Published as a conference paper at ICLR 2024
Lisa Anne Hendricks, Zeynep Akata, Mareus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In European conference on computer vision, pp. 3-19. Springer, 2016.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12111 12119, 2022.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018a.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. In Proceedings of the European conference on computer vision (ECCV), pp. 563-578, 2018b.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Lo, et al. Segment anything, arXiv preprint arXiv:2304.08 02643, 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338 5348. PMLR, 2020.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661-2677 2019.
Alexander Levine, Sahil Singla, and Soheil Feizi. Certifiably robust interpretation deep learning. arXiv preprint arXiv: 20055/2. /2/05, 2019.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12559, 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing, 56(4):2183-2195. doi: 10.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4774, 2017. URL htRt:ttttp/p/rccednnggee neurips cc. /pppe//p///// html
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188-5196, 2015.
Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa. Discrete representations strengthen vision transformer robustness. In International Conference on Learning Representations.
11

---

Published as a conference paper at ICLR 2024
Chengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang, Junfeng Yang, and Carl Vondrick. Generative interventions for causal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3947-3956, 2021.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprini arXiv:2212.07011 2022a.
Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Wang, and Carl Vondrick. Doubly right object recognition: why prompt for visual rationales. arXiv preprini arXiv:2212.06202, 2022b.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
Anh Nguyen, Jeff Clune. Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug play generative networks: Conditional iterative generation of images in latent space. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 20/7 pp. 35200. IEEE Computer Society, 2017. doi: 10.1 109/CVPR.2011..374 URL /ttt::://dii.rg/100.. wttp://...111///PPP... 2017. .374
Anh Mai Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthe- sizing the preferred inputs for neurons in neural networks via deep generator networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike Luxburg, Isabelle Guyon, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3387-3395, 2016. URL httpt:ttps///eet / htt.ttpss//proceednggge www.nngg....rrrrree pps cc cc/paper/2011emm 6/hash/ 5d7 www3999fcdf499999 0ww.0/99..04999111777777999
64a- ract. html
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 112399//distill000//
Sérgio Percira, Raphael Meier, Victor Alves, Mauricio Reyes, and Carlos A. Silva. Automatic brain tumor grading from MRI data using convolutional neural networks and quality assessment. In Danail Stoyanov, Zeike Taylor, Seyed Mostafa Kia, Ipek Oguz, Mauricio Reyes, Anne L. Martel, Lena Maier- Hein, Andre Marquand, Edouard Duchesnay, Tommy Löfstedt, Bennett A. Landman, M. Jorge Cardoso, Carlos A. Silva, Sérgio Pereira, and Raphael Meier (eds.), Understanding and Interpreting Machine Learning in Medical Image Computing Applications First International Workshops MLCN 2018, DLF 2018, and iMIMIC 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16-20, 2018 Proceedings, volume 11038 of Lecture Notes in Computer Science, pp. 106- 114. Springer, 2018. doi: 10. 00107777788303000200 8 12. URL 030-0000-22220 02628 12
http::://6oi.orr//0..00000 1007
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13018-13028, June 2021.
Jean Ponce, Tamara L Berg, Mark Everingham, David A Forsyth, Martial Hebert, Svetlana Lazebnik, Marcin Marszalek, Cordelia Schmid, Bryan C Russell, Antonio Torralba, al. Dataset issues in object recognition. Toward category-level object recognition, pp. 29-48, 2006.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. 8748-8763 PMLR, 2021.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, San Francisco, CA. USA, August 13-17, 2016, pp. 1135 1 1144, ACM, 2016. doi: 10.114529396722293777 URL https 7/doi.orge / 1t..0////....11/91111
12

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems, 32, 2019.
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Andrew Jackson. Fine-tuning image transformers using learnable memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12155-12164, 2022.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Alek- sander Madry. Editing a classifier by rewriting its prediction rules. Advances in Neural Information Processing Systems, 34:23359-23373, 2021.
Ramprasaath R. Selvaraju, Abhishek Das. Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient- based localization. CoRR, abs/1610.02291,, 2016. URL http:://axxi...o////
1610. 02391
Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk--uantifying and controlling the effects of context classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8218-8222 2019.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume of Proceedings of Machine Learning Reseurch, pp. 3145-3153. PMLR, 2017. URL
igs ml wtw...mr....ss....m
www.sprsssvv....... /vwp///0///// shr kumarrl 7a html
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. In Workshop at International Conference on Learning Representutions, 2014.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 35638-15650, 2022.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, abs/1706.03825, 2017. URL /ttp::/axxvvvorr... abs/1705 03825
Antonio Torralba and Alexci A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109. /4084, 2021.
Yi Yang and Shawn Newsam. Bug-of-vissullwordd and spatial extensions for land-use classification. In ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM GIS), 2010.
Wei Ying, Yu Zhang, Junzhou Huang, and Qiang Yang. Transfer learning via learning to transfer. In International conference on machine learning, pp. 5085-505094 PMLR, 2018.
13

---

Published as a conference paper at ICLR 2024
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-1 2014, Proceedings, Part I 13, pp. 818 818-833. Springer, 2014.
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Puro- hit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00520 2022.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Andy Zou, Long Phan. Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. Representation engineering: top-down approach to ai transparency, 2023.
A USING OUR INTERPRETATION TO UNDERSTAND ADVERSARIAL ATTACK
Given image X of class c, we conduct adversarial attack on CLIP by finding e such that CLIP makes wrong zero-shot classification with X as class e'. We find such through optimizing following loss through gradient descent:
argmin CrossEntropy (M(X + +e) (tex)() ea) + Alle||2
where ee is a one-hot vector with c'-th location equals 1, M is CLIP image encoder, T(text) is CLIP's encoding of shot class prompts. We used regularization coefficient 0.01.
We conduct such attack on an image of forest class as showin in o There's 's no visible difference for image before and after attack, but CLIP misclassifies 66b as beach.
(a) Image before attack (X)
(b) Image after attack (X + e)
Figure 6: Comparison of forest images before and after the attack. There's no visible difference.
CLIP shot classifies 6a forest and 6b to beach.
14

---

Published as a conference paper at ICLR 2024
Table 5: Example interpretation of latent tokens before and after adversarial attack.
Layer
Token
Before Attack
After Attack
5
8
vegetation
ultramarine rain
12
7
lorest
12
texture
12
11
forest
pool
15
rattan
algac
60
Attack
No Attack
50
Interprets
30
20
%
0
9
10
11
13
Layer
Figure 7: Percentage of tokens interpreting to Forest Beach before and after adversarial attack each layer. Adversarial attack impacts model most starting around layer 10.
To qualitatively examine effects of adversarial attack from latent tokens, we first interpret CLIP latent tokens before and after attack with open vocabulary. We provide a few examples in Table
We also quantitatively study effects of adversarial attack. We interpret latent tokens with This is forest and This is beach and record the percentage of tokens that interpret to each prompt on every layer. Result is shown in Figure 7 Adversarial attack impacts model most starting around layer 10.
B MITIGATING DISTRIBUTION SHIFT IN TOKEN INTERPRETATION VIA RANDOM SMOOTHING
In model control experiments, we have found that adding random smoothing noise to each layer's output when forward propagate without attention increases model control performance. We reason that random smoothing might attain this improvement by addressing potential distribution shift resulting from disabling attention.
Our algorithm involves intervening in the original inference procedure of the transformer model and constraining perform computations on individual tokens. Once we disable the Multi-Head Self-Attention (MSA) operation, the calculations are limited to the MLP (Multi-Layer Perceptron) and BN (Batch Normalization) layers starting from the layer where token of interest situates. The MLP and BN layers move the token toward the final image-text representation space without additional training.
Since a transformer is trained with attention enabled, it achieves best performance when token is combined with other tokens. Our method of disabling cross-attention creates different distribution of token embeddings.
B. DISTRIBUTION SHIFT ANALYSIS
To understand this distribution shift, we employed t-SNE (Van der Maaten & Hinton 2008 analysis on the original and intervened token embedding distributions at different layers. Figure shows that the token embeddings resulted from removing attentions largely reside the same regions as those obtained from attentions enabled. The result also reveals that attention mechanism might plays
15

---

Published as a conference paper at ICLR 2024
Table 6: Average Drift
CLS
Other
CelebA
0,08264
0.10314
VAW
0.07910
0.08703
UC Merced Land Use (Forest)
0.09039
0.11492
UC Merced Land Use (Highway)
0.09020
0,10192
larger role in the penultimate layer to compose different concepts into high-level understandings as removing attentions create larger distribution shifts.
Over the VAW and CelebA dataset, we mea- sure the distribution shift between with attention and without attention with L2 norm between the center of the distributions. We obtain the Lg difference between the mean vector of tokens produced from a previous layer with attention and that from a previous layer without attention. Note that we only measure the drift for layer to layer 12 since interpreting layer 13 does not require disabling attention. In Table 6 we com- pared drift of CLS token and non- non-CLS token by averaging across layers. The result shows that CLS tokens results in smaller drifts than other tokens, agreeing with the histogram in Figure g
0.230
Non CLS
0.170
0.130
0.820
0.075
0.050
0.020
0.000
9.0
0.1
0122
0.3
0.4
0.5
0.6
Distisnne
Figure 8: A histogram of L2 distance of individual tokens with and without attention. We show the histogram of the shifts for CLS tokens and non- CLS tokens separately. CLS tokens tend exhibits smaller drifts.
We acknowledge the distribution shifts intro- duced by removing attentions, especially for the penultimate layer. However, if the feed-forward model is robust to such distribution shifts and can extrapolate, accurate interpretations can still be obtained even with shifted embeddings. One crucial characteristic of a robust model under distribution shifting is smoothness, often indicated by small Lipschitz constraint. For instance, linear models with regularization tend to extrapolate well out of the distribution and possess a small Lipschitz constraint. contrast, deep models tend have larger Lipschitz constraints and exhibit less smoothness. Since the operations from from layer 12 to the output layer is shallow and nearly linear, reasonable interpretations may still be retrieved despite the relatively larger gap observed before and after removing attentions.
B.2 MITIGATING DISTRIBUTION SHIFT WITH RANDOM SMOOTHING
We propose incorporating random smoothing to further enhance the robustness of the feed-forwart process against the distribution shift. Previous studies (Levine et al. 2019 Salman et al. 2019 Cohen et al. 2019) have shown that introducing random smoothing can reduce the Lipschitz constant and increase model smoothness. In our approach, we adopt random noise to smooth the latent tokens, mitigating the effects of distribution shift and maintaining the model's performance.
When applying random smoothing, for j-th token on layer i, we sample from a normal distribution with mean of zero and standard deviation of the obtained average L2 drift for j-th tokens on i-th layer, where each component is independent of each other. We then pass the latent tokens added with this noise through the layer 100 times and obtain the token for the next layer by taking an average.
C VOCABULARY AND DETECTION WORDS
For fixing typographical attacks, we detect the text in latent representation by identifying tokens that interpret to following words: [wrdd, text, word, line of word, line of text, line of text white background, text over white background, black text, black text over white background, black alphabets, alphabets, letters, black letters).
16

---

Published as a conference paper at ICLR 2024
Cayer
2ayer
lout
MM 100 (tta)
KEA MAS of
vi MESI IO
wit MSA in
Laper
Loper
Lapuc
Figure 9: Distribution shifts introduced by removing cross attention. The blue/red dots are the original/our tokens. The purple color indicates they overlap in space. Token embeddings labeled as Ours on layer: are obtained by disabling attention operations between layer - 1 and i. This visualization is created from the CelebA dataset.
17

---

Published as a conference paper at ICLR 2024
Original Image
L2T34: columnar
L3T45: dietrddderbulledd
L4T20: floor
L4T20: wasteland
LST3:
LST40: blue-green
L7T27: columnar
L7150:
L7145: buildng
LST32:
L8T38: soll
Figure 10:
Example interpretations of tokens from earlier layers. Earlier layers often interpret to lower level features that reveal the model's 's reasoning process from more abstract concepts.
For intervening model reasoming experiment, we replace all tokens that interpret to car related tokens to airplane related tokens, which are randomly sampled from tokens that interpret to airplane in all airplane class images. Words used detect airplane token: [plane, aeroplane, aircraft, airplane, carrier, jet, airliner. flight.. Words used to detect car tokens: [car, automobile, motor vehicle, auto, truck, van, SUV, crossover, sedan, coupe, convertible, wagon, motorcycle, bike, bus, couch, lorry, truiler, caravan, camper, motorhome, RV, vehicles).
For reducing the gender bias, we conduct classification of gray hair vs non gray hair on CelebA dataset through fine-tuning linear layer on CLS token embedding. We fine-tuned the linear layer for one epoch with Adam optimizer (Ir 10 10-3). We first fine-tune a model without intervention and measure accuracy for each class. Then, we fine-tune model while replacing all tokens layer 12 that interpret to words other than a list of word related to hair (hair, gray hair, gray, not gray hair, hairstyle, curl hair, straight hair) with zero vector. At inference, the same replacement is done. In addition to the hair words, the vocabulary includes (1) 5000 randomly sampled unique nouns and adjectives extracted from GPT description for ImageNet; (2) words related gender and features that model might use to decide gender: [face, gender, male, female, man, men, woman, women, sex, boy, girl, man face, woman face, nose, eyes, breast, shoulder, mouth, ears, shoulders, shirt, jacket, tie, scarf, watch, hat, eyebrows, lashes, iris, pupils, nostrils, lips, teeth, tongue, temples, cheeks, cheekbones, chin, throat, arms, elbows, wrists, hands, fingers, chest, sternum, ribs, blouse, sweater, hoodie, blazer, pendant, watch, beanie, headband). We also manually avoided replacement of CLS token to avoid too significant of information loss.
D VISUALIZATIONS
We provide additional examples of our method's interpretation and associated token visualizations for earlier layers in models in Figure I0 In Figure [T] show interpretations of part of model on an airport image and a highway image. We also illustrate the procedure of extracting and replacing token in our intervention of model reasoning.
18

---

Published as a conference paper at ICLR 2024
Lapera
Layert
Token
CBl colca
sicor
Tekan 18
cellig
Sirvisy
Token
oniig
Teken 20
vigradis
Tokes
Takin 22
Tokes
k.5
Takmn 24
vegrasd
Sia om4
lorgan
Token
K309 lnd
parage
Tekmn 26
vmkky
Token
Token 28
ang frit
517
Token
Amal
Token 30
Insendd
aTMC
Extract
Replace
Figure 11: An example interpretations of part of model and demonstration of intervening process on reasoning procedure by extracting aircraft token from the airport image and replacing the automobile token in the highway image. The model then combines the information of concrete road environment and airplane object to classify the image as airport.
19

---

Published as a conference paper at ICLR 2024
Table Average Number Tokens Replaced per Layer (RS). We show the average number of latent tokens that are interpreted to contain the semantic of interest. For the 50 tokens in each layer of our studied transformer, we find the tokens are replaced sparsely, but the higher layers tends to be replaced more. The task for debias is slightly different in nature than the first two tasks. In debias, we only hope to focus on one feature (hair color) remove other potentially spurious features, while in the first two tasks we hope to remove/edit small details and focus information of the entire image. Therefore, the number tokens replaced in the last task is much larger. To prevent over-damage of image features, only removed tokens on the final layers (which contains best quality interpretations for object-level concepts and directly contribute to CLS) and left tokens on earlier layers intact.
Layer
5
9
10
12
Typographical Attack
0.04
1.04
0.24
0.27
0.52
0.46
0.65
0.76
0.73
0.53
0.67
0.48
Entity Editing Debias
1.52
2.22
1.58
1.15
1.36
0.94
1.73
2.46
2.41
0
0
0
0
43.58
9,015
Storot Diriiorry
500
12m11
41.15
8,6ch
0U90)
6:297
16:21
0.78
3300
44,42
$1.02
42.97
42/12
40.25
6:291
6.7%
0:775
3001
45.50
5000
Number Oljjeet Words
Number Objjeet
Weighted Average Accuracy.
(b) Number of Tokens Replaced.
Figure 12: Ablation study on changing vocabulary/object word list size. (a) Balanced, larger vocabulary and object word list sizes improve performance. (b) Those sizes affect the number of tokens replaced.
E TOKEN REPLACEMENTS
In Table 7 we show the average number of tokens replaced on each layer in different experiments.
F EFFECTS OF VOCABULARY SIZE
Our approach produces natural language descriptions by retrieving the closest match from set of provided vocabulary. During model editing, our method selects tokens that interpret list of object
words of interest to determine which tokens to edit. We conduct an ablation study on the effect of
vocabulary and object word list sizes over removing gender bias in CelebA dataset in Figure 12
where we find (1) larger vocabulary and object word list can generally improve performance; (2) performance is optimal when vocabulary and object word list sizes attain balance, so that information is not over-removed or under-removed as shown in 12b These findings provide guidance on choosing appropriate vocabulary when using our framework for model editing.
20


Reviews:
[{"review_id": "6k4SpB8VE5", "reviewer": "Reviewer_jxBW", "summary": "The paper proposes a method to interpret the “latent” representations of vision transformers. To interpret a visual token representation of layer K, the paper proposes to disable the self-attention for layers higher than layer K, and then use the final layer representation to calculate text-token similarity. The ability to find such latent tokens enables several applications: fixing typographical attacks, intervening in the reasoning procedure, and reducing spurious correlations.", "questions": "I am not sure I get the motivation behind the Intervening in the Reasoning Procedure experiment. Under what settings would this be useful?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Such an interpretation approach is definitely new; it is a novel observation that one could simply disable the self-attention from above a certain layer (K) and then use the representation as a representation for latent token at layer K. The paper uses causal intervention and saliency map overlap to verify the effectiveness of the approach.", "weaknesses": "My main concern is that I do not quite see the method’s advantage compared to gradient-based methods that find important input regions (e.g., Grad-CAM) given a text description.\n\n(1). The first question is why we want to find latent tokens but not salient regions?\n\nConceptually, the biggest advantage is that there might exist “high-level” and “abstract” latent tokens. For example, in Figure 3, using Grad-CAM to find regions corresponding to “overpass” might result in a lot of matched image patches while using the proposed method can find one single latent token corresponding to the concept.\n\nHowever, this is not reflected in the quantitative experiments. For the fixing typographical attacks and reducing spurious correlations experiments, ideally a gradient-based baseline could be included, where we seek to use grad-cam to find and zero-out a few important image patches. Then one could compare whether the proposed method can achieve the same performance but zeroing out less tokens. Otherwise, it is hard to claim that the method can find “high-level” latent tokens.\n\n\n(2). Suppose we wish to find latent tokens corresponding to a text description, why should we resort to the proposed method but not a gradient-based method where we calculate the gradient with respect to each latent tokens of every layer? This seems like a more principled way to obtain important latent tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCPGnS2ugY", "reviewer": "Reviewer_jZ36", "summary": "This paper introduces a method to interpret intermediate representations (referred to as *latent tokens*) in vision-language Transformer models (e.g. CLIP) by retrieving relevant natural language descriptions for individual latent tokens. The authors do so by removing the self-attention operations after a latent token, and using the last layer token representation to retrieve a text description. The authors posit that the retrieved text for a latent token provides an interpretation for it. They further posit that manipulating these latent tokens can be used to edit and control model behavior.", "questions": "- What is the benefit of the proposed method over the saliency maps that are presented in Figure 3?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The experiments are very convincing. The author use their methodology to not only do model interpretation (which would have been sufficient imo), but also for model controllability. The results for interpretability (causality and saliency map overlap) are very convincing. \n\n- The controllability experiments are also very well designed, and demonstrate consistent results across three different kinds of model editing -- typography attacks, entity editing and gender debiasing.\n\n- The paper is well-written overall. Some of the experiment setups are a little hard to understand, because the setting is slightly artificial,", "weaknesses": "- I am skeptical of the motivation behind the methodology. Specifically, typically the CLS representation from the vision encoder is used as the query to do text retrieval in CLIP, but do we know that latent tokens corresponding to other image patches from the final layer retrieve meaningful text concepts? There is no theoretical motivation -- which isn't strictly needed, but I would be much likelier to trust the method beyond just the empirical results.\n\n- It's unclear what the benefit of these natural language descriptions for latent tokens is, or how these descriptions should be leveraged. The examples in Section 4.3 (\"Our Interpretations Reveal Visual Reasoning\") do not seem very convincing, and are more about how one chooses to interpret the tokens' NL descriptions (e.g. I would not think \"motor vehicle\" + \"valley\" = \"overpass\" is necessarily a correct reasoning, let alone whether that's actually how the model went about the reasoning process).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2j4rFXBGIK", "reviewer": "Reviewer_aYwS", "summary": "The paper proposes a method to interpret the semantic meaning of vision foundation models by retrieving the closest text description. Specifically, the method turns off the inter-token cross attentions in the latter layers and only keeps the self-attention, in order to get the final CLS representation that corresponds to only the target token. Saliency attention visualizations and quantitative results are provided to show the correctness of the interpretations. To demonstrate the application of the proposed method in terms of controlling and intervening the model behaviors, the paper runs experiments on three tasks: fixing topographical attacks, intervening object entities, and removing gender bias.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper runs experiments on three practical tasks to showcase the applications of the derived interpretations. It is exciting to see that the interpretations can improve the model’s robustness towards text/topographical attacks and gender bias, as well as intervening the model’s decision in object classification in satellite images.\n\n2. Abundant visualizations and numbers are reported to show the advantage of the proposed method over random intervention.", "weaknesses": "1. The major limitation is that the method can only be applied on the CLIP model (at least only CLIP is shown in the paper). Since CLIP aligns the visual embeddings (CLS) with the texts, the method can retrieve texts based on the CLS embedding after turning off the cross-attentions. Otherwise, if the model’s token embeddings are not trained to be aligned with the texts, it is doubtful whether the method can still work or not. It would be best if the authors can show that the method can workin on other non-text trained models like MAE, DINOv2, etc.\n\n2. Most of the comparisons are against the “random baseline”, which is not a very strong baseline. For example, additional stronger baselines should be compared with, like the ones listed in the “model interpretation” in the related works sections. Moreover, it’s better to include related work [1].\n\n3. The method is based on text-retrieval, which assumes a finite set of candidate texts (closed world). In related work, the authors criticize that Koh et al (2020) (concept bottleneck network) is limited to a closed vocabulary, but isn’t this work also closed vocabulary? \n\n4. The method is training free. As an extension, can the method be extended for improving model training? \n\n[1] (arxiv 2301.13081 STAIR: Learning Sparse Text and Image Representation in Grounded Tokens)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "klZe7hVqZ9", "reviewer": "Reviewer_ZFTq", "summary": "This paper proposes a novel method to interpret the latent tokens in pretrained vision-language models like CLIP using natural language descriptions. The key idea is to map the latent token embeddings to the final output space by disabling self-attention and propagating through the feedforward layers. This allows retrieving the closest text description for each token from the model's vocabulary. The authors demonstrate how these interpretations can provide insights into the model's reasoning and enable controlling model behaviors like fixing adversarial attacks, reducing biases, and replacing entities.", "questions": "Can you provide more details on how the distribution shift introduced by disabling attention affects interpretation quality? Is performance very sensitive to this?\n\nWhat processes did you use to create the vocabularies for retrieving token interpretations? How important is vocabulary size and coverage?\n\nHow do your interpretation results compare qualitatively to other methods like saliency maps or concept activation vectors?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel method to interpret transformer tokens with language; doesn't require retraining or new data.\n\n2. Solid motivation,clear methodology, extensive experiments across multiple datasets.\n\n3. Well-written paper with clear explanation of the approach and results.\n\n4. Interpretability is valuable for ML model transparency and trust. Controlling models via token editing has useful applications.", "weaknesses": "1. More analysis could be provided on how distribution shifts from removing attention affect interpretation quality.\n\n2. More comparisons to related interpretation methods would further validate advantages.\n\n3. Significance could be boosted by showing applications beyond the demonstrated tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)