Towards Interpreting Visual Information Processing in Vision-Language Models
Authors: Clement Neo
Luke Ong
Philip Torr
Mor Geva
David Krueger
Fazl Barez
Venue: ICLR 2025 Poster (2025)
Keywords: Interpretability
Vision-Language Model
Mechanistic Interpretability
Multimodal
Primary area: interpretability and explainable AI
TL;DR: Visual information of objects in VLMs is highly localised and visual representations in the LM are refined towards interpretable text representations

Abstract: Vision-Language Models (VLMs) are powerful tools for processing and understanding text and images. We study the processing of visual tokens in the language model component of LLaVA, a prominent VLM. Our approach focuses on analyzing the localization of object information, the evolution of visual token representations across layers, and the mechanism of integrating visual information for predictions. Through ablation studies, we demonstrated that object identification accuracy drops by over 70\% when object-specific tokens are removed. We observed that visual token representations become increasingly interpretable in the vocabulary space across layers, suggesting an alignment with textual tokens corresponding to image content. Finally, we found that the model extracts object information from these refined representations at the last token position for prediction, mirroring the process in text-only language models for factual association tasks. These findings provide crucial insights into how VLMs process and integrate visual information, bridging the gap between our understanding of language and vision models, and paving the way for more interpretable and controllable multimodal systems.

Published as a conference paper at ICLR 2025
TOWARDS INTERPRETING VISUAL INFORMATION PROCESSING IN VISION-LANGUAGE MODELS
Clement Neo Luke Ong', Philip Torr", Mor Geva", David Krueger', Fazl Barez
Nanyang Technological University UUniversity of Oxford TTe Aviy University MIILA Taageettcc
ABSTRACT
Vision-Language Models (VLMs) are powerful tools for processing and under- standing text and images. We study the processing of visual tokens in the lan- guage model component of LLaVA, a prominent VLM. Our approach focuses on analyzing the localization of object information, the evolution of visual token rep- resentations across layers, and the mechanism of integrating visual information for predictions. Through ablation studies, we demonstrated that object identifica- tion accuracy drops by over 70% when object-specific tokens are removed. We observed that visual token representations become increasingly interpretable the vocabulary space across layers. suggesting an alignment with textual tokens corresponding image content. Finally, we found that the model extracts object information from these refined representations at the last token position for pre- diction, mirroring the process in text-only language models for factual association tasks. These findings provide crucial insights into how VLMs process and inte- grate visual information, bridging the gap between our understanding of language and vision models, and paving the way for more interpretable and controllable multimodal systems.
INTRODUCTION
Vision-Language Models (VLMs) take an image and text as input, and generate a text output. They have become powerful tools in processing and understanding text and images, enabling a wide range of applications from question answering to image captioning (Liu et al., 2023b; Li et al., 2023; Alayrac et al., 2022). Among VLM architectures, the "adapter" style has demonstrated impressive state-of-the-arr performance, notable for its simplicity. These models combine a pre-trained image encoder, a pre-trained language model (LM), and a learned adapter network that maps image encoder outputs to soft prompts for LM inputs (Merullo et al., 2022; Liu et al., 2023b;a). Despite their importance and potential, the inner workings of VLMs remain poorly understood as compared to their text-only counterparts. While significant progress has been made in understanding language models (Elhage et al., 2021; Wang et al., 2023; Bricken et al., 2023) and, to a lesser extent, vision transformers (Palit et al., 2023; Vilas al., 2023; Pan et al., 2024), there is notable gap in our understanding of VLMs, where both modalities operate in the same space. deeper comprehension of VLMMs could lead to practical advances in safety, robustness, and functionality, similar to the progress made in language models, such as model editing (Meng et al., 2022) or identifying specific components responsible for certain behaviors (Arditi al., 2024).
In this paper, we investigate the LM component of VLMs as they serve as the "brain" of the system, comprising more than 95% of the total parameters for models like LLaVA 1.5 7B. LLMs pre-trained on large text datasets are likely to remain the foundation of VLMs because of their powerful reason- ing capabilities. Therefore, understanding how these LMs process and integrate visual information is crucial for advancing VLM interpretability.
Many open questions remain regarding the inner workings of VLMs. The nature and structure of visual representations fed into the LM remain unclear, as these visual inputs are soft prompts that
Work done during FRA-Krueged AI Safety Lab internship.
Author contributions detailed in $6. Correspondence Clement Neo <element@clemenneeo.cmm

---

Published as a conference paper at ICLR 2025
do not correspond to language tokens and cannot be interpreted such (Lester et al., 2021; Merullo et al., 2022). This raises questions about how visual information is encoded in these representations. Furthermore, the spatial distribution of information within the image encoder's 's feature maps remains unclear, leaving open the question of whether and how much object-specific details are localized or dispersed across the entire representation.
Furthermore, the mechanisms by which the language model processes these visual inputs are not well understood. A significant modality gap exists between visual and textual input (Jiang et al., 2024), and it is unclear whether our mechanistic understanding of text processing in language models generalizes to the processing of visual input in VLMs.
To understand the representations in the visual inputs for VLM and how the VLM processes them, we study LLaVA 1.5 7B (Liu et al., 2023a), popular open-source VLM with competitive perfor- mance that uses CLIP (Radford et al., 2021) as an image encoder and Vicuna 13B (Chiang et al., 2023) as its language model. We also test on focus on a set of object identification tasks and our main contributions are as follows:
and Qwen2- VL (Wang et al., 2024). We
1. Using ablation techniques, demonstrate that the information for object is highly localized to the token positions corresponding to their original location in the image.
2. By extending the logit lens technique primarily used for language models (Nostalgebraist, 2020), we find that the representations of the visual input in the LM are refined towards the embedding of interpretable tokens in the vocabulary, despite the LM not being explicitly trained to do so.
3. By blocking the attention flow between tokens, show that the model extracts object information from the object tokens in the middle to late layers.
Our findings are first steps in understanding the internal mechanisms of VLMs, paving the way for more interpretable and controllable multimodal systems. The code for our experiments is available at http:://gthbbbccmmmmmme////aaaaa
2 BACKGROUND
Transformer Architecture. Transformer-basee autoregressive Large Language Models (LLMs) process sequences of input tokens to predict the next token in the sequence (Vaswani et al., 2017). These models achieved state-of-the-art performance across various natural language processing tasks (Brown et al., 2020), even though they are merely trained on next-token prediction (Radford al., 2019). A LLM takes input sequence X (k, and outputs probability distribution over the vocabulary V to predict the next token In+1-
To do so, the model refines token representations layer by layer through series of computations. Each token z; is initially represented by an embedding vector hy of dimension dm. obtained from lookup operation in an embedding matrix We E RlV|xdm, This representation is then updated through successive layers using multi-head self-attentio (MHSA) and feed-forward (FF) sublayers:
h h h (-1 + d;
(1)
where fy is the representation of token x, layer i, af is the output from the MHSA sublayer, and ft is the output from the FF sublayer. All vectors h1,, a/. f! E Rdm.
The MHSA sublayer consists of H attention heads working parallel, each implemented as follows:
(OK" VaLe + M)V
Attention K)V = softmax
where Q, K, K,K, E Raxde are query, key, and value matrices derived from learned linear projections of the input, and M E Rux is causal mask. The causal mask M is set to -oo in the upper right triangle, meaning My 0 if i 2 j. else co. This ensures that each position can only attend to previous positions and itself, thus preserving the autoregressive property in language models. These
SSurred from the xtuner HuggingFace model.
2

---

Published as a conference paper at ICLR 2025
LM
"USER:
firesk into patahes
GLIP VIT
MLP adapter
30
dog
dog
Image Tokions
Festuum Map
"Whhet neat the trew?? ASSISTANT:
"dog"
Ob iderriiication baak
1 Ablate visual tolkens to wrong answers
Methods
Analyse component ourputs embedoing space [aiso known Logit Lens)
Stop attention heads from attending to the image tokens known Attention Knockaut)
In adapter-style Vision-Language Models (VLMs), the visual tokens (in green) are soft prompts for the language model (LM), and are not interpretable through the vocabulary embedding. Through a set of object identification tasks, we find that (1) object information can be localized to subset of visual tokens, (2) the representations of the visual tokens evolve towards interpretable text embeddings, and (3) the model extracts some information from the visual tokens to the last token position in the middle late layers, identify the object.
Figure I:
outputs are then concatenated and linearly projected to produce the final output of the multi-head attention sublayer.
The FF sublayer applies linear transformations with element-wise non-linear function between them:
W (W +b))++++++) +
where Wi. W by. and by are learned parameters. Finally, the representation of the last token on the final layer, A is projected into probability distribution over V using unembedding matrix Wo dx and a softmax operation.
Image Encoders.
The transformer architecture has been successfully adapted for state-of-the-an image encoders, demonstrated by Dosovitskiy al. (2021). In this approach, input images divided into fixed-size patches, flattened, and linearly embedded before being processed as a se- quence of tokens, analogous word embeddings in NLP tasks. Unlike text-based models, the attention mechanism in these image encoders is bidirectional (i.c.. no masking is applied). The in- put sequence includes a special class token, whose final embedding typically used for image classification tasks. The representations at the other token positions form what is called the feature map of the image.
Building upon this foundation, Radford et al. (2021) introduced Contrastive Language- Pre- training (CLIP), which jointly trains vision transformer and text trasnsformer to produce similar embeddings for matching image-text pairs. CLIP's image encoder is currently being used for state- of-the-art performance on many downstream tasks like zero-shot image classification and vision- language models like LLaVA.
Connecting Text and Image Models. Tsimpoukelli et al. (2021) pioneered the concept of training an image encoder such that its output can be used by frozen language model for multimodal few- shot reasoning. Merullo et al. (2022) advanced this approach by freezing both the pre-trained image encoder and pre-trained language model, training only linear map that converts image features into inputs for the text model. This introduced the "adapter style" approach, utilizing pre- pre-trained components for both modalities.

---

Published as a conference paper at ICLR 2025
Building this foundation, Liu et al. (2023b) introduced LLaVA, which combined CLIP ViT- L/14 image encoder with a Vicuna-13B language model, connected by a trained linear projection, or an MLP for improved performance (Liu et al., 2023a). Crucially. LLaVA's training process focuses solely on fine-tuning for multimodal conversation, without any next-token pretraining on image-text pairs. This approach allows LLaVA leverage the strong pre-trained capabilities of both the vision and language models while efficiently adapting to multimodal tasks.
2.1 NOTATION
In this section, we describe how an image and query processed through LLaVA.
Image Processing. The input image I is processed by first being cropped into a square, then suitably resized before being divided into 576 image patches. The CLIP ViT-L/14 image encoder fcup processes this image to produce feature map E = fCuF(() E R NNduur where N = 576 is the number of image patches and deL.p is the dimensionality of CLIP's embeddings. An adapter network A then maps these CLIP embeddings the language model's input space, yielding set of visual tokens ΣA = A(E)) E R-Vxd, where dm is the dimensionality of the language model's 's input embeddings. We call EA visual tokens to distinguish them from the image tokens.
Text Processing and Combined Input. For the text input, given a tokenized prompt sequence T' (t1 (.)) the embedding layer of the language model ÉLM maps these tokens to embeddings: Ej = (T) E IR Mxdxx The full input to the language model is the concatenation of the adapted image embeddings and the text embeddings: X = [Ex; Er] E INN++) from which the language model can generate output autoregressively.
2.2 OPEN QUESTIONS AND HYPOTHESES
One might expect that the output of the adapter EA would produce embeddings corresponding to image information. For example, if the image contained car, we might expect the adapter to produce tokens that correspond to the embedding of "car". However, the output of the adapter forms soft prompts that have no semantic meaning when decoded through the vocabulary (Merullo al., 2022; Bailey et al., 2023).
2.2.1 LOCALIZATION OF OBJECT-SPECIFE INFORMATION
A key question of our investigation is: Where is object-specific information located in visual tokens? Two conflicting hypotheses emerge from recent research:
1. Object-Centric Localization: Joseph & Nanda (2024) found that in classical vision trans- formers, token representations corresponding specific object locations tended align with those objects' class embeddings in the late layers, even without explicitly training for this behavior. This suggest that object information might be localized the tokens corre- sponding to the spatial location of the object in the original image. However, their findings were on a vision transformer trained on ImageNet classes, and not the vision transformer in CLIPP
2. Global Information in Register Tokens: contrast, Darcet et al. (2024) identified reg- ister tokens in vision transformers, including CLIP variant. These background patch tokens with unusually high norms that appear to encode global image features. It follows that the LM in an adapter-style VLM could primarily rely on these information-rich register tokens for its output.
We investigate these hypotheses in Section using ablation techniques in a of object identifica- tion tasks.
2.2.2 PROCESSING OF SPECCICSPECIIIC REPRESENTATION
The processing of visual tokens may differ from text tokens. Soft prompts are very different from normal prompts (Bailey et al., 2023), and there is modality gap between textual and visual rep- resentations in VLMs (Jiang et al., 2024). Furthermore, LLaVA is fine-tuned exclusively on visual question-answering (VQA) prompt-response pairs, without pre-training on next-token prediction.
4

---

Published as a conference paper at ICLR 2025
This may affect how the model processes visual information compared to text. In Section 4.1, we attempt the logit lens method to see how the representations visual tokens evolve through the layers.
Recent studies have shed light on how LMs process information. Geva et al. (2023) identified a then--tten-ettract process in factual association prompts, where the model first refines information at subject tokens through MLP sublayers before transferring it to the last token position via atten- tion sublayers. Similar mechanisms have been observed in arithmetic tasks (Stolfo et al., 2023), suggesting that this (sequencing of functions) might be general mechanism in LMs.
However, Basu al. (2024) adapted the factual association setup VLMs and found preliminary evidence that the LM may first summarize image information in text tokens, instead of extracting the image information the last token position from the image tokens directly, While we do study factual association rather we choose to focus simpler object identification task, we study how information flows from the image tokens in Section 4.2.
INVESTIGATING THE REPRESENTATIONS IN MAPPED VISUAL TOKENS
In this section, we use ablation experiments test if object information is concentrated in specific visual tokens. By ablating selected tokens and observing degradations in the model's ability to identify the object, we map how object information distributed across the visual tokens.
Dataset. We use images from the COCO Detection Training set (Lin et al., 2014). To ensure the reliability of our results, we employ two filtering steps:
(1) Choosing Simpler Images. As heuristic to focus on simpler images, we choose images where (a) there is an object whose size is between 1,000-2,000 square pixels, or about 2-4% of the image by area, (b) only one instance of the that object is present in the image, and (c) there are less than 4 annotated objects. We present few examples of the images in Figure 4.
(2) Controlling for Hallucination. Model can sometimes hallucinate objects based on context, even when the object's visual information removed. To address this, we create two versions of each image: the original and one where the target object is masked with noise. We only keep images where the model correctly identifies the object in the original image but fails to identify it in the masked version. This ensures that the model's object identification relies the object's visual information.
After applying both filtering steps, our final dataset comprises 4,318 images.
Method. We present a high level overview of our methodology in Figure Let EA = (e eN} be the set of visual tokens. We define subset C {1. NN consisting of the indices of tokens hypothesized to contain information about particular object o in the image.
For each ablation experiment, create modified set of embeddings E'A:
if i E S
E otherwise
where ē= EN e; is the mean embedding across all visual tokens from 50,000 images from the ImageNet validation split (Deng et al., 2009). This replaces the hypothesized object-relevant tokens with an average token, effectively ablating their specific information. We do mean ablation to preserve the norm of the image token and keep them in-distribution, as their norms are typically much higher than the norm of text tokens (Bailey et al., 2023). We then evaluate the impact of token ablation using three methods:
1. Generative Description: We prompt the model with "Describe the image" using both the original (Ex) and ablated embeddings (E')). We then ascertain whether the target object o is produced verbatim in the model's generated description before and after ablation. If is mentioned with EA and not EA,, it indicates that the ablated tokens were crucial for the model to identify and include the object in its description.
2. Binary Polling: We ask the model "Is there a [o] in this image?" using both E and Eyg- We then compare whether the model's next generated token changes from "Yes" to
5

---

Published as a conference paper at ICLR 2025
Ablate visual tokens in different ways
Before Ablation
After Ablation
This ls photo of park There is dog nox to the tic...
T piilo ol pani. There bee in the fu pookligg shade...
Breek ano
paiches
CLIP
adapter
MLP
Image Tokens
Fextune
Map
ablated)
LM
(1) Generative
22) Polling
(3) VOA
Lika
Dascnii imaag
Thare
Image?
2 Ask object identification questions
Compare answers before and after ablation
Figure 2: Overview of our ablation experiments, (1) We ablate some visual tokens that potentially contain object-specific information, (2) prompt the model describe the image, or answer object- specific questions, then (3) measure the impact of token ablation by calculating the percentage of initially correct object identifications that become incorrect after ablation.
"No" after ablation, which indicates that the ablated tokens were crucial for identifying the object.
3. Visual Question-Answering: We manually curate a set of 100 images with specific ques- tions about objects, such as "What is on the bed?". We choose questions with unambigu- ous answers and avoid objects directly related to the question (e.g., avoiding "pillow" for bed-related question). We prefill the model's response with "It is a" and compare the next generated token before and after ablation. A set examples in Figure 5.
We choose the subset S of tokens to ablate in five ways:
(1) Object Tokens. We choose the visual tokens whose positions correspond to the image patches that originally contained that object. (2) Object Tokens with Buffer. In addition to the object to- kens, we include neighboring tokens. We test with two buffer sizes: Buffer (including immedi- ately adjacent tokens) and 2 Buffer (including tokens up to 2 positions away from object tokens). (3) Register Tokens. We select visual tokens with norms more than standard deviations from the mean norm. These correspond to the register tokens identified by Darcet et al. (2024), which are thought to encode global image features. (4) Random Tokens. As a baseline, we ablate n random to- kens. (5)
Tokens. As stronger baseline, method (Sundararajan et al., 2017) to identify tokens most important for the model's decision. For given object o, we prompt the model with "Is there [o] in the image?" and compute integrated gradients with respect to the logit for the "Yes" token over 50 steps. We then ablate the n tokens with the highest attribution scores.
the Integrated Gradients attribution
Results. Our results are presented in Table 1. We find that ablating the object tokens significantly impairs the model's 's ability to identify the object. For comparable numbers of ablated tokens, object token ablation consistently results in larger performance decreases across all settings as compared the gradient-based and random baselines. This suggests that the information about that object is localised to the region of the object token. Furthermore, these findings are consistent across all three experimental settings and applicable to both LLaVA and LLaVA-Phi:
4 INVESTIGATING VISUAL INFORMATION PROCESSING
4.1 ANALYZING THE RESIDUAL STREAM THRROGGH EMBEDDING SPACE
Can we interpret how the representations at the visual token positions evolve through the layers? To do so, we employ the logit lens technique (Nostalgebraist, 2020): for each layer, we decode the activation at each token position using the unembedding. Formally, for each hidden state h each token position and layer we project it into a probability distribution over using We and take the token with the highest logit. We do this for LLaVV 1.5.

---

Published as a conference paper at ICLR 2025
Table 1: Performance degradation after token ablation. lower percentage indicates greater impact of the ablation, meaning the model is more likely to answer incorrectly, thus suggesting that the ablated tokens contain more localized object information. Average token counts apply to both the generative and polling settings for LLaVA 1.5. We find that ablating the object tokens with adjacent tokens (33.4 tokens on average, bold) significantly causes object identification performance to drop, more so than the integrated gradients and random baselines for a comparable number of tokens (40 tokens, in bold).
LLaVA 1.5
LLaVA-Phi Polling Decrease (%)
Ablation Type (Avg Token Count)
Generative Decrease (%)
Polling Decrease (%)
VQA Decrease (%)
Generative Decrease (%)
VQA Decrease (%)
Object (12.6)
33.33
15.38
33.33
47.46
28.48
41.51
Buffer (33.4)
71.79
51.28
86.67
82.89
86.48
96.23
Buffer (60)
76.92
64.10
92.22
89.56
95.49
100.00
Register tokens (3.1)
5.13
0.00
3.33
7.63
0.00
1.89
Int. Gradients (5)
2.56
0.00
6.67
17.11
1.23
28.30
Int. Gradients (10)
20.51
2.56
6.67
25.26
4.30
24.53
Int. Gradients (20)
33.33
20.51
17.78
41.14
9.63
37.74
Int. Gradients (40)
48.72
35,90
50.00
61.49
26.64
56.60
Gradients (60)
58.97
48.72
71.11
70.61
40.98
69.81
Int. Gradients (100)
76.92
51.28
84.44
81.23
61.68
83,02
Int. Gradients (250)
76.92
69.23
97.78
91.93
90.16
98.1 11
Random (5)
0.00
0.00
1.11
3.68
0.00
0.00
Random (10)
4,88
0.00
0.00
3,95
0.00
1.89
Random (20)
0.00
2.22
5.88
0.00
0.00
Random (40)
2.44
0.00
0.00
8.51
0.00
0.00
Random (60)
7.32
0.00
1.11
9.39
0.41
3.77
Random (100)
7.32
0.00
4.44
11.23
0.20
7.55
Random (250)
24.39
0.00
16.67
20.61
1.84
15.09
Results. We find that the activations in the late layers at each visual token position correspond token embeddings that describe its original patch object in corresponding image token. We provide some case studies in Figure and interactive demo more examples.
To quantify this phenomenon, we analyzed 170 COCO validation images with objects of sizes be- tween 20,000 and 30,000 square pixels (approximately 1/2 width and 1/3 height of the image). We found that in the best-performing layer for each image, an average of 23.7% the object patch token positions correspond to the correct object class token. The best-performing layer occurs average at layer 25.7 (out of 33 layers), confirming that mid-to-late layers tend to develop the strongest object-token correspondences (Figure 6).
We found that the tokens decoded often correspond to concepts more specific than the object-level. For example, in Figure 3a, the tokens go beyond identifying the sweater worn ("swe") and describe the specific patterns on the sweater with tokens like (ond) and "cross".
In instances, tokens correspond their corresponding concepts in other languages, like in Figure 3c where the tokens correspond to "year" and "month" but in Chinese and Korean instead. This is somewhat surprising given that previous work has shown that Llama used for non-English languages tend to have English tokens in their intermediate activations when interpreted through the logit lens (Wendler et al., 2024).
Overall, it is surprising that the logit lens work on VLMs! This technique works auto-regressive LLMs because they are pretrained on next-token prediction, meaning their activations will be iter- atively refined towards the unembedding of the predicted next token. However, VLMs are merely fine-tuned on multimodal tasks. The effectiveness of the logit lens here suggests that the hypothesis that transformers build predictions by promoting concepts in vocabulary space (Geva et al., 2022) may generalise to VLMs, even when only fine-tuned on multimodal tasks.
Testing Other Models.
We extended our logit lens analysis to Qwen2VL-2B to verify if these
findings generalize beyond the LLaVA family. Following the same methodology, we analyzed 253 COCO validation images and found that in the best-performing layer, average of 6.5% of the visual tokens map to the correct object class token. The best- perfforinn layer occurs at layer 25.1 (out of 29 layers), confirming the middle-to-lat layer pattern observed in LLaVA (Figure 7). Inter-

---

Published as a conference paper at ICLR 2025
lys
Trees:
Towe Yaler
"ontrols" "Whells"
We Weteeriing wheel
Mack: ymy
Garge:
pullamm "deiig
SHERIEE
Whell, 766"
(bari
hand": "gropp"
(a) An image of a lady in the sweater. The logit lens identifies tokens that correspond to specific detail of the sweater, as "pattern" "diam" "diam"(ond).
(b) An image of child in go-kart. The repre- sentations sometimes encode specific details, such as "look" and "gaz"(e instead of just "face".
Ttackgronn"
Tadkgroand; "wht
Tabel "iick jen
Tighr:
Two" "bur"" "Sa"
+ Nharin Chineeal
"Apple"
(c) An image of a switch. In the intermediate layers, the year and month tokens are encoded in non-English characters.
(d) An image of bunch of apples. Global features, like count, show up in hackground tokens, though this may be artifact the I.M processing.
Figure 3: Examples of tokens and their positions that the logit lens yields in the late layers. The labelled tokens come from a range of layers around the middle-to-lee to-dle-to--lte layers. our labelling for an image, tokens that are part of word are completed with an educated guess e.g. "diam" "(ond)
estingly, Qwen2VL shows non-zero correspondence even in early layers, possibly due its cross- attention adapter providing better initial mapping compared to LLaVA's simpler MLP adapter. Both models demonstrate increasingly aligned token representations with object tokens in later layers, suggesting this refinement pattern is general property of adapter-style VLMs.
Case Study: Global Features. We sometimes find global-level features, such numbers possibly referring to object counts, appearing in unexpected background tokens (Figure 3d). Ablating these specific tokens doesn't change the model's description, and the numbers persist in the logit lens even after ablation. This suggests these global features may be artifacts of the language model's processing rather than direct information from visual tokens potennilll explaining why CLIP- based VLMs often underperform CLIP itself classification tasks (Zhang et al.. 2024).
Potential Applications. This may be useful for getting coarse segmentation maps for little to no extra computation, which could lead to downstream applications. For example, Liu al. (2024) suggests to adjust the attention weights towards the visual tokens in the LM reduce hallucinations. We speculate that directing attention towards specifie objects may improve this method, though leave further exploration to future work.
4.2 TRACING ATTENTION FLOW
To understand how visual information is processed and integrated within the language model, we in- vestigate the flow of critical information through the network. Specifically, examine whether the
8

---

Published as a conference paper at ICLR 2025
model directly extracts information from relevant visual tokens, whether it aggregates information from visual tokens before transferring it to the final output token.
Method. We employ the attention knockout technique introduced by Geva et al. (2023). This method involves selectively blocking attention between specific tokens at different layers of the model, allowing to assess the importance of various connections for the model to identify the object. Our process as follows:
(1) For given imput and target layer L, artificially block specific attention connections in the multi-head self-attention (MHSA) sublayer by setting the corresponding attention mask values to negative infinity: M layer C 1, and and are the indices of the tokens between which we want to block attention.
Vj E [1, H] where M is the attention mask for head j
(2) We apply this blocking over a window of consecutive layers, including early layers (LI-10), early to middle layers (L5-14), middle layers (L11-20), middle to late layers (L15-24), and late layers (L.21-31).
(3) For each window, we measure the decrease in accuracy and prediction probability for the correct token when blocking attention under three scenarios: (i) From object tokens and their surrounding buffer to the final token. (ii) From all non-object tokens to the final token. (iii) From all visual tokens except the last row the last row of visual tokens*.
We conduct this experiment using LLaVA 1.5 on our curated Visual Question Answering (VQA) dataset of 100 images.
Findings. Our results are presented in Table 2.
We find that blocking attention from the object tokens (and their buffers) to the final token in mid-late layers leads to noticeable performance degradation. This suggests that the model directly extracts object-specific information these later stages. Blocking attention from non-object tokens to the final token in early layers also causes some performance drop. This indicates that contextual infor- mation from the broader image processed and integrated in the earlier stages of the model.
Interestingly, blocking attention from visual tokens to the last row of visual tokens has minimal effect on performance. This contrasts with findings by Basu et al. (2024), who suggested that the model might summarize image information in the last of visual tokens before using it. Our results indicate that, at least for our object identification tasks, the model doesn't ''t rely on such a summarization step.
Table 2: Results of blocking attention between different token groups across various layers of the I.L. VA 1.5 model. The values represent the relative performance on the correct token being pre- dicted when blocking attention, with 1.00 being no impact and 0.00 being all questions wrong post- ablation.
Attention Blocking
Layer
From
To
Early
Early-Mid
Mid
Mid-Late
Late
All
O
LTP
1.00
1.00
0.96
0.88
0.93
0.82
O+1
LTP
1.00
0.99
0.90
0.82
0.89
0.67
O+2
LTP
1.00
1.00
0.91
0.80
0.91
0.68
I-(+1)
LTP
0.88
1.00
0.97
0.96
0.98
0.82
O+1
LVR
1.00
1.00
1.00
1.00
1.00
1.00
1-LRR
LVR
1.00
1.00
1.00
1.00
1.00
1.00
From: O= Object Tokens, O+n = O + Buffer,
1-(0+1) = All visual tokens except O+1, I-LVR = All visual tokens except last row
"To: LTP Last Token Position, LVR Last Visual Token Row
We do this as recent work by Basu et al. (2024) suggests that the model may summarise visual information in the last row of visual tokens. though they study this in different VQA task. If the model does use the last row of visual tokens as 'working summary', then blocking the attention from all the visual tokens to the last should prevent it from answering the question.

---

Published as a conference paper at ICLR 2025
Testing Other Models. To verify that our attention blocking results are not specific to LLaVA-1.5, we conducted additional experiments with LLaVA-Phi-3. Rather than using fixed layer groupings, employed sliding window of 10 layers to obtain finer- fine-grained results actoss the network. We find that the results (Figure 8) closely mirror our original results: blocking attention from object tokens to the last token position shows the strongest effect in middle-to-late layers, while blocking attention from visual tokens to the last row continues to show minimal impact.
RELATED WORK
Explainability. Classical techniques like GRAD-Cam (Selvaraju et al., 2017) and Integrated Gradi- ents (Sundararajan et al., 2017), focus input attribution, while recent tools like I. LVI.M-Interpree (Stan et al., 2024) extend these methods to VI.Ms. Our work complements these by investigating the model's internal processing mechanisms.
Mechanistic Interpretability (MI) for LLMs. Techniques such as causal tracing (Meng et al., 2022) and sparse coding (Bricken et al., 2023; Huben et al., 2023; Marks et al., 2024) have advanced our understanding of how LLMs perform tasks despite being primarily trained for next-token pre- diction, but VLMs are not trained on next-token prediction for visual inputs (Liu et al., 2023b). Our work extends some of these techniques to understand visual information processing in multimodal contexts.
MI for Multimodal Models. In CLIP, researchers have discovered multimodal neurons (Goh et al., 2021), decomposed image representations (Gandelsman et al., 2023; 2024), identified interpretable network subgraphs (Rajaram et al., 2024), and extracted interpretable features using sparse coding (Rao al., 2024; Daujotas, 2024; Fry, 2024). In contrast, our work focuses on generative VLMs.
For generative VLMs, Schwettmann et al. (2023) found neurons corresponding to visual concepts, while Palit et al. (2023) adapted causal tracing for BLIP. While these studies help us understand the role of specific components in the VLM, we investigate the representations of the visual inputs.
Basu et al. (2024) used causal tracing and attention contributions in visual question-answeringg setup to find that VLMs such as LLaVA retrieve information from earlier causal layers of the LM (i.e. layers 1-4 vs layers 4-7 in an LLM) when answering a multi-modal question. Their analysis suggests that the visual information may be summarised in a consistent subset of late visual tokens. In contrast, we study VLMM in a more straightforward object-identificaiion setting. We also examine their information transfer hypothesis through our attention blocking experiments in Section 4.2.
Parallel Work. Concurrent research by Jiang et al. (2025) independently arrived at similar logit lens findings and developed a hallucination reduction method based on these insights. Wu et al. (2024) observed comparable logit lens patterns across both visual and audio modalities.
6 CONCLUSION
Our investigation into visual information processing in LLaVA reveals that object information is highly localized within visual tokens corresponding to the object's spatial location, challenging the hypothesis that VLMs primarily rely on global features in register tokens. Visual token representa- tions evolve to align with interpretable textual concepts across layers, suggesting VLMs refine visual information toward language-like representations even without next-token prediction pretraining for visual inputs. Our attention blocking experiments suggest that VLMs may extract object information from relevant visual tokens mid-to-late layers. These findings bridge the gap between our under- standing of language and vision models, providing a foundation for more interpretable multimodal systems.
Limitations & Future Work. We focused on LLa LLaVtttype models as a starting point for interpreting more complex VLMs, but our results may not generalize to significantly different architectures. Our study concentrates on object identification tasks, providing simpler setup compared to multi-step reasoning. While we tested object identification in three ways, our findings may not fully represent VLM behavior in complex tasks like reasoning open-ended question answering. Future work should explore practical applications reducing hallucinations and enhancing factual accuracy, while examining whether these findings extend to broader task types and model architectures.
10

---

Published as a conference paper at ICLR 2025
AUTHOR CONTRIBUTIONS
Clement Neo conceived the study, designed and performed the experiments, and led the writing of the manuseript. Luke Ong contributed to the writing and literature review. Philip Torr, Mor Geva and David Krueger provided advice and comments. Fazl Barez served the primary advisor for this work and helped significantly with the design and write-up of the paper.
ACKNOWLEDGMENTS
We thank the Torr Vision Group for providing compute and hosting Clement Neo during the intern- ship.
We thank Sarah Schwettmann, Ashkan Khakzar, William Rudman, Joseph Miller, Alex Spies for discussing and reading drafts of the paper.
REFERENCES
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction, 2024. URL https: //arxiv. wwt./rrg///.........../........e /aabbs//406 2406 11717.
Luke Bailey, Gustaf Ahdritz, Anat Kleiman, Siddharth Swaroop, Finale Doshi-Velez, and Weiwei Pan. Soft prompting might be a bug, not a feature. 2023.
Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Mas- siceti. Understanding Information Storage and Transfer in Multi-modal Large Language Models, jun 2024. URL http://///// www.ivvorrgaoooo.ooooooo abs /2006/46.0000 04236.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer- circuits.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL attps::/ppp htttps:s//proceeddggg. neurips wwtpaper_peeeeeeeeeeeeeeeeeeeeee farper_fflle wwtilles/pppe/e// wwwpp..ppp..ee
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https:/vicunu.lmsysccom (uccessed 14 April 2023), 2(3):6, 2023.
Timothée Darcet. Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference Learning Representations, 2024.

---

Published as a conference paper at ICLR 2025
Gytis Daujotas. Case study:
Interpreting, manipulating, and controlling clip with sparse autoen- coders, 2024. URL bttps://www..essttt wwtppostss//////ooo/oo/o/// 6GgM case-study- Suudy-interee
6/
Ac-
cessed: 2024-09-24.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Putlern Recognition, pp. 248.255, 2009. doi: 10. 101/0//P..2000.5000000
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: /ttppe/wewww.neteeeeeeeeeeeeeeee www.:nnen/toimdddiieeem.m aNTTy.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodss, Danny Hernandez, Andy Jones, Jackson Keriion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, http://eeicciiiipiipp//////////a
2021.
Hugo Fry.
Towards interpretable features in vision transformers, 2024.
multimodal interpretability:
Learning
sparse URL
https:://// //www. lesswrong httm/p/otts/bbbbbttaaarttaaaaaa towards-mul i modall htp.ppttabbbllllllll.....--eeeee Work. Accessed: 2024-06-28.
Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition, 2023.
Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting the second-order effects of neurons in clip, 2024.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pp. 30 40-445,7 2022.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual asso- ciations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12235, 2023.
Gabriel Goh, Nick Cammarata t, Chelsea Voss t. Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks, Distill, 2021. doi: 10.23915/distill.oo 00030..
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2023.
Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model, 2024. URL https: //aaxi/aa.iiiia//aa org/ abs 06968.
Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations, 2025. URL htttps://ariiss org/abs/2411 2410 02762.
Sonia Joseph and Neel Nanda. multimodal mechanistic interpretability https://wwet wwtp:/wwwww.m..oooru..... org ptstts/////p//ppoppssss/sssssess httpp/pssssskoooooooooop http://wwowyyyybbbbb/bbbbbb iWF /WFKe/ URL
Laying the foundations for vision and open problems, 2024.
Accessed: 2024-06-28.
12

---

Published as a conference paper at ICLR 2025
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 12118653/112221emmmll hwtp://00...mmmmmmaa..... 243. URL wtg//02111 emm.12m22..m.pp......mmmmmmmm ma.n.243.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-19742. PMLR, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCC 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023b.
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for wrgg 2207.217777
alleviating hallucination in Ivlms, 2024. URL
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint (1t.:::2003.994444 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Advances in Neural Information Processing Systems, 36, 2022. 2522v2220..5226..
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In The Eleventh International Conference on Learning Representations, 2022.
Nostalgebraist. Interpreting gpt: The logit lens. post /t/////KKKBBBBBBB www.Daal66.r///ttepeerr.... whtpriinnng-tttt---o-----------e forum. .org//rrr////ooo..... Aug 2020.
Accessed: 23 Sep 2024.
Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision- Dision-language mech- anistic interpretability: A causal tracing tool for blip. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2856-2861, 2023.
Xu Pan, Aaron Philip, Ziqian Xie, and Odelia Schwartz. Dissecting query-key interaction in vision transformers. ICML MI Workshop, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https::/arxis org/ alss 2000.00020......
Achyuta Rajaram, Neil Chowdhury, Antonio Torralba, Jacob Andreas, and Sarah Schwettmann. Au- tomatic discovery of visual circuits, 2024. URL httttp/se//arri....r///////// 143499
Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery, 2024. URL https://arxiv.orrt abs/2401 14499.
Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pp. 2862-2867, 2023.
13

---

Published as a conference paper at ICLR 2025
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626, 2017.
Gabriela Ben Melech Stan, Estelle Aflalo, Raanan Yehezkel Rohekar, Anahita Bhiwandiwalla, Shao- Yen Tseng, Matthew Lyle Olson, Yaniv Gurwicz, Chenfei Wu, Nan Duan, and Vasudev Lal. Lvim-intrepret: An interpretability tool for large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8182-8187, 2024.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. mechanistic interpretation of arith- metic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7035-7052, 2023.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-212, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Martina G Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference Learning Representations, 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing lasoun-lagguage model's perception of the world at any resolution. arXiv preprint arXiv:2409.122 12191, 2024.
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do Ilamas work in En- glish? on the latent language of multilingual transformers. In Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15366-15394, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18633///22222aacooooo https://aclantie
URL
org/ a:1- 820.
Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities, 2024. URL httttt:t//aaatteeeeeeeeeeeettttt
.rg.orgggooo........oo brgg/bb///// /2l///41111 2411.04986......
Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. Why are visually- language models bad at image classification? arXiv preprint arXiv:2405. 10.224055.45.... 2024.
14

---

Published as a conference paper at ICLR 2025
A
FILTERED DATASET EXAMPLES
Image ID: 21571-d00 dog
Image ID: 27562 trisbes
201
300
400
250
100
200
27674
100
200
900
302
Figure 4: Example of dataset images for the object identification tasks.
15

---

Published as a conference paper at ICLR 2025
B CURATED IMAGES AND QUESTIONS EXAMPLES
JAMAICA
STU
(a) "What is on the bench? It is a" + book
(b) "What is below the street sign? It + bottle
(c) "What is the truck? It + dog
(d) "What the seat? is a" cup
Figure 5: Examples of curated images and their questions. its answer with "It is a"" The correct answer is underlined.
We ask the model the question and prefill
16

---

Published as a conference paper at ICLR 2025
C
OBJECT CORRESPONDENCES
LLaVA Layer-wise Average Correspondence
12
Average Coneepandence (%)
10
10
15
20
25
Figure 6: Token-to-object correspondence in LLa across transformer layers. Analysis of 170 COCO images shows peak correspondence (23.7%) occurs later layers (average layer 25.7/33).
Qwen Layer-wise Average Correspondence
Average Correspondence (%)
3.5
2.5
2.0
1.
10
20
25
Layer
Figure 7: Token-to-object correspondence in Qwen2VL-2B showing increasing alignment in deeper layers, peaking at layer 1/21 with 6.5% correct mapping. Unlike LLaVA, Qwen exhibits non-zero correspondence early layers.
17

---

Published as a conference paper at ICLR 2025
D LLAVA-PHI-3 BLOCKING
Impact of Attention Blocking Across Model Layers
Arcuraty
1.0
0.0
0.6
10
Layer [Center of 10 laye Window
O+1 - LTP
Delative Accurey
1.0
0.8
10
15 Layer [Center of 10-layer Window!
20
O+2 - LTP
Belative Accurayy
10
15 [Center 10 Window!
20
1-(0+1) - LTP
Acturary
10
Layer layer
O+1 LVR
Accurscy
Layer [Center of 10-laye Window!
I-LVR LVR
Accurary
Layer (Center of 10 layer Window!
Figure 8: Attention blocking results for LLaVA-Phi-3 using a sliding window of 10 layers. We find that similar to LLaVA-1.55 blocking attention from object tokens to the last token position has the strongest effect in middle-to-late layers, while blocking attention from visual tokens to the last visual row shows minimal impact throughout the network.
18


Reviews:
[{"review_id": "rN7wJZqrxB", "reviewer": "Reviewer_UGCm", "summary": "This work applies mechanistic interpretability techniques to understand how visual information is localized and processed in VLMs. This work:\n\n- Tries to localize where information processing happens within VLM transformers\n- Applies the “logit lens” (Nostalgebreist 2020) to understand how representations of visual tokens evolve across layers of the transformers\n- Applies attention knockout to see how blocking attention between different subsets of tokens in the transformer causes the accuracy of the VLM to decrease\n\nFor localization, they find that ablating object tokens with a buffer has the most impact on model performance on the 3 ablations, with two LLaVA VLMs. For the logit lens, they find that patch embeddings at different layers can be mapped to corresponding meaninful tokens in the tokenizer vocabulary.\n\n**EDIT 11/25/2024:** I am raising my score from 6 to 8, and soundness from 3 to 4, after the author response.", "questions": "In Figure 3, what layer do these unembeddings correspond to? Are these all from a specific layer, or did you select instances from different layers that matched human interpretations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Important problem: The authors address an important problem of understanding how visual knowledge is localized and processed in transformer-based VLMs.\n\n2. The formulation of each sub-question is well-motivated, the method and metrics are well-designed, the analysis of each sub-question is really thorough, and the findings are well summarized.\n\n3. The paper is very well-written and easy to follow.", "weaknesses": "1. **Evaluated insufficient VLMs:** As the authors acknowledged, the analysis is performed on only two VLMs from the same (LLaVA) family. However, the methodologies should be easily extendable to other transformer-based VLMs. \n\n    A key sticking point of mechanistic interpretability methods is that we do not know how the findings generalize to other models, so it is hard to take any findings at face value when they have only been performed on one model (especially when some of the findings contradict findings from other works, e.g. VLMs rely on features stored in register tokens). \n\n    I suggest including findings from other VLMs such as InstructBLIP, QwenVL, InternVL, and seeing if your findings generalize to more models.\n\n2. **Results of \"logit lens\" analysis are hard to interpret:** While the authors find some interesting mappings between visual patches and output tokens, it's hard to tell if these are meaningful correspondences or an artifact of cherry-picking/confirmation bias, since we see mappings for only a handful of patches, and only at one layer (although it is unclear which one). \n\n    One way to make this analysis seem less cherry-picked is to annotate patch-token pairings manually for whether they are a \"matching\" pair or not, and analyze what % of patches map to a matching token. You could also look at how these mappings evolve across layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "37joiOVMLS", "reviewer": "Reviewer_c4uP", "summary": "This paper studies how VLMs, particularly LLaVA, process and integrate visual information. Through ablation studies, the authors show that object information is localized in specific visual tokens, which evolve into interpretable text-like representations across layers. Attention flow experiments reveal that the model extracts object-specific information in mid-to-late layers while processing contextual information in earlier layers.", "questions": "Is it easy to apply the methods and findings in this paper to downstream tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well organized. There are effective diagrams and explanations that make complex concepts accessible.\n2. The paper presents an innovative approach to understanding how visual tokens are processed in the language model component of VLMs. The insights on visual token alignment could inform practical techniques to improve VLM robustness and reduce hallucinations in multimodal tasks.\n3. The authors design comprehensive experiments and ablation studies to support the findings.", "weaknesses": "1. The experiments only focus on the LLaVA model. The authors can consider other prevailing VLMs.\n2. The authors can provide more direct evidence (experiments) to support the potential applications of the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1cBih1twoA", "reviewer": "Reviewer_nkbz", "summary": "This paper presents a study of the processing dynamics of visual tokens within adapter-style VLMs, particularly focused on investigating where object information may be stored and how it is used throughout the model. \n\nThe study is anchored around 3 main methods: \n\n1. The ablation of tokens pertaining to the locations of objects within images (by replacing with a globally averaged token). This is compared against the ablation of register tokens, random tokens, and high-gradient tokens (determined using the Integrated Gradients method for selecting tokens most influential in a given model decision). Experimental results (Table 1) show that  the ablation of object level tokens results in a significantly higher degradation in performance when compared against ablating comparable amounts of random or high-gradient tokens. \n\n2. Applying a Logit Lens style analysis to intermediate layers' representations to qualitatively study what information is encoded in intermediate representations of tokens as they flow through the model. A (I believe qualitative) study over this shows that activations of visual tokens in later layers of the model tend to be decoded by the unembedding matrix into tokens pertaining to concepts related to the image (e.g. an \"eye\" on a face). \n\n\n3. Attention knockout to better understand the importance of certain token group dyads for model performance. Results show that \"knocking-out\" attention from object tokens to the last visual token (which is conjectured to be used by VLMs to reason about outputs) results in a greater performance degradation when compared against knocking out all visual tokens except for those in the last row to the visual tokens in the last row. \n\n**Recommendation**\nI think the paper is generally very well written, easy to understand, and generally supports its claims about the locality of object-level information to tokens pertaining to object regions in images. I have a few questions and concerns (see Weaknesses and Questions below) which I would value the authors addressing before I finalize my score (and which I think addressing could help strengthen the clarity of the paper). I am generally leaning towards recommending acceptance.", "questions": "* L257 - 259: This is likely not necessary to clear up, but I'm curious if the authors tried ablating tokens by zeroing out embeddings rather than using the global average token? I imagine this might just generally hurt performance potentially adding a confounding factor to the experiments. Depending on the answer, perhaps mentioning the reason behind this design choice could add clarity. \n\n* L262-L266: How is it ascertained whether the target object o is mentioned in the generated description? Is this done manually or automatically? \n\n* L242-L246: How is correct identification determined? Is this automated or manual? \n\n* L329-:330: I'm wondering what the precise definition of \"comparable number of tokens\" is here? Is this compared to the average number of object tokens per image in the dataset? \n\n* L350-L352: How are correspondences identified, is this done by hand? If so, could the authors please comment on the statistics of how many images were evaluated, and if some measure of percentage of correspondences found was measured? In other words, I'm trying to get a sense of how reliably this behavior is observed in the model. Given the current level of detail provided, readers may assume that these results were cherry picked. \n\n* L369: \"such numbers seemingly counting applies in background tokens.\" -- the wording is a little awkward, consider revising.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Presentation**: The paper is very clearly written and the provided figures do an excellent job of communicating the experimental setup and results. I have some questions about implementation details that appear omitted, but everything that was included in the paper was generally very easy to follow due to the quality of the writing. \n\n**Token Ablation Experiments**: The experiments presented in section 3 appear quite thorough and soundly motivated/constructed. The filtration strategy selected to simplify the experimental setup makes sense, and the selection of baselines to compare object token ablation against provides good coverage of alternative explanations to the papers' hypothesis. As a result, the provided experimental results appear to clearly show the presence of object information within the tokens of object regions in images.", "weaknesses": "**Attention Flow Experiments**: The attention flow experiments appear somewhat inconclusive. The provided experimental results do indeed show a distinct performance degradation when knocking out attention from object tokens in later layers to the last visual token when compared against knocking out attention from all rows to the last row. However, the authors state in L219-L228 that there are competing explanations about where information is aggregated (e.g. just within visual tokens vs. also within text tokens). I would have therefore expected an additional condition in Table 2 showing the performance of knocking out some variant of text token attention. I would appreciate it if the authors could comment on this. \n\n**Missing Details**: This is minor, but I think there's a number of experimental details missing whose inclusion could stand to strengthen both the clarity and reproducibility of the paper (see Questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g6PAwPkuR3", "reviewer": "Reviewer_LYyP", "summary": "This work proposes some interpretable experiments to investigate the flow of visual information processing in LLaVA-based VLMs. The authors find 3 synthetic tasks to test whether the blurred object tokens will affect the prediction accuracy of VLMs. Secondly, the attention blocking experiments investigate whether the VLMs rely on visual information summarization to predict the correct answer. The findings are interesting and vital to VLM development.", "questions": "1. The definition of last row of visual tokens in line 435 is very confused to me. For example, CLIP-L has 16*16 visual tokens, does it mean the last [-1,;] 16 tokens? It is recommend to provide a lot of explanations on that and why such experiments on last row visual tokens are required? \n\n2. The experiments in Table 2 is highly related to how you split the early/mid/late layers. Since Phi-3 has less layers than the LLM used in LLaVA, vicuna-7b, is there any difference on your findings when you adopt LLaVA-Phi-3 as investigation VLMs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This work provides very important insights to the visual information processing of VLMs. Even if previous phenomena like the correlation between high-resolution SFT and good performance in LLaVA-1.6 has too some degree revealed that the object visual tokens can be very significant to the correct image understanding of VLMs, the quantitive analysis on such effects are not conducted.", "weaknesses": "1. The contributions on the interpretability of visual information processing in VLMs are well recognized, while the contributions on how we can adopt such information processing pattern on better prompting or training for VLMs are limited. For example, if the object visual tokens are vital for the answer prediction, is it possible that simply repeat them or augment them can lead to significant performance improvement? And maybe some text prompts or soft-prompts can well utilize such object visual tokens for better model understanding?\n\n2. The investigations focus on object only, which are only important to the general VQA, captioning, and some specific object detection tasks. Beyond object visual tokens, the OCR tokens, text rendering tokens, and knowledge-related tokens can also be your study object. For example, what is the effect of blocking the important text tokens to the tasks of TextVQA, OCRVQA, MME, POPE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)