Towards Interpreting Visual Information Processing in Vision-Language Models

## Abstract

Abstract Vision-Language Models (VLMs) are powerful tools for processing and understanding text and images. We study the processing of visual tokens in the language model component of LLaVA, a prominent VLM. Our approach focuses on analyzing the localization of object information, the evolution of visual token representations across layers, and the mechanism of integrating visual information for predictions. Through ablation studies, we demonstrated that object identification accuracy drops by over 70% when object-specific tokens are removed. We observed that visual token representations become increasingly interpretable in the vocabulary space across layers, suggesting an alignment with textual tokens corresponding to image content. Finally, we found that the model extracts object information from these refined representations at the last token position for prediction, mirroring the process in text-only language models for factual association tasks. These findings provide crucial insights into how VLMs process and integrate visual information, bridging the gap between our understanding of language and vision models, and paving the way for more interpretable and controllable multimodal systems.

## 1 Introduction

Vision-Language Models (VLMs) take an image and text as input, and generate a text output. They have become powerful tools in processing and understanding text and images, enabling a wide range of applications from question answering to image captioning (Liu et al., ; Li et al., ; Alayrac et al., ). Among VLM architectures, the “adapter” style has demonstrated impressive state-of-the-art performance, notable for its simplicity. These models combine a pre-trained image encoder, a pre-trained language model (LM), and a learned adapter network that maps image encoder outputs to soft prompts for LM inputs (Merullo et al., ; Liu et al., ; ). Despite their importance and potential, the inner workings of VLMs remain poorly understood as compared to their text-only counterparts. While significant progress has been made in understanding language models (Elhage et al., ; Wang et al., ; Bricken et al., ) and, to a lesser extent, vision transformers (Palit et al., ; Vilas et al., ; Pan et al., ), there is a notable gap in our understanding of VLMs, where both modalities operate in the same space. A deeper comprehension of VLMs could lead to practical advances in safety, robustness, and functionality, similar to the progress made in language models, such as model editing (Meng et al., ) or identifying specific components responsible for certain behaviors (Arditi et al., ).

In this paper, we investigate the LM component of VLMs as they serve as the “brain” of the system, comprising more than 95% of the total parameters for models like LLaVA 1.5 7B. LLMs pre-trained on large text datasets are likely to remain the foundation of VLMs because of their powerful reasoning capabilities. Therefore, understanding how these LMs process and integrate visual information is crucial for advancing VLM interpretability.

Many open questions remain regarding the inner workings of VLMs. The nature and structure of visual representations fed into the LM remain unclear, as these visual inputs are soft prompts that do not correspond to language tokens and cannot be interpreted as such (Lester et al., ; Merullo et al., ). This raises questions about how visual information is encoded in these representations. Furthermore, the spatial distribution of information within the image encoder’s feature maps remains unclear, leaving open the question of whether and how much object-specific details are localized or dispersed across the entire representation.

Furthermore, the mechanisms by which the language model processes these visual inputs are not well understood. A significant modality gap exists between visual and textual input (Jiang et al., ), and it is unclear whether our mechanistic understanding of text processing in language models generalizes to the processing of visual input in VLMs.

To understand the representations in the visual inputs for VLM and how the VLM processes them, we study LLaVA 1.5 7B (Liu et al., ), a popular open-source VLM with competitive performance that uses CLIP (Radford et al., ) as an image encoder and Vicuna 13B (Chiang et al., ) as its language model. We also study LLaVA-Phi(^1^11Sourced from the [xtuner HuggingFace model](https://huggingface.co/xtuner/llava-phi-3-mini-hf).) for our ablation experiments. We focus on a set of object identification tasks and our main contributions are as follows:

- 1.
Using ablation techniques, we demonstrate that the information for an object is highly localized to the token positions corresponding to their original location in the image.
- 2.
By extending the *logit lens* technique primarily used for language models (Nostalgebraist, ), we find that the representations of the visual input in the LM are refined towards the embedding of interpretable tokens in the vocabulary, despite the LM not being explicitly trained to do so.
- 3.
By blocking the attention flow between tokens, we show that the model extracts object information from the object tokens in the middle to late layers.

Our findings are first steps in understanding the internal mechanisms of VLMs, paving the way for more interpretable and controllable multimodal systems. The code for our experiments is available at [https://github.com/clemneo/llava-interp](https://github.com/clemneo/llava-interp).

Figure: Figure 1: In adapter-style Vision-Language Models (VLMs), the visual tokens (in green) are soft prompts for the language model (LM), and are not interpretable through the vocabulary embedding. Through a set of object identification tasks, we find that (1) object information can be localized to a subset of visual tokens, (2) the representations of the visual tokens evolve towards interpretable text embeddings, and (3) the model extracts some information from the visual tokens to the last token position in the middle to late layers, to identify the object.
Refer to caption: x1.png

## 2 Background

Transformer Architecture. Transformer-based autoregressive Large Language Models (LLMs) process sequences of input tokens to predict the next token in the sequence (Vaswani et al., ). These models achieved state-of-the-art performance across various natural language processing tasks (Brown et al., ), even though they are merely trained on next-token prediction (Radford et al., ). A LLM takes an input sequence $X=(x_{1},...,x_{n})$ and outputs a probability distribution over the vocabulary $V$ to predict the next token $x_{n+1}$.

To do so, the model refines token representations layer by layer through a series of computations. Each token $x_{i}$ is initially represented by an embedding vector $h^{0}_{i}$ of dimension $d_{m}$, obtained from a lookup operation in an embedding matrix $W_{E}\in\mathbb{R}^{|V|\times d_{m}}$. This representation is then updated through successive layers using multi-head self-attention (MHSA) and feed-forward (FF) sublayers:

$$ $h^{l}_{i}=h^{l-1}_{i}+a^{l}_{i}+f^{l}_{i}$ (1) $$

where $h^{l}_{i}$ is the representation of token $x_{i}$ at layer $l$, $a^{l}_{i}$ is the output from the MHSA sublayer, and $f^{l}_{i}$ is the output from the FF sublayer. All vectors $h^{l}_{i},a^{l}_{i},f^{l}_{i}\in\mathbb{R}^{d_{m}}$.

The MHSA sublayer consists of $H$ attention heads working in parallel, each implemented as follows:

$$ $\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}+M \right)V$ (2) $$

where $Q,K,V\in\mathbb{R}^{n\times d_{k}}$ are query, key, and value matrices derived from learned linear projections of the input, and $M\in\mathbb{R}^{n\times n}$ is a causal mask. The causal mask $M$ is set to $-\infty$ in the upper right triangle, meaning $M_{ij}=0\text{ if }i\geq j,\text{ else }-\infty$. This ensures that each position can only attend to previous positions and itself, thus preserving the autoregressive property in language models. These outputs are then concatenated and linearly projected to produce the final output of the multi-head attention sublayer.

The FF sublayer applies two linear transformations with an element-wise non-linear function $\sigma$ between them:

$$ $f^{l}_{i}=W^{l}_{v}\sigma(W^{l}_{k}a^{l}_{i}+b^{l}_{k})+b^{l}_{v},$ (3) $$

where $W^{l}_{v}$, $W^{l}_{k}$, $b^{l}_{k}$, and $b^{l}_{v}$ are learned parameters.
Finally, the representation of the last token $x_{n}$ at the final layer, $h^{L}_{n}$, is projected into a probability distribution over $V$ using an unembedding matrix $W_{U}\in\mathbb{R}^{d\times|V|}$ and a softmax operation.

Image Encoders. The transformer architecture has been successfully adapted for state-of-the-art image encoders, as demonstrated by Dosovitskiy et al. (). In this approach, input images are divided into fixed-size patches, flattened, and linearly embedded before being processed as a sequence of tokens, analogous to word embeddings in NLP tasks. Unlike text-based models, the attention mechanism in these image encoders is bidirectional (i.e., no masking is applied). The input sequence includes a special class token, whose final embedding is typically used for image classification tasks. The representations at the other token positions form what is called the *feature map* of the image.

Building upon this foundation, Radford et al. () introduced Contrastive Language-Image Pre-training (CLIP), which jointly trains a vision transformer and a text trasnsformer to produce similar embeddings for matching image-text pairs. CLIP’s image encoder is currently being used for state-of-the-art performance on many downstream tasks like zero-shot image classification and vision-language models like LLaVA.

Connecting Text and Image Models. Tsimpoukelli et al. () pioneered the concept of training an image encoder such that its output can be used by a frozen language model for multimodal few-shot reasoning. Merullo et al. () advanced this approach by freezing both the pre-trained image encoder and pre-trained language model, training only a linear map that converts image features into inputs for the text model. This introduced the “adapter style” approach, utilizing pre-trained components for both modalities.

Building on this foundation, Liu et al. () introduced LLaVA, which combined a CLIP ViT-L/14 image encoder with a Vicuna-13B language model, connected by a trained linear projection, or an MLP for improved performance (Liu et al., ). Crucially, LLaVA’s training process focuses solely on fine-tuning for multimodal conversation, without any next-token pretraining on image-text pairs. This approach allows LLaVA to leverage the strong pre-trained capabilities of both the vision and language models while efficiently adapting to multimodal tasks.

### 2.1 Notation

In this section, we describe how an image and query is processed through LLaVA.

Image Processing. The input image $I$ is processed by first being cropped into a square, then suitably resized before being divided into 576 image patches. The CLIP ViT-L/14 image encoder $f_{\text{CLIP}}$ processes this image to produce a feature map $E_{I}=f_{\text{CLIP}}(I)\in\mathbb{R}^{N\times d_{\text{CLIP}}}$, where $N=576$ is the number of image patches and $d_{\text{CLIP}}$ is the dimensionality of CLIP’s embeddings. An *adapter network* $A$ then maps these CLIP embeddings to the language model’s input space, yielding a set of visual tokens $E_{A}=A(E_{I})\in\mathbb{R}^{N\times d}$, where $d_{m}$ is the dimensionality of the language model’s input embeddings. We call $E_{A}$ visual tokens to distinguish them from the image tokens.

Text Processing and Combined Input. For the text input, given a tokenized prompt sequence $T=(t_{1},\ldots,t_{M})$, the embedding layer of the language model $E_{\text{LM}}$ maps these tokens to embeddings: $E_{T}=E_{\text{LM}}(T)\in\mathbb{R}^{M\times d_{\text{LM}}}$. The full input to the language model is the concatenation of the adapted image embeddings and the text embeddings: $X=[E_{A};E_{T}]\in\mathbb{R}^{(N+M)\times d_{\text{LM}}}$, from which the language model can generate output autoregressively.

### 2.2 Open Questions and Hypotheses

One might expect that the output of the adapter $E_{A}$ would produce embeddings corresponding to image information. For example, if the image contained a car, we might expect the adapter to produce tokens that correspond to the embedding of “car”. However, the output of the adapter forms soft prompts that have no semantic meaning when decoded through the vocabulary (Merullo et al., ; Bailey et al., ).

#### 2.2.1 Localization of Object-Specific Information

A key question of our investigation is: Where is object-specific information located in visual tokens? Two conflicting hypotheses emerge from recent research:

- 1.
Object-Centric Localization: Joseph & Nanda () found that in classical vision transformers, token representations corresponding to specific object locations tended to align with those objects’ class embeddings in the late layers, even without explicitly training for this behavior. This suggest that object information might be localized to the tokens corresponding to the spatial location of the object in the original image. However, their findings were on a vision transformer trained on ImageNet classes, and not the vision transformer in CLIP.
- 2.
Global Information in Register Tokens: In contrast, Darcet et al. () identified *register tokens* in vision transformers, including a CLIP variant. These are background patch tokens with unusually high norms that appear to encode global image features.
It follows that the LM in an adapter-style VLM could primarily rely on these information-rich register tokens for its output.

We investigate these hypotheses in Section 3 using ablation techniques in a set of object identification tasks.

#### 2.2.2 Processing of Object-Specific Representation

The processing of visual tokens may differ from text tokens.
Soft prompts are very different from normal prompts: (Bailey et al., ), there is a modality gap between textual and visual representations in VLMs (Jiang et al., ). Furthermore, LLaVA is fine-tuned exclusively on visual question-answering (VQA) prompt-response pairs, without pre-training on next-token prediction. This may affect how the model processes visual information as compared to text. In Section 4.1, we attempt the *logit lens* method to see how the representations of visual tokens evolve through the layers.

Recent studies have shed light on how LMs process information. Geva et al. () identified a refine-then-extract process in factual association prompts, where the model first refines information at subject tokens through MLP sublayers before transferring it to the last token position via attention sublayers. Similar mechanisms have been observed in arithmetic tasks (Stolfo et al., ), suggesting that this (sequencing of functions) might be a general mechanism in LMs.

However, Basu et al. () adapted the factual association setup to VLMs and found preliminary evidence that the LM may first summarize image information in text tokens, instead of extracting the image information to the last token position from the image tokens directly. While we do not study factual association – rather we choose to focus on a simpler object identification task, we study how information flows from the image tokens in Section 4.2.

## 3 Investigating the representations in mapped visual tokens

In this section, we use ablation experiments to test if object information is concentrated in specific visual tokens. By ablating selected tokens and observing degradations in the model’s ability to identify the object, we map how object information is distributed across the visual tokens.

Dataset. We use images from the COCO Detection Training set (Lin et al., ). To ensure the reliability of our results, we employ two filtering steps:

(1) Choosing Simpler Images. As a heuristic to focus on simpler images, we choose images where (a) there is an object whose size is between 1,000-2,000 square pixels, or about 2-4% of the image by area, (b) only one instance of the that object is present in the image, and (c) there are less than 4 annotated objects. We present a few examples of the images in Figure 4.

(2) Controlling for Hallucination. Model can sometimes hallucinate objects based on context, even when the object’s visual information is removed. To address this, we create two versions of each image: the original and one where the target object is masked with noise. We only keep images where the model correctly identifies the object in the original image but fails to identify it in the masked version. This ensures that the model’s object identification relies on the object’s visual information.

After applying both filtering steps, our final dataset comprises 4,318 images.

Method. We present a high level overview of our methodology in Figure 2. Let $E_{A}=\{e_{1},...,e_{N}\}$ be the set of visual tokens.
We define a subset $S\subset\{1,...,N\}$ consisting of the indices of tokens hypothesized to contain information about a particular object $o$ in the image.

For each ablation experiment, we create a modified set of embeddings $E_{A}^{\prime}$:

$$ $e_{i}^{\prime}=\begin{cases}\bar{e}&\text{if }i\in S\\ e_{i}&\text{otherwise}\end{cases}$ $$

where $\bar{e}=\frac{1}{N}\sum_{i=1}^{N}e_{i}$ is the mean embedding across all visual tokens from 50,000 images from the ImageNet validation split (Deng et al., ). This replaces the hypothesized object-relevant tokens with an average token, effectively ablating their specific information. We then evaluate the impact of token ablation using three methods:

Figure: Figure 2: Overview of our ablation experiments. (1) We ablate some visual tokens that potentially contain object-specific information, (2) prompt the model to describe the image, or answer object-specific questions, then (3) measure the impact of token ablation by calculating the percentage of initially correct object identifications that become incorrect after ablation.
Refer to caption: x2.png

- 1.
Generative Description: We prompt the model with “Describe the image” using both the original ($E_{A}$) and ablated embeddings ($E_{A}^{\prime}$). We then ascertain whether the target object $o$ is mentioned in the model’s generated description before and after ablation. If $o$ is mentioned with $E_{A}$ and not $E_{A}^{\prime}$, it indicates that the ablated tokens were crucial for the model to identify and include the object in its description.
- 2.
Binary Polling: We ask the model “Is there a [o] in this image?” using both $E_{A}$ and $E_{A}^{\prime}$. We then compare whether the model’s next generated token changes from “Yes” to “No” after ablation, which indicates that the ablated tokens were crucial for identifying the object.
- 3.
Visual Question-Answering: We manually curate a set of 100 images with specific questions about objects, such as “What is on the bed?”. We choose questions with unambiguous answers and avoid objects directly related to the question (e.g., avoiding “pillow” for a bed-related question). We prefill the model’s response with “It is a” and compare the next generated token before and after ablation. A set of examples is in Figure 5.

We choose the subset $S$ of tokens to ablate in five ways:

(1) Object Tokens. We choose the visual tokens whose positions correspond to the image patches that originally contained that object. (2) Object Tokens with Buffer. In addition to the object tokens, we include neighboring tokens. We test with two buffer sizes: 1 Buffer (including immediately adjacent tokens) and 2 Buffer (including tokens up to 2 positions away from object tokens). (3) Register Tokens. We select visual tokens with norms more than 2 standard deviations from the mean norm. These correspond to the register tokens identified by Darcet et al. (), which are thought to encode global image features. (4) Random Tokens. As a baseline, we ablate $n$ random tokens. (5) High-Gradient Tokens. As a stronger baseline, we use the Integrated Gradients attribution method (Sundararajan et al., ) to identify tokens most important for the model’s decision. For a given object $o$, we prompt the model with “Is there a [o] in the image?” and compute integrated gradients with respect to the logit for the “Yes” token over 50 steps. We then ablate the $n$ tokens with the highest attribution scores.

Results. Our results are presented in Table 1. We find that ablating the object tokens significantly impairs the model’s ability to identify the object. For comparable numbers of ablated tokens, object token ablation consistently results in larger performance decreases across all settings as compared to the gradient-based and random baselines. This suggests that the information about that object is localised to the region of the object token. Furthermore, hese findings are consistent across all three experimental settings and applicable to both LLaVA 1.5 and LLaVA-Phi.

**Table 1: Performance degradation after token ablation. A lower percentage indicates a greater impact of the ablation, meaning the model is more likely to answer incorrectly, thus suggesting that the ablated tokens contain more localized object information. Average token counts apply to both the generative and polling settings for LLaVA 1.5. We find that ablating the object tokens with adjacent tokens (1 Buffer) significantly causes object identification performance to drop (in bold), more so than the integrated gradients and random baselines for a comparable number of tokens (in bold).**
|  | LLaVA 1.5 | LLaVA-Phi |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Ablation Type | Generative | Polling | VQA | Generative | Polling | VQA |
| (Avg Token Count) | Decrease (%) | Decrease (%) | Decrease (%) | Decrease (%) | Decrease (%) | Decrease (%) |
| Object (12.6) | 33.33 | 15.38 | 33.33 | 47.46 | 28.48 | 41.51 |
| + 1 Buffer (33.4) | 71.79 | 51.28 | 86.67 | 82.89 | 86.48 | 96.23 |
| + 2 Buffer (60) | 76.92 | 64.10 | 92.22 | 89.56 | 95.49 | 100.00 |
| Register tokens (3.1) | 5.13 | 0.00 | 3.33 | 7.63 | 0.00 | 1.89 |
| Int. Gradients (5) | 2.56 | 0.00 | 6.67 | 17.11 | 1.23 | 28.30 |
| Int. Gradients (10) | 20.51 | 2.56 | 6.67 | 25.26 | 4.30 | 24.53 |
| Int. Gradients (20) | 33.33 | 20.51 | 17.78 | 41.14 | 9.63 | 37.74 |
| Int. Gradients (40) | 48.72 | 35.90 | 50.00 | 61.49 | 26.64 | 56.60 |
| Int. Gradients (60) | 58.97 | 48.72 | 71.11 | 70.61 | 40.98 | 69.81 |
| Int. Gradients (100) | 76.92 | 51.28 | 84.44 | 81.23 | 61.68 | 83.02 |
| Int. Gradients (250) | 76.92 | 69.23 | 97.78 | 91.93 | 90.16 | 98.11 |
| Random (5) | 0.00 | 0.00 | 1.11 | 3.68 | 0.00 | 0.00 |
| Random (10) | 4.88 | 0.00 | 0.00 | 3.95 | 0.00 | 1.89 |
| Random (20) | 2.44 | 0.00 | 2.22 | 5.88 | 0.00 | 0.00 |
| Random (40) | 2.44 | 0.00 | 0.00 | 8.51 | 0.00 | 0.00 |
| Random (60) | 7.32 | 0.00 | 1.11 | 9.39 | 0.41 | 3.77 |
| Random (100) | 7.32 | 0.00 | 4.44 | 11.23 | 0.20 | 7.55 |
| Random (250) | 24.39 | 0.00 | 16.67 | 20.61 | 1.84 | 15.09 |

## 4 Investigating visual information processing

### 4.1 Analyzing the Residual Stream through Embedding Space

Can we interpret how the representations at the visual token positions evolve through the layers? To do so, we employ the *logit lens* technique (Nostalgebraist, ): for each layer, we decode the activation at each token position using the unembedding.
Formally, for each hidden state $h_{i}^{l}$ at each token position $i$ and layer $l$, we project it into a probability distribution over $V$ using $W_{U}$ and take the token with the highest logit. We do this for LLaVA 1.5.

Results. We find that the activations in the late layers at each visual token position correspond to token embeddings that describe its original patch object in its corresponding image token. We provide some case studies in Figure 3, and an [interactive demo](http://clementneo.com/llava_logit_lens/) on more examples.

We found that the tokens decoded often correspond to concepts more specific than the object-level. For example, in Figure 3(a), the tokens go beyond identifying the sweater worn (“swe”) and describe the specific patterns on the sweater with tokens like “diam”(ond) and “cross”.

In some instances, okens correspond to their corresponding concepts in other languages, like in Figure 3(c) where the tokens correspond to “year” and “month” but in Chinese and Korean instead. This is somewhat surprising given that previous work has shown that Llama used for non-English languages tend to have English tokens in their intermediate activations when interpreted through the logit lens (Wendler et al., ).

Figure: (a) An image of a lady in the sweater. The logit lens identifies tokens that correspond to specific detail of the sweater, such as “pattern” and “diam”(ond).
Refer to caption: x3.png

Overall, it is surprising that the logit lens work on VLMs! This technique works on auto-regressive LLMs because they are pretrained on next-token prediction, meaning their activations will be iteratively refined towards the unembedding of the predicted next token. However, VLMs are merely fine-tuned on multimodal tasks.
The effectiveness of the logit lens here suggests that the hypothesis that transformers build predictions by promoting concepts in vocabulary space (Geva et al., ) may generalise to VLMs, even when only fine-tuned on multimodal tasks.

Case Study: Global Features. We sometimes find global-level features appearing in unexpected places, such numbers seemingly counting apples in background tokens (Figure 3(d)). When we tried ablating these specific visual tokens, the model’s description remained largely unchanged. Interestingly, the numbers persisted in some token positions in the logit lens even after ablation. These findings suggest that the global features revealed by the logit lens could be artifacts of the language model’s processing rather than direct information from the visual tokens. We hypothesise this to be one of the reasons as to why VLMs based on CLIP often have worse performance on image classification tasks than CLIP itself (Zhang et al., ).

Potential Applications. This may be useful for getting coarse segmentation maps for little to no extra computation, which could lead to downstream applications. For example, Liu et al. () suggests to adjust the attention weights towards the visual tokens in the LM to reduce hallucinations. We speculate that directing attention towards specific objects may improve this method, though we leave further exploration to future work.

### 4.2 Tracing Attention Flow

To understand how visual information is processed and integrated within the language model, we investigate the flow of critical information through the network. Specifically, we examine whether the model directly extracts information from relevant visual tokens, or whether it aggregates information from visual tokens before transferring it to the final output token.

Method. We employ the *attention knockout* technique introduced by Geva et al. (). This method involves selectively blocking attention between specific tokens at different layers of the model, allowing us to assess the importance of various connections for the model to identify the object. Our process is as follows:

(1) For a given input and target layer $\ell$, we artificially block specific attention connections in the multi-head self-attention (MHSA) sublayer by setting the corresponding attention mask values to negative infinity: $M^{\ell+1,j}_{rc}=-\infty\quad\forall j\in[1,H]$ where $M^{\ell+1,j}$ is the attention mask for head $j$ at layer $\ell+1$, and $r$ and $c$ are the indices of the tokens between which we want to block attention.

(2) We apply this blocking over a window of consecutive layers, including early layers (L1-10), early to middle layers (L5-14), middle layers (L11-20), middle to late layers (L15-24), and late layers (L21-31).

(3) For each window, we measure the decrease in accuracy and prediction probability for the correct token when blocking attention under three scenarios: (i) From object tokens and their surrounding buffer to the final token. (ii) From all non-object tokens to the final token. (iii) From all visual tokens except the last row to the last row of visual tokens.
We conduct this experiment using LLaVA 1.5 on our curated Visual Question Answering (VQA) dataset of 100 images.

Findings. Our results are presented in Table 2.

We find that blocking attention from the object tokens (and their buffers) to the final token in mid-late layers leads to noticeable performance degradation. This suggests that the model directly extracts object-specific information in these later stages. Blocking attention from non-object tokens to the final token in early layers also causes some performance drop. This indicates that contextual information from the broader image is processed and integrated in the earlier stages of the model.

Interestingly, blocking attention from visual tokens to the last row of visual tokens has minimal effect on performance. This contrasts with findings by Basu et al. (), who suggested that the model might summarize image information in the last row of visual tokens before using it. Our results indicate that, at least for our object identification tasks, the model doesn’t rely on such a summarization step.

**Table 2: Results of blocking attention between different token groups across various layers of the LLaVA 1.5 model. The values represent the relative performance on the correct token being predicted when blocking attention, with 1.00 being no impact and 0.00 being all questions wrong post-ablation.**
| Attention Blocking | Layer |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| From^* | To^† | Early | Early-Mid | Mid | Mid-Late | Late | All |
| O | LTP | 1.00 | 1.00 | 0.96 | 0.88 | 0.93 | 0.82 |
| O+1 | LTP | 1.00 | 0.99 | 0.90 | 0.82 | 0.89 | 0.67 |
| O+2 | LTP | 1.00 | 1.00 | 0.91 | 0.80 | 0.91 | 0.68 |
| I-(O+1) | LTP | 0.88 | 1.00 | 0.97 | 0.96 | 0.98 | 0.82 |
| O+1 | LVR | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| I-LVR | LVR | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |

## 5 Related Work

Explainability. Classical techniques like GRAD-Cam (Selvaraju et al., ) and Integrated Gradients (Sundararajan et al., ), focus on explaining model outputs based on inputs. Recent tools like LVLM-Interpret (Stan et al., ) extend these methods to VLMs. While these approaches are valuable for understanding input-output relationships, they primarily focus on input attribution. Our work complements these by investigating the model’s internal processing mechanisms.

Mechanistic Interpretability (MI) for LLMs.
Recent techniques such as causal tracing (Meng et al., ) and sparse coding methods (Bricken et al., ; Huben et al., ; Marks et al., ) have advanced our understanding of how LLMs perform tasks despite being primarily trained for next-token prediction. However, VLMs are not trained on next-token prediction for visual inputs (Liu et al., ), raising questions about the generalizability of LLM processing mechanisms to visual inputs in VLMs. Our work extends some of these interpretability techniques to the multimodal domain to understand how they process visual inputs.

MI for Multimodal Models. In CLIP, (Goh et al., ) discovered multimodal neurons in the image encoder that respond to concepts across different modalities. Gandelsman et al. (; ) developed methods to decompose CLIP’s image representation, revealing property-specific roles for attention heads and polysemantic behavior in neurons. Rajaram et al. () introduced a method to identify network subgraphs in CLIP that had interpretable functions. Rao et al. (); Daujotas (); Fry () used sparse coding methods to yield interpretable features in the image encoder. However, CLIP’s architecture and training objectives differ substantially from generative VLMs, which our work focuses on.

For generative VLMs, Schwettmann et al. () found neurons in the LM that corresponded to visual concepts, while Palit et al. () adapted causal tracing for BLIP. While these studies provide valuable insights to specific components in the VLM, our work aims to investigate the representation in visual inputs and how those representations evolve in the LM.

Basu et al. () used causal tracing and attention contributions in a visual question-answering setup to find that VLMs such as LLaVA retrieve information from earlier causal layers of the LM (i.e. layers 1-4 vs layers 4-7 in an LLM) when answering a multi-modal question. Their analysis suggests that the visual information may be summarised in a consistent subset of late visual tokens. In contrast, we study VLMs in a more straightforward object-identification setting. We also examine their information transfer hypothesis through our attention blocking experiments in Section 4.2.

## 6 Conclusion

In this work, we investigated the processing of visual information in Vision-Language Models, focusing on the LLaVA architecture. Our study reveals that object-specific information is highly localized within visual tokens corresponding to the object’s spatial location in the original image. This challenges the hypothesis that VLMs primarily rely on global image features encoded in register tokens.

Surprisingly, we found that representations at visual token positions evolve through the model’s layers to align with interpretable textual concepts, suggesting that VLMs may refine visual information towards language-like representations even without explicit pretraining on next-token prediction for visual inputs. Our attention blocking experiments further showed that VLMs directly extract object-specific information from relevant visual tokens in mid to late layers, while also processing broader contextual information from non-object tokens in earlier layers.

These findings advance our understanding of VLMs, bridging the gap between our knowledge of language and vision models. Our work provides a foundation for more interpretable and controllable multimodal systems. We hope to further investigate how our insights can be used to improve VLM performance, particularly in reducing hallucinations, model editing, and enhancing factual accuracy.

## Limitations

We focused on LLaVA-type models, testing our findings on two variants (LLaVA 1.5 and LLaVA-Phi). We chose LLaVA for its simple architecture, viewing our findings as a starting point for interpreting more complex VLMs. However, our results may not generalize to VLMs with significantly different architectures or training procedures. Further research is needed to confirm whether similar mechanisms exist in other models.

Our study primarily investigates object identification tasks. We chose this focus as it provides a simpler, more direct setup compared to tasks requiring multi-step reasoning, such as multimodal factual association. While we attempted to diversify our experimental setup within this domain by testing the model’s ability to identify object through three distinct ways, our findings may not fully capture VLM behavior in more complex tasks like reasoning or open-ended question answering. The applicability of our results to these broader task types remains an open question.

## Author Contributions

Clement Neo conceived the study, designed and performed the experiments, and led the writing of the manuscript. Luke Ong contributed to the writing and literature review. Philip Torr, Mor Geva and David Kreuger provided advice and comments. Fazl Barez served as the primary advisor for this work and helped significantly with the design and write-up of the paper.

## Acknowledgments

We thank the Torr Vision Group for providing compute and hosting Clement Neo during the internship.

We thank Sarah Schwettmann, Ashkan Khakzar, William Rudman, Joseph Miller, Alex Spies for discussing and reading drafts of the paper.

## Appendix A Filtered Dataset Examples

Figure: Figure 4: Example of dataset images for the object identification tasks.
Refer to caption: extracted/5914255/21571.png

## Appendix B Curated Images and Questions Examples

Figure: (a) “What is on the bench? It is a” $\rightarrow$ book
Refer to caption: extracted/5914255/000000302035.jpg

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: In Figure 3, what layer do these unembeddings correspond to? Are these all from a specific layer, or did you select instances from different layers that matched human interpretations?
soundness: 4
strengths: 1. Important problem: The authors address an important problem of understanding how visual knowledge is localized and processed in transformer-based VLMs.

2. The formulation of each sub-question is well-motivated, the method and metrics are well-designed, the analysis of each sub-question is really thorough, and the findings are well summarized.

3. The paper is very well-written and easy to follow.
summary: This work applies mechanistic interpretability techniques to understand how visual information is localized and processed in VLMs. This work:

- Tries to localize where information processing happens within VLM transformers
- Applies the “logit lens” (Nostalgebreist 2020) to understand how representations of visual tokens evolve across layers of the transformers
- Applies attention knockout to see how blocking attention between different subsets of tokens in the transformer causes the accuracy of the VLM to decrease

For localization, they find that ablating object tokens with a buffer has the most impact on model performance on the 3 ablations, with two LLaVA VLMs. For the logit lens, they find that patch embeddings at different layers can be mapped to corresponding meaninful tokens in the tokenizer vocabulary.

**EDIT 11/25/2024:** I am raising my score from 6 to 8, and soundness from 3 to 4, after the author response.
weaknesses: 1. **Evaluated insufficient VLMs:** As the authors acknowledged, the analysis is performed on only two VLMs from the same (LLaVA) family. However, the methodologies should be easily extendable to other transformer-based VLMs. 

    A key sticking point of mechanistic interpretability methods is that we do not know how the findings generalize to other models, so it is hard to take any findings at face value when they have only been performed on one model (especially when some of the findings contradict findings from other works, e.g. VLMs rely on features stored in register tokens). 

    I suggest including findings from other VLMs such as InstructBLIP, QwenVL, InternVL, and seeing if your findings generalize to more models.

2. **Results of "logit lens" analysis are hard to interpret:** While the authors find some interesting mappings between visual patches and output tokens, it's hard to tell if these are meaningful correspondences or an artifact of cherry-picking/confirmation bias, since we see mappings for only a handful of patches, and only at one layer (although it is unclear which one). 

    One way to make this analysis seem less cherry-picked is to annotate patch-token pairings manually for whether they are a "matching" pair or not, and analyze what % of patches map to a matching token. You could also look at how these mappings evolve across layers.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Is it easy to apply the methods and findings in this paper to downstream tasks?
soundness: 4
strengths: 1. The paper is well-written and well organized. There are effective diagrams and explanations that make complex concepts accessible.
2. The paper presents an innovative approach to understanding how visual tokens are processed in the language model component of VLMs. The insights on visual token alignment could inform practical techniques to improve VLM robustness and reduce hallucinations in multimodal tasks.
3. The authors design comprehensive experiments and ablation studies to support the findings.
summary: This paper studies how VLMs, particularly LLaVA, process and integrate visual information. Through ablation studies, the authors show that object information is localized in specific visual tokens, which evolve into interpretable text-like representations across layers. Attention flow experiments reveal that the model extracts object-specific information in mid-to-late layers while processing contextual information in earlier layers.
weaknesses: 1. The experiments only focus on the LLaVA model. The authors can consider other prevailing VLMs.
2. The authors can provide more direct evidence (experiments) to support the potential applications of the proposed methods.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: * L257 - 259: This is likely not necessary to clear up, but I'm curious if the authors tried ablating tokens by zeroing out embeddings rather than using the global average token? I imagine this might just generally hurt performance potentially adding a confounding factor to the experiments. Depending on the answer, perhaps mentioning the reason behind this design choice could add clarity. 

* L262-L266: How is it ascertained whether the target object o is mentioned in the generated description? Is this done manually or automatically? 

* L242-L246: How is correct identification determined? Is this automated or manual? 

* L329-:330: I'm wondering what the precise definition of "comparable number of tokens" is here? Is this compared to the average number of object tokens per image in the dataset? 

* L350-L352: How are correspondences identified, is this done by hand? If so, could the authors please comment on the statistics of how many images were evaluated, and if some measure of percentage of correspondences found was measured? In other words, I'm trying to get a sense of how reliably this behavior is observed in the model. Given the current level of detail provided, readers may assume that these results were cherry picked. 

* L369: "such numbers seemingly counting applies in background tokens." -- the wording is a little awkward, consider revising.
soundness: 3
strengths: **Presentation**: The paper is very clearly written and the provided figures do an excellent job of communicating the experimental setup and results. I have some questions about implementation details that appear omitted, but everything that was included in the paper was generally very easy to follow due to the quality of the writing. 

**Token Ablation Experiments**: The experiments presented in section 3 appear quite thorough and soundly motivated/constructed. The filtration strategy selected to simplify the experimental setup makes sense, and the selection of baselines to compare object token ablation against provides good coverage of alternative explanations to the papers' hypothesis. As a result, the provided experimental results appear to clearly show the presence of object information within the tokens of object regions in images.
summary: This paper presents a study of the processing dynamics of visual tokens within adapter-style VLMs, particularly focused on investigating where object information may be stored and how it is used throughout the model. 

The study is anchored around 3 main methods: 

1. The ablation of tokens pertaining to the locations of objects within images (by replacing with a globally averaged token). This is compared against the ablation of register tokens, random tokens, and high-gradient tokens (determined using the Integrated Gradients method for selecting tokens most influential in a given model decision). Experimental results (Table 1) show that  the ablation of object level tokens results in a significantly higher degradation in performance when compared against ablating comparable amounts of random or high-gradient tokens. 

2. Applying a Logit Lens style analysis to intermediate layers' representations to qualitatively study what information is encoded in intermediate representations of tokens as they flow through the model. A (I believe qualitative) study over this shows that activations of visual tokens in later layers of the model tend to be decoded by the unembedding matrix into tokens pertaining to concepts related to the image (e.g. an "eye" on a face). 


3. Attention knockout to better understand the importance of certain token group dyads for model performance. Results show that "knocking-out" attention from object tokens to the last visual token (which is conjectured to be used by VLMs to reason about outputs) results in a greater performance degradation when compared against knocking out all visual tokens except for those in the last row to the visual tokens in the last row. 

**Recommendation**
I think the paper is generally very well written, easy to understand, and generally supports its claims about the locality of object-level information to tokens pertaining to object regions in images. I have a few questions and concerns (see Weaknesses and Questions below) which I would value the authors addressing before I finalize my score (and which I think addressing could help strengthen the clarity of the paper). I am generally leaning towards recommending acceptance.
weaknesses: **Attention Flow Experiments**: The attention flow experiments appear somewhat inconclusive. The provided experimental results do indeed show a distinct performance degradation when knocking out attention from object tokens in later layers to the last visual token when compared against knocking out attention from all rows to the last row. However, the authors state in L219-L228 that there are competing explanations about where information is aggregated (e.g. just within visual tokens vs. also within text tokens). I would have therefore expected an additional condition in Table 2 showing the performance of knocking out some variant of text token attention. I would appreciate it if the authors could comment on this. 

**Missing Details**: This is minor, but I think there's a number of experimental details missing whose inclusion could stand to strengthen both the clarity and reproducibility of the paper (see Questions below).

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. The definition of last row of visual tokens in line 435 is very confused to me. For example, CLIP-L has 16*16 visual tokens, does it mean the last [-1,;] 16 tokens? It is recommend to provide a lot of explanations on that and why such experiments on last row visual tokens are required? 

2. The experiments in Table 2 is highly related to how you split the early/mid/late layers. Since Phi-3 has less layers than the LLM used in LLaVA, vicuna-7b, is there any difference on your findings when you adopt LLaVA-Phi-3 as investigation VLMs?
soundness: 3
strengths: 1. This work provides very important insights to the visual information processing of VLMs. Even if previous phenomena like the correlation between high-resolution SFT and good performance in LLaVA-1.6 has too some degree revealed that the object visual tokens can be very significant to the correct image understanding of VLMs, the quantitive analysis on such effects are not conducted.
summary: This work proposes some interpretable experiments to investigate the flow of visual information processing in LLaVA-based VLMs. The authors find 3 synthetic tasks to test whether the blurred object tokens will affect the prediction accuracy of VLMs. Secondly, the attention blocking experiments investigate whether the VLMs rely on visual information summarization to predict the correct answer. The findings are interesting and vital to VLM development.
weaknesses: 1. The contributions on the interpretability of visual information processing in VLMs are well recognized, while the contributions on how we can adopt such information processing pattern on better prompting or training for VLMs are limited. For example, if the object visual tokens are vital for the answer prediction, is it possible that simply repeat them or augment them can lead to significant performance improvement? And maybe some text prompts or soft-prompts can well utilize such object visual tokens for better model understanding?

2. The investigations focus on object only, which are only important to the general VQA, captioning, and some specific object detection tasks. Beyond object visual tokens, the OCR tokens, text rendering tokens, and knowledge-related tokens can also be your study object. For example, what is the effect of blocking the important text tokens to the tasks of TextVQA, OCRVQA, MME, POPE?

Meta-review: This paper applies several mechanistic interpretability techniques to understand how visual information is localized and processed in VLMs such as LLaVA. After rebuttal, it received scores of 6688. All the reviewers are happy about the paper, commenting that the paper is well written, with comprehensive experiments. The well-designed ablation studies and thorough analysis are helpful to understand VLM information processing. Given all the positive scores this paper has received, the AC would like to recommend acceptance of the paper.

Decision: Accept (Poster)