Visual Representations inside the Language Model
Authors: Benlin Liu
Amita Kamath
Madeleine Grunde-McLaughlin
Winson Han
Ranjay Krishna
Venue: COLM 2025 (2025)
Keywords: multimodal language model
visual representation
mechanistic interpretability
vision-language
TL;DR: Introducing a new approach to Multimodal LLM interpretability: studying intermediate visual representations in Key-Value Caches. We find they encode sufficient information for perception tasks, and can be intervened on to improve model performance.

Abstract: Despite interpretability work analyzing VIT encoders and transformer activations, we don't yet understand why Multimodal Language Models (MLMs) struggle on perception-heavy tasks. We offer an under-studied perspective by examining how popular MLMs (LLaVA-OneVision, Qwen2.5-VL, and Llama-3-LLaVA-NeXT) process their visual key-value tokens. We first study the flow of visual information through the language model, finding that image value tokens encode sufficient information to perform several perception-heavy tasks zero-shot: segmentation, semantic correspondence, temporal correspondence, and referring expression detection. We find that while the language model does augment the visual information received from the projection of input visual encodings---which we reveal correlates with overall MLM perception capability---it contains less visual information on several tasks than the equivalent visual encoder (SigLIP) that has not undergone MLM finetuning. Further, we find that the visual information corresponding to input-agnostic image key tokens in later layers of language models contains artifacts which reduce perception capability of the overall MLM. Next, we discuss controlling visual information in the language model, showing that adding a text prefix to the image input improves perception capabilities of visual representations. Finally, we reveal that if language models were able to better control their visual information, their perception would significantly improve; e.g., in 33.3% of Art Style questions in the BLINK benchmark, perception information present in the language model is not surfaced to the output! Our findings reveal insights into the role of key-value tokens in multimodal systems, paving the way for deeper mechanistic interpretability of MLMs and suggesting new directions for training their visual encoder and language model components.

Published as a conference paper at COLM 2025
Visual Representations inside the Language Model
Benlin Liu', Amita Kamath Madeleine Grunde-McLaughlin,
Winson Han Ranjay Krishnal
University of Washington UUniversity of California Los Angeles 9 Allen Institute for Al liublêcs. washington. edu
Abstract
Despite interpretability work analyzing VIT encoders and transformer ac- tivations, we don't yet understand why Multimodal Language Models (MLMs) struggle on perception-heavy tasks. We offer under-studied per- spective by examining how popular MLMs (LLaVA-OneVision) Qwen2.5- VL, and Llama-3-LLaVA- first study the flow of visual information through the language model, finding that image value tokens encode sufficient information to perform several perception-heavy tasks zero-shot: segmentation, semantic corre- spondence, temporal correspondence, and referring expression detection. We find that while the language model does augment the visual information received from the projection of input visual encodings correlates with overall MLM perception capability- contains less visual information on several tasks than the equivalent visual encoder (SigLIP) that has not undergone MLM finetuning. Further, we find that the visual in- formation corresponding to input-agnostic image key tokens in later layers of language models contains artifacts which reduce perception capability the overall MLM. Next, we discuss controlling visual information in the language model, showing that adding text prefix to the image input im- proves perception capabilities of visual representations. Finally, we reveal that if language models were able to better control their visual information, their perception would significantly improve; e-g., in 33.3% of Art Style questions in the BLINK benchmark, perception information present in the language model is not surfaced to the output! Our findings reveal insights into the role of key-value tokens in multimodal systems, paving the way for deeper mechanistic interpretability of MLMs and suggesting new directions for training their visual encoder and language model components.
process their visual key-value tokens. We
we reveal
1 Introduction
Multimodal language models (MLMs) still struggle on perception tasks (Fu et al., 2024; Tong et al., 2024b), particularly those that require reasoning over relative depth, object localization, identifying object segments, and spatial understanding (Kamath et al., 2023; Hu et al., 2024). Contemporary studies indicate that such perceptual capabilities can be improved with specialized task-designs or datasets (Ray et al., 2024; Bigverdi al., 2025). Despite empirical improvements on benchmarks with specialized designs, we lack the scientific tools dissect what causes these improvements. We conjecture that the answer to this question requires understanding how MLMs represent and reason over visual inputs.
Efforts to interpret vision-language models have traditionally focused on two main direc- tions. The first direction analyzes ViT-based encoders such as CLIP (Radford et al., 2021) and DINO (Caron et al., 2021), aiming to understand how visual features are extracted and mapped into semantic space (Dosovitskiy et al., 2020). For instance, one method iden- tifies the role of each attention head within CLIP, then reduces spurious correlations by removing specific attention heads (Gandelsman et al., 2024). The second direction examines the outputs of the transformer blocks that process visual tokens in MLMs (Liu et al., 2024; Wang et al., 2024), e.g., finding that visual representations gradually align with textual

---

Published as a conference paper at COLM 2025
2
Solmmax
Segmentation
Semantic Correspondence
Norm
90
V vddos
Norm
Nx
Attention
Norm
reshape
Referring Exp. Detection
Temporal
Correspondence
Image value tokens encode information sufficient zero-shot..
HOW many monkeys are in the imagee?
Figure 1: We study visual representations in the key-value cache within Mutimodal Lan- guage Models (MLMss, they are uninfluenced by text (due to the causal nature of cross- modal attention) and directly contribute to the MLM output. Despite MLMs struggling on perception (Fu al., 2024; Tong et al., 2024b), find that their intermediate image value tokens encode sufficient information for various zero-shot perception lasks-calling for research understand and surface the visual information already present within MLMs.
concepts across layers (Neo et al., 2024). While insightful, these two directions overlook critical feature that distinguishes MLMs from traditional ViTs: how visual information is represented and used inside the language model.
We argue that the key-value (KV) tokens corresponding to the visual inputs are a crit- ical-and understudied-lens for understanding how language models process visual information. In MLMs, an image is typically encoded by an encoder (with an optional lightweight projection) and then fed into a language model as visual tokens. Cross-modal attention draws on keys and values to determine what information to retrieve about an image. Analyzing these KV tokens can clarify how MLMs integrate visual details alongside linguistic context. Although interpretability approaches for text-only language models have studied the KV cache (Ge al., 2024), they focus on improving efficiency and do not translate over when studying visual tokens inside a language model. Moreover, the cache may reveal factors that either enhance or degrade visual fidelity in MLMs, giving us handles to improve perception-oriented tasks.
We investigate the internal KV representations of three popular and distinct models: LLaVA- VieVision 7B (Li et al., 2025a), Qwen2.5-VL 7B (Bai et al., 2025), and Llama-3-LLaVA-Nels 8B (Li et al., 2025b). We first find that the models' image value tokens encode sufficient in- formation perform zero-shot perception- segmentation, co-segmentation, referring expression segmentation, semantic correspon- dence, and temporal correspondence), and that their performance on these tasks correlates with perception performance of the overall MLM. We then use these tasks to measure perception capability of the visual representations in different layers of the MLM, shedding light on the flow of visual information through the language model. We find that, similar to semantic information in text-only language models, the segmentation capability of the visual representations gradually builds over the first two-third layers, then drops off steeply in the later layers, likely as the model begins to focus on generating the text output.
tasks (foreground segmentation, semantic
However, while the language model layers do initially augment the input visual information received from the projection of the input visual encodings, we show that on several tasks, they have less perception capability than encodings from the equivalent visual encoder (SigLIP) that has not undergone MLM finetuning, which agrees with findings from con- current work (Fu et al., 2025). Further, find that visual information corresponding to input-agnostic image key tokens in later layers of language models contains artifacts which actually reduce perception capability of the overall MLM.
2

---

Published as a conference paper at COLM 2025
Having built an understanding of the visual representations in the language model and the impact of the visual information they contain on perception capabilities of the MLM, next discuss controlling the visual information within the language model. We first introduce promising method to improve this visual information: adding a textual prefix to the image. The promptable nature of visual representations in the language model enables them to dynamically adapt to the prefix via causal attention, which we show improves their perceptual capabilities on three tasks: referring expression segmentation, semantic correspondence and domain adaptation for zero-shot semantic segmentation. Finally, we reveal that if language models were able better control the visual information they contain, their perception capabilities would significantly improve; e.g., in 33.3% of Art Style questions in the BLINK benchmark (Fu al., 2024), the correct perception information is present in the language model, but is not surfaced to the final output.
Our work lays the foundation for further studies in mechanistic interpretability of visual representations in MLMs, as well as how to leverage the same to improve model perfor- mance on perception-heavy
pars--aaricuularyy in the cases we reveal, where the visual information is already present within the model. Our findings also suggest new directions for training both the visual encoder and language model components of MLMs.
2 Preliminaries
MLMs typically consist of three main components: an image encoder, a lightweight adapter, and language decoder. The image encoder, often a ViT backbone, processes raw image inputs into a set of embeddings. The connector, usually a linear layer, then projects these embeddings into space compatible with the language model. Finally, the language decoder, typically a Transformer-basee information to generate responses. In this section, we discuss how images are processed by the visual encoder, and contrast with how they are processed by the language model.
autoregressive model, integrates both textual and visual
2.1 Image processing by the visual encoder
Each image is divided into N patches, flattened, and projected into D-dimensional embeddings. After adding positional encodings, these embeddings form a sequence: X = xx,,..,NN| where X; is the representation of the 7th patch. These patch embeddings are then passed into Multi-Head Self-Attention mechanism (Vaswani et al., 2017).
Each patch representation X; is linearly transformed into three separate vectors: Query (Q), Key (K), and Value (V). This is done using learnable weight matrices: Q XWo, XWR, V- XWy, where Wo, Wx, Wy are learned weight matrices of shape D dg and Q,K, are of shape N x d2, where dy is the dimension of each attention head.
Self-attention is then computed using the scaled dot-product attention mechanism:
((()T
Attention( (,,nttionQ,,KVV V) softmax
V.
VTve
ViT uses a multi-head attention mechanism in which h self-attention operations, called "heads", are run in parallel and concatenated before projection to the output space. Following the attention block, the token representations are passed through feedforward network (FFN), usually consisting of two-layer MLP with nonlinearity between layers. Layer normalization and residuals are applied before and after attention and FFN respectively.
Every visual token attends to every other token dense global self-attention mechanism. As ViTs compute Q K, V on the fly for every token in fully tensorized operation, they do not store representations separately for later querying.
2.2 Image processing within the language model
The key differences between image processing in MLMs versus ViT are: (1) an explicit key- value storage mechanism; (2) causal attention; and (3) cross-attention between modalities. We detail each below, along with how they relate to text generation by the language model.
3

---

Published as a conference paper at COLM 2025
Explicit key-value storage. In standard transformers, key-value (KV) cache is used to efficiently store and retrieve token embeddings during self-attention: rather than recom- puting attention scores for every token at every step, transformers store keys and values for previously seen tokens. In MLMs, this is operationalized by storing pre-computed visual information static key-value tokens in the cache, allowing text tokens to selectively retrieve them via causal cross-modal attention.
Causal attention. In autoregressive decoding, causal mask M is applied, preventing tokens from attending to future positions:
Attention( Q K, V) = softmax (Q(K) Vdx + +M)v
In MLMs, the image is generally positioned before the text in the input. Hence, causal attention ensures that text tokens only attend to previous text and visual tokens, and that visual tokens do not attend to text tokens (i.e., there is one-way querying), allowing them to be pre-computed and stored statically in the KV cache (c.f. Appendix for details).
Cross-modal attention. Due to the positioning of images before text in the input, the LLM can query visual features when generating text via causal attention, i.e,, they can "look up" relevant visual information: Attention Qtext, Kvision Vvision)- Vision tokens do not influence text generation directly, i.e. they do not modify the text keys or values (similar to how text token does not modify successive text tokens in the input).
Group query attention. In our experiments, we use models that adopt the Group Query Attention (GQA) (Ainslie et al., 2023) mechanism. Unlike standard multi-head attention, where each attention head has its own independent query, key, and value projections, GQA shares keys and values across heads while maintaining distinct queries. Specifically, all attention heads in a group share the same key and value matrices but compute attention independently using their own query projections (c.f. Appendix for details):
head;
Attention Q;, Kshared, Vshared)
Text generation by the language model. The weighted sum of values omitted by the attention mechanism is passed through the remainder of the transformer block to the next layer, successively until the final layer, when it is normalized, passed through a linear layer and softmax, and generates a prediction probability distribution over the vocabulary, predicting each token of the output auto-regressively.
To understand visual representations within the language model, we study the basee keys and values in the attention mechanism (c.f. Figure 1). First, due to causal attention and the typical position of image before text, these key-value pairs are unaffected by the text input and thus form purely visual representation. Second, once computed, they are stored and fixed, allowing all text tokens to attend to image keys and retrieve values that directly contribute to the output. As such, they serve a proxy for the input image, remaining accessible throughout any downstream language task. Understanding what is encoded in these representations ticoossttaiiosssei is therefore essential for improving perception in MLLMs.
3 Visual Information within the Language Model
In this section, we study how visual information flows through the language model, as well as its perceptual capability as measured by six perception-heavy probing tasks. We discuss how this capability compares to popular visual encoders, and that it correlates with overall MLM performance on perception-heavy tasks. We then reveal that certain visual information in the language model actually degrades MLM perception.
3.1 Flow of visual information through the language model
Despite the use of strong vision encoders (Zhai et al., 2023), MLMs continue to perform poorly on vision-centrrc benchmarks (Fu et al., 2024; Tong et al., 2024b). This suggests that the language model may not correctly process visual information passed into it by the vision
4

---

Published as a conference paper at COLM 2025
Segmentation
Correspondence
Pascal-5i (mIOU)
MSRC
ImageNet-S (mlOU)
RefCoco (mIOU)
SPair7IK (PCK)
DAVIS
(Jm)
(J&FF))
(3)
(Fmm)
Visual Encoder (pust-proj) LM Image Value (max)
58.4
51.1
59.6
36.9
57.4
56.8
58.0
26.2
61.6
54.6
64.8
46.1
65.8
62.6
67.2
SigLIP without MLM FT
27.3
62.0
59.1
61.7
41.6
59.8
58.9
60.7
Table 1: Performance of different visual representations on segmentation and correspon- dence probing tasks. The highest-performing image value in the LM outperforms the input visual representation from the visual encoder (post-projection) but falls short of SigLIP without MLM finetuning on the first three segmentation tasks.
encoder. We study the flow of visual information through the language model by probing the image value tokens! in the KV cache of each layer of the language model. We discuss LLaVA-One Vision 7B in this section, and equivalent experiments for Qwen2.5-VL. 7B and Llama-3-LLaVA-NeXT 8B in the Appendix. Our observations are consistent across models.
Why values? As discussed in Section 2.2, image value tokens serve as a proxy for the input image at each layer within the language model. Further, individual image values contribute to MLM language generation by aggregating spatially distributed information, per the position encoding added to the corresponding image key. Therefore, image values should encode well-localized semantics of the image, while maintaining cross-modal alignment.
Six probing tasks. To assess whether image values contain aforementioned visual informa- tion, we probe them on tasks requiring the same. We discuss below each task alongside its experimental setup, where all tasks are performed zero-shot unless otherwise specified.
Few-shot foreground segmentation. We evaluate on Pascal-5i (split 0) (Shaban et al., 2017). We train a linear classifier on top of each image value to perform 5-shot binary segmentation.
Co-segmentation. We evaluate on MSRC (Shotton et al., 2006), which contains sets of images across which the model must segment a similar object. For each image value token, we follow Amir et al. (2022) and perform spatial clustering to segment objects across images.
Semantic segmentation. We evaluate on ImageNNet-S (Gao et al., 2022). We project the text of the class name into the representational space of the image value tokens using the MLM (independent of the image), then choose the final text token as the class representation. We select image values with highest dot product with this text token as the segmented region.
Referring expression segmentation. We evaluate on RefCOcO (Kazemzadeh et al., 2014), which requires grounding language to an object in the image. We follow the same method as semantic segmentation, but with text of the referring expression rather than the class name.
Semantic correspondence. We evaluate on SPair71K (Min et al., 2019), in which models must establish semantic correspondences between keypoints annotated on pairs of images. We find nearest neighbors between images for each image value corresponding to keypoint.
Temporal correspondence. We evaluate on DAVIS (Pont-Tuset et al., 2017), in which models propagate the mask of an object across video frames. We follow the same method as for semantic correspondence.
Image value tokens are capable of perception-heavy tasks. We evaluate each image value token in the language model (i.e,, across all heads and layers), and report the highest perfor- mance among them in Table 1 as "LM Image Value (max)". These visual representations show the capability to perform wide variety of perception tasks, from different types of segmentation to different types of correspondence across images.
Visual information is refined in the language model. We compare the results of the highest- performing image value token in the language model to the performance of the input visual representation received by the language model, i.e., the representation produced by
We use "image (key, value} token" interchangeably with "image (key, value)".
5

---

Published as a conference paper at COLM 2025
Testayen/laalmmmm Performanc Theers Segmentat Layers
PerlayesMammmmm Nerfermnnce Stare Comespendecc Aeroas Lapan
Figure 2: Performance of each image value (bottom), and maximum per-layer image value performance (top). On segmentation tasks (left), visual information builds gradually in the first two-thirds of the language model (LM), then drops steeply. On correspondence tasks (right), visual information builds sharply after the first third of the LM, then drops steadily (note the scale of the Y-axis). The LM builds upon the input visual representation (in red).
72.5
LLAVA OV -728
Visual Representation
SPair-71K PCK
DAVIS
70.0
J&Fm
Im
Fm
MLM Downstrean
Percegtion Perf.
67.5
LLAVA-OV -7B
Qwen 2.5 -7B
CLIP
41.4
62.5
60.6
64.4
65.0
DINO
36.7
71.4
67.9
74.9
62.5
Qwen-2
LM Image Value (max)
46.1
65.8
62.6
67.2
60
LLAVA- OV -0.78
57.5
Table 2: Performance of visual representations on correspondence probing tasks. The highest- performing image value in the LM compares fa- vorably to strong vision encoders.
55.0
47.5
50.0
52.5
60.0
Segmentation Perf. of Image Values
Figure 3: Segmentation performance of image values correlates with MLM perception downstream tasks.
the projection layer after the visual encoder. The outperforms the input visual representation across all six probing tasks. This shows that the language model refines the visual information it receives, building upon the information contained by the visual representations.
image value token
Visual information builds, then drops, across language model layers. We now study how visual information evolves across the layers of the language model. Figure 2 (bottom) depicts the average segmentation and correspondence performance of the image value tokens corresponding to each head in each layer of the language model. Figure 2 (top) tracks the maximum performance per layer. For the segmentation task, the early layers (1-15) gradually refine visual information, which peaks at layers 14-15, after which drops off steeply, likely as later layers of the model begin to focus on generating the text output. This is similar to the processing semantic information by language models (Tenney et al., 2019). For the correspondence lask, the information is initially retained, then refined in layers 7-10, after which drops off steadily. This is likely due to the fact that the correspondence task abstracts away pixel appearance, which is needed for segmentation and for most multimodal tasks. c.f. Appendix for per-task observations.
Visual information of the MLM visual encoder is less than that of the equivalent visual encoder LSigLIP) without MLM finetuning. LLaVA-OneVision 7B uses SigLIP v1-384 as its visual encoder, finetuning during multimodal training. We compare the performance of the original SigLIP v1-384 model with the finetuned visual encoder in LLaVA-One Visioon ppost-projection). to determine how the visual information captured by the visual encoder changes during the MLM finetuning process. As shown in Table 1, we find that the original SigLIP model actually outperforms the finetuned visual encoder across all probing tasks. Further, while the language model refines the visual information it receives from the fine- tuned visual encoder, it underperforms the original SigLIP model on 3 of 6 probing tasks that the
(foreground segmentation, co-segmentation, and semantic segmentation)-hintee language model is not always able to compensate for the lost information. This agrees with concurrent observations that MLMs can under-perfomn visual encoders (Fu et al., 2025).

---

Published as a conference paper at COLM 2025
Images
Feature maps of keys
Layers
Figure PCA visualization of all image keys in LLaVA-OneVision 7B for two COCO images. The input-agnostic image keys (highlighted) alone are nearly constant across images.
Visual information in the language model compares favorably to that of other visual encoders. In Table 2, we compare the results of the the language model in LLaVA-OneVision 7B to the performance of CLIP (Radford et al., 2021) and DINO (Caron et al., 2021) visual representations on our correspondence probing tasks. We find that the highest-performing image value outperforms both CLIP and DINO on semantic correspondence, and outperforms CLIP on temporal correspondence (but not DINO). This can be partially attributed the benefits of cross-modality of the image value, and partially attributed to the training procedure of LLaVA-One Vision, which includes multi-image and video data-similar to the pairs of images in correspondence tasks.
image value in
Visual information in the language model correlates with downstream perception perfor- mance of the MLM. We assess MLMs' end-to-end perception reasoning by evaluating them on perception-heavy tasks in CV-Bench (Tong et al., 2024b) and BLINK (Fu et al., 2024), and compare their performance that of their highest-performing image value tokens averaged across the segmentation probing tasks. As shown in Figure 3, we observe that MLMs of different sizes and versions exhibit the same trend: the stronger the perception capability of their image value tokens, the better the MLMs themselves perform on downstream perception-heavy tasks. This correlation highlights the importance of visual information within the language model. Further details about this experiment are in the Appendix.
3.2 Certain visual information in the language model degrades MLM perception
Having studied the perception capability of visual representations in the language model, we now ask: why do MLMs still perform poorly on perception-heavy tasks? We show, via intervention study, that certain visual information in the language model actually degrades perception capability of the MLM. We focus on LLaVA-One Vision 7B in this section, and discuss equivalent experiments for Qwen2.5-VL 7B in the Appendix. Our observations are consistent across models.
Our study inspired by Darcet al. (2024), who reveal that in ViTs, tokens in middle layers corresponding to background patches tend to encode input-agnostic artifacts rather than useful information about the image. We investigate whether similar phenomenon occurs in MLMss by identifying input-agnostic visual representations in the language model, and determining whether they, too, encode artifacts instead of helpful visual information.
Input-agnostic image key tokens dominate in early layers of the language model, and re-emerge in late laters. We sample 1000 images from MSCOCO (Lin et al., 2014) and obtain intermediate visual representations for each sample from LLaVA-OneVision 7B. We compute the variance of image key tokens in each head and layer across the samples. We find that the variance forms bimodal distribution, which we split with threshold to define input-agnostic keys (details in Appendix). As shown in Figure 4, we find that when setting this threshold, input-agnostic keys dominate in earlier layers of the language model, but, critically, re-emerge in later layers. We test whether these input-agnostic keys in later layers too, contain input-agnostic artifacts in ViTs with an intervention study.
7

---

Published as a conference paper at COLM 2025
Blocking visual information corresponding to input- agnostic image key tokens in later layers improves MLM perception. We conduct an interventional study, blocking text token queries to input-agnostic image keys in the last 10 layers via attention knockout (Geva et al., 2023). Notably, we do not block text keys that belong the same attention head as these input- agnostic image keys, ensuring a focus on understand- ing the impact of visual representations on model output. We evaluate on two benchmarks: POPE (Li
Model
POPE
MME
LLVVA-OneVision 7B
78.1
173.3
intervention
81.2
179.0
Table 3: Our intervention reveals im- age keys that contain artifacts, as blocking them improves MLM per- formance on POPÉ and MME.
et al., 2023), and subsets of MME (Fu et al., 2023) which together address hallucination and general perception capabilities. As shown in Table 3, we find that blocking image-agnostic image keys (and thus, image values) in later layers improves model performance on both benchmarks. In the Appendix, we show that blocking these keys in early layers significantly reduces model performance, and discuss experimental controls. By showing that removing the influence of these visual representations on the language output improves the MLM per- formance, we prove that image values corresponding to input-agnostic image keys in later layers of the language model likely encode artifacts rather than helpful visual information.
We posit that input-agnostic image keys in earlier layers play an essential role in aggregating contextual information consistent across different inputs, whereas middle-to-later layers of the model gradually shift focus toward understanding semantics (Neo et al., 2024), similar to CNNs (Krizhevsky al., 2012) and pure-text language models (Tenney et al., 2019). Hence, the input-agnostic image keys in later layers of the model are likely to encode artifacts, similar to ViTs (Darcet et al., 2024). Our findings shed light on visual information in the language model that degrades MLM perception, calling for further research in identifying why this phenomenon occurs, and how to mitigate the same.
4 Controlling Visual Information in the Language Model
Thus far, we have discussed how visual information flows through the language model, and how it impacts MLM perception performance positively and negatively. In this section, discuss controlling this visual information: leveraging the language model mechanism of causal attention to improve visual information via textual prefixing, and then revealing the significant scope for improvement in MLM perception if language models could better control their visual information.
4.1 Improving visual information by leveraging causal attention in language models
Visual representations in the language model have an interesting capability thanks to their cross-modal nature: they can be made promptable by prefixing textual prompts to the visual input. Causal cross-modal attention then allows the model to dynamically adapt its visual representations the fly to better align with specific tasks. We study this in the context of three tasks: referring expression segmentation, semantic correspondence, and domain adaptation for semantic segmentation. We present results in Table 4.
Prefixing improves referring expression segmentation. We evaluate image values from LLaVA-OneVisio Vision 7B RefCOCO. Without text prefixing, the experiment is as described in Section 3.1. When textual prefix is added to the image input (here, the image caption from COCO (Chen et al., 2015)), the image values achieve modestly improved segmentation results. This shows that the visual representations were able to incorporate the text caption- which contains information about the image, and may contain information about the object to segment-to improve their textual grounding and segmentation capabilities.
Prefixing improves semantic correspondence. evaluate image value tokens from LLaVA- OneVision 7B on SPair71K. Without text prefixing, the experiment is as described in Section 3.1. The textual prefix we add for this task is the class name of the object in the image. As with referring expression segmentation, the semantic correspondence performance improves slightly with prefixing, showing that the visual representations incorporate textual information from the prefix to improve textual grounding and segmentation capabilities.
8

---

Published as a conference paper at COLM 2025
BDD-10K day+street
BDD-10K night-highway
RefCOCO
SPair7IK
No prefixing Prefixing
64.8
46.1
69.7
63.4
65.3
46.5
70.4
68.2
Prefixing-random Prefixing-imcorrect
64.4
45.8
68.9
63.2
44.9
68.5
62.1
Table 4: Performance of image values segmentation and correspondence tasks improves with prefixing of a relevant text prompt (top), and not a random or incorrect one (bottom).
Prefixing improves domain adaptation for zero-shot semantic segmentation. We evaluate 7B on BDD-10K (Yu et al., 2020) on two domains: city
image values from street during daytime, and highway during nighttime. Without prefixing, the experiment is similar to semantic segmentation described in Section 3.1. By prefixing the image with text specifying the domain ("The image is taken at (daytime, nighttime) and it is from {city street view, highway) "), performance improves on both domains, but more significantly on the more challenging one. This shows that the visual representations successfully incorporate the text information identifying the domains, then adapt on the fly to each.
Controls. We run two controls for this experiment, to ensure the content of the text prefix is what improves the visual representation quality, not the addition of a text prefix itself. For all tasks, we prefix a random string of text (" " rustic wooden table filled with freshly baked croissants, dripping honey, and a steaming pot of Earl Grey tea beside bowl of ripe figs." ") to the image and calculate performance ("Prefixing-random "). Across all tasks, this does not improve performance compared to no prefixing, and rather lowers it slightly. For the semantic correspondence task, we add a second control wherein we prefix the name of the incorrect class "'Prefixing-incorredd prefix the name of the incorrect domain ("The image is taken at daytime and it is from city street view" for an image of night+highway, and vice versa). These numbers are even lower than the random prefix for both tasks. These results collectively show that it is indeed the prefix content that improves visual representation quality.
We do the same for domain adaptation, in which we
These results suggest that image value tokens are not only effective for visual understand- ing, but are also inherently adaptable to linguistic conditioning. Further, they set forth promising method to improve visual information by leveraging mechanisms already present in the language model. Our findings call for further research improving control of visual information to improve perception capabilities within the language model.
4.2
Better control of visual information in the language model could improve MLM perception significantly
We have shown in Section 3.1 that visual representations in the language model are capable of perception-heave tasks-however, our motivation to study them was rooted in the poor performance of MLMs on end-to-end perception reasoning (Fu et al., 2024; Tong et al., 2024b). This hints that MLMs are not optimally utilizing the perceptual capabilities of the visual representations in the language model. We study whether better control of the visual information within the language model could improve MLM perception: specifically, how much perception capability is present in the language model but not currently being surfaced to the final text output?
Three paired tasks. Perception-heavy tasks can be framed as visual question answering (VQA); they can also be evaluated directly from the visual representations, i.e., the image value tokens (as in Section 3.1). We refer to these as paired tasks. We posit that the image value tokens being able to perform a task on an example is a sufficient (if not necessary, in case the language model can compensate with its reasoning capabilities) condition for the MLM answering the VQA corresponding to the example correctly. We perform this evaluation on three such paired tasks: semantic correspondence, art style and visual similarity, discussed below. We calculate the fraction of instances where the MLM predicts the VQA answer incorrectly, despite its image value tokens predicting the answer correctly for the same input.

---

Published as a conference paper at COLM 2025
Semantic Correspondence (%)
Art Style (%)
Visual Similarity (%)
MLM V
36
53.8
79.3
Value v MLM X
7.5
33.3
18.5
Value X MLM V
2.5
6.0
6.7
(MLM Value) V
43.5
87.1
97.8
Table 5: Instances in 3 paired tasks with different correctness of the MLM and its image values ("Value"), MLMs under-utilize visual information contained in their image values.
Semantic correspondence. We frame semantic correspondence in SPair71K as a VQA task, as in BLINK (for which image coordinates are not public), generating 200 samples. We then evaluate image value tokens on semantic correspondence as an image pair, as in Section 3.1.
Art style. We use the Art Style subset of the BLINK benchmark (Fu et al., 2024), which consists of 234 examples that require selecting the image option that shares its art style with the image in the question. We perform average pooling of image value tokens in each image, selecting the image option with highest cosine similarity with the image in the question.
Visual similarity. We use the Visual Similarity subset of BLINK (Fu et al., 2024), which consists of 271 examples that require selecting the image option that most visually similar to the image in the question. We follow the same method as for the art style task.
Language models significantly under-utilize the visual information they contain. As shown in Table 5, language models under-utilize visual information present in the model on all four tasks. the case the art style task, this is particularly egregious, with performance increasing by 33.3% if the model had successfully surfaced the perception capability of its visual representations. Across all tasks, the percentage of instances where the image value token was incorrect but the MLM prediction was correct is very low (well below random chance, which is 25% for semantic correspondence and 50% for art style and visual similarity), showing the importance of visual representations for these tasks.
Our finding reveals that language models under-utilize their visual information, agreeing with the concurrent Fu al. (2025) and underscoring the need for further research in better controlling visual information within the language model to improve MLM perception.
5 Discussion and Future Work
Our investigation sheds light on an as-yet under-explored component of MLMs: the visual representations stored in the key-value cache and attended to during text generation; in contrast to existing work, which has studied the transformer block outputs within MLMs (Neo et al., 2024; Zhang et al., 2024). We find that: (1) language models refine visual information in earlier layers, then lose it in later layers, with the highest performance being less than that of the equivalent visual encoder that has not undergone ML. finetuning on multiple tasks; (2) visual information corresponding to input-agnostic image key tokens in later layers of the language model actually degrades MLM perception; (3) causal attention of language models can be leveraged to improve their visual information via text prefixing; and (4) better control of visual information in the language model could significantly improve MLM perception. We contextualize our findings with relevant work in the Appendix.
equivalen
Our work calls for research in three main directions: first, improving finetuning of visual encoders like SigLIP in multimodal training so that the visual information provided to the language model after the projection layer not inferior that of the original visual encoder without multimodal finetuning; second, improvement of visual representations within the language model by leveraging traditional text-only methods such as prompting; third, improving language model control of existing visual information, potentially by preventing the occurrence of the artifacts we find in later language model layers, or by using additional tokens (Bigverdi et al., 2025). Our work highlights the importance of interpreting image processing in MLMs by studying visual representations in language models, and shows promise for the future of perception reasoning in MLMs.
10

---

Published as a conference paper at COLM 2025
References
Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint ar xrXi::2305.12255, 2023.
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. ECCVW What is Motion For?, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar XrX:::230 12966, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL hitpps///are /ttp:v///axvvv.rrrr/ orw.li.orr/aaaae orggobs////// 10002.11922 13923.
Mahtab Bigverdi, Zelun Luo, Cheng-Yu Yu Hsieh, Ethan Shen, Dongping Chen, Linda G Shapiro, and Ranjay Krishna. Perception tokens enhance visual reasoning in multimodal language models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3836-3845, 2025.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015. URL https:://aaiiiii
arrg/abs/150d 00325.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3);, 2023.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Tweelfth International Conference on Learning Representations, 2024. URL https:// wwwp/neeteeeeneeeeeeee nett/forummid=22n33LLL11
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 245, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. Xro preprint ar arXio:2010.11929, 2020.
AI Alignment Forum. Interpreting gpt: the logit lens - ai alignment fo- rum, 2024.
URL https://w..aliggeettfrrrr wwtoorooooor////ooooo.oooooooerr wowgposts///ceeccoeoooooooo http///ss/ccs///////// postts//AcRBBBDDdaav66rrr interpreting-en soppt-the-logi--ee
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint ar arXiv:2306.1333 133944 2023.
Stephanie Fu, Tyler Bonnen, Devin Guillory, and Trevor Darrell. Hidden in plain sight: Vims overlook their visual representations. ar preprint ar arXie:2506.08008 2025.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pp. 148-166 Springer, 2024.
11

---

Published as a conference paper at COLM 2025
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting CLIP's image repre- sentation via text-based decomposition. In The Twelfth International Conference on Learning Representations, 2024. URL https: http///v/ppeeeeeee//////ee/ nett/foormm1dd5CC9s5suul
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025. URL https://
openreview.ner
Shanghua Gao, Zhong- Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. 2022.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelffh International Conference on Learning Representations, 2024. URL forum??d==NNrFoDPMyy.
net/
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023. URL F1G7y94K82.
Yash Goyal, Tejas Khot, Douglas Summers -tay,, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. Llama scope: Extracting millions of features from Ilama-3.1-8h with sparse autoencoders. ar preprint ar Xrto:2241..0026,, 2024.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint ar xrii:2246.009400 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, and Xuming Hu. Mmneuron: Discovering neuron-level -opeif- interpretation in multimodal large language model. Xiv preprint wwxx::2406.11999 2024.
Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. ar Xiv preprint drXiv:2410.02762, 2024.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, 2023.
What's "up" with vision-language models?
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg, Referitgame: Refer- ring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25, 2012.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-onevision: Easy visual task transfer. Transactions on Machine Learning Research, 2025a. ISSN 2835-8856. URL https://openreve wttpre/ww..neeoorr//////aaaa
12

---

Published as a conference paper at COLM 2025
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun MA, and Chun- yuan Li. -LaVA-neXT-mttrreavee Tackling multi-image, video, and 3d in large multimodal models. In The Thirteenth International Conference Learning Representations, 2025b. URL https://openrent ew.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluat- ing object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 292-305, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/ /v2l w0w.//222mmmmmmmii..22.... //aclanthology.orgg// hwwphholloo..rgg/////0 httpg//.........eeeeeee emnlp-main.28
URL https:
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26300 2024.
Yiming Liu, Yuhui Zhang, and Serena Yeung-Levy. Mechanistic interpretability meets vision language models: Insights and limitations. In The Fourth Blogpost Track at ICLR 2025, 2025.
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. arviv preprint arXiv:1908. 10543, 2019.
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. arXiv preprint arXie:2410.07149, 2024.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- and Luc Van Gool. The 2017 davis challenge on video object segmentation. arxiv preprint ar xxXi::704400665, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language su- pervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Interna- tional Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 1748-8765 PMLR, 2021. URL http://proceedings httpa/..........///eee.ee wttpss/v////e//eeeeepeppp v333///ddfodo211111 html.
Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Anirud- dha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat Spatial aptitude training for multimodal language models. arXiv preprint ar arXi::2412.07755, 2024.
Amireeza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In British Machine Vision Conference (BMVC), 2017.
Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost: Joint ap- pearance, shape and context modeling for multi-class object recognition and segmentation. In Computer Visi 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part 9, pp. 1-15. Springer, 2006.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunning- ham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https: https//////rsssffeerrrrriiiiiiie tttp://////iiiiiooooootttatttttt html. html.
13

---

Published as a conference paper at COLM 2025
lan Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Annual Meefing of the Association for Computational Linguistics, 2019. URL https://appp 155992064.
semanticscholar.c wwtpanttissccoola.orrrrrrruuuuue
Shengbang Tong, Ellis L Brown II, Penghao Wu, Sanghyun Woo, ADITHYA JAIRAM IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: http:///ppppevvieeeeee net/forum?id= Vi8AepAXGy.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pp. 9568-9578, 2024b.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXio, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. ar xrv preprint ar Xix::22409 1it.::299...111,,, 2024.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636 2665,264,,p0 2020.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pp. 11975-11986, 2023.
Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis-Philippe Morency, Katia P Sycara, and Yaqi Xie. Self-correcting decoding with generative feedback for mitigating hallucinations in large vision-language models. In The Thirteenth International Conference on Learning Representations, 2025.
Kaichen Zhang, Yifei Shen, Bo Li, and Ziwei Liu. Large multi-modal models can interpret features in large multi-modal models. Xiv preprint ar XrXi::2411.1490 2024.
14

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Related work
We discuss three fields of research most relevant to our work.
Multi-modal language models. MLMs (Liu et al., 2024; Bai et al., 2023) achieve an end- to-end percepiionTeasoningg system by integrating vision encoders (Radford et al., 2021) with large language models (LLMs) (Chiang et al., 2023; Touvron et al., 2023), leading to significant breakthroughs various vision-language tasks such as image captioning (Lin et al., 2014) and visual question answering (VQA) (Hudson & Manning, 2019; Goyal et al., 2017). However, recent studies have pointed out that improvements in VQA tasks often stem from the strong language priors of the language model rather than precise perception of the input image (Fu et al., 2024; Tong et al., 2024b). As a result, MLMs perform poorly on vision-centric benchmarks. Most existing studies on perception focus only the impact of the vision encoder. In contrast, we investigate how the language decoder influences perception. Concurrent to our work, Fu et al. (2025) reveal that MLMs under-utilize visual information captured by their visual encoders and rely on language priors.
Mechanistic interpretability. Although optimizing large language models through next- token prediction has led to many emergent behaviors and significant improvements in various language and vision-language tasks, their internal mechanisms remain largely unknown due to their complexity. This lack of understanding also limits our insights into issues like hallucination. Mechanistic interpretability aims to uncover the internal workings of language models, such as information flow, by using tools like the logit lens (Forum, 2024) and sparse autoencoders (SAEs). However, most existing studies focus on understanding purely anguage-based models (Templeton et al., 2024; He et al., 2024; Gao et al., 2025). Recently, some works have begun exploring the mechanistic interpretability of multimodal language models (Liu et al., 2025), primarily investigating cross- mods- information fusion- iaiion-secifficlllyy whether the language modality effectively leverages visual information through the attention mechanism (Huo et al., 2024; Jiang et al., 2024; Zhang et al., 2024). However, language models not only involve cross-modal fusion but also process visual information further. While a few studies have explored how language models process visual inputs (Neo et al., 2024), they have not examined the visual representations in the KV cache. Our work focuses on understanding the visual representation stored in the language model and how these representations influence the generated language responses.
Visual representation learning. Visual representation learning one of the core problems in computer vision, where a visual encoder transforms raw pixels into representation that can be applied to various downstream tasks (Dosovitskiy representation is typically evaluated using probing tasks to assess the model's perception capabilities, on tasks such as ImageNet classification (Deng et al., 2009). In MLMs, commonly used visual encoders include CLIP (Radford et al., 2021) and DINO (Caron et al., 2021). However, in MLMs, the visual encoder is not the only component that processes visual inputs. The cross-modal decoder that follows the encoder not only handles language tokens but also processes visual tokens. Therefore, it is necessary to use probing tasks to analyze the intermediate visual representations produced by the cross-modal decoder.
al., 2020). The quality of this
A.2 Group Query Attention
In our experiments, we use models that adopt the Group Query Attention (GQA) (Ainslie et al., 2023) mechanism. Unlike standard multi-head attention, where each attention head has its own independent query, key, and value projections, GQA shares keys and values across heads while maintaining distinct queries. Specifically, all attention heads in a group share the same key and value matrices but compute attention independently using their own query projections:
head; =
Attention (Q; Kshared, Vshared).
This design has two main advantages in our setting. First, most recent state-of-the oft multimodal language models (MLMs) adopt GQA, making our analysis directly applicable
15

---

Published as a conference paper at COLM 2025
to widely used architectures. Second, by reducing the number of key-value pairs stored in the attention mechanism, GQA greatly simplifies the analysis of attention heads and visual representations, making it more tractable for interpretability research.
In practice, the models we study all adopt this GQA-based architecture. Specifically, both LLaVA-One-Wision and Qwen2.5-VL 5-VL-7B use 28 queries per layer that share 4 key-value pairs, while
8B employs 32 queries per layer sharing key-value pairs.
A.3 Per-Task Performance of Image Value Tokens Across Layers
We now discuss the performance of image value tokens in LLaVA-OneVision 7B on each of the six probing tasks evaluate.
On foreground segmentation (Pascal-5i), as shown in Figure 5, the segmentation perfor- mance relatively high in very early layers, then drops gradually over the first 10 layers. It builds again over the next 7 layers, then falls in the remaining layers.
On co-segmentation (MSRC), as shown in Figure 6, the trend in segmentation performance similar to foreground segmentation: being quite high in early layers, dipping and rising around layer 17, then falling steeply in later model layers.
On semantic segmentation (ImageNet-S), as shown in Figure 7, the segmentation perfor- mance continuously builds over the first 18 layers, staying above the performance of the input visual representation (post-projection). It fluctuates slightly in layer layers.
On referring expression segmentation (RefCOCO), shown in Figure 8, the segmentation performance follows a similar trend as semantic segmentation, building gradually till layer 17 and then dropping gradually, remaining above the input visual representation (post-projection) performance.
On semantic correspondence (SPair71K), as shown in Figure 9, the correspondence perfor- mance is about the same as that of the input visual representation (post-projection) initially, then sees a sharp increase layers 7-10 and sharp decrease in layers 10-13, then steady decline in the succeeding layers of the model. The variance in image value performance much greater than that of the segmentation tasks (note the Y axis of the line plot). An extremely similar trend is shown on temporal correspondence (DAVIS), as shown in Figure 10. Both correspondence tasks need to abstract away pixel appearance information, which may explain the steep decline in performance of later language model layers.
Per-Layer-Maxiul Performance of Value Tokens on Pascal-Si Across Layers
Inout
22
Performance Value Tokens on Pascal-5i Layers
Seg. Perf
Input Vis. Rep.
22
20
Layer
Figure 5: Foreground segmentation performance on Pascal-5i for LLaVA-OneVissin 7B. (top) Maximum-per-layee performance. (bottom) Performance per head.
16

---

Published as a conference paper at COLM 2025
Per-Layer-Maxiiumm Performance of Value Takens on MSRC Across Layers
Pert.
60
oout Vis
Rep Pert
5558
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Performance of Value Takens on MSRC Across Layers
Seg. Part.
60
Head
Input Vis.
58 Rep.
56
10
13
15 Layer
14
16
18
19
20
22
25
26
28
Figure 6: Co-segmentation performance MSRC for LLa LLaVA-OneVision 7B. (top) Maximum- per-layer performance. (bottom) Performance per head.
Per-Layer-Mxximu Performance of Value Tokens on ImageNet-S Across Layers
54.0
53.0
G55.5
01.00
VIS. Rep. Pert
$1.0
Performance Value Tokens ImageNet-S Across Layers
Seg.
Head
53
52
10
11
12
13
15
16
18
19
20
21
23
24
25
26
27
26
Layer
Figure 7: Semantic segmentation performance ImageNet-S for LLaVA-OneVision 7B. (top) Maximum-per-laye performance. (bottom) Performance per head.
Per-Layer-Maximum Performance of Value Tokens on RefCOCO Across Layers
64
Seg:10
62
61
3
9
10
11
12
13
15
15
17
18
19
20
21
22
23
24
25
26
27
28
Performance of Value Tokens on RefCOCO Across Layers
Seg. Parf.
Head
63
62
15 Layer
21
24
27
Figure 8: Referring expression segmentation performance RefCOCO for LLaVA-
OneVision 7B. (top) Maximum-perrlayer performance. (bottom) Performance per head.
17

---

Published as a conference paper at COLM 2025
Per-Layer-Maximum Performance Value Tokens on SPair71K Layers
Corr. Pert.
30
Performance of Value Tokens on SPair71K Across Layers
Corr.
45
40
Inpur Vis
35 Rep. Perf.
13
16 Layer
22
25
28
Figure Correspondence performance on SPair7IK for LLaVA-OneVision 7B. (top) Maximum-per-layer performance. (bottom) Performance per head.
Maximum Performance Per-LayerMaximum
Tokens on DAVIS Across Layers
Cort. Pert.
Inout Rep.
Performance of Value Tokens DAVIS Across Layers
Corr.
60
Rep. Pert:
50
28
Layer
Figure 10: Correspondence performance on DAVIS for -laVA-OneVision 7B. (top) Maximum-per-layer perforancce (bottom) Performance per head.
18

---

Published as a conference paper at COLM 2025
A.4 Segmentation Results for All Models
We report segmentation results of all three MLMs in Table 6. Qwen2.5-VL 7B performs the best, closely followed by LLaVA-OneVision 7B. Llama3-LLaVA-NeXT 8B performs less well, as its vision and text components both exhibit lower performance than those of the other two models.
We also show the performance of image value tokens averaged across the four segmentation tasks of each of these models: ALaVA-OneVisiie 7B (Figure 11), Qwen2.5-VL 7B (Figure 12), and Llama-3-LLaVA-Ner the performance of their value tokens on segmentation increases gradually over the first two-third layers, then drops steeply in the later layers.
8B (Figure 13). All three models exhibit similar trend:
Visual representations
Pascal-5i
MSRC
ImageNet-S
RefCOCO
Average
LLAMAS-LLaVA-NeTTT 8B
21.6
54.4
51.4
62.6
47.5
LLaVA-OneVision 7B Qwen-2.5VL.78
26.2
61.6
54.6
64.8
51.8
63.9
55.9
53.5
Table 6: Segmentation results of the best value in MLMs, split across 4 tasks. Results are in mIOU but for MSRC co-segmentation, which reported in Jm-
LLAVA-OV Per-Layer-Maximmm Perf. of Value Tokens on Segmentation Across Layers
51.5
0 51.0
550.0
49.0
48.5
48.0
LLAVA-OV Perf. of Value Tokens Segmentation Across Layers
Aug. Seg.
Head
49
43
2
9
10
11
12
13
15 Layer
14
16
17
18
19
20
21
22
23
24
25
26
27
28
Figure 11: Average segmentation across tasks for LLLVAA-OeVVise 7B. (top) Maximum- per-layer performance. (bottom) Performance per head.
19

---

Published as a conference paper at COLM 2025
Qwen2.5 Ber-Layer-Maximaee Perf. of Value Tokens on Segmentation Across Layers
Avg. Seg. Perf.
52
Qwen2.5 Perf. Value Tokens Segmentation Across Layers
Avg. Sog.
48
46
9
11
12
14
15 Layer
17
18
20
23
24
25
26
Figure 12: Average segmentation across 4 tasks for Qwen2.5-VL. (top) Maximum-per-aaye performance. (bottom)
LLAMA3 Mar-LayerMaxxium Perf. Value Tokens on Segmentation Across Layers
Avg. Seg. Perf. 43
5
24
30
LLAMA3 Perf. of Value Tokens Segmentation Across Layers
Avg. Sey.
46
Head
44
10
12
13
14
15
17 Layer
16
18
19
20
21
22
23
24
25
26
28
29
30
31
32
Figure 13: Average segmentation across 4 tasks for LLL.M-3-LLaVA-NNXXT (top) Maximum- per-layer performance. (bottom) Performance per head.
20

---

Published as a conference paper at COLM 2025
A.5
Correlation of Perception Capability of Image Value Tokens with that of the overall MLM: Additional Details
We assess MLMs' end-to-end perception reasoning by evaluating them on the perception- heavy vision-language reasoning tasks in CV-Bench (Tong et al., 2024a) and BLINK (Fu et al., 2024). This allows us to investigate a key question: Does improved perception of visual representations in the language model correlate with better perception reasoning of the MLM itself? If so, this would motivate the need to further study visual representations within the language model.
Experiment. To calculate MLMs' end-to-end perception reasoning capability, we evaluate on two vision-centric VQA benchmarks: CV-Bench, and the perception-heavy tasks in BLINK. CV-Bench consists of 2,637 test samples and assesses MLMs' 2D and 3D perception capabilities in a single image, covering four major categories: counting, relations, depth estimation, and distance estimation. BLINK, on the other hand, contains single-image and multi-image questions. Since certain question types in BLINK, such as art style, are less relevant to perception, we select subset of BLINK consisting of four perception- related tasks: counting, object localization, relative depth, and spatial relations, totaling 1,021 test samples. We evaluate MLMs across these 3,658 test samples to estimate their perception reasoning ability. We compare this to MLM image value tokens' segmentation capabilities averaged across four segmentation probing tasks: foreground segmentation, co-segmentation, semantic segmentation and referring expression segmentation.
We evaluate two MLMs with different configurations: (1) models of the same version but
different sizes: ManV--On/l Vision (0.5B, 7B, 72B); and (2) models of the same size but
different versions: Qwen2-VL 7B and Qwen2.5-VL 7B. The LLaVA-OneVision 0.5B model stores 2 key-value pairs per layer, the 7B model stores 4 key-value pairs per layer, and the 72B model stores 8 key-value pairs per layer.
Results. As shown in Figure 3 and discussed in Section 3.1, we observe that MLMs of different sizes and versions exhibit the same trend: the stronger the perception capability of their image value tokens, as estimated by their segmentation performance, the better the MLMs themselves perform on downstream perception reasoning vision-language tasks. This finding shows that improved perception of visual representations in the language model goes hand-in-hand with better perception reasoning capabilities of the MLM itself perception-heavy vision-language wesks-showing the importance of high--rrnuulartty visual representations, and further emphasizing the need to study the same in MLMs.
A.6 Input-Agnostic Visual Representations: Additional Details
A.6.1 Identifying Input-Aguostic Image Key Tokens
As discussed in Section 3.2, we study the variance in image keys per head per layer across 1000 COCO images. The variance is shown in Figure 14 as distribution with the threshold depicted and in Figure 15 as a heatmap.
We find that when plotting the variance of each key, bimodal distribution forms, with the first mode centered at variance of 200, and the second mode centered at variance of 600. We set the threshold at the lowest point between these two modes, at variance of 450. Keys with variance below the threshold are labeled input-independent, and are labeled input-dependent otherwise. Note: to ensure our experiments are not specific to a single image distribution, we calculated variance across validation set of ADE20K (which consists of 2000 images). The pattern in variance was consistent.
We find that when setting this threshold: 20 keys are input-independdnt in the first 8 layers (including all 16 keys in the first four layers); 8 keys are input-independent in the next 10 layers; and 14 keys are input-independent in the last 10 layers. We call these the early layers, middle layers and late layers respectively.
21

---

Published as a conference paper at COLM 2025
Continuous Bimodal Distribution Curve for COCO
0.0020
1000
Derety
0
12
0.00100
600
19
23
200
0.0000
200
400
600
800
1000
1200
26
Value
0 0:222 Head Index
Figure 14: Distribution of L.2 variance of image keys in
7B across 1000 COCO images. We set threshold at variance=450 (ver- tical dashed line) to determine whether keys are input-agnostic input-dependent
Figure 15: Heatmap depicting the L2 variance of each image key token LLaVA-One 7B across 1000 COCO images.
A.6.2 Intervention Input-Agnostic Image Key Tokens
We then run the intervention by blocking: (1) only the input-independent keys at early layers; (2) only the input-independent keys at middle layers; and (3) only the input-independent keys at late layers.
We perform this experiment for both LLa Vision 7B and Qwen2.5-VL. 7B. We show results POPE in Table 7. We also show results another popular hallucination bench- mark: MME. Following the setup from Zhang et al. (2025), we evaluate on four subsets of MME: count, color, position and existence. In all four of these, the model is asked Yes/No question about the image (similar to POPE), which address hallucination across various axes and also measure general model capabilities. Results on MME are shown in Table 8.
Our results clearly show the applicability of our findings to multiple benchmarks and multiple models: specifically, the existence of input-agnostic image key tokens in later layers of the model, which (similar to the finding from Darcet et al. (2024)) encode artifacts. Note: the "existence" subset, which the closest equivalent to POPE, is much smaller than POPE (60 data points compared to 9000 in POPE). The models seem to achieve perfect existence scores both before and after intervening, hinting that this existence subset both smaller and easier than POPE.
Model
Intervention
POPE F1
LLaVA-OV
none (base model)
78.1
early layers
65.1
middle layers
77.9
late layers
81.2
Qwen2.5-VL
none (base model)
87.9
early layers middle layers late layers
69.1
82.4
89.2
Table 7: Intervention study on LLaVA-One Vision 7B and Qwen2.5-VL 7B on the POPE benchmark. Intervening at middle-to-late layers mitigates halllucination.
22

---

Published as a conference paper at COLM 2025
Model
MME Subset
Score of Base Model
Score after Intervention
LLaVA-OV
count
170.0
175.0
color
178.3
187.5
position
145.0
153.3
existence
200.0
200.0
Qwen2.5-VL
count color
182.5
182.5
185.8
193.3
position
160.0
165.8
existence
200.0
200.0
Table 8: Intervention study on LLaVA-OneVision 7B and Qwen2.5-VL 7B on the MME benchmark. Intervening at middle-to-late layers mitigates hallucination and improves general model capability.
A.6.3 Controls for the Intervention Experiment
To ensure that artifacts are present in input-agnostic representations alone, we additionally perform this intervention on: (1) the same number of input-dependent keys in later layers, randomly selected; and (2) the same number of all keys in later layers, randomly selected.
Results from controls on POPE are shown in Table 9, and results from controls MME are shown in Table 10. In all cases, the controls under-perform the main experiment, showing that only representations corresponding input-agnostic keys contain artifacts.
Model
Intervention
POPE
LLaVA-OV
none
78.1
input-agnostic
81.2
input-dependent
77.4
random
78.6
Table Control for the intervention study on LLavA-OneVision 7B on POPE. In each experiment, the same number of keys were blocked in the same layers of the model. Blocking input-agnostic keys improves performance, while blocking input-dependent keys reduces it. Blocking random combination of them performs midway between the two. Clearly, only representations corresponding to input-agnostic keys contain artifacts.
MME subset position
Model
Intervention
count
color
existence
LLaVA-OV
none
170.0
178.3
145.0
200.0
input-agnostic
175.0
187.5
153.3
200.0
input-dependent
165.8
172.5
141.7
197.5
random
169.2
175.0
146.7
199.2
Table 10: Control for the intervention study on LLaVA-OneVisile 7B on MME. In each periment, the same number of keys were blocked in the same layers of the model. Similar to POPE, blocking input-agnostic keys improves performance, while blocking input-dependent keys reduces it. Blocking random combination of them performs midway between the two. Clearly, only representations corresponding to input-agnostic keys contain artifacts.
A.7 Image Key Token Visualizations
Figures 16, 17 and 18 show key visualizations for samples from COCO for L.L. LLaVA-OneVision 7B, Qwen2.5-VL 7B and Llama3-LLaVA-Nex" 8B respectively.
23

---

Published as a conference paper at COLM 2025
Figure 16: Image keys in LLaVA-OneVision 7B for a random sample of COCO images, with input-agnostic keys highlighted.
24

---

Published as a conference paper at COLM 2025
Figure 17: Image keys in Qwen2.5-VL 7B for a random sample of COCO images.
25

---

Published as a conference paper at COLM 2025
Figure 18: Image keys Llama-LLaVA-NeXT 8B for a random sample of COCO images. This model stores eight KV pairs per layer. Their quality is less than that of LLa L.LVAA-OeVision 7B, as their vision and text components are both weaker.
26


Reviews:
[{"review_id": "xp7jBuWY4A", "reviewer": "Reviewer_RSnv", "summary": "The motivation for this paper is that we don’t know why MLMs struggle with semantic segmentation and object localization.  Previous work looked at ViTs or the Transformer blocks.   KV tokens are less studied here.  These are studied across LLaVA-OneVision and other models and input independent and dependent keys are identified.  It is found that value tokens contain enough information for these tasks and outperform clip and Dino.  Notably perception information doesn’t necessarily make it through to the final output, even when the information is present.  It is found that blocking text queries to image keys mitigates hallucination in the POPE object identification task by 3.2%.  Section 2 describes related work.  Section 3 describes the preliminaries, including MLMs and an image encoder, lightweight adapter and language decoder.  Three key differences in the image processing in the language model are 1) explicit key-value store 2) causal attention and 3) cross-attention between modalities.  Input independent keys have very low variance across input images while input dependent ones have much higher variance.  These tend to dominate mid-to-later layers of the model.  Each question in the POPE benchmark asks whether a particular object appears in the image (9000 examples).\n\nAn intervention is performed to block text tokens on input independent keys in 1) the first four layers and then 2) the last 10 layers.  Section 5 describes image values.  The authors hypothesize that image values contain well localized semantics while maintaining cross-modal alignment: segmentation and correspondence are looked at.  All tasks are zero-shot.  Few shot segmentation, semantic segmentation, referring expression segmentation and semantic and temporal correspondence are all evaluated.\n\nSection 5.2 indicates that visual representations are promotable by prefixing language.  Prefixing improves performance.  Section 5.3 shows mixed results though the authors are investigating the power of the cross modal representations.  They finally evaluate things on reasoning tasks.  The better the values on image tasks (segmentation), the better on the final output.  Conclusion is that MLMs should be doing better at perception than they are.  The authors conclude with future work.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, there are potentially nice insights from the paper, though I found it challenging to read at times.  The experiments could be interesting, and understanding the workings of MLMMs is valuable work.", "reasons_to_reject": "Not discussed in the paper is that MLMs are trained using CLIP and DINO as their visual encoders (via a connector) and that MLMMs have way more parameters, so you’d expect them to be even better at vision tasks, if not just as good.  In some sense, the results aren’t that surprising as a result.  I also didn’t find the PCA outputs (e.g., Figure 8) very readable.\n\nI was particularly surprised that the authors didn't use Siglip as well, as it's what was used in LLaVA Onevision, which they compared against in the paper.", "questions_to_authors": "* Can you please clarify the way that you think about the interaction between visual encoders (SigLip, CLIP, DINO) and MLMMs?  And how that factors into the results?  I would expect at least a discussion in the paper on this.  MLMMs should be strictly better.\n* What does the table look like if you add SigLip to Table 3?\n* Input-dependence and input-independence in this setting wasn't really defined until much later in the paper than needed (Section 4)\n* I don't understand the caption (or value) of Table 1.  What is the average segmentation performance of a DINO or ClIP when trained for that task?  What is state of the art?\n* Some examples of the before and after in Table 2 would be nice.  What is state of the art for Table 2?\n* In Section 5.5 it's not clear to me whether it's the value tokens or the initial encoder that matters here.  What is the performance of the visual encoder alone?\n* Isn't Figure 3 just saying that if you have more parameters, you do better at vision?  What am I missing here?\n* Why was the \"intervention\" done on the particular layers as described?  It seems like there are many ways this could have been done.", "extra_scores": {}}, {"review_id": "gsyn8KGAeO", "reviewer": "Reviewer_SMbp", "summary": "* The work addressed the issue of Multimodal Language Models (MLMs) performing poorly on perception-heavy tasks. \n\n* Instead of improving the architecture, the paper proposes investigating key-value (KV) tokens to understand how MLMs process visual information.\n\n* The work first distinguishes between input-dependent and input-independent keys and finds that it correlates with specific hallucination issues. Then, the paper investigates the image value tokens that perform descent on image-only tasks.\n\n* The findings offer more discussion into MLM functioning, which aims to acquire deeper mechanistic interpretability.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written and easy to follow.\n\n* The preliminaries in section 3.1 help understand the paper's following claims. Also, it provides a good comparison between different architectures and their purposes.\n\n* Visualization helped understand the low variance claim.", "reasons_to_reject": "This work tries to lead a discussion with several findings. However, most of the claims lack experimental support.\n\n1. In Section 4.1, the work tried to bound the connection between feature variance and hallucination. However, the motivation is unclear, and it is not apparent that they are related. In addition, the experiment of blocking independent keys is insufficient. Since the paper investigated three models, why not show the results of the three? Also, it was performed on a single benchmark. It is hard to claim the independent key is the actual cause of hallucination.\n\n2. In section 5.1, the paper tried to make a claim that MLM can perform on image tasks. Table 1 should be compared with existing segmentation models to show how close the models can achieve. For the prefixing setting, it is evident that writing a context-related prompt may improve MLM performance, which is the reason why MLMs were developed. The new insight is limited.\n\n3. Same for section 5.2, the finding is the motivation of MLMs\n\n4. In section 5.3, the work tried to compare MLM with CLIP and DINO, which is not a fair comparison considering different training data and model architectures.", "questions_to_authors": "* In section 5.4, I don't get what the point the author is trying to make. Could you elaborate more on this?", "extra_scores": {}}, {"review_id": "qiegYOlLX2", "reviewer": "Reviewer_GABW", "summary": "This paper delves into the visual representations within multimodal language models (MLMs), focusing on the visual key-value (KV) tokens in the LLM. Through experiments on three popular MLMs, the authors differentiate between input-dependent and input-independent keys, revealing that blocking attention to input-independent keys in middle-to-later layers can reduce hallucinations in object identification tasks by 3.2%.  Furthermore, they demonstrate that value tokens encode sufficient information for perception-intensive tasks such as segmentation, semantic correspondence, and temporal correspondence. These value tokens even outperform strong ViT-based models like CLIP and DINO on certain tasks. The study underscores the potential of KV tokens for improving perception capabilities in MLMs and highlights the importance of cross-modality for vision-only tasks. The paper's contributions are significant, offering new insights into the interpretability of MLMs and providing a novel perspective for enhancing their perception performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Innovative Research Perspective: The paper focuses on the KV tokens within LLMs, a relatively underexplored area. By analyzing these tokens, the authors provide a fresh perspective on understanding how MLMs process visual information, distinguishing their work from prior studies that primarily focused on vision encoders or transformer block outputs.\n2. Strong Empirical Evaluation: The paper includes extensive experiments on three different MLMs, covering tasks such as segmentation, semantic correspondence, and temporal correspondence. The results demonstrate that value tokens possess substantial perceptual capabilities, providing solid empirical support for the claims.\n3. Practical Significance: The findings reveal that cross-modality can benefit vision-only tasks, offering valuable insights for improving perception performance in MLMs. For instance, the paper shows that prefixing textual prompts to images can enhance the performance of value tokens on tasks like semantic segmentation, highlighting the potential for leveraging cross-modality to boost perception capabilities.", "reasons_to_reject": "The authors categorize visual key tokens into input-dependent and input-independent keys based on whether the key variance across 1K COCO images exceeds a fixed threshold. The following issues may limit the generalizability of the findings:\n1.  How does blocking attention to input-independent keys in middle-to-later layers affect POPE under different threshold settings?\n2. As all input images are from COCO, do input-independent keys relate to image distribution?\n3. Given that the intervention studies on image keys are conducted on a relatively small scale, do input-independent keys remain consistent when scaled up?\n4. Will blocking input-independent keys improve performance on other benchmarks beyond the illusion benchmark? We suggest that the authors also evaluate their approach on general-purpose benchmarks like MMBench.", "questions_to_authors": "See 'Reasons To Reject' section above", "extra_scores": {}}, {"review_id": "PETQXVj0K2", "reviewer": "Reviewer_6is2", "summary": "The paper presents the results of a number of analyses focusing on the visual key-value store of three multimodal language models (MLM).\nThe paper identifies keys which do not depend on the visual input (i.e. they are nearly constant across input images) and shows that by blocking attention to these keys in upper layers of MLMs can reduce \"hallucinations\". \nThe next analysis shows that visual value tokens encode information sufficient for several vision task, and correlated with model performance on these downstream tasks. However, this information is not always successfully surfaced by the model to correctly perform the downstream task.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is focused study of a specific, little studied aspect of MLM behavior. The analyses are appropriate for the task and carried out properly. The results are of interest to the vsion and language field and may prove useful to improve the understanding and performance of MLMs.", "reasons_to_reject": "The quality of presentation in some places could be improved. The details of several tables and figures are hard to interpret. The contribution of the paper is very focused and therefore somewhat narrow.", "questions_to_authors": "You may want to be explicit about what exactly  the key visualizations depict, i.e. what does each pixel in the \"feature map\" stand for.\nTable 3 contains some symbols and terms which are not referred to in the caption or the text (I_m, F_m, αbbox).\nIt is not clear how the numbers shows in figure 3 are derives from the performance scores on the tasks described in the text.\nThe paragraph in section 2 named \"Mechanistic interpretability\" would likely be better named just interpretability as there is little in covered in there that's particularly mechanistic.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on the key-value representations corresponding to image tokens in 3 multimodal language models (MLMs). The authors locate keys that are deemed to be independent of the input image (based on variance) and find that blocking the attention going to these keys (particularly in middle-to-later layers) leads to a decrease in hallucinations in object identification. The study then reveals that the MLMs are not able to utilize the information encoded in the value tokens for perception-heavy tasks such as segmentation, semantic correspondence and referring expression detection. The outcomes indicate the importance of key-value tokens for understanding the information flow in MLMs as they process visual inputs.

**Pros**

Quality: The reviewers appreciated the extensive experiments and appropriate analyses. 

Originality: The paper offers unique insights on visual key-value tokens in MLMs in contrast to work exploring vision transformer-based encoders and the outputs of transformer blocks.  
 
Significance:  This is an interesting paper that uncovers significant features of visual key-value tokens, and the consequences of altering them, with implications for improving perception-centric performance, interpretability and hallucination reduction.

**Cons**

The paper could benefit from clarifying the claims and improving the presentation of tables, figures, symbols and terms. 

One main question was whether blocking the attention to keys that are deemed to be input independent would have similar outcomes in general-purpose benchmarks. The authors claim that their results reflect general capabilities, listing tasks such as semantic segmentation and referring expression segmentation as well as including results from an extra hallucination benchmark that covers multiple axes such as color, position and existence. Reviewers were not very convinced of the claims regarding the implications for general-purpose tasks, indicating that some claims lack experimental support.

The authors ran further intervention tests on a larger subset from another dataset, evaluated another vision encoder, and provided results on an additional hallucination benchmark. I suggest incorporating these in the paper. 

I think the responses were reasonable and adequately addressed the majority of the concerns. The findings would be valuable as a focused contribution on vision-centric tasks, hinting at ways to mitigate hallucination as well as improve performance in MLMs.
Decision: Accept