Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
Authors: Lin Long
Changdae Oh
Seongheon Park
Sharon Li
Venue: ICLR 2026 Poster (2026)
Keywords: large vision-language models
multimodality
language prior
Primary area: foundation or frontier models, including LLMs
TL;DR: A formal framework for understanding and quantifying the language prior in LVLMs by contrasting the chain-of-embedding between visual and blind contexts.

Abstract: Large vision-language models (LVLMs) achieve strong performance on multimodal tasks, yet they often default to their language prior (LP)---memorized textual patterns from pre-training while under-utilizing visual evidence.
Prior analyses of LP mostly rely on input–output probing, which fails to reveal the internal mechanisms governing when and how vision influences model behavior. To address this gap, we present the first systematic analysis of language prior through the lens of chain-of-embedding, which examines the layer-wise representation dynamics within LVLMs. 
Our analysis reveals a universal phenomenon: each model exhibits a Visual Integration Point (VIP), a critical layer at which visual information begins to meaningfully reshape hidden representations and influence decoding for multimodal reasoning.
Building on this observation, we introduce the Total Visual Integration (TVI) estimator, which aggregates representational discrepancy beyond the VIP to quantify how strongly visual query influences response generation. Across 60 model–dataset combinations spanning 10 contemporary LVLMs and 6 benchmarks, we demonstrate that VIP consistently emerges, and that TVI reliably predicts the strength of language prior. This offers a principled toolkit for diagnosing and understanding language prior in LVLMs.

Published as a conference paper at ICLR 2026
UNDERSTANDING LANGUAGE PRIOR OF LVLMs BY CONTRASTING CHAIN-OF-EMBEDDIIN
Lin Long" Changdac Oh"
Seongheon Park
Sharon Li*
University of Wisconsin Madiison
{llong, changdae, seongheon park, sharonlijecs.wise.. "Equal contribution
*Corresponding author
ABSTRACT
Large vision-language models (LVLMs) achieve strong performance on multi- modal tasks, yet they often default their language prior (Lei-memorized tex- tual patterns from pre-training while under-utilizing visual evidence. Prior analy- ses of LP mostly rely on mechanisms governing when and how vision influences model behavior. To ad- dress this gap, we present the first systematic analysis of language prior through the lens of chain-of-embedding which examines the layer-wise representation dy- namics within LVLMs. Our analysis reveals a universal phenomenon: each model exhibits Visual Integration Point (VIP), critical layer at which visual informa- tion begins to meaningfully reshape hidden representations and influence decod- ing for multimodal reasoning. Building this observation, introduce the Total Visual Integration (TVI) estimator, which aggregates representational discrepancy beyond the VIP quantify how strongly visual query influences response gener- ation. Across 60 model-dataset combinations spanning 10 contemporary LVLMS and benchmarks, we demonstrate that VIP consistently emerges, and that TVI reliably predicts the strength of language prior. This offers principled toolkit for diagnosing and understanding language prior in LVLMS.
probing, which fails to reveal the internal
Code: ()
INTRODUCTION
Modern large vision-language models (LVLMs) (OpenAI. 2025; Comanici et al., 2025; Bai al., 2025; Zhu et al., 2025) have extended the boundaries of Al applications at unprecedented rate. Their remarkable capability in solving highly complex vision-language tasks originated from the internalized rich unimodal knowledge during the pre-training (Radford et al., 2021; Oquab et al., 2024; Brown et al., 2020) and also from the strong multimodal alignment (Liu et al., 2023; Dai et al., 2023; Zhu et al., 2024). Despite their successes, a central challenge remains: LVLMs are prone to over-relying on their language prior (LP) the statistical patterns memorized during large-scale language model pretraining while under-utilizing the actual visual evidence (Fu et al., 2024; Lee et al., 2025; Luo et al., 2025). This imbalance often results in hallucinations, shortcut reasoning. and brittle generalization when tasks truly demand visual grounding. For example, when asked "What color is the banana?", an LVLM may confidently answer "yellow" even if the banana in the image green, demonstrating that the model defaults to its LP. Recent studies ( (Yin et al., 2024; Liu al., 2024d; Lee et al., 2025) further show that such LP reliance persists across diverse tasks.
Understanding and quantifying LP in LVLMs is thus critical, both for diagnosing their limitations and for guiding the design of more reliable multimodal systems. However, current approaches to analyzing LP primarily rely on input-output probing. For instance, Lee et al. (2025) and Luo al. (2025) constructed datasets with counterfactual visual input to measure models' performance under challenging visual grounding scenarios, while Deng et al. (2025) evaluate models on modality- conflicting queries to assess modality preference. While useful, such coarse input-output analyses have fundamental limitation to investigate LP of LVLMs in-depth, because: (1) they ignore the rich latent representations inside the model, which may inform how textual and visual signals inte- grated, and (2) they cannot disentangle where in the model the LP begins to interfere with effective visual integration, leaving per-sample mechanistic interpretation (Bereska & Gavves, 2024) elusive.
Motivated by this, we propose a new framework for understanding and quantifying language prior, which leverages the chain-of-embedding the sequence of hidden representations across LVLM

---

Published as a conference paper at ICLR 2026
Comparing representation distances acruss chain-efrruroolinng
Spatting Iniegration
each Fg
Coll, distnnes un Pivs
What color the banana?
Layel 11
Layer'
Col distases Py
> di Giad
Lec!
Hoad Prograttion Mow
Layer
deverge
Doodin
aipia Noeel
layer
Lays!
Layer
Rolu Aded
6+
Qunatifylng
Layer1
estimating Food Fisaof Integration
TVRC-X LP(
LVLM
CoEl
Law TVI
artmonk titted Londscape with ziver Eiguns (dstadl))
In lobe baakethell
Yellow.
(x,, Pvr
layer
Figure 1: Framework Overview. For data from two distributions Pvr (vision-dependent) and P visson-indeependent, we extract ehaiedii- for two queries w/ and w/o visual input, and the expected representation distance to spot visual integration point l" Then, estimating total visual integration based on 1" allows us to quantify LP of an LVLM per sample.
layers. Making use of these latent representations is essential, because they provide direct insight into the inner mechanisms LP, beyond surface-level outputs. Specifically, framework contrasts embeddings from vision-text inputs (Zijs) with those from vision-removed inputs (Zhind). ). at each layer L. Based on the contrastive chain-of-embedding,, exhibit a Visual Integration Point (VIP), a layer at which visual information begins to meaningfully influence the LVLM's decoding process. At and beyond VIP, the distance between Z and Zind increases substantially for vision-dependent tasks, signaling that the model has begun to actively integrate visual evidence to solve the task. In contrast, iision-independent tasks show smaller such shift. Thus, VIP captures critical point where visual input begins to exert meaningful influence on inference, revealing the extent which the model relies on vision falls back on language priors.
reveal a striking phenomenon: LVLMs
Inspired by observations from VIP, we propose quantifying LP through Total Visual Integration (TVI), which measures the effective amount of visual integration that affects the answer decoding of LVLM. Specifically, TVI aggregates distance between contrastive embeddings Z and Z actoss all post-VIP layers to measure the cumulative strength of visual integration. Intuitively. TVI is inversely related the magnitude of LP: models with strong reliance on language priors exhibit low TVI, while those that leverage vision more deeply exhibit high TVI. Through extensive experiments covering 10 contemporary LVLMs and datasets (60 settings combined), of the existence of VIP, and that TVI can be a reliable indicator of LP. Morcover, we demonstrate that TVI strongly correlates with performance benchmarks requiring visual reasoning, outperforming other proxies such as visual attention weights or output divergence. Then, we provide a theoretical interpretation of our measure well as analytic bounds of it for broader in practice. We illustrate the overall framework in Figure 1, and summarize our contribution as follows:
show the universality
1. We present a novel framework that contrasts the ofain--f-embedding of an LVLM for fine- grained analysis of the visual integration and language prior of LVLMs.
2. Based on this framework, we show that there is a specific layer, VIP, where an LVLM's behavior undergoes a dramatic change, and observe that pIst-VIP layers' representations play a key role in quantifying the amount of language prior of an LVLM.
3. Across representative LVLMs and datasets, we consistently demonstrate the existence of VIP, show how we can use it to predict the strength of language prior of an LVLM certain sample through TVI, and further present thcoretical analyses on our framework.
PROBLEM STATEMENT
Basic notations. Let D {(x,, ),N, denote a dataset of N image-text queries (Ie, 22)). sam- pled from population distribution P. Each tuple (æ.. consists of a visual input 2.. and a natural language query z₁, expressed in prompt form. We distinguish X, from x, to denote a random vari- able and its observation (similarly for X,). Then, define the data structure as follows.
2

---

Published as a conference paper at ICLR 2026
Definition 2.1. We define Pyr the vision-dependent distribution, consisting of examples where resolving the textual query requires access the associated visual input. In contrast, Pr is the vision-independent distribution, containing examples where the textual query can be answered cor- rectly without visual information (i.e. the text alone suffices). A sample dataset 'D is constructed with Dvr and Dr, each containing at least one element from populations Pvr and Pr, respectively:
D (Dvr U UD: min((Dvrl, (Drl) 2 21}.
where || denotes the cardinality of a set. See examples of DDv and Dy in Figure!
Meanwhile, we have an LVLM, FB fn a fr. ft.- ..... fi for parameterized with A. Here, So denotes the composition of the visual encoder, modality connector, and text embedding layer; (Ji....:JIr JT.) corresponds to the L stacked decoder layers of the LLM; and fa is an output head. The LVLM maps the multimodal input query (Iv. ) to a di-mensioo probability distribution over the vocabulary space V, from which the most likely answer ÿ is obtained via the argmax operator.
Language prior (LP). An LVLM has a vast amount of knowledge in its parameters obtained during unimodal pretraining and visual instruction tuning of entire model components. Since the pre-training of LLM backbone is far more extensive in quantity and diversity of data, and total com- puting budget, LVLMs are prone to over-reliance on memorized statistical textual patterns withour integrating visual information during inference. Given an input x and an LVLM Fg. we define the model's reliance on statistical textual patterns as the language prior, F(, Note that LP is more like a latent property that lacks gold-standard measurement. Therefore, previous work typically approximates how robust an LVLM against LP through its performance on carefully curated datasets. In contrast, we propose novel approach that (I) does not require any annotations or careful data curation, (2) tries to quantify LP a more direct manner, which enables flexible and fine-gre fine-grained, sample-wis wise diagnosis for LP of LVLMs. Refer to Appendix A for additional context.
Our position. Although there have been recent attempts to analyze LP in LVLMs, they primarily focus on evaluating model predictions curated datasets (Lee et al., 2025; Luo et al., 2025; Vo et al,,, 2025), without offering a well-defined or generalizable formulation. We argue that such coarse input-output analysis is insufficient: cannot reveal how LP manifests within the model nor how it can be rigorously quantified. In particular, prior approaches overlook the rich latent information encoded inside
representations that inform how visual and textual signals are integrated and how biases emerge. Making use of these latent representations is essential because they provide direct insight into the inner mechanisms of LP, beyond surface-level outputs. With this motivation, we pose the following research question: Can we derive formal framework to understand and quantify the language prior of LVLMs through the lens of their internal states?
METHODOLOGY
3.1 BHAIN-OF-EMBEDDDIE AND REPRESENTATION DISTANCE
In contrast to previous approaches that focus on LVLM output (Rabmanzadehgervi et al., 2024; Vo et al., 2025; Lee et al., 2025; Luo et al., 2025), we leverage the chain-of-embedddng for fine-graing -hniggr analysis of LVLM, which is defined as a sequence of hidden states across layers, i.e., (Z), ........ Zz),, where Z'_ fi( (X,. u: X,) E dRd=1 denotes the last-token embedding I {1,-.., T.} contextual summary vector". Notably, contrast embeddings from two different input constructions as below.
Z = X.)
(embedding from both visual and textual inputs)
Z == X.)
(embedding from textual input only)
Now, given a distance metric d, we analyze the difference between these two embeddings per layer by defining an expected representation distance and its maite-sample estimator,
Di(P., Fb) E(x..x.)>P, [d(Z'n, Zemma), Di(D.,
Σ
d( -.it))))
(2)
D..
(Do.Re),ED,
where D, is Dvr or Dr, and P. is Pvr or Pr.
Although 7' = f((... f2 (f,(((X,,) (X,, X2))) is more precise, we slightly abuse the notation for clarity.
Such last-token embeddings integrate information from all preceding tokens and are widely used to inves- tigate model's hehavior when generating the next token (Jiang et al., 2024; Tian al., 2024; Li et al., 2025b).
3

---

Published as a conference paper at ICLR 2026
We adopt the cosine distance by default, though other distance functions, including non-metric dis- tances (Deza & Deza, 2009), can also be valid. An ablation study with alternative metrics is provided in Section 4. Intuitively, Zvis should encode distinctive visual semantics that cannot be inferred from text alone, whereas Aind primarily reflects the model's default linguistic expectations. However, the degree of this discrimination can depend on how visual information contributes differently to different data, and across different layers of the model. We elaborate on this in the next section.
3.2 VISUAL INTEGRATION POINT HYPOTHESIS
Deep neural networks are known to develop hierarchical representations across layers (Chen al,, 2023; Fan et al., 2024; Jin et al., 2025), where each layer has different types and resolutions of information (Joseph & Nanda, 2024; Skean et al., 2025; Artzy Schwartz, 2024; Jiang et al., 2025). In this paper, we hypothesize that an LVLM has Visual Integration Point (VIP) 1", a critical layer where the model begins to actively leverage visual information to perform task-specific reasoning. Prior to this point, the model primarily engages in general purpose processing of visual and textual inputs-visure features may be "seen," but not yet "used" to guide inference, and the interactions between modalities remain shallow. This behavioral shift can be reflected in the representation distances: at and beyond VIP, the distance between Z and Z increases markedly for vision- dependent tasks (PTv), signaling that the model has started utilize visual information to solve the task, while vision-independent tasks (Pr) show smaller such shift. Thus, the notion of VIP captures a key behavior transition inside LVLMs. If such a specific point I exists, identifying allows us to localize where the differences between language- language-prior-done prior-dominated and visually grounded inference start to manifest within the model's internal processing. We formalize this hypothesis below.
perprool
Hypothesis 3.1 (Existence of the visual integration point). Given a distance metric d(..-) Z x 2 + R. distributions Pry and Pr (Eq. 1), and an LVLM Fg with L. layers which produces a chain- of-embeeding (Z² ...,, = given input, let Dj be an expected representation distance defined as Eq. Then, there exists a visual integration point (" that discerns D2 between Pvr and Py, that is,
Di(r,, (6) >T, 4IS 1"
31' (l,..., L
Di Di(P,, Fo) 00, VI<r
where T E R. denote a model-dependee constant threshold for each data distribution
Gamma-3-48 MME
Gamme-3- 3-48 on MMBBanch
Gamma-3-48 VLind-Banch
0.62
6
0:53
6.03
0.00
0.50
0.00
Layer
Layer
Qwwn2.55V/L-22 on MME
Swen22.5-/L-72 on MMBaacc
Qween.55-VL-22 on VLind Bench
0.325
O-
0.350
Dr
00000
6
0.000
0.600
6.635
F120
Figure 2: Visual Integration Point. We consistently observe that there is a specific layer 1" that clearly distinguish the distance between Z and Zilind across two groups Dvr and Dr.
In Figure 2, we plot the representation distance estimated for two groups: Dvr (vision-dependent) in red and Dr vissioninddeennennt in blue, across all layers in Qwen2. 5-VL-7B (Bai et al., 2025) and Gemma- 3-4B (Team et al., 2025). We evaluate three representative datasets: MME (Yin et al., 2024), MMBench (Liu et al., 2024d), and VLind-Bench (Lee et al., 2025). Since these datasets do not explicitly annotate the degree of visual dependency for each instance (PVT vs. PT), we par- tition each dataset 2 into two auxiliary groups: Dvt {x., 2x.) E D Fo( Iz) # Fe (2, xx)} and Dj = {(I,, 2x) E D Fo(x,, z2.) = (Z xx)) This split leverages the prediction agree- ment between multimodal and text-only inputs as a proxy for task type: if two predictions differ, the sample must have demanded visual information to the model, suggesting membership in Pvr likely.
We manually select the and VIP for each model over the observed distances Dj for analysis convenience (see Appendix B and for details on this manual selection and automatic selection method well).
4

---

Published as a conference paper at ICLR 2026
From Figure 2, we make four key observations: (1) Existence of VIP. Representation divergence between Dyr and Dr does not show from the beginning. Instead, for both models, we observe a clear visual integration point (1*), where the representation distance for the Dvr group rises more sharply compared to Dr group, marking the onset of genuine multimodal integration: (2) Behavioral shift across VIP. We observe notable increase in the standard deviation of representation distances across VIP. Specifically, before VIP, the model exhibits relatively uniform representation distances across samples, suggesting general-purpose information processing. After VIP, the model's usage of visual information becomes more diverse and instance-dependent to solve a specified task for each query; (3) VIP is dataset-agnostic. Within each model, the location of the VIP relatively con- sistent across all datasets. For . 5 5-VL-BB the transition consistently occurs around layers 18-20, and for Gemma- 3-4B, the transition around layers 20-22. This stability suggests that the VIP is primarily the LVLM's intrinsic property, not one driven by dataset-specific biases; and (4) Model-specific patterns. Despite the shared existence of the VIP, the shape of distance across lay- differs across models. In Qwen2. 5-VI- representation distance grows relatively smoothly before peaking near the middle-to-late layers and then declines. In contrast, Gemma- - -3-43 exhibits flat trajectories for many early layers, followed by steep and monotonic rise after VIP. This sug- gests that each model has a distinctive hierarchical representation derived from its unique designs.
Overall, these findings highlight not only the universality of the VIP existence, which distinguishes vision-centric decoding (post-l*) from general information-gatheriin behavior (pre-!"), ) but also the variability in how different LVLMs distinctively integrate visual information across depth.
3.3 QUANTIFYING LANGUAGE PRIOR OF LVLMS THROUGH TOTAL VISUAL INTEGRATION
Although the visual integration point detects the birth of LP(x, Fo), we are also (or even more) interested in how strong LP(x, Fo) is. To quantify this, we define a total visual integration (TVI) estimator in Def. 3.2, which measures the total amount of visual integration that effectively affects the answer decoding of LVLM,, and thus is inversely related to LP in nature.
Definition 3.2 (Total visual integration estimator). For observed imput = (Iv, 22). define
Iwis := (Iv. 22) and Fblind := (0,z,). Given an LVLM Fe with L decoder layers which produces two sets of bhain-of-embedding (2y ..., zHis) and (Zhinds ... >bbma)) we define the empirical estimator for the per-sample total visual integration as follows,
1 TVI(U;;; L --
Σ [dd =blind))
where z fe((vv)), Abrind fe( and -)..-) denotes a distance metric.
Here marks the VIP layer, where visual information begins to meaningfully influence the model's internal states for groundd decoding. The TVI score then measures the cumulative con- tribution of visual information by averaging representation distances across all subsequent layers (1 2 1*). The idea behind TVI is that once the model passes the VIP, its internal representations increasingly reflect effective visual grounding, rather than shallow alignment or language-drive driven statistical patterns. A higher TVI indicates that visual information is more effectively utilized dur- ing the response decoding phase, while a lower T'I suggests that the model is more likely to remain text-dominated even after 1" In this sense, TVI provides holistic measure of how much the model truly uses vision for actual problem solving: strong LP corresponds to weak or shallow visual integration (low TVI), while effective multimodal reasoning corresponds high TVI.
To investigate the distinction between pre-I" and post- I' phases visual integration, we analyze Spearman's 's rank correlation between them and answer correctness on VLind- Bench (Lee et al., 2025), which requires vi- sual reasoning. The results in Table show that corre- lations are weak and statistically less-significant when TVI is computed over pre-I* layers. In contrast, the post-l" aggregation yields remarkable correlations with the prediction correctness, indicating that only after the VIP, the representation distance becomes strongly associated with task performance, thereby serving as a reliable indicator of effective visual integration. In this paper, we stick with post-l" aggregation in Definition 3.2.
Table 1: Spearman's 's rank correlation between prediction correctness and TVI aggregated from different layers.
Model
pro-1"
post-l"
0,1489
0.7241
Qwen2 5-VL-78 78
0.002)
0.001)
0.4659
0.7174
Gemma 3-4B
0.001)
0.001)
5

---

Published as a conference paper at ICLR 2026
Cemera 48 20MP
Sweed N.VL 76 MPM
LLANK VS. MPM
Intendt 55 MME
0.08
Lowr
Leee
No tio no Ri0
Nu Okey Nikil 1411
Ny Jknp Talak (011)
0.01
Lever
G6mma-3-46 en NiSSter
5- Hetiter
LL VONSTA
0,60
Whwd eno
1.41
layer MLP
Queeriz 2000-222441-18
3. Gky (ahelddd
4. Slyp Mikeolii
Figure 3: VIPs of different models observed across different datasets. Our novel framework, fueled by contrasting chain-of-embeeddingg allows us to consistently observe VIP across multiple models and datasets, and further enables us to estimate TVI to measure language prior.
Taken together, results from Figure 2 and Table highlight two key insights: (1) the existence of the visual integration point 1", where effective representational shifts starts to happen by integrating vi- sual information, and (2) the strong relationship between post-!" TVI and downstream performance on vision-dependent tasks. These findings demonstrate that VIP and TVI provide a principled toolkit for analyzing visual integration and language prior in LVLMs. We summarize our findings below.
Summary of preliminary findings
1. The layer-wise expected representation distance between Dyr and Dr, i.e., (D Fe) (Drr,, Fo), shows a sudden bump up after a specific layer 1", while marginal before 1*.
2. The aggregated distance -- Eísr [d(('hss =b(ma))] over post-l" layers serves as a reli- able indicator of language prior, particularly for datasets requiring visual reasoning.
4 EXTENDED EXPERIMENTS
Building on the visual integration measurement introduced in the previous section, conduct ad- ditional experiments to assess its empirical validity. Furthermore, we designed a set of in-depth analyses explore the relationship between visual integration and the language priors in LVLMs.
VIP consistently emerges across different datasets and models. We extend the experi- mental setups described in Section 3 to a broader range of 6 datasets and 10 LVLMs, including 2wen2.5-m - (Bai et al., 2025), InternVI.3-B8 Gemma-3-44 (Team et al., 2025), LLaVA-v1.5-22 (Liu et al., 2024a), Eagle2.5 5- (Chen (Liu et al.,
(Zhu et al., 2025),
et al.. 2025a), 3.2-ma-...-11B-iis- 2024b), LLavA-oV-Qwen2--23 (Li et al., 2025a) SmolVLM (Marafioti et al., 2025), and (Dai et al., 2023). For the datasets, we consider general VQA benchmarks including MME (Chaoyou et al., 2023), MMBench (Liu al., 2024d), MMStar (Chen et al., 2024), and MMMU (Yue et al., 2024). We also incorporate two benchmarks specifically de- signed for language prior evaluation, which are VLind-Bench (Lee et al., 2025) and ViLP (Luo al.,
InstrrcctBL.P--icnn--
hnttps://hugging wtt.pe..c//...... co/meta---eeeooo...e..........aa 1ama / 1. ama- 3.2-1 2-111 1B-1----- Vision- Instruct

---

Published as a conference paper at ICLR 2026
2025). This results in a combination of 60 experimental settings. Implementation details, including data statistics, generation configuration, strategy for VIP selection, etc., are provided in Appendix C. As illustrated in Figure 3, the emergence of VIP is remarkably consistent across all settings: for each model, there exits a clear transition layer I where the distance between embeddings Zijs and Ziid increases more significantly for vision-dependent group (Div), compared to the vision-independent group (Dr). These results highlight the universality of the VIP existence. Due to the space limit, we defer the complete experimental results Appendix D.
Qwen2.5-VL-7B MMBencháVILP
LLaVA-v1.5-7B MMBench&ViLP
17.5
Weak (MMBencch)
20.0
Weak (MMBsnch) Strong L? (VILP)
15.0
Strong (MILP)
Percentage (8)
Percentage (1)
15.0
7.5
5.0
5.0
2.5
0.0
0.02
0.04
0.05
0.38
0.10
0.32
0.14
0.0
0.05
0.10
0.15
0.20
0:25
TVi
TVI
Figure 4: TVI under language priors of different strengths. We see that TVI effectively discerns the differences in strength of LP, thereby standing for reliable measure for LP.
TVI reliably differentiates strong vs. weak language prior. To examine whether TVI (c.f. Defi- nition 3.2) reliably reflects the strength of the language prior, we contrast results on two complemen- tary datasets: ViLP and MMBench. ViLP intentionally constructed to induce a strong LP on the data side by designing queries where plausible answers can often be inferred from textual patterns or statistical correlations without the need for visual grounding. In contrast, MMBench represents weak LP setting, with less misleading questions that encourage stronger visual grounding for task success. As shown in Figure 4, our analysis reveals that datasets with stronger language priors (e.g., ViLP) yield lower TVI values, indicating weaker visual integration in the model, whereas less biased datasets (e.g., MMBench) produce higher TVI values, reflecting stronger use of visual information. This confirms that TVI serves as a reliable quantitative indicator of LP.
Interventional validation for TVI. To further ver- ify whether TVI robustly quantifies LP under differ- ent inference setups, we conduct a small interventional study. Specifically. applied based hallucination mitigation method PAI (Liu et al., 2024c) Qwen2.5-V1.- 5-VI.-7B as inference-time in- tervention, which promotes the model to pay more at- tention to visual features, implicitly increasing visual integration. As shown from the results in Table 2, the intervention not only improves task performance but also yields substantial increase in TVI. This observation demonstrates that TVI faithfully reflects changes in the model's degree of visual integration, thereby providing robust evidence that it is a reliable metric for quantifying LP.
Table 2: Downstream performance and TVI before and after intervention.
Accuracy (%)
TVI
ceentionncorrection.
Before intervention 50,00
0.038
After intervention
52.33
0.144
Comparison to existing proxy for lan- guage prior. There are alternative ap- proaches to explain LP proposed in previ-
Table 3: Spearman's 's rank correlation between dif- ferent metrics and answer prediction correctness.
Owen2 S-VL-78
Into==Vi--3-8
ous works, which rely on output-based or attention-based heuristics by assuming (1)
Metric
VLind
ViLP
VLind
Vil.P
0.7155
0.6335
Iu6727
0.5709
LP manifests high similarity between output tokens generated with and with- out visual input (Chen et al., 2025b; Xie et al., 2024), or (2) LP arises due to insuf- ficient attention being allocated to visual tokens (Liu et al., 2025). In Table 3, we compare TVI with two existing approaches (see Appendix C for detailed formulation), average
TVI
(p 0.001)
(p 0.001)
(p 0.001)
(p < 0.001)
-0.3344
0.4967
Attention
(p 0.075)
(p 0.330)
0.001)
(p 0.197)
Output Divergence
0.2978
0,5084
0.1627
0.5615
(p 0.001)
(p 0.001)
(p 0.001)
(p < 0.001)
7

---

Published as a conference paper at ICLR 2026
visual attention and output divergence, by conducting the Spearman's 's rank correlation analysis be- tween these measures and the correctness of model predictions on two datasets, which all require integrating visual information to produce correct answers. Our TVI consistently exhibits stronger correlation with output correctness across all datasets and models, suggesting that TVI stands for a reliable indicator of effective visual integration of LVLMs. In contrast, the other approaches show weak and inconsistent correlations in different scenarios.
We argue that both existing approaches fail to directly capture the true impact of visual integration on the model's generation. In the case of visual attention, the model may assign high weights to ir- relevant regions of the image rather than the areas required for correct reasoning, and ultimately fall back on its language prior to generate the answer- resulting in inflated attention scores but weak correlation with language prior. Meanwhile, solely measuring output-level discrepancy does not fully capture fine-grained behavior exhibited in internal representation dynamics-diiffreenee differences that are more fundamental in nature than what can be observed from final outputs. It shows the signifi- cance of procedural aggregation in TVI. We provide additional visualization analysis in Appendix E.
Ablations on distance metries. To investigate how different choices of distance metric d affect our ability to capture model behavior, we conduct ablation studies with alternative formu- lations of TVI. As shown in Table 4, TVI remains strong indicator of model correctness when computed using the L.2 distance between latent embeddings. However, when we ap- ply the logit-lens technique (nostalgebraist, (022)-projcctinng hidden states at each layer into the output token space and computing divergence between the resulting distributions the effectiveness of T'I drops significantly. This degradation sug- gests that such a projection distorts or suppresses the interme- diate behavioral differences that occur during decoding. The output space, shaped by the language modeling head, inher- ently filters latent representations through decoding-biand -iased lens, which may obscure subtle but meaningful cross-modal integration patterns. These observations reinforce our central contribution: to faithfully capture the behavioral dynamics of vision-language models, it is essen- tial to examine the internal processing trajectory within the latent representation space, rather than relying on surface-level discrete outputs or their immediate projections. Additional visualization analysis is provided in Appendix E.
Table 4: Spearman's rank cor- relation between correctness and TVI under different distance metrics. Results are based on eval- uations using 1--e2..5--V--- All p-values are < 0.001.
Metric
VLind
ViLP
Embedding-based Cosine Distance
0.7155
0.6335
1.2 Distance
0.7123
0.6578
Output-based (w/ logit-lens)
KI Divergence
-0.1693
0.2901
IS Divergence
-0.2261
0.2942
Varying model scales. We fur- ther examine whether our find- ings generalize across models of different scales. As shown in Figure 5, the VIP consistently emerges across models of varying sizes (4B, 12B, and 27B), under- scoring the robustness and gener- ality of our proposed behavioral analysis framework. Interestingly, also find that the VIP tends to appear at a similar relative depth, which is approximately 60% of the total number of layers, regardless of model size. In addition, after normalizing by the dimensionality of hidden states, we observe that the average normalized TVI is con- sistently higher in larger models on both Dvr and Dr. This suggests that larger models are more effective at leveraging visual information in a uniform manner across diverse input types, thereby exhibiting greater robustness to misleading language priors. These
SWIME P-LE
and tWV Mommaa PVti 1914
boo TV olie Inveerteed 9913
PVi- 2014
Figure 5: Varying model scales. VIP and the dimension- normalized TVI analysis results for three variants of Gemma-3 model family.
8

---

Published as a conference paper at ICLR 2026
observations collectively reinforce the broader applicability of our framework in analyzing visual integration behavior across model scales.
Practical utiliiy. We also investigate the practical applicability of TVI here by providing a crete example to make use of our findings to actually improve LVLMs. Specifically, we leverage TVI additional regularization term along with the vanilla cross-entropy loss for next-token prediction during instruction tuning. That is, given the input 2 = (Iv, 22) and instruction y, we adjusted the original LLaVA training objective (Liu et al., 2023) to:
(5) where the strength of the regularization is controlled by À (we set as 0.03). Due to resource limi- tations, we trained the model on a 60k randomly sampled subset of alava_vl. _lava_vi___m _1_5_mix6656 All other hyperparameters used for visual instruction tuning remain the default of Liu et al. (2023).
(i,, y; 0) = log Fo(v/z) X TV,
The result is shown in Table 5, where the performance improvement indicates that ex- plicitly encouraging stronger visual integra- tion (via TVI) leads better downstream task performance. This highlights TVI's po- tential as helpful training regularizer for improving the visual perception and reason- ing of LVLMs in practice.
Table 5: Effect of TVI regularization on down- stream performance, MME dataset (Chaoyou et al., 2023) with L...A.v1.5-7B.
Perception
Reasoning
LLa
1369.75
298.21
LLaVA-v1.5 w/ TVI
1400.44
321.43
Additional empirical analyses. In addition, we deepen our understanding of the proposed frame- work by providing additional analysis Appendix E, including different aggregation strategies for TVI calculation, instruction- levell perturbation, image-text vs. image-only chain comparison, VIP and TVI evolution across training stages, and case studies on TVI failure cases.
THEORETICAL ANALYSIS
Next, introduce new interpretation for measure, Di Fe) Dr Fe). that locates VIP (Theorem 5.1) and discuss how we can practically employ the expected representation distance (Theorem 5.2) through theoretical analyses. All the proofs and an additional theorem that justifies the use of our empirical representation distance (Lemma F. F.1) are given Appendix F.
Information- interpretation representation divergence. By recasting the represen- tation distance measurement as a density estimation problem, i.e., di d(Z Zilina) x - log PT (Z') (please Lemma F.2), show that the difference in expected representation distances, (P Fo) (Prr Fo), which we call representation divergence here, can be interpreted as relative distributional discrepancy that measures how far the density estimator fr(ZZ), defined by di d(Z.,,. Zllma). from population distribution Pvr(Z') compared to pr Pr(Z") in Theorem 5.1.
Theorem 5.1. Let X (Xv, X,) X be random variable from Pvr Pr, and fi X - Z be a layer stack from an LVLM Fg. For Pr. define a density estimator pr( := NN': f((,),, and denote Dvr(Z') (resp. Pr 7))) the population distribution Z' = fi(X) derived from Pvr (resp. Pr). Then, given di 1,, Zz) : 1 2 the difference in the expected representation distances between Pvr and Pr, i.e., (P Fb) (P Fo), can be expressed as follows,
KL(pvr 2 [(r((Z)) KL(pr(Z) (p(((ZZ)) (lrr)) H
where H is constant H (pvr(Z')) - - (pr()))) and KL(-II-) denotes the KL divergence.
Implication. Theorem 5.1 tells that Dr Fo) Dr Fb) can be interpreted as relative proximity of the density estimate ůr to each distributions pvT and pr with an additive constant H. Intuitively, the first term, KL(pvrl (L(pvrllpr) can be understood how fo (estimate of PT) far from the true representation distribution on Pvr while the second term, KL(Pr)) (r), is quality of the density estimation with PT approximate Pr. This expression converts the expected representational distance of the LVLM Fo over Pyr and pr into an information- theoretic divergence, the amount of surprise if approximate the distribution over pvt via binnd-epreeentaaioo-ccneerrr Gaussian estimator pr(Z') (Z') = Ni; 2 1).

---

Published as a conference paper at ICLR 2026
Analytic bounds representation divergence for practical use. We have assumed a fixed model F6 so far. If one is willing to adapt the model to improve its effective visual integration, the analytic bounds in Theorem 5.2 described with H- H-divergence (sce Def. F.4) can be useful.
Theorem 5.2. Let X = (X.,, X,) X be random variable of multimodal input query. Given stuck of LVLM layers Jt X + and a distance metric d : 2 x 2 + [0, 1}, define a hypothexis h d(f((x,; Xc), fi(X))) X + [0,1] and set of these hypotheses H that has a pseudo- dimension Then, for (i(P., Fo) == Exwp, -Exxp_h(XX) with any Pvr. Pr, and PM := Pu and
the empirical distributions Dvr ~ Pvr and Dr ~ Pr of N samples for each, we have the following bounds wap. at least 1 8 for 0 < 5 1,
i) (Dr,, Fo) 2 dx (Prr. , Dr) - Ös - Di(Prr,FF) Fo),
(7)
1
1 + dx (Dvr, Dr) + +08
ii) 2 (D(r Dr) Ös s (P Fe) - 2
(8)
where H ((IxX))-((X)I h E HH s 1) and Ö₂ Olv / (log $ + +clog ))
Implication. The first inequality (Ineq. 7) reveals a relationship between two expected rep- resentation distances across Pvr and Dr ~ Pr with dy (Dvi, Dr) as a bridge. This tells us that if we want to increase visual integration for unknown data distribution that require vi- sual reasoning (Pvt). we can pursue a greater lower bound of it by decreasing (Dr, F9) and dg (Dvr, Dr) with empirical samples have. Meanwhile, in case where we encountered unknown mixture distribution PM. the second inequality (Ineq. 8) tells us we can broaden the effective range of Dr on PM by pursuing greater value of (DvT, Dr).
CONCLUSION
In this work, we present a formal framework for understanding and quantifying the language prior in LVLMs by contrasting the framework, we identify the consistent existence of the Visual Integration Point (VIP), a specific layer at which the model begins meaningfully incorporate visual context for task-solving beyond the shallow information gathering. Building on this observation, we propose a new metric named Total Visual Integration (TVI), which estimates the degree of effective visual integration and there- fore language prior. We conduct comprehensive experiments across 9 LVLMs and 6 datasets, and the results demonstrate that our framework robustly works across models and tasks, providing consistent and interpretable signals about the presence and strength of language prior. Finally, we present some theorems for better understanding and utilization of our framework. We hope that this work sheds light on the internal mechanisms of multimodal models and provides a foundation for diagnosing the language prior, ultimately guiding the development of reliable and responsible LVLMs.
between visual and blind contexts. Through this
Limitations. We set the language prior to LVLMs our sole target of analysis here, and devel- oped our method based on the representation dynamics of LVLMs. However, there are many other potential biases and vulnerabilities originating from query distribution shifts in the wild, which may induce remarkable changes in the representation space and thus degradation of downstream perfor- mance (Verma et al., 2024; Oh et al., 2025a;b; Kim al., 2025). Reliability of TVI-based language prior estimation should be further validated under realistic distribution shifts.
Besides, our method requires white-box access the model's internal hidden states and attention patterns. This restricts its applicability to open-weight models and excludes commercial APIs or closed-source systems. However, our framework is primarily designed for model analysis and inter- pretability research of white-box models, rather than serving as a versatile tool, aiming to shed light on how and when visual information integrated during inference.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
The authors would like to thank Boyi Li, Haobo Wang, Junbo Zhao, Samuel Yeh, and Jiatong Li for their insightful feedback and helpful discussions throughout the development of this work. Their suggestions greatly contributed to improving the quality of the draft. Changdae Oh, Seongheon Park, and Sharon Li are supported in part by the AFOSR Young Investigator Program under award num- National Science Foundation under awards IIS UIS-2237037 and IIS-2331600 Office of Naval Research under grant number 400014-23-1-2664 1-2443,, Schmidt Sciences Foundation, Open Philanthropy, Alfred P. Sloan Fellowship, and gifts from Google and Amazon.
ber 1-9550-23-1-0188
REFERENCES
Amit Artzy and Roy Schwartz. Attend first, consolidate later: On the importance of attention in dif- ferent IIm layers. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 177-184, 2024.
Saketh Bachu, Erfan Shayegani, Rohit Lal, Trishna Chakraborty, Arindam Dutta, Chengyu Song. Yue Dong, Nael B. Abu-Ghazalch. and Amit Roy-Chowdhury. Layer-wise aligmment: Exam- ining safety alignment across image encoder layers vision language models. In Forty-second International Conference on Machine Learning, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 1rXiv:2502.13923, 2025.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Shai Bev-d, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1)1151-175 2010.
Leonard Bereska and Stratis Gavves. Mechanistic interpretability for Al safety review. Transac- tions on Machine Learning Research, 2024. ISSN 2835-8856.
Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, Bingjie Wang, and Chenliang Xu. Unveiling visual perception in language models: An attention head analysis approach. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4135-4144 4144, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamerr. Advances in neural information processing systems, 33:1 33:1877-1901, 2020.
2020.
Fu Chaoyou, Chen Peixian, Shen Yunhang, Qin Yulei, Zhang Mengdan, Lin Xu, Yang Jinrui, Zheng Xiawu, Li Ke, Sun Xing, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.133: 1 13394, 3, 2023.
Guo Chen, Zhiqi Li, Shihao Wang. Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Eagle 2.5: Boosting long-context post-training for frontier languae models. arXiv preprint arXiv:2504.15271, 2025a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056-27087, 2024.
Yangyi Chen, Hao Peng, Tong Zhang, and Heng Ji. Prioritizing image-related tokens enhances vision-language pre-training. arXiv preprint orXiv:2505.0m 08877, 2025b.
Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp. Se- bastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active for- getting. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 31543-3155577 2023.
11

---

Published as a conference paper at ICLR 2026
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general purpose vision- lsioonlanguage models with instruction tuning. Advances in neural information processing systems, 36:49250-49267, 2023.
Ailin Deng, Tri Cao, Zhirui Chen, and Bryan Hooi. Words or vision: Do vision- vision-language mod- els have blind faith in text? In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3867-3876, 2025.
Michel Marie Deza and Elena Deza. Encyclopedia of distances. In Encyclopedia of distances, pp. 1-583. Springer, 2009.
Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang. Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of ilms are necessary during inference. arXiv preprint arXiv:2403.02188, 2024.
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination trol by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14303-14312, 2024.
Stephanie Fu, Tyler Bonnen, Devin Guillory, and Trevor Darrell. Hidden in plain sight: Vlms overlook their visual representations. arXiv preprint 2rXi::2506.08008, 2025.
Xingyu Fu, Yushi Hu, Bangzheng Li. Yu Feng. Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pp. 148-166. Springer, 2024.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversare training of neural networks. Journal of machine learning research, 17(59):1-35. 2016.
Che Jiang, Biqing Qi, Xiangyu Hong, Dayuan Fu, Yang Cheng, Fandong Meng, Mo Yu, Bowen Zhou, and Jie Zhou. On large language models' hallucination with regard to known facts. arXiv preprint arXiv:2403. 200,00 2024.
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucina- tions via attention lens. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pp. 25004-25014 2025.
Mingyu Jin, Qinkai Yu. Jingyuan Huang, Qingcheng Zeng, Zhenting Wang. Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, et al. Exploring concept depth: How large language models acquire knowledge and concept at different layers? In Proceedings of the 31st Interna- tional Conference on Computational Linguistics, pp. 558-573 2025.
Sonia Joseph and Neel Nanda. Laying the foundations for vision and multimodal mechanistic inter- pretability & open problems. In A/ Alignment Forum, volume 2, 2024.
Haeji Jung, Changdae Oh, Jooeon Kang, Jimin Sohn, Kyungwoo Song, Jinkyu Kim, and David R Mortensen. Mitigating the linguistic gap with phonemic representations for robust cross-lingual transfer. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pp. 200-211, 2024.
Eunsu Kim, Junyeong Park, Na Min An, Junseong Kim, Hitesh Laxmichand Patel, Jiho Jin, Ju- lia Kruk, Amit Agarwal, Srikant Panda, Fenal Ashokbhai Ilasariya, et al. World in frame: Understanding culture mixing as a new challenge for vision- lsioualanguage models. arXiv preprint 1oXiv:2511.22788 2025.
12

---

Published as a conference paper at ICLR 2026
Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. Vlind-bench: Measuring language priors in large lasounlaaguaae models. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 4129-4144, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. Trans. Mach. Learn. Res., 2025, 2025a.
Haoxi Li, Sikai Bai, Jie Zhang, and Song Guo. Core: Enhancing metacognition with label-free self-evaluation in Irms. arXiv preprint rrXiv:2507.06087 2025b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chengzhi Liu, Zhongxing Xu, Qingyue Wei, Juncheng Wu, James Zou, Xin Eric Wang, Yuyin Zhou, and Sheng Liu. More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models. arXiv preprint arXiv:2505.21522 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34911 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 26286-26296. IEEE, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https:// ht/blo///////oeoooooooooooe/ee// 200e://224-00000 -13---01--------eee 1lava-nexa/
1lava-vl.gittae
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: training-fri method for alleviating hallucination in Ivims. In European Conference on Computer Vision, pp. 125-140. Springer, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: your multi-modal model an all-around player? In European conference computer vision, pp. 216-233. Springer, 2024d.
Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, and Honglak Lee. Probing visual language priors in VLMs. In International Conference on Machine Learning, 2025.
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Za- kka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvlm: Redefining small and 2025.
efficient multimodal models. arXiv preprint arXiv:2504.0522
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards inter- preting visual information processing in vision-language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net, 2025.
nostalgebraist.
Interpreting gpt: The logit lens. https: htops::/www...r lesswrong www/post./...eeoooooopooooooo.t. BoKKBBWwwwaNorrrr//tterrreigooo ght-the-logee wwwtthhe-ootttteee -enn--leitttnnnnn.nooeee 2020. LessWrong.
Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision- language models. Advances in Neural Information Processing Systems, 17:12777-127777, 2024.
Changdae Oh, Zhen Fang, Shawn Im, Xuefeng Du, and Yixuan Li. Understanding multimodal LLMs under distribution shifts: An information-theoreee approach. In Forty-second International Conference on Machine Learning, 2025a.
Changdae Oh, Jiatong Li, Shawn Im, and Sharon Li. Visual instruction bottleneck tuning. arXiv preprint a00. 13946, 2025b.
13

---

Published as a conference paper at ICLR 2026
OpenAl. Gpt-5 system card. Technical report, OpenAL, August 2025. URL https: //dn..
openai , Syst---psstte---e---ss-- card. pdf.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacidin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. Featured Certification.
Misra,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pp. 18- 34, 2024.
Lisa Schut, Yarin Gal, and Sebastian Farquhar. Do multilingual LLMs think in english? In ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025.
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. In Forty-second International Conference on Machine Learning. 2025.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega: A question answering challenge targeting commonsense knowledge. arXiv preprint 1rXiv:1811.009. 2018.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint a50ivv:503. 19786, 2025.
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self-improvement of Ilms via imagination, searching, and criticizing. Advances Neural Information Processing Systems, 37:52723- -27448, 2024.
Peter Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: 1: A fully open, vision-centric exploration of multimodal 1lms. Advances in Neural Information Processing Systems, 37:87310-88 01-8111--736666 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024b.
Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, and Neel Nanda. How visual rep- resentations map to language feature space in multimodal llms. arXiv preprint arXiv:2506. 1976. 2025.
Aayush Atul Verma, Amir Sacidi, Shamanthak Hegde, Ajay Therala, Fenil Denish Bardoliya, Na- garaju Machavarapu, Shri Ajay Kumar Ravindhiran, Srija Malyala, Agneet Chatterjee, Yezhou Yang, et al. Evaluating multimodal large language models across distribution shifts and augmenta- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5314-5324, 2024.
An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang. Anh Totti Nguyen, and Daeyoung Kim. Vision language models are biased. arXiv preprint arXiv:2505.2394/ 2025.
Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yer Kan. V-DPO: mitigating hallucination in large vision language models via rision-guide direct preference optimization. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA. November 12-16. 2024, pp. 13258 13273. Association for Computational Linguistics, 2024.
14

---

Published as a conference paper at ICLR 2026
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. National Science Review, ((11))mwae443, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556 9556-9567, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pp. 11975-11986, 2023.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gor- don. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31, 2018.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
Related Work
20
B Discussion on Visual Integration Point
20
B.1 Note on VIP determination
20
B.2 Algorithmic Estimation Visual Integration Point
21
C Implementation Details
21
Full Results of the Main Experiment
24
E Further Analysis
24
Details on Theoretical Analysis and Proofs
28
F. Justification and Interpretation Representation Divergence
................................
28
F2 Notations and Problem Setup for Theorem F.6
29
F3 Proof for Theorem R.6
30
G Impact Statement
31
Disclosure of LLM Usage
I Ethics Statement
31
Reproducibility Statement
32
16

---

Published as a conference paper at ICLR 2026
LLaVA v1.5 78 on MME
LLaVA NeXT Vicuna 78 on MME
LLaVA OV Qwen2 78 on MME
Br (hop TVI-5-060)
Br (Avg TVt-0 075)
0.25
Dr ldeg TV6-0 (Rog Tol.6.00))
0.30
0.20
0.15
0.35
0.15
0.10
0.45
0.00
0.01
0.40
0.00
0.00
Layer
LLaVA-v2.5-78 MMBench
LLaVA-MeXT--ccuaa-D MMBench
LLaVA Ww-Qwen2 MMBencch
Dar (lig Toloo 1413
Dvr (Aop 7VI-9.315)
Dur (Avg Tvl-0.099)
0.15
0.10
0.19
0.05
0.03
20
24
15
20
Layar
Layer
Layer
LLaVA-r1.5-7B on MM4ttr
LLaVA-NeXTWVcuun-7B MMStar
LLaVA- DV- DV-Qwen2-78 on MHStta
0.20
Dr (Axp TV1-5-3001
0.15
(Sog T.1--6475)
0.350
TVi escl
Dur llvg TVieo 0533
0.35
0.10
0.380
0.075
0.05
0.000
0.50
0.00
0.000
Layer
LLaVA-v1.5-78 MMMU
U.V--NoTTViiuma-78 MMMU
0.35
Dyy (6og TVI-5.105)
0.30
0.00
TVi-3-550)
D. lAep 1V0- Tvi-0.651)
0.00
0.05
6.00
0.00
0.00
20
25
Layer
Layer
Layer
LLaVA v1.5 7B on VLind Bench
LLaWA-WeXTVVccuma-B on VLind-Bench
LLaVA-OV-- WW--wwn--70 7: on VLind-Bench
0.20
Der (Kog TVt-o 1983
0.20
D- CAug TVt-0 678)
Dr. ldep TV6-o (82) Byn (Aog (1l-6.155)
0.35
0.15
0.15
0.30
0,30
0.08
0.01
0.00
32
50
Layer
LLaVA-v1.5.. 5-
N on VILP
LLaVV OW-Qwen2-28
(Aog TVI-9.8721
Dyr (drg TVI-6.129)
TVl-0.087)
Dy ldvp TV 0.032)
0.35
0.12
0.55
0.00
0.07
24
20
Layar
Layer
Layer
Figure 6: Complete experimental results. (Part 1) LLaVA-v1.5-7B. LLaVA-NeXT- Vicuna-7B,
and LLaVA-OV-QWen2-77B on six datasets.
17

---

Published as a conference paper at ICLR 2026
Gamma-3-48
SmolVLM MME
Llama-3. 2-11B-iiinnn
0.02
By (dog TVl=0.020
(dog
0.25
By (Aug TWt-0 10sg
B. (dog 41-0
0.20
5.02
0.15
0.0
12
18
30
Layor
Gamma-3-48 MMBBanch
SmolVLM MMBench
LIama-3.2 1amma-3.-11B-vissii MM4ench
0.43
(áve 1Viud.0311
0.15
Ovr (Aug Tv 1833
0.6
(Sag TVies 4991
0.4
9.32
0.05
3.50
0.05
Layer
Layer
Layer
Gemma-3-48 MMStar
SmolVLM on MKStar
Liama 3 (aamm-332-11--iiiiaeo on MMStter
0.025
Dr ldvg TVoo
0. (0og TWte 270) De TVI-1.3761
Der lAvg TVi-0 433)
Dyr (Aug Tvi-6.079)
0.320
1.00
0.018
0.330
059
12
1.5
24
50
Layer
Gamme-3-48
SmoVVLM
0.15
Pi (Avg To0-5.c01)
(evg TVi=0,203)
D. (igg
0.015
0.6
TVI-8.660
0.03
0.800
0.00
24
Layer
Layer
Gamma-3-48 VLLndd VLLindBeech Bench
SmolVLM VLLnddBBenh
Liama 3.2 11B Vision on VLindi Bench
Tet-o.600)
0.03
Dur (Avg TV--o 0313
Our (Svg (V1-5-380)
Dur (kop TVi-8-495)
0.02
0.01
0.05
0.00
1.
24
30
10
155
25
Layer
Gemma-3-4B
SmotVLM on VILP
Liama-3.2-11B-Wttten VILP
Pi (Avg TVl-0.118) (10)
0.18
Di (lug TVt- 2013
0.010
0.005
0.000
0.00
0.0
Layer
Layor
Figure 7: Complete experimental results. (Part 2) Gemma-3-444m SmolVLM, and Llama-3. 1.2ma-3.2-111 3.2-11B. Vision on six datasets.
18

---

Published as a conference paper at ICLR 2026
Swen2.55VV--7 MME
Eagie2.5
InternVL3 MME
Our lovg 1V1.0-3743
0.30
0.25
Dur (dep 190.0.350)
0.25
Dus (4og Tat-o 169)
0.15
0.350
0.05
0.00
0.00
10
15
25
25
1.5
20
25
Layer
Layer
Layer
Qwen2.5-VVL 2- on MMBench
Eagle2.5-58 on MMBench
InternVL3--8 MMBanch
0.125
By (Avg (40)
By (Aug Tet-o 1111
By laxg (i1u
0.15
0.10
0.05
0.00
0.00
Layer
Layer
Layer
Qwen2-5-VE-70 on MiSStar
Eagle2,5-88 on MMStar
InternVL3 BB on MMStar
0.30
D- lAxp TV1=0-254)
By dlog Tut-o 681)
Dr lhg TV--0 (70) Dur (Aog To 677)
0.00
Bur (Avg (Vi-0-166)
0.33
9.00
0.30
0.04
0.02
0.05
0.01
0.30
Layer
Qwen2.S-VL-7B MMMU
Eagle2.5
InternVL3-88 MMMU
0.08
0.00
Der (Rng TVt-o 028)
TViub-10d) 1661
0.10
Dur dkip Tvt-o 648)
0.34
0.30
0.00
0.00
20
20
25
Layer
Layer
Qwen2.5-VL-78 VLind-Bench
Eagle2.5-80 on ViLind-Bench
InternVL3-BB on VLind Bench
d- (Aug TVl-
D. Dlvg TVt-0 144)
0.25
Dy ldeg TVi-0 039)
0.100
0.20
0.075
0.15
025
0.05
0.000
0.00
Layer
Layer
Layer
Qwen2.5-VL-78 on VILP
Eagle2.5-48 on VILP
InternVL3-88 VILP
0.30
Our (fing o o43)
(Avp TVI=9.8271
0.20
Dy (log Tvt-o 1031
Byr (Hog TVi.8.120)
0.25
Pu. daug Tat- 1145
(Vi-0,020)
0.15
6
0.50
0.00
3.5
20
25
Figure 8: Complete experimental results. (Part 3) Qwen2.5-VL-7B,, 7agge.--.5. SBB, and Intern VL3- 8B six datasets.
19

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Visual perception in LVLMs. Most mainstream LVLMs (Liu et al., 2023; 2024a;b; Bai et al,, 2025; Dai et al., 2023) adopt backbone, a language model that processes both image and text tokens, and a modality adapter that aligns visual representations with the language space. The visual understanding capabilities of these models largely depend on the perception quality of pre-trained vision encoders (e.g., CLIP (Radford al., 2021 1), SigI.IP (Zhai al., 2023)) and the reasoning ability of large-scale language models.
architecture comprising three key components: vision
Despite promising results on some multimodal benchmarks, this paradigm has been recently chal- lenged due its poor performance on vision-centric tasks, suggesting that these models often fail to truly see the image (Tong et al., 2024b;a; Vo et al., 2025). A growing body of work has sought to un- cover how visual perception operates internally within LVLMs. For example, Bi al. (2025): Jiang et al. (2025); Neo et al. (2025) examine the layer- layer-wise attention patterns and identify distinct phases of visual integration, typically emerging trained sparse autoencoders (SAFs) as analytical tools to show that visual representations gradually align with language representations over depth, converging in the later layers. Complementarily, Fu et al. (2025) apply probing techniques and argue that language decoders in existing LVLMs struggle to effectively leverage the visual features produced by their vision backbones. Their findings suggest that the bottleneck lies not in the availability of visual information but in feature misalignment.
the mid-to-late layers. Venhoff et al. (2025) utilize pre-
Language prior in LVLMs One of the most prominent limitations of current LVLMs is their tendency to overly rely on language priors, often generating plausible outputs without grounding in the visual context. This behavior- commonnly referred to as the language prior problem- drawn increasing attention. Recent studies attempt to evaluate this phenomenon by designing datasets that stress-test visual grounding. For example, Lee et al. (2025) and Luo et al. (2025) construct datasets with counterfactual visual inputs to assess whether models can disentangle visual signals from misleading linguistic cues. Similarly, Deng et al. (2025) test modality conflict scenarios to evaluate the model's 's preference between text and image inputs.
While these works help reveal the presence of language priors, they offer limited insight into the underlying causes. Most current understandings of language prior are based on two widely adopted assumptions: (1) it manifests high similarity between outputs with and without visual input (Chen et al., 2025b; Xie et al., 2024), and (2) it arises due to insufficient attention allocated to visual to- kens (Liu al., 2025). Building on these assumptions, several works propose methods to miti- gate language priors such as contrastive decoding (Favero et al., 2024) or inference-time attention reallocation (Liu et al., 2024c). Others, like Chen et al. (2025b) and Xie et al. (2024), explore training- time interventions that penalize outputs overly aligned with the model's 's default language predictions.
B DISCUSSION ON VISUAL INTEGRATION POINT
B. NOTE ON VIP DETERMINATION
In Eq. 3, we defined the VIP I* based on the pre-I* and post-I" representation divergences (P Fo) - -D,(P,, Fe) where the pre-l" chain-of-embeddings shows nearly zero representation divergence whereas the post-l" chain-of-embedddin vergence defined by positive т. In practice, we can not access the population distribution Pyr and Pr. therefore we can not compute the truth expected representation distance D; Fo). What we do in practice is estimate that expected representation distance with finite samples Dvr and Dr, and see the evolution of representation distance gap to manually pick the VIP (". We observed that this empirical estimator of representation distance (Eq. 2) works well as measure for determining VIP in general, yet there can be some cases where the fitness of the estimator is bad, e.g., Figure 10. However, even in that case, the point I*, where the empirical representation divergence exceeds positive constant for all subsequent layers, consistently emerges, and the TVI (Eq. 11) is calculated based on that 1* becomes strong indicator of I.P.
exhibits an effectively large representation di-
20

---

Published as a conference paper at ICLR 2026
Table Comparison of VIP detection methods and their effectiveness (Spearman correlations between the resulting TVI and downstream correctness).
Model (Layer #)
VIP Detection
MMBench
VIP (p)
ViLP VIP (p)
18
18
(wen2.5-Vi.-2 (28)
Manual
(0.6335)
(0.6335)
18 (0.6335)
19 (0.6336)
Owen2. 5 (28)
Variance-based
20
20 (0.7970)
Gemma-3-4B (30)
Manual
(0.7970)
16
16 (0.7973)
Gemma-3 -3-43 (30)
Variance-based
(0.7973)
16
16 (0.5709)
InternVL. 8B (32)
Manual
(0.5709)
17
20
InternVi.3 88 (32)
Variance-hased
(0.5749)
(0.5949)
B.2 ALGORITHMICC ESTIMATION ON VISUAL INTEGRATION POINT
As discussed in $3.2, identifying the VIP provides valuable insights into when visual information begins to influence answer decoding. In most cases, we rely on empirical observation of the repre- sentation distance curves manually determine 1", which proves be straightforward and inter- pretable across a wide range of models. However, in situations where manual inspection imprac- tical c.g., for large-scale model comparisons automated pipelines estimate the VIP in a data-driven, automatic way. To this end, we introduce an algorithmic rule that can automatically estimate the VIP given model and dataset. While not required for our core analysis, this estimation method serves as practical tool in settings where manual identification of is infeasible.
may be desirable to
Below is an estimation strategy based on the test statistic we discussed in Eq 12 of Lemma F.1 by specifying arbitrary significance levels that user prefers. To specific, given pooled sample
ofx standard deviation â = P24
oi Dell
we can define the estimated VIP as follows,
Di Fo) - (Dr,
B
DD Fe) (D Fe)
viP(D,FF) arg min IELLL
where oî,1 Exery (d( (d(( FoD,
is sample variance on Dr and of is similarly de- fined. Intuitively, the quantity on the left-hand side denotes a deviation-normalizee normalized representation distance in the layer I, whereas the right-hand side means the historical average of those distances with weighting coefficient 3, which was set 2.0 in our case. The above algorithm can be applied to any given dataset, and we conduct comprehensive experiments to show its robust performance across different models and datasets in Table and Figure 9.
[Dil-1
From Table 6 and Figure 9, we can see that VIP estimated by the proposed algorithm are quite close to the result of the manually selected one and clearly mark where two curves start diverge (based the divergence plots), and the Spearman correlations between TVIs computed by those estimated VIPs and the downstream prediction correctness are robust across different datasets.
C IMPLEMENTATION DETAILS
Models. We evaluate our framework on 10 publicly available LVLMs, covering a diverse range of architectures and training paradigms. For all models, we use the official estruction-tuued check- points available on Hugging Face". To ensure consistent comparison across models, we set the generation temperature to 0.
hhttpe://huggingee co/ htt//veen/veeeee w-w.://ww.......-1--2 (ew.il-8-Insttraan Inst https://huggii ngface wow.eaaaooo/////aaaaaaaa -15--6--11
21

---

Published as a conference paper at ICLR 2026
Py: Llog (V-00.08))
Pu (Reg Tried 0111
0.20
Ru. (6ng (Voe6.81)
0.10
0.00
20
20
32
20
Gamma-3-1 3-48 on VILP
InternVL3-88 on VILP
0.025
O- lAvg TVp-0,036)
200
D. (kap Tei-00079)
Dur lóvg TVied 043)
0.30
Layer
Layer
Manuelly Selected
Automatically VIP (Variance-based Detection)
Figure 9: Comparison between the manually selected VIPs and automatically selected VIPs.
Datasets. Our evaluation spans 6 benchmark datasets, each consisting of either binary ('Yes/No') multiple-choice questions. This design ensures that the hidden state of the final token- used for representation distance calculations- iss closely tied to the model's reasoning process. For MMMU, use the validation set and filter out samples that involve more than image or open-ended output to ensure consistency in the evaluation setting. Also, when constructing the prompt, we do the provided explanations for fair comparison. For ViLP, we consistently select image3 and answer3 to curate our (counterfactual) VQA pair. For VLind-Bench, we convert each annotated counterfactual statement into a binary "Yes/No' question. To perform this transformation, we use the advanced language model G with the following prompt:
Generate a question based on the counterfactual information in the given statement. The question should be answered by yes.
Here are some examples:
Statement: The Statue of Liberty is holding sword instead of a torch. Question: Is the Statue of Liberty holding sword?
Statement: The Sydney Opera House is illustrated as underwater aquarium, with fish swimming around its structures. Question: the Sydney Opera House underwater?
Statement: The Leaning Tower of Pisa is perfectly vertical in the image, without any tilt. Question: Is the Leaning Tower Pisa perfectly vertical? Now generate question for the following statement: (statement)
The models are instructed to directly generate the answer under a zero-shot setting, without involving any reasoning steps. Additional statistics of each dataset and corresponding splits provided in Table 7.
To further validate the reliability of the agreement-based separation introduced in $3.2, and to demonstrate that our analysis can generalize to scenarios with known visual dependencies, we con- struet a controlled baseline dataset. Specifically, we take questions from CommonsenseQA (Talmor et al., 2018), which are inherently language-only and pair each question with randomly selected, irrelevant image from COCO 2017-val (Lin et al., 2014), forming a synthetic VQA setting that does not require visual input. We treat this as our vision-independent group Dr. For the vision-
Instruet
22

---

Published as a conference paper at ICLR 2026
Qwen2.5-VL-7B MME
Qwen2. 5-8L-78 on MMBench
Qwen2.5 L/7L-7 LB on VLind-Bench
0.20
Bazelina (hog TVI-6.064)
Baseiine (dvg TVI-6,66G)
0.15
(eop 19110.081)
9.10
0.13
0.30
0.41
0.00
0.00
0.00
Layer
Layer
Gemma 3-48 on MME
Gemma 4B on MMBencc
Gemma 3 44 on VLind Banch
0.010
Easuliss (leg TViv0_0333
Bxsdlice (fng TVl-8-5333
0.000
Bsesline chep Toleo,
0,025
0.023
9.915
0.015
0.01
9.055
0.003
0.000
0.003
30
Layer
Layor
Layerr
Figure 10: Experimental results under controlled settings. We use a synthetic baseline con- structed from CommonsenseQQ questions paired with irrelevant images as Dr wrision-ndeeendenn)) while standard VQA benchmarks (MME, MMBench, and VLind- hind-Bench) are used as Dvr (vision- dependent).
dependent group DyT, we samples from standard VQA benchmarks such MME, MMBench, and VLinn-Bench, which typically require more grounding in visual content. As shown in Figure 10, the average TVI is significantly lower for the baseline Dr compared to the general VQA datasets, confirming that the model does not extract meaningful information from irrelevant visual input. In contrast, even in the presence of strong language priors, the model still benefits from image content in Dvr. It is also worth noting that there is a reverse trend before VIP for Qwen2. 5-, indicating representation distance before VIP is not associated with the actual meaningful visual integration during decoding. These findings are consistent with our previous results and further sup- port the effectiveness of our proposed separation framework. Nonetheless, ensure better control over data attributes such format and context length, and to reveal clearer trends, we continue to use the agreement-best based separation strategy in the main text.
Table 7: Dataset statistics. M stands for multiple-choiee and B stands for binary-choice (Yes/No). Qwen2. 5-V----L77- VB used as an example here.
Statistics
MME
MMBench
MMStar
MMMU
Veind-Bench
ViL.
Question Type
B
M
M
M
B
M
[D]
2374
4377
1500
805
418
300
'Dvll
546
2782
1057
446
144
177
Drl 1828
1595
443
359
274
123
Dvrl/|Drl
0.30
1.74
2.39
1.24
0.53
1.44
Metrics. All TVI values reported in our experiments computed based on empirically de- termined Visual Integration Points (VIPs) specific to each model. The following VIPs are (16), Gemma-3-4B (20), Gemma-3-----m 33-1BB (26), Gemma-3-277 (35), LLaVA-v1. (9), Eagle2 5-8 (15), L1 2itaa-..-11B-visio. (12), -LLaVA--eeT-vicund
used: Qwen2. 5-VI (18), InternVL3-
(12), -LaVA-OV-Qwe:2-777 (15) and SmolVIM (15)". We
also provide an automatic method for estimating VIP in Appendix B.2 for potential practical usage. Representation distances are computed using the hidden states corresponding to the last generated token. The metrics introduced in $4 are computed as follows:
Visual Attention
(L.) Output Divergence = dd( Zlind)
(10)
LH
'It should be noted that these manually selected VIPs are not necessarily optimal: however, they already achieve strong and robust effectiveness and are sufficient for analytical purpose. The truly optimal VTP is likely lie in their vicinity.
23

---

Published as a conference paper at ICLR 2026
Table 8: Summary of evaluated models and their architectural specifications.
Model
Fusion
Layers
Hidden Size
MLP
28 64 80
3584 5120 8192
Gemma-3-4B/12B/277
Q-Former
34 34/48/ 62
2560
3840 5376
InternVL3-8B
MLP
28
3584
uLaVA-v1.5-7B
MLP
32
4096
Baglc2.5-88)
MLP
28
3584
Llama-3,
X-Attention
40
4096
LlaavA-NeeX--Viccna-----
MLP
32
4096
1LaVA--V--Qwan2-77
MLP
28
3584
SMolVLM
MLP
24
2048
Instruct www.uuccLLI-VViiiie
Q-Former
32
4096
the mi-ddes:
they rel-catt
thee
the Teeeka
0.00
Figure 11: Extended experiments across models of varying scales and fusion architectures.
where a (1.b) denotes the total attention from the final generated token to all preceding visual tokens in head at layer and - represents the hidden state at the final layer.
D FULL RESULTS OF THE MAIN EXPERIMENT
We provide the complete experimental results on 9 LVLMs and 6 datasets in Figure 6, 7, 8. Across all models and datasets, a consistent existence of the VIP can be observed. However, for some models, such SmolVLM, the divergence between Dyr and Dy is less pronounced, likely due to the model's limited capacity and thus less promising visual integration.
Additional architectures & scales. We further validate findings models with diverse with cross-attention based multimodal
chitectural designs, including fusion and Instructin wwwooonn---- wiw.-i--iiuna---- with Q-Former-style fusion mechanism, well as on larger-scale models such as Qwen2. 5-VL-33 32B and Qwen2. 2.5 5-VL 722.. As shown in Fig- ure 11, the results consistently corroborate our conclusions across both architectural variants and model scales. Comprehensive statistics for all evaluated models are provided in Table 8.
E FURTHER ANALYSIS
Analysis on the limitations of attention-based and output-based LP analysis. First, for attention-baue based methods, we argue that a higher visual attention weight does not necessarily im- ply better visual grounding. As shown in Figure 12, under weak language priors, the model is able to correctly attend to the key areas that are semantically related to the given instruction. However, under strong language priors, we observe a pathological attention pattern in which the model's atten- tion becomes abnormally concentrated in limited region of the image. We refer to this phenomenon as an attention sink. In such cases, although the aggregated visual attention weight appears high, does reflect meaningful visual processing. Instead, the model is effectively bypassing genuine visual understanding by fixating on irrelevant or static regions, thereby undermining the utility of attention-ban -aased metrics reliable indicators of visual integration.
24

---

Published as a conference paper at ICLR 2026
Weak Language Prior
Strong Language Prior
Orgaddeep
Sernerkee
Oriiial langg
Instartin
cupe dancu tox odd cudpated
preeepin
evea hoadits saade aarne caetic boad Only
Antrottee
(,,5
(sunt())
peu when hot one slamn whet
Ced senver durepton
Map
May
Allecian May
Figure 12: Visualization of visual attention maps under weak and strong language priors.
Qwen2.5-VL-7B MMBench&.ViLP
Qwen2.5-VEL B-VLL-B on MMBench&ViLP
Weak (MMBench) Strong LP (VILP)
Weak LP (MMBench)
Strong LP (ViLP)
Lagh kre up
SapusSuc
10
20
10
20
25
Layer
Layer (b)
Figure 13: Layer-wise representation distances in latent space vs. output space. We apply the logit-lens to project hidden states at each layer into the output space. In (a), distances are computed over the entire output vector, while in (b), they are restricted the top-k token positions correspond- ing to candidate answer options.
To better understand why output-based representations are less effective in capturing language prior behavior, we visualize how representation distances vary across the latent and output spaces. As shown in Figure 13, the projection from the latent space to the output token space tends to obscure semantic distinctions that otherwise indicative of the model's underlying behavior-such as whether is performing effective visual grounding or defaulting to language priors. This observation aligns with our earlier argument: surface-level outputs alone may not faithfully reflect the internal decision-making process of LVLMs. Instead, meaningful behavioral signals often reside in the deeper latent representations, emphasizing the importance of analyzing internal dynamics rather than relying solely output-level comparisons.
Ablation aggregation strategies for TVI caleulation. In our main experiments, adopt simple aggregation strategy for computing TVI by averaging the representation distances across all Post-VIP layers. To assess whether more sophisticated aggregation may improve the metric,
25

---

Published as a conference paper at ICLR 2026
additionally experiment with standard -eriedddeviii-bbbeieeeeeee based weighted TVI, defined as:
I -- +
or dizvis <bhina]]
(11)
where 0i denotes the standard devia- tion of representation distances layer I. This weighting scheme emphasizes lay- whose distance distributions exhibit higher variability, i.c., layers that are more sample-specific and potentially more in- formative, while down-weighting contri- butions that may arise from less discrim- inative layers. As shown in Table 9, this reweighting provides a slight improve- ment in effectiveness over the simple aver- aging strategy across both VLind and ViLP. However, the gains are modest and come at the cost of additional computation. These findings suggest that our original simple averaging approach already serves as a strong and robust summary statistic, while the weighted variant may offer incremental refinement in specialized scenarios.
Table 9: Comparison of aggregation methods for computing TVI.
Model
Aggregation
VLind
ViLP
Owen2.5-VI--
Simple Average
0.7155
0.6335
Qwen2. Swen2.5-VL-D 5-VL-7B
Std Reweighting
0.7164
0.6348
InternVL.3-B InternVL3 8B
Simple Average
0.6727
0.5709
Std Reweighting
0.6739
0.5723
Analysis instruction-level perturbation. To strengthen study of the contrasting chain-of-embeddingg we compare embedding chains produced by image-text inputs versus image- only inputs (Figure 14 (a)). As expected, the two curves exhibit only negligible differences, likely due to the limited semantic con- tent of image-only inputs. This phenomenon can be attributed to two main factors: (I) The hidden states from image-only inputs do not reveal anything about the model's answer prediction, considering that the instructions provided. The result- ing representations thus are not directly comparable in the same behavioral space. (2) Most current LVLMs are not trained to handle image-only inputs for question answering, which makes the model's behavior on image-only inputs unpredictable. Nonetheless, even under these constraints, we still observe that the revised TVI, which now specifically captures the contribution of textual content to the decoding process, is marginally higher for ieiion-ideeeendent samples than for vision-dependent ones. This further corroborates the robustness of our analysis framework and the consistency of our conclusions.
Owen2,5 VL. 78 on VILP
Qwen2,5 VL 78 VLand Bench
B. dlee 3331 Yo0-6.300)
B. (Avg 2000
0.0
Qwen2255 VL 78 en WLind Bench
Qwen2,5VVL 78 on VILP
0.12
Levs Niesort devs 0521 Valcerrric
6,125
Lare kioced bes (lng Riioeettee (001))
1.00
0.375
0.00
(b)
Layer
Figure 14: Impacts of instruction-level perturbation. (a) Representation distances obtained by contrasting chain-of- embedding sequences produced with and without textual in- structions. (b) TVI scores under instructions with different styles
To further isolate the influence of the textual component in multimodal inputs, we investigate how different instruction styles affect model behavior. Specifically, contrast misleading instructions (those originating from VLind-Bee "Lind-Bench or ViLP that elicit strong LP) with generic, vision-centti prompts such as "Describe the image in detail." As results in Figure 14 (b) show. TVI under these vision-centric instructions significantly higher than under misleading ones. This confirms that TVI is sensitive to the model's bias toward language priors under different instruction regimes.
Analysis on VIP & TVI across different training stages. To better understand how VIP and TVI evolve during model training, we analyze checkpoints from multiple stages of LLaVA-v1.5 visual instruction tuning. As shown in Figure 15 and Figure 16, our evaluation reveals two key
26

---

Published as a conference paper at ICLR 2026
LANEAAL D-F DNGFORT
LLAMANS SUB DOOOLRIN
Dnd
Figure 15: Evaluation results across different stages of LLaVA-v1.5 visual instruction tuning.
Checkpoints of LLaVA-v1.5 ViLP
0,100
0.095
0.090
Avg
0.000
0.075
0.065
500
1000
1500
2000
2500
3000 Steps
3500
4000
4500
5000
Figure 16: Average TVT on ViLP across different stages of visual instruction tuning.
observations. First, the VIP position remains remarkably stable across training stages (around the 12th layer). This suggests that the mechanism governing where visual information is integrated is largely established during pretraining and persists throughout subsequent finetuming. In other words, VIP appears to reflect an intrinsic architectural property rather than a behavior shaped by instruction tuning. Second, TVI exhibits a clear upward trend training progresses, indicating that the degree to which the model effectively incorporates visual information improves gradually during instruction tuning. These findings align with our previous conclusion that VIP is a model-specific property as well as our expectation that visual instruction tuning incrementally enhances the model's multimodal fusion capability.
Analysis on when TVI fails. Althrough TVI shows strong correlation with answer correctness as shown Table 3, there are still some non-neglectable portion on failure cases that TVI can not properly prediet the model's 's answer quality. Conceptually, TVI is designed to capture the degree of effective visual integration during decoding. While higher TVI generally implics that the generated answer is more grounded in visual input and thus more likely be correct, there are natural regimes where this relationship weakens, for example: (1) Some questions are not strongly visually demand- ing; the model can answer them correctly with relatively little visual integration. In such cases, the answer may be correct even when TVI is small, leading to "false negatives" from TVI's perspective. (2) For very challenging visual questions (e.g., fine-grained recognition, subtle spatial reasoning, or interpreting small/occluded objects) the model may still fail even after substantial effort to integrate visual information (high TVI). This yields "false positives" in terms of Co-coorresssess alignment. These factors introduce inherent noise into any evaluation based on TVI vs. correctness.
To further characterize these patterns, we visualize the TVI distribution for ViLP samples that Qwen2.5-VLL-7 answers correctly and perform a focused analysis on those with unexpectedly low TVI. As shown in Figure 17, while the majority of correctly answered samples exhibit relatively high TVI, a small subset attains low TVI despite being answered correctly. Upon inspection, many of these cases can indeed be solved without relying on visual input, either due to knowledge leakage or lucky guessing. This suggests that such questions intrinsically less visually demanding, and the low TVI observed in these cases does not constitute a failure of the metric but instead reflects the fundamental properties of the task itself.
27

---

Published as a conference paper at ICLR 2026
Bees collect honey, What do bees as in picture?
Qwen2.5-VL-7B on ViLP
A. Honey B. Fork Nectar
15-
Visual (TVI=0.0223): C
Blind: C
Count
100
A girafle's s neck usually straight. What shape giraffe's neck in this image?
Straight
Zigzag
Visual (TVI- 0.0338): C Blind: C
0.02
0.03
0.04
0.05
0.06
Figure 17: Visualization of TVI distribution and case studies on TVI failure cases.
F DETAILS ON THEORETICAL ANALYSIS AND PROOFS
F.I JUSTIFICATION AND INTERPRETATION ON REPRESENTATION DIVERGENCE
We first show that empirical estimate for the difference in the expected representation distances (Eq. 3) can be viewed as two-sample test statistic with asymptotic normality in Lemma F.1.
Lemma F.1. Let X vX.. X2) X be a random variable sampled from Pvr or Pr. and denote Dvr Pvr and Dy Pr as empirical distributions with N and M i.i.d. samples, respectively. Given stuck of LVLM layers fi X - Z from Fg and distance metric d Z Z R with a finite second moment, the difference in the expected representation distance estimates Di(.,) between Dyr and Dy is a two-sample test statistic with asymptotic normality, that is,
o7 + ovr M
Di(Drv, Fe) - Di((D,, Fb) N( LVT.
as N,M + 00, co.
(12)
N
where NMVT = PPr, Fo), pT = Di(Pr, Fo) and our = (urp, X.), f((x.)) o7 = Varp, Xt), f(X:)) co.
< 00,
Proof. Let IvT and Jr be the index sets N and M i..d. samples from Pvr and Pr, respectively. If samples from Pvr and Pr are independent, with finite second moments, we have,
DD Fo) Di(r,, Fo)
E [d(f((z), 1t), f((i)))
(13)
N
M
of
~N(PT PVT:
(by Central Limit Theorem as N, M - oo)
M
N
Note that, even though the samples from Pvr and Pr are not independent, the asymptotic normality still holds by considering covariance terms between the two distributions.
Now, in Lemma F.2, provide interpretation of our representation distance measure between CoE by casting the distance measurement as a density estimation problem.
Lemma F.2. Let X = (X., X₂) E X be random variable sampled from PP or Pr, and ft X + Z be a stack of LVLM layers. For Pr. define a density estimator (r(ZZ) = (z): fi(X.), I). Given
a squared l2 distance d(Z,, Z2) 1 1 Zzl 2. the representation distance (X X,), f((X))
is the negative log-likes -iikelihhodd estimate of Z' from P7, denoted pr(Z'), up additive constant. That is,
d(f((X,, X1), f(X.)) A logg + log C,
(14)
where C = (2TT) is normalizing constant for the d, dimmennsioa unit-variance Gaussian.
28

---

Published as a conference paper at ICLR 2026
Proof. is easy check,
2T(2') = N/Z'; Zolind 1)
(15)
Pr(Z²) C. C.xpp( 11 7 11
(16)
log - log C 11 -hhtiall
(17)
2 logrr(Z') = log C d(Z', Zhiina)
(18)
where Zilind (x(X.).
In other words, the distance between the original representation z'= fx X,) and the blind one Ziind (ix, can be expressed as probability density estimate of Z LivenN((s Zoind I), which has mean Zilin with the isotropic variance, as our estimator. On top of this new framing, i.e., distance measurement as a density estimation problem, we further provide an information-theoreti interpretation of the difference in expected representation distance in Theorem F.3.
Theorem F.3 (Restatement of Theorem 5.1). Let X = (X.. X.) X be a random variable from Pvr or Pr. and fi : X + Z be a layer stack from an LVLM Fg. For Pr. define density estimator pr ) == (Z fi(X,), 1), and denote Pvr(Z") (resp. pr( )) as the population distribution on
Z fi(X) derived from Pvr (resp. Pr). Then, given [1, Z2) == 1 1 Z2ll2, the difference
in the expected representation distances between Pvr and Pr, i.e. D (PTT. Fa) D,(PT, Fo), can
be expressed as follows,
(Z) (I((p(()))1)(((()) H,
(19)
where is constant H (pvr( - - (pr(Z))) and kL(,Il-) denotes the KL. divergence.
Proof.
D: (P(T, - (P,,,
(20)
-Epr(z))| log prl (z)) log C7 - - + log C7
(21)
(Fpr((2))- /o (Z'))
(22)
= (.(PT(Z"),TT(2))
(ipp(('),T((2))
(23)
= [KLL(PVT( (Z' (z)rr + H (pvr(Z))
+ (pr(Z'))
(24)
where H(-) and (..-) denote entropy and cross-entropy, respectively. Here, Eq. 21 holds by Lemma F22, and the remaining equality is trivial by the definitions d() and information theoretic measures.
This simple theorem gives us a new interpretation on the measure of representation divergence, (P Fo) DP(,, Fo): the amount of expected excess surprisal when assume that the sample representation follows blind representation-centerd normal distribution compared to the true population distribution PVT. compensated by estimation quality (L(pr
F.2 NOTATIONS AND PROBLEM SETUP FOR THEOREM F.6
We recast the problem of measuring the representation distance di d(Zvis. Zuliia) as a binary classi- fication task, where we want classify the sample (Zviss Zisina) into if it originates from the distribution Pvr while 0 for the samples from Pr.
To be specific, let X, Z, and y denote input, LVLM representation, and output, respectively. We have a multimodal input query X = (X.,, X,) E X, stack of LVLM layers fi X - Z, and distance metric d: x Z + [0. 1]. With that, we define a hypothesis d(f((Xe,, f((,))
X' > J0, 1] real value function to measure the relative likelihood that the input X is sampled from Pvr rather than Pr, and we also define the labeling function h" X + {0, 1] that maps the input into its ground-truth membership. ie., 1 if it's from PyT and 0 if it's from Pr. Then,
29

---

Published as a conference paper at ICLR 2026
formulate expected error (a.k.a. risk) of hypothesis h w.r.t the labeling function h* distribution P as follows: Ep(h, h*) == (x-xxll(()) - (x)I]..
Besides, in Def. F.4, introduce measure of discrepancy between two distributions, H- divergence, which has been widely adopted in domain adaptation literature (Ben-Date Deen-David et al., 2006; 2010; Ganin et al., 2016; Zhao et al., 2018 8), and also VLM fine-tuning regimes (Oh et al., 2024).
Definition F.4 (H-divergence, (Ben-David al., 2006; 2010)). Let P and P' be probability distri- butions on the input domain X, and H be hypothesis class for X. Denote An := =1)=(11 E H} as a collection of subsets of X that are the support of some hypotheses in H. Then, the distance between P and P' based on H is defined as follows:
(P, P') =2 sup
[PP|A] Pp [A]]
(25)
AEAx
Now, we are ready to present the proof for Proposition 5.2 in the next subsection.
F.3 PROOF FOR THEOREM F.6
Theorem F.5 (Restatement of Theorem 5.2). Let X = (X X2) E X be a random variable of a multimodal input query. Given a stack of LVLM layers ft : + Z and a distance metric d: Z x Z + [0, 1]. define hypothesis h e these hypotheses H that has a pseudo-dimension c. Then, for Dr(P., Fo) == Exxp, [h(X)) with any Pvr, Pr, and PM == Por+Pr and the empirical distributions Dvr ~ Pvr and Dr ~ Pr of N samples for euch, we have the following boundy w.p. at least 1 for 1,
v, X1), fi()))) X + [0, 1] and set of
1 i) 1 - DDD,,Dr, Fo) 2 (Dvr Dr) Ös s Di(Pvr,Fo))
(26)
1
1 + dn (DvT, Dr) + +08
i) 2 (Dvr. Dr) Ö, < (PMM Fo) s 2 4
(27)
where H := ([x(X)) h, E H,O < t < 1} and Ô₂ Olv (log 1/2 + c ))
Proof. Note the lemma below that provides connection between the difference in the expected errors across two distributions and their distributional discrepancy.
Lemma E.6 (Zhao et al. (2018)). For h W' E H. := {h : X - [0, 1]} assume that H. has a finite pseudo dimension d. For any distribution P and P' over X,
1
(Ep(h, h') Ep. (h, h')l < - dR (P.
(28)
where H : [[ h, E H,O
See Lemma of Zhao et al. (2018) for the proof. We start our derivation of Proposition 5.2 from the ineq. F.6 as below,
5 du (PTT, (e h1)) Sp, (h,
(29)
= =|Dii(PTT Fo) - 1| - [Di(PT, Fb) - 0l|
(30)
= (P Fo) Di Fo)l
(31)
2 - Pi Fo)) - (Pt,,
(32)
=1 =1 Di Fe) Di(Pr,Fo):
(33)
where the first equality holds by definition, the first inequality holds by the reverse triangular in- equality, and the second and fourth equality hold given 0 D, Fb) < 1.
In the meantime, for the empirical distributions DVT ~ Pvr and Df ~ Pr of N samples for each, given 0 < 8 < have the following approximation error bounds with probability least 1 5
30

---

Published as a conference paper at ICLR 2026
for any h H. (See Lemma 5 and Lemma 6 of Zhao et al. (2018)).
N )) O N (log + c clog
Ep. Ee
ED. Eb. (h,
(34)
N (PT. Pr) S dR (DV,, Dr) + 0(( N (log + c clog -
(35)
where Pdim(H)) c.
Then, by plugging the above inequality (Ineq. 35) into the Ineq. 31, we have,
1
1
N ~) = Di(Pvr, F6) Di(Pr, Fo),
1
2 (Dvi, Dr) Dr)
O(V N 8 + clog
(log
(36)
1 2 dd Dr) + O(V (log + clog
N -) Pi Fb) Di
1
(37)
where we derive the first statement of Proposition F.5 from the Ineq. 36, and the second statement of that by combining both Ineq. 36 and Ineq. 37, that complete the proof.
G IMPACT STATEMENT
Language prior represents pathological behavioral pattern in LVLMs, where the model overly relies on its linguistic knowledge and fails to properly ground its predictions in the visual input. This phenomenon underlies critical issues such as hallucination, modality misalignment, and failure cases in vision-centric reasoning. It also suggests that current LVLMs may not be operating in the modality-awar manner we expect-even when their outputs appear plausible (as the result of the vanilla next-token- -rreeinedeiiiiiinn training paradigm). One of the main challenges in mitigating language prior lies in its vague and subjective nature: there exists no clear definition or quantitative measure of "language prior" in dataset or task. Consequently, efforts to balance visual and textual information during training or fine-tuning often rely on heuristics or manual annotations.
Our work sheds light on this issue by proposing formal framework to characterize and quantify the language prior through the model's own behavior. This makes the problem not only more visible but also more measurable. If the degree of language prior can be reliably estimated from within the model, we can begin to incorporate this signal directly into training objectives or inference strategies in principled way. In this way, framework provides principled foundation for deeper understanding and offers practical tools for improving real-world multimodal systems.
In addition to the ultimate goal, i.e., understanding and quantifying LP of LVLM, our novel method, contrastive chain-of-emaln embeddings, on the path to pursue that goal can also create a rich inspira- tion for line of works layer-wise representation analysis (Skean et al., 2025), layer-specific adaptive training approach for LVLMs (Bachu et al., 2025; Oh et al., 2025b), and inclusive AI applications with unbiased multimodal alignment (Kim al., 2025) or representation-centri multi- linguality (Jung et al., 2024; Schut et al., 2025), which ultimately contribute to building a trustworthy multimodal AI system for everyone.
H DISCLOSURE OF LLM USAGE
Some portions of this paper were polished and refined with the assistance of LLM tools (e.g., Chat- GPT) improve clarity, fluency, and consistency writing. We also harnessed coding agent (e.g., Cursor) to write some simple utility functions after double-checking. All technical content, experi- mental results, and analytical conclusions were independently developed by the authors without the use of LLMs.
I ETHICS STATEMENT
This work conducts empirical and analytical studies on the internal behavior of LVLMs, with the goal of understanding and quantifying their reliance language priors and the extent of visual
31

---

Published as a conference paper at ICLR 2026
information integration during inference. To pursue high standards of scientific excellence, pro- pose a formal framework with clear definitions of all used terms and conduct validation at scale, e.g.. 60 combinations of models and datasets, and further provide theoretical analyses framework. Our study does not involve any human subjects, personally identifiable information, sensitive data. All experiments datasets that are widely adopted in the multimodal learning community. Our proposed metrics and analyses are intended for research and diagnostic purposes. By providing tools diagnose when LVLMs rely on text versus vision, we aim to support more accountable model development and contribute positively the responsible advancement of AI. We encourage future work further validate these findings under more diverse real-world conditions.
conducted using publicly available models and benchmark
J REPRODUCIBILITY STATEMENT
All of the models and datasets used in this work are publicly available. To further ensure the reproducibility of our findings, we provide comprehensive descriptions of all experimental settings, including dataset preprocessing, model configurations, metric definitions, and evaluation protocols, in Section and Appendix C. Our framework does not require model re-training or fine-tuning, and all evaluations are conducted in a zero-shot setting using publicly available model checkpoints, which minimizes computational and hardware requirements. We release the complete codebase for analysis framework, including tools for data preparation, TVI computation, and visualization, the following repository: ?
32


Reviews:
[{"review_id": "ZbjNtHRIO5", "reviewer": "Reviewer_zCDF", "summary": "This paper introduces a framework to analyze the internal mechanisms of MLLMs, specifically their over-reliance on language priors instead of visual evidence. The authors propose a \"chain-of-embedding\" analysis, which contrasts layer-wise hidden states from vision-text inputs with those from text-only inputs. This analysis reveals two key findings. First, all models exhibit a Visual Integration Point, a specific layer where visual information begins to meaningfully influence representations. Second, the paper introduces the Total Visual Integration estimator, which aggregates representation distances after the VIP to quantify the strength of visual integration per sample. A high TVI indicates effective visual grounding, while a low TVI suggests a strong LP reliance. This framework was consistently validated across 54 model-dataset combinations.", "questions": "1. Can this solution be used for data filtering to screen for sample pairs that are strongly visually relevant?\n\n2. Looking at Figures 2 and 3, the difference in representation distance after VIP is only on the order of 0.001. Is this difference not significant enough?\n\n3. Why only use the last token hidden states for calculation? Can all embeddings be included?\n\n4. Is the VIP layer different at different model training stages? What would the trend be?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Novel Analysis of Language Prior. The paper proposes a novel framework for analyzing Language Prior in LVLMs. Instead of relying on traditional input-output probing , it introduces a fine-grained, white-box methodology that examines the internal, layer-wise \"chain-of-embedding\". This provides a new and insightful perspective on how and when models integrate visual information.\n2. Sufficient Experimental Validation. The authors demonstrate the robustness of their claims through extensive experiments. The existence of the Visual Integration Point (VIP) is not an isolated finding; it is consistently observed across 9 different LVLMs and 6 distinct benchmarks, totaling 54 experimental settings. This large-scale validation strongly supports the claim that the VIP is a universal phenomenon in these architectures.", "weaknesses": "1. Lack of Clarity in Methodology. The main body of the paper omits critical methodological details, hindering clarity and reproducibility.\n\t- Undefined Symbols in Formulas. In Section 3.1, Equation (2) introduces the notations $P_{star}$ and $D_{star}$ without explicitly defining what the wildcard $star$ represents. While context implies it refers to the $P_{VT}$ and $P_T$ distributions defined earlier, this omission makes the formulation imprecise.\n\t- Ambiguous Distance Calculation. The paper's core distance metric, $d(z_{vis}^l, z_{blind}^l)$, is underspecified in the main text. Both $Z^l_{vis}$ and $Z^l_{blind}$ are sequences of token embeddings, not single vectors. The crucial methodological detail—that the distance is only computed using the hidden state of the last token—is only mentioned in the appendix. This choice is non-trivial, as it impacts the interpretation of the results, and should be explicitly defined and justified in Section 3.1.\n2. Lack of Interventional Validation for TVI. The paper's validation for the Total Visual Integration metric is purely diagnostic. The authors demonstrate that TVI correlates strongly with task correctness on vision-dependent benchmarks . However, this is not sufficient proof of its reliability as a proxy for Language Prior. A much stronger validation would be an interventional experiment. The authors mention existing methods for mitigating LP (e.g., contrastive decoding, attention reallocation) in their related work . They should have applied these mitigation techniques and then measured the TVI score. If TVI is a reliable metric, it should significantly increase (reflecting lower LP) when these mitigation methods are active.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DvYtI7bNn3", "reviewer": "Reviewer_2B1a", "summary": "Vision-Language Models (VLMs) often rely overly on language priors. This paper examines this issue from a representation-level perspective by contrasting the internal embeddings of models with and without visual input. The authors propose two diagnostic metrics. The Visual Integration Point (VIP) identifies the layer where visual features begin to significantly influence textual representations. The Total Visual Integration (TVI) quantifies the overall magnitude of visual influence across layers. Experiments on multiple VLMs and benchmarks show that VIP positions are consistent within each model, while TVI correlates with both visual grounding strength and the dominance of language priors. The paper further analyzes the choice of distance metrics, the effect of model scale, and provides a theoretical justification.", "questions": "1. The paper introduces a threshold to determine the VIP but provides no clear method for how it is chosen or tuned for different models. It is stated that VIP is data independent but not model-dependent.\n2. Although the anonymous code is provided, the shared link currently returns \"The requested file is not found\". \n3. The use of the term \"ablation\" appears inaccurate. For example, the ablation on model scale is more of a sensitivity analysis, as it varies model size instead of removing the components. The authors might want to recap the research methodology.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The key idea, contrasting the internal embeddings with and without image input, is simple yet effective. It provides a direct way to study how visual information propagates within VLMs, offering a view of cross-modal interaction that complements existing output-based analyses.\n2. The experiments produce consistent results across models and datasets, supporting the reliability of both proposed metrics. The findings are both insightful and well-evidenced through multiple analyses and visualizations.\n3. The experiments cover a broad set of VLMs and benchmarks, and include examinations of distance metrics, model scale, and theoretical interpretation. This completeness strengthens the empirical grounding of the paper.", "weaknesses": "1. It states that \"a higher TVI indicates that visual information exerts a stronger and more persistent influence throughout the later layers\". However, the current TVI definition measures the mean visual effect after the integration point, which can overestimate models with brief but strong fusion peaks. Because it ignores the layer-wise distribution of visual influence, a single large spike can inflate TVI even if vision effects fade quickly. A refined version, such as a weighted or variance-penalized TVI, would better reflect sustained multimodal reasoning by emphasizing persistence and uniformity of visual influence.\n2. Some statements, such as the claim that VIP corresponds to the onset of \"effective problem-solving\", are not fully supported by the evidence. Since the paper does not evaluate task performance at the layer level, VIP should be interpreted as marking representational change rather than functional reasoning. A more cautious phrasing is needed to improve accuracy and credibility.\n3. The current analysis focuses primarily on the language-prior effect, contrasting text–image input with text-only input. A more complete triple-wise  comparison, including text-only, image-only and text-image pairs, would allow a deeper understanding of how each modality contributes individually and jointly to the representation flow, offering a fuller picture of multimodal integration.\n4. While the work provides a valuable diagnostic for analyzing language priors, it stops short of offering guidance or methods to reduce such bias. Without follow-up analysis or strategies for mitigation, the practical utility of VIP and TVI remains limited to interpretation rather than its application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5pbkCnxVpX", "reviewer": "Reviewer_PjBx", "summary": "The authors introduce a chain-of-embedding (CoE) analysis that examines layer-wise hidden representations of LVLMs. By comparing embeddings produced with and without visual inputs across model layers, they identify a consistent Visual Integration Point (VIP) — the layer where visual information begins to meaningfully affect model behavior. Building on this, they define a quantitative measure called Total Visual Integration (TVI), which aggregates representation differences after the VIP to estimate how strongly the model integrates visual signals. A low TVI indicates a stronger language prior (i.e., text dominates), while a high TVI reflects effective multimodal grounding. The key findings are as follows: Every tested LVLM (9 models, 6 benchmarks) exhibits a distinct VIP layer where visual information starts influencing predictions. TVI strongly correlates with model correctness on visual reasoning tasks, outperforming traditional metrics like visual attention or output divergence. Theoretical analysis connects representation distance to KL divergence between visual and text-only distributions, offering an interpretable, information-theoretic view of LP..", "questions": "Questions are mentioned in the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces chain-of-embedding analysis, a new internal probing method that captures layer-wise representation dynamics between visual and textual modalities — a perspective largely missing in prior language prior studies.\n2. he discovery of the Visual Integration Point (VIP) provides an intuitive and mechanistic explanation of when LVLMs start to rely on vision, enhancing understanding beyond input–output probing.\n3. The proposed Total Visual Integration metric offers a principled, continuous measure of multimodal reliance, showing strong correlation with model performance across tasks.\n4. The authors test 9 LVLMs across 6 benchmarks (54 settings total), showing strong consistency and robustness of the findings.\nTheoretical grounding: The work connects empirical findings to information-theoretic interpretations (via KL divergence), lending mathematical rigor to an otherwise empirical domain.", "weaknesses": "1. While the framework effectively analyzes where and how LP arises, it does not propose concrete strategies to reduce language prior or improve multimodal reasoning.\n2. The method requires access to intermediate activations and model internals, making it impractical for closed-source LVLMs or large-scale commercial APIs.\n3. While broad in scope, the work is heavily diagnostic, with limited discussion of causal factors or practical interventions to improve LVLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j09BguS9sI", "reviewer": "Reviewer_mW7M", "summary": "This paper begins with reporting their observation of Visual Integration Point (VIP), the layer where visual input starts to affect hidden representations. Based on this observation, the paper proposes Total Visual Integration (TVI), a metric quantifying visual grounding (and, inversely, language-prior strength). TVI is distinct from prior approaches that rely on output comparisons using carefully curated datasets. Across nine LVLMs and six benchmark datasets, the authors find that VIP typically emerges at a similar depth of the decoder (agnostic to models) and that higher TVI correlates with better accuracy on vision-dependent samples. The authors also show that TVI outperforms previously proposed metrics like visual attention and output divergence in predicting model correctness on visually grounded tasks. Finally, they argue that TVI provides a more direct and interpretable measure of an LVLM’s visual grounding.", "questions": "See the above weaknesses.\n\nBesides, while all evaluated models are relatively small, they still span a range of parameter scales (2B to 27B). Do the authors observe any trend in VIP location or TVI magnitude with respect to model size? It may be difficult to compare across different architectures, but evaluating models of varying sizes within the same family in future work could help clarify this relationship.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper goes beyond conventional output level evaluations of language priors by examining layer-level hidden representations. It provides a new and interpretable perspective on how LVLMs integrate visual information and develop visual grounding.\n- The newly proposed metric, TVI, offers a more robust and empirically and theoretically grounded way to quantify visual grounding (or, inversely, language prior strength) than prior metrics such as visual attention or output divergence. It directly measures the degree of vision’s influence on internal representations.\n- The approach is easy to implement (requiring only paired forward passes with and without images) while yielding illuminating insights into when LVLMs begin to use visual input.\n- The study evaluates nine LVLMs across six benchmarks, demonstrating that the observed VIP and TVI behaviors are overall consistent and generalizable across different architectures, dataset types, and model scales.", "weaknesses": "- The identification of VIP relies on dataset-level statistics and a tau threshold derived from pre-VIP variance, which introduces potential dependence on dataset composition (eg, the ratio of visually-dependent to -independent samples). The paper does not analyze how sensitive VIP detection is to this choice or whether tau threshold could be defined in a dataset-agnostic manner.\n\n- Most of the 9 evaluated LVLMs use variations of late-fusion (adapter-based) architectures, with the exception of Gemma. This architectural bias may partly explain why the detected VIP consistently appears around the middle or later layers. The paper would be stronger if it included models with more diverse fusion architectures (eg, Flamingo-style, Q-Former, or Perceiver-based) to test whether or not the VIP phenomenon generalizes beyond late-fusion setups.\n\n- It would also be valuable to test larger models. While API models cannot be easily analyzed, several powerful open-source LVLMs (e.g., 70B-scale models) could be evaluated to examine whether the observed trends in VIP and TVI scale to higher capacity models.\n\n- In Table 2, TVI clearly outperforms other metrics, with correlation values ranging from 0.57 to 0.71. However, this still leaves a notable fraction of cases where TVI fails to capture visual grounding. An analysis of these failure patterns or outlier samples would have provided deeper insight into when TVI succeeds or breaks down, further strengthening the validity of the TVI metric.\n\n- [Minor] While VIP and TVI are conceptually interesting, the paper does not clearly articulate how these metrics could guide future LVLM development. Including concrete examples of potential applications would make the work more actionable.\n\n- [Minor] Although the paper emphasizes the number of evaluated models and benchmarks, these are not well-documented, even in the appendices. I recommend adding summary tables for both datasets and models. For models, include details such as fusion strategy and number of attention layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)