A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models
Authors: Lixin Xiu
Xufang Luo
Hideki Nakayama
Venue: ICLR 2026 Poster (2026)
Keywords: Interpretability
Multimodal Learning
Large Vision-Language Models
Partial Information Decomposition
Information Theory
Primary area: interpretability and explainable AI

Abstract: Large vision-language models (LVLMs) achieve impressive performance, yet their internal decision-making processes remain opaque, making it difficult to determine if the success stems from true multimodal fusion or from reliance on unimodal priors. To address this attribution gap, we introduce a novel framework using partial information decomposition (PID) to quantitatively measure the ``information spectrum'' of LVLMs---decomposing a model's decision-relevant information into redundant, unique, and synergistic components. By adapting a scalable estimator to modern LVLM outputs, our model-agnostic pipeline profiles 26 LVLMs on four datasets across three dimensions---\emph{breadth} (cross-model \& cross-task), \emph{depth} (layer-wise information dynamics), and \emph{time} (learning dynamics across training). Our analysis reveals two key results: (i) two task regimes (synergy-driven vs.\ knowledge-driven) and (ii) two stable, contrasting family-level strategies (fusion-centric vs.\ language-centric). We also uncover a consistent three-phase pattern in layer-wise processing and identify visual instruction tuning as the key stage where fusion is learned. Together, these contributions provide a quantitative lens beyond accuracy-only evaluation and offer insights for analyzing and designing the next generation of LVLMs. Code and data are available at \url{https://github.com/RiiShin/pid-lvlm-analysis}.

Published as a conference paper at ICLR 2026
A COMPREHENSIVE INTORMATTIO-DEECOMPSSTIIOI ANALYSIS OF LARGE VISION-LANGUAG MODELS
Lixin Xiu', Xufang Luo², Hideki Nakayama' 'The University of Tokyo Microsoft Research
ABSTRACT
Large vision-language models (LVL.Ms) achieve impressive performance, yet their internal decision-making processes remain opaque, making it difficult de- termine if the success stems from true multimodal fusion or from reliance on uni- modal priors. To address this attribution gap. we introduce a novel framework using partial information decomposition (PID) to quantitatively measure the "in- formation spectrum" of decomponcnn a model's decision-relevant in- formation into redundant, unique, and synergistic components. By adapting a scalable estimator to modern LVLM outputs, our model-agnostic pipeline profiles 26 LVLMs four datasets across three dimensions breadth crooss-model & cross-task), depth (layer-wise information dynamics), and time (learning dynam- ies across training). Our analysis reveals two key results: (i) two task regimes (synergy-driven vs. knowledge-dee level strategies (fusion-centric vs. language-centric). We also uncover consistent three-phase pattern in layer-wise processing and identify visual instruction tuning as the key stage where fusion learned. Together, these contributions provide a quantitative lens beyond accuracy-only evaluation and offer insights for analyz- ing and designing the next generation of L.V.Ms. Code and data are available www.i/SShiipppppp. lvlm ana ysis
and (ii) two stable, contrasting family-
Intps: / tit.ub..u...
INTRODUCTION
Large vision-language models (LVLMs) achieve remarkable success across a wide range of multi- modal tasks, including visual question answering (Chen et al. 2024), image captioning (Bai et al. 2025a), and open-ended reasoning (Zhu et al. 2025a). However, the internal mechanisms driving these impressive results remain largely opaque. Accuracy and other aggregate performance metrics only reflect the final outcomes of model predictions, but not the underlying processes through which these outcomes are obtained. While prior research has begun analyze large language models (LLMs) to isolate factors shaping predictions (Jain & Wallace 2019; Meng et al., (022). LVLMs pose a distinct challenge because they must process and integrate multiple modalities. In partic- ular, understanding whether model's prediction is primarily driven by visual evidence, language priors, or the interaction between the two is critical for interpreting its behavior. However, existing interpretability efforts often adopt "micro-scope or introduce ad hoc metrics that lack firm theoretical support. Consequently, the field still lacks comprehensive, quantitative tools capable of dissecting the internal strategies by which LVLMs use multimodal information.
focus-anale analyzing one modality in isolation
To address this challenge, we introduce a framework built on partial information decomposition (PID) to comprehensively and quantitatively analyze LVLM behavior. PID is rigorous information- theoretic methodology that decomposes the mutual information between set of inputs and an output into distinct components. Originally developed in neuroscience and complex systems (Williams & Beer 2010), PID characterizes how multiple information channels jointly influence a target variable. We extend this perspective to LVLM inference by treating vision features Xi and language fea- tures X2 inputs and the model prediction Y as the output, partitioning decision-relevant informa-
Correspondence to: Xufang Luo <nakayama@ci. .u-tokyo.ac, jp>.
<x uo@mi crosoft. com> and Hideki Nakayama

---

Published as a conference paper at ICLR 2026
(1). Proposed framework for PID estimation in LVLM scenario
Image:
Language feature X2
Language
Mutlimoda prediction (..3)
BATCH
U/
vord and
Model.
10:1 9:2% 0.81% 1:1))
A.fon plant
Vislarionl prediction P(V)X))
Redundancy
25% 8i 255 2.25% 0.25%)
U4 Vision Uniqueness
the splin Sarmwar cunee bene offle
Ug Language Uniqueness Synergy
Language-onty prediction P())Xg)
[A:155
0:40%)
Erlodding Layer
(2). 3-dimentional infernaiinndeccomosiiion analysis
(a). Cross-model crose-tank comparison
(b), Layer-wise information
(c). Learning dynamics
LLaVA LLaVA.onnvision anevision
Model: LLaVA-1.5
Owen2-VV.
Layer i (Laxi)
Stoge
Stage 2
Qwen2.5-V 5-VL
InternVL.55
Vesin-anguage algrment pretraning
Visual
11 families &
InternVL3
Layer 15
Instruction Tuning
Gemman
models
Liama-3. 2-vision
Cambrian-1
A MLP
MLP LM
LLaVA-1.5 1.5
Layer
InstructLIP
Fuyu
Legit lens
MMBench (Ganenal) POPE (Hallucinatio))
datasets
Lager (F
Reefknot (Halldination)
0
0
0
PMC-VQA (Domain-spcccc
11 1/
Figure 1: Overview of this research. The first part the framework of PID estimation for LVLMs. Given an image-text pair, we extract image and text embeddings as two features, run standard multimodal forward pass and collect two unimodal predictions by masking the other modality. PID values are estimated with BATCH estimator. The second part reveals three analysis dimensions: (I) cross-model and cross-task comparison, (2) layer-wise information dynamics, and (3) learning dynamics training. To our knowledge, this is the first comprehensive LVLM analysis through the lens of information decomposition.
tion into four non-negative terms: redundancy R (shared by both), vision uniqueness UI, language uniqueness U2, and synergy (emerging only from their combination). We refer to {R, U1, U2 S} as the model's information spectrum. It provides principled lens for probing LVLM internals and, unlike "micro-scoue scorroe analyses more empirical approaches, enables quantitative separation of the model's core information information-processing strategies.
To apply PID to modern LVLMs, we adapt the PID estimator proposed by Liang al. (2023a) for visual aantrerng (VQA) tasks and propose a model-agnostic pipeline that requires no ar- chitectural changes or retraining. Using this pipeline, we analyze LVLMs along three axes: (1) cross-model, cross-task comparison spanning 26 models and four benchmarks; (2) layer-wise infor- mation dynamics via a logit-lens view; and (3) the evolution of multimodal fusion across training stages. The full framework and analysis dimensions are summarized in Figure
Our study yields several actionable insights. We identify two task regimes- synenny---snee showledge-driven--e and show that model families themselves adopt two stable, contrasting strategies fatrggiiss-fuioon-ccettt
vs. language-centric. We clarify how fusion develops, observing a three- phase pattern layers and finding that visual instruction tuning is the key stage where S is un- locked. Taken together, these results provide a quantitative basis for moving beyond accuracy-onjy evaluation toward more principled understanding of multimodal processing LVLMs.
2 RELATED WORK
The evolution of vision-language models. Vision-language models (VLMs) have shifted from contrastive dual-encoder paradigms to generative paradigms. While early dual-encoder models like CLIP (Radford et al.) (02T)) and ALIGN (Jia et al., (221)) focused on joint representation learning, the current approach parameter-effictent generative architecture, typically comprising vi-
2

---

Published as a conference paper at ICLR 2026
sion encoder, projector, and large language model (LLM) backbone (Timpoukelli et al. 2021 Alayrac et al. 2022 Merullo et al. 2022 Li et al. 2023a).
The evolution of large langonnl models (LVLMss) is driven by advances in LLM backbones 2024). Architecturally, backbones have shifted from encoder-
and training methodologics (Dai ct al.
decoder models like T5 (Raffel et al. 2020 to decoder-only models such Llama (Tovrron al.,
2023). Methodologically, performance has been greatly enhanced by various training stages on vast,
high-quality datasets, strategy pioneered by LLaVA (Liu et al., 2023b) and MiniGPT-4 (Zhu al.
2023). State-of-the-art LVLMs integrate the most powerful backbones (e.g.. L (Grattatiori
al. 2024) and Qwen2.5 (Qwen al. (2025) with advanced training recipes (Liu et al. [023a Li
et al., 2024 Wann et al.) |024]/Cher et 20224] Tong et al. 2024] Meta Al 2024 Bai et al. 2025a
Zhuu al. 2025a). resulting in more capable yet more complex systems.
Probing the black box: interpretability in VLMs. To address the opacity of VLMs, researchers
have adapted interpretability techniques from transformer-based language models (Vaswani et al.)
2017]. One line of work generates post-hoe explanations to identify influential inputs using attri-
bution heatmaps (Schulz et al., [020 Wang et al. (023), attention maps (Abnar& & Zuidema 2020
Chefer et al. 2021 Gandelsman et al. 2024), and activation analysis (Conmy et al.) [023]] Arditi
cet al.) 2024). Complementary approaches probe a model's internal representations to understand
its learned knowledge (Meng et al. (022), for instance by training linear probes (Alain & Ben-
gio 2016) to decode features in LLMs (Hewitt & Manning. [209 Tenney al.) (019). applying
the logit lens (nostalgebraist 2020 Belrose et al. 20.3) to inspect LLMs' intermediate computa-
tions (Cywińskii et al.) [025] Nco al. 2024)). or identifying "multimodal neurons" corresponding
to guman--nterpretable concepts (Goh et al. 2021 Schwettmann et al. (023).
With the rise of LVLMs, research is shifting toward understanding their deep multimodal fusion. Recent work in this area includes analyzing and manipulating the visual token representations that bridge the two modalities (Jiang et al.) [222bb [Basu et al. [2024] [Liu et al. 2025)) ) as well as quanti-
fying vision's 's contribution through analyses of visual attention sinks (Kang et al. 2025) and cross-
modal information-foor tracing (Zhang et al. 2025 Nikankin et al. 2025 Yang et al.) 2024).
An information thhorretcc lens on multimodal learning. Information theory offers a quantitative framework for analyzing information flow in neural networks. Foundational concepts like mutual information (MI) (Shannon 1948) and the subsequent information bottleneck (IB) principle (Tisbyy let al.) 2000 have been widely used in multimodal learning. Applications range from using MI for
interpretability (Oh et al.,
2025) to employing the IB framework for both guiding representation
learning (Almudévar al. 2025 Jiang et al. 20224a] Xiao et al. 2024 Wu al. 2025 Bai al.,
2025b) and enhancing model transparency (Wang et al. 2023 Zhu ct al. 2025b).
While mutual information can quantify the total information from a source, it cannot disentangle the complex interactions between multiple inputs, such as vision and text. To address this, partial information decomposition (PID) (Williams & Beer 20100) decomposes the information about tar- get variable into redundant, unique, and synergistic components. The application of PID to machine learning a nascent field (Ehrlich al., 2022] Dissanayake et al., [025] Choi et al.) 2025 Shan et al., 2025), though recent studies have begun apply it within multimodal learning contexts (Liang et al., 2023a b) 2024). However, its use for analyzing the composition, flow, and evolution of multimodal information within modern LVLMs remains unexplored, which is gap this paper aims to fill.
METHODOLOGY
This section details our methodology for applying PID to analyze LVLMs in three parts. We first review the fundamentals of PID and the specific estimator our work adapts in Section 3.1 We then present framework, including three key adaptations that make this estimator robust for the unique context of modern LVLMs, in Section Finally, we outline the experimental design that leverages this framework to conduct comprehensive analysis across three dimensions in Section 3.3
3

---

Published as a conference paper at ICLR 2026
3.1 PRELIMINARIES
Partial information decomposition. Mutual information (Shannon 1948) measures the statistical dependence between two variables. However, in three-variable system comprising two source variables X1, X2 and a target variable Y. with respective state spaces X, X2. and V the standard interaction information / (X1;; Y) can be negative, limiting its interpretability.
To address this, partial information decomposition (PID) (Williams Beer 2010) reframes the problem by decomposing the total mutual information I 1(X1, X2; Y) into 3 non-negative atoms: redundancy (information common to both sources), uniqueness (information exclusive to each source), and synergy (new information emerging from their combination). Following Bertschinger et al. (2014), the components are defined on the set of distributions Ap = {Q E A Q(...)) P( y), Va; E X, y E V i E {1,2} }, which contains all joint distributions Q over (X1, X2, Y) that preserve the ttrget marginals of the true distribution P. The atoms are given as:
R = max 12;
QEAr
U1 min QEár
IQ(X;;Y |X2),
U2 min 1Q(X2; | Xx))
(3)
QEAr
S = X2; min lo X2;
(4)
QEAp
This decomposition provides a principled framework for quantifying how individual and joint sources of information contribute to target variable, but estimating these atoms from data is non-trivial task that requires specialized estimators.
Estimating PID for multimodal inputs. To estimate PID for the high-dimensional representa- tions within modern LVLMs, we adapt the scalable estimator from Liang et al. (2023a). This work introduces two methods: a convex programming baased estimator CVX for discrete features, and an approximate estimator BATCH designed for continuous, high- nigh-dimensional modalities.
Our work builds upon the BATCH estimator, as well-suited for analyzing the continuous vec- torial embeddings produced by LVLMs. This method uses neural networks to parameterize the required probability distributions. It then optimizes an coormation-theorett objective over mini- batches, employing a variant of the Sinkhorn algorithm (Cuturi 2013) to enforce the marginal- matching constraints defined Ap.
See Appendix A for more details on PID and the BATCH estimator.
3.2 A PID ESTIMATION FRAMEWORK FOR LVLMS
We propose a PID estimation framework for LVLMs tailored multiple-choice visual question answering (MC-VQA) tasks. This focus is a deliberate methodological choice: BATCH requires a finite y. thus the natural set of choices in MC-VQA (e.g., (A, B, C, DJ) allows for a clean analysis while avoiding the noisy and potentially biased process of manually clustering open-ended answers, or training auxiliary projection heads map LVLM representations into pre-defined clusters. Such additional components make PID estimates sensitive to clustering and projection hyperparameters, making it unclear whether the estimated quantities primarily reflect the LVLM's 's original end-to-end behavior or the added mapping, which is not how these models are typically used.
As illustrated in Figure D (1), our pipeline begins by defining the source variables- vision (X₂) and language from an L'LM's internal embeddings. We then conduct multimodal and unimodal inference runs to obtain three conditional probability distributions: P(Y X1, X2). P(Y X1), and P(Y X2). These distributions, along with the source features X and X2, are then fed into the BATCH estimator to compute the final PID values {R, U1, U2. S}. Notably, this estimation relies only on the model's predictive distributions and input representations; used when computing the PID components. This makes PID a process-level descriptor of model behavior, complementary to standard accuracy metrics.
ground-truth labels
4

---

Published as a conference paper at ICLR 2026
Input representation and unimodal conditioning. We define the source variables X₁ and X2 as the mean-pooles visual and textual token embeddings, respectively, and ablate alternative summa- rization strategies (last-hidden and max pooling) in Appendix B
Estimating the unimodal conditionals P(Y X1) and P(Y X2) for an integrated LVLM requires carefully designed probe. We approximate unimodal conditioning by masking one modality the embedding level: following the corruption scheme of Meng et al. 2022), we replace the entire em- bedding sequence of the other modality with noise. Each vector in this noise sequence is drawn i.i.d. from N (t,, diag(o)))) where
E I denote the dimension-wisg mean and standard deviation of that modality's 's embeddings, pre-computed across the dataset. This calibrated noise removes the other modality while keeping the embedding scale in-distribution.
Confidence thresholding for renormalization. For both unimodal and multimodal conditioning, we extract a categorical predictive distribution over the finite candidate set V. We first compute a token-length normalized candidate score (Y I)) from the model log-likelihood of the candidate answer string, and then renormalize across candidates to obtain P(Y (-).
However, renormalizing over a restricted candidate set can artificially inflate confidence when the model assigns low scores to all candidates under the full vocabulary distribution. To mitigate over- confidence from a restricted candidate set, we compute the total candidate-set score and apply a confidence threshold:
P(Y (PCY 1 if Eyey Sang (Y= 2 ((K) otherwise
where K = | and ((K denotes the uniform distribution over V. This prevents low-confidence guesses from contributing spurious structure the PID computation. We also ablate the confidence threshold - E (0.2, 0.3, 0.4); see Appendix for details.
Soft aggregation for the marginal output distribution. A final adaptation addresses the estima- tion of the marginal output distribution P(Y). Discretizing predictions via argmax and then com- puting frequencies can introduce a measurement artifact: for a totally uncertain (uniform) output, argmax resolves ties in a fixed manner (e.g.. always selecting the first label). artificially converting uncertainty into sharp peak upon aggregation. To avoid this, we use soft aggregation and estimate P(Y) by averaging the regularized predictive distributions across all N samples:
1 P(Y) = N
P.(Y).
(6)
where P 1), denotes the regularized categorical distribution for sample i. This pre- serves the model's output statistics and leads to a more faithful PID analysis.
3.3 ANALYSIS DIMENSIONS & EXPERIMENTAL. SETTINGS
To conduct a comprehensive three dimensions: (1) a large-scale comparison across wide range of models and tasks, (2) the layer-wise information flow inside representative models, and (3) the learning dynamics by examin- ing model checkpoints throughout the training process. For reproducibility. all experimental settings, including inference details and key hyperparameters, are provided Appendix C
cermmation-dccomossttiin analysis, we design experiments across
3.3.1
TROSS-MODES AND PROSS-TASE COMPARISON
Models. To assess how architecture and scale affect information use, analyze 26 models (0.5B
to 90B parameters) from 11 open-source LVL.M families Our selection prioritizes recent, state-
of-the-art families including LLLVA--One Vision (Li et al. 2024), Qwen2-VI. (Wang et al. 2024
Qwen2.5-VL (Bai et al. (2025a). Gemma-3 (Team et al. 2025). Intern [Chen et al. 2024
Intern (Zhu et al. 2025a). and Llama-3.2-Visiie
(Meta All (0244). We also include Cambrian-
1 (Tong et al. 2024) for its multi-viss vision encoder design and established models like LLa (Liu
al. 2023a). InstructBLIP (Dai et al. 2023), and Fuyu (Bavishi al. 2023) to serve as baselines.
All model checkpaints downloaded from Hugging Face: https:///hut ///gg/ ng face wwwace.cco///aaa. co/model1
5

---

Published as a conference paper at ICLR 2026
Tasks, We evaluate all models four diverse MC-VQA datasets: MMBench ('en_dev') Liu
et al. 2024) for general reasoning, POPE ("COCO14 adversarial") (Li et al. 2023b) and Reef- knot PPPerception & MCQ') (Zheng al. 2024) for hallucination evaluation, and PMC-VQA ("test clean") (Zhang et al. 2023) for domain-specifie medical knowledge. Table | summarizes the characteristics.
Table I; Details of the datasets used for evaluation. The listed training and test splits are not for LVLM fine-tuning; they are created by randomly partitioning each dataset (3:1 ratio) for the PID estimation, the BATCH estimator requires separate sets to train networks and estimate PID values.
# of # of training of test
Dataset
Task
options
samples
samples
MMBench General visual reasoning
2-4
3246
1083
POPE
Hallucination evaluation
2
2250
750
Reefknot
Hallucination evaluation
4
1612
538
PMC-VQA
Domain Domain-specific knowledge
4
1500
500
Image-removal intervention. As behavioral validation, remove the image to obtain text- only baseline and measure the accuracy drop Deision which we relate to the PID-based information spectrum assess models' visual reliance.
3.3.2 LAYER-WISS INFORMATION DYNAMICS
To trace the internal information flow, we conduct layer-wise PID analysis on three representa- tive model families: VL322VL3-2B/8BB Qwen2.5-VL-33B7BB, and LLaVA-L.5-7B/BBB We analyze them on MMBench and PMC-VQA to observe dynamics on general and domain-spectio tasks. By applying the logit lens (nostalgebraist, 2020), we project the hidden state each transformer block through the LM head to obtain layer-speciñic output distribution for our PID analysis.
3.3.3 LEARNING DYNAMICS OF MULTIMODAL FUSION
To understand how fusion capabilities evolve, we analyze the two-stage training process of repre- sentative model, LLaVA-1.5 (7B/13B), and reproduce its training using the original data and official settings. This process involves (1) vision-language alignment pretraining, where only the projector is trained to align frozen vision and language embeddings, followed by (2) visual instruction tuning. which fine-tunes both the projector and the LLM.
We save four equidistant checkpoints from each stage and evaluate each checkpoint's ' full PID profile on MMBench and MMC-VQA to create a temporal trace of its learning trajectory.
4 RESULTS AND FINDINGS
We treat PID components signals: R (overlap), U, (vision-only cues), U2 (language-side knowl- edge), and S (combined use). Across breadth (models x datasets), depth (layers), and time (training), ask which signal most consistently shapes LVLM behavior and generalization.
4.1 DIMENSION 1: CROSS-MODEL & CROSS-TASK COMPARISON
Because redundancy R and vision uniqueness U1 are consistently small, we focus on synergy and language uniqueness U2 characterize task demand and model strategy (see full spectra in Appendix
4.1.1 TWO REGIMES OF INFORMATION USE ACROSS TASKS
Task-level patterns: two regimes of evidence use. Question: Do datasets push LVLMs to rely more on combining inputs, or on what they already know from text? To understand how different tasks challenge L.LMM, first investigate whether they systematically demand different kinds of

---

Published as a conference paper at ICLR 2026
information. Our analysis examines how models allocate information between S and U2 each dataset, revealing recurring differences that we summarize as two regimes of information use.
Figure plots, for each dataset, the dataset-level mean shares of S and U2 averaged across all models, with 95% bootstrap Cls. A clear split emerges, driven mainly by S: MMBench and POPE form one cluster characterized by high S, while Reefknot and PMC-VQA form second cluster with markedly lower and higher U2- These are empirical profiles of how LVLMs be- have on these datasets, not labels of the datasets themselves. In this second cluster, synergy ap- pears to have a practical ceiling, suggesting that while fusion beneficial, it cannot fully compen- sate for missing language-side knowledge; cor- respondingly, accuracies are typically 20-30% lower than in the high-S group.
Syergydttoon
Figure 2: Share of synergy S and language uniqueness U2 across four datasets.
Finding 1: Across the four benchmarks, we observe two recurring information use regimes in experiments. A synergy-driven regime appears tasks where models rely more strong cross-modal fusion. contrast, a knovledgeddriven regime appears on tasks where performance reflects stronger dependence language-side knowledge and priors.
Information correlates of accuracy across task regimes. Within each regime, we test which component tracks model accuracy. We compute Spearman's 's between accuracy and each PID term across 26 LVLMs per dataset. Note that PID is computed from model predictions only; labels are used solely to report accuracy.
Table 2: Spearman correlations (p) and p- p-values across datasets.
Dataset
S
U2
T(X1, X2; Y)
T(X1; X2; Y) p-val
p-val
0
p-val
p-val
P
MMBench
0.750
0.001
0.194
0.343
0.632
0.001
-0.757
< 0.001
POPE 0.742
< 0.001
-0.009
0.964
0.157
0.445
-0.701
< 0.001
Reefknot 0.357
0.073
0.313
0.119
0.266
0.196
-0.348
0.081
PMC-VQA 0.432
0.027
0.406
0.040
0.559
0.003
-0.587
0.002
Table shows a consistent pattern. On synergy-driven benchmarks (MMBench, POPE), S is the strongest positive correlate of accuracy (p 0.75, p < 0.001), whereas I(X,, Y) is less sistent across datasets 2 This implies that top-performing models are not those with simply "more" information, but those that translate overlapping cues into effective cross-modal modal
On knowledge-driven benchmarks (Reefknot, RMC-VQA) the picture shifts: Uz becomes compar- atively more informative (significant PMC-VQA), while S remains positively related to accuracy but is no longer dominant. These results suggest that fusion is beneficial in both regimes, but gains bounded when language-side knowledge becomes the primary bottleneck.
Finding 2: In synergy-driven tasks, accuracy is most strongly associated with synergy S. In knowledge-driven tasks, language uniqueness Uz becomes more predictive, while synergy S remains helpful but less dominant.
"The interaction information T(X1; X2:Y) R S is strongly negative here, largely mirroring the -

---

Published as a conference paper at ICLR 2026
Intervention-basee validation: image removal. We further validate this interpretation via simple intervention: removing the image and measuring the accuracy drop Dvision- Across mod- els, Vision correlates strongly with S on synergy-driven benchmarks (MMBench/POPE: p 0.809/0. /0.44, both with < 0.001), and more weakly on knowledge-driven ones (Reefknot/PMCC- VQA: p 0.459/0.400, p (0043).043). This confirms key prediction: models with higher $ are more sensitive to visual ablation, indicating that S captures decision-relevant visual reliance.
Qualitative examples illustrating S-dominant (MMBenchhPOPE) and U2-bounded (Reefkno//PMC- VQA) cases are provided in Appendix [ reflect consistent model-level strategies.
We next ask whether such accuracy-relevant components
4.1.2 INFORMATION STRATEGIES ACROSS MODEL ARCHITECTURES
Model families exhibit stable, contrasting information strategies. Do model families lean ward combining imputs or toward language knowledge-and and does that preference hold across set- tings? We summarize each family's behavior by its median S and U2 within each regime, shown in Figure
Svergyydiive
tevig
120
Figure 3: Family-level strategies: median versus median U2 per family, computed across model sizes within each task regime. Points show the family medians for each regime. Outliers (Instruct- BLIP, Fuyu) are omitted for clarity.
Families occupy two clearly separated regions, corresponding to two information-use strategies: a fusion-centric group (e.g.. Intern Qwen2/2.5-VL.) with relatively high S and lower U2, and a language-centric group (e.g., Gemma3, Cambrian) with lower S and higher Uz. Although absolute S drops knowledge-driven tasks, the relative positions of families remain similar across regimes, suggesting that this preference a stable family-level tendency.
Finding 3: LVLM families exhibit two stable information strategies: fusion-centric families rely more on synergy S, while language-centric families rely more on language uniqueness U2. This strategic identity persists across task regimes.
Scaling effects on synergy-driven tasks. If family identity is stable, scaling should reinforce the same leaning. A common expectation is that larger models rely more on Uz: we test this on synergy- driven tasks. To assess how information use changes with scale, we compare Small (S), Mid (M), and Very-Large (VL) models within representative families.
As shown in Table 3 the share of language uniqueness U2 does not systematically increase with size and often decreases. By contrast, the accuracy differences between sizes tend to co-vary with changes in the share of synergy S: larger checkpoints that improve more in accuracy also exhibit larger increases S. This is consistent with Finding 2, where on synergy-driven tasks performance is more closely tied to S than to Uz.
8

---

Published as a conference paper at ICLR 2026
Table 3: Scaling on synergy-driven tasks: changes in accuracy (AAce) and PID shares (AS, AU2)
for S-MM and M-+VL within representative families.
Family
Sizes (B)
S
MM
(%)
M-VL
(%)
AAcc
AS
AU2
AAcc
AS
AU-
LLaVA- Aeee Vision
0.5-77-772
11.9
11.9
-6.5
3.1
14.8
-9.7
Qwen2-VE
2-47-$72
5.7
0.6
8.4
-3,9
0.4
-0.6
Qwen2.5-VI.
3-47-472
1.3
-6.3
1.0
2.5
5.5
2.5
InterVV1.2.
2-48-+78
7.3
36.8
-55.6
3.6
10.6
3.8
Intern VL3
2-48-778
2.7
2.5
-6,2
6.4
4.6
-10.3
Finding 4: On synergy-drives tasks, within each family the accuracy differences between avail- able sizes are more closely associated with changes in S than with changes in U2. Larger checkpoints therefore tend to benefit more from stronger multimodal fusion than from further amplifying language-side priors.
4.2 DIMENSION 2: LAYER-WISE INFORMATION DYNAMICS
Because stable family preferences and gains with scale in Dimension 1 tracked increases in S, the next question where arises in stack, We therefore analyze layer-wise information with the logit lens; full results are in Appendix E.2
Figure 4: Layer-wise PID dynamics for representative models on synergy-driven (MMBench, top) and knowledge-driver (PMC-VQA, bottom) tasks. A consistent three-phase pattern appears across models and datasets.
For Qwen2.5-VL. 5-LB and Intern VL3-8B, since the logit lens does not yield meaningful intermediate predictions, we omit them here. Figure 4hsows a consistent three-phase profile across models and datasets. S typically emerges and peaks in the middle-to-late layers, often softens near the output, then spikes at the final layer. U2 generally builds through the stack, peaking at the second-to-last layer before sharp final drop. R and Un remain small throughout. An exception is Intern VL.3-BB, where U2 does not exhibit the final drop.
Finding The layer-wise dynamics reveal a shared three-phase pattern information flow. Information emerges in the middle-to-latee layers, then moves from language- baataageeeseddddde representa- tion building in the later layers to a decisive, synergistic fusion of modalities in the final layer.
4.3 DIMENSION 3: LEARNING DYNAMICS OF MULTIMODAL FUSION
While layer-wise snapshots reveal where S arises, they do not show when it emerges. We therefore turn to the training trajectory. We trace PID through the two-stage training L. (7B, 13B). Full results in Appendix

---

Published as a conference paper at ICLR 2026
The results in Figure show a clear separation be- tween the two training stages in our reproduced LLa pipeline. Throughout alignment pre- training (Stage 1), both S and U2 remain low and relatively stable. Once visual instruction tun- ing begins (Stage 2), both components increase markedly. The effect of model scale also differs across components: the 7B model shows a more pronounced increase in S, whereas the 13B model exhibits stronger increase in U2, indicating that larger models in this setting place greater em- phasis on language-side priors during fine-tuning. Overall, both trends suggest that fine-tuning ben- efits from scale.
PHOHOA
25
04
24.
Figure 5: Evolution of S and U2 during two- stage training of LLa (7B, 13B).
This aligns with observations from prior work such as MiniGPT-4 (Zhu al., 2023), where the second-stage visual instruction tuning was found to be crucial for improving generation quality. Our PID analysis provides an information-theorriie view of this phenomenon, showing how fusion and language priors U2 emerge and diverge across stages and model scales.
Finding 6: In the two-stage training of LLaVA-1.55 multimodal fusion S remains negligible during alignment pretraining and increases primarily during visual instruction tuning. Across the available sizes, the 7B model shows relatively larger gains in S, whereas the 13B model exhibits comparatively stronger growth in U2 during this stage.
Summary of empirical findings. Taken together, results show that PID provides coherent view of LVLM behavior across three complementary axes. At the task level, benchmarks fall into recurring information-use regimes, and at the family level, model families adopt contrasting strate- gies characterized by different balances S and U2; at the layer level, S and U2 follow a shared three-phase pattern of information flow; and across training stages, multimodal fusion S emerges primarily during visual instruction tuning and interacts with model scale.
5 CONCLUSION
LVLMs are typically evaluated by accuracy, which tells us what they get right but not how different modalities are utilized. In this work, we introduce PID-based framework that yields a process-level decomposition of cecision-relevann information in LVLMs and, to our knowledge, provides the first systematic application of PID at this scale. By adapting a scalable PID estimator to LVLM outputs and applying it to 26 models across four benchmarks, we offer an information- complements accuracy-only evaluation and supports more targeted analysis of multimodal behavior.
lens that
This study has several limitations. (1) PID estimation assumes a discrete target space, so we do not cover fully open-ended generation tasks. (2) Our unimodal probes are approximate: masking modality with calibrated noise stabilizes estimation, but U1. U2. and S are measured under this probe rather than under truly natural unimodal inputs. (3) PID correlational: the components derived from model predictions and inputs, and their relationships accuracy interventions reflect associations rather than full causal mechanisms.
Future work extend this study in several directions:
1. Methodology:
developing PID estimators and output encodings that handle richer generative settings and additional modalities, and exploring complementary unimodal probes.
Model and training design: using (U1, U2, S) as diagnostic signals during scaling and instruc- tion tuning, and potentially auxiliary objectives to balance fusion and language priors.
3. Evaluation: using PID-based analyses to guide the construction of benchmarks that explicitly require high synergy S or isolate language priors U2.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by JST-SPRING Grant Number JPMISP2108, JST-CRONOS Grant Num- ber JPMJCS24K88, JSPS KAKENHI Grant Number JP23K28123 12, and the Institute of Al and Beyond of the University of Tokyo.
ETHICS STATEMENT
All authors have read and follow the ICLR Code of Ethics. Our study analyzes publicly available LVLM checkpoints and datasets (MMBench, POPE, Reefknot and PMC-VQA); no new data were collected, no human subjects were involved, and no personally identifiable information used. We respect dataset/model licenses and cite original sources; do not redistribute proprietary content. The work focuses on interpeeabilliyy/aallysiss (PID of information use) rather than deployment, and is intended to improve transparency of multimodal systems. Potential risks (e.g., inherited dataset or model biases) are acknowledged; we report cases in the paper'appendix. The authors declare no conflicts of interest and no sponsorship that influenced the results.
REPRODUCIBILITY STATEMENT
Our proposed PID framework and experimental design described in Section 3 Full tables/plots and additional analyses appear in Appendix [ and robustmess checks in Appendix B The public repository linked in the abstract contains code and configuration files needed rerun the study, including dataset splits, prompts, preprocessing steps, model versions, and random seeds, as well as scripts to compute PID.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190-4 019004499,1100 2020.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprini
2016.
Mann-Bapistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Antonio Almudévar, José Miguel Hernández-Lobato, Sameer Khurana, Ricard Marxer, and Alfonso Ortega. Aligning multimodal representations through an information bottleneck. arXiv preprint arXiv:2500..04870 2025.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. Advances in Neural Infor- mation Processing Systems, 37:136037-136083 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502 13923, 2025a.
Shuanghao Bai, Wanqi Zhou, Pengxiang Ding, Wei Zhao, Donglin Wang, and Badong Chen. Re- thinking latent redundancy in behavior cloning: An information bottleneck approach for robot manipulation. arXiv preprint orXiv:2502.028833, 2025b.
Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Mas- siceti. Understanding information storage and transfer in multi-modal large language models. Advances in Neural Information Processing Systems, 37:7400-7426, 2024.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşrllar Introducing our multimodal models, 2023. URL ht tstps: :/s:://www //ww.. adept. a ai/blog/ folloo-//uuuubb
11

---

Published as a conference paper at ICLR 2026
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint 2rX1v:2303.08112, 2023.
Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 16(4):211122883 2014.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi- modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pp. 397-406, 2021.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
Young Sang Choi, Vincent Jeanselme, Pierre Elias, and Shalmali Joshi. Icym2i: The illusion of multimodal informativeness under missingness. arXiv preprint arXiv:2505.1695 2025.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318- 2023.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Bartosz Cywiński, Emil Ryd, Senthooran Rajamanoharan, and Neel Nanda. Towards eliciting latent knowledge from Ilms with mechanistic interpretability. arXiv preprint arXiv:2505. /4352, 2025.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpos vision-language models with instruction tuning. Advances in neural information processing systems, 01492200-9261, 2023.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang. Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier-class multimodal 1402, 1lms. arXiv preprint arXiv:2409. 1400: 2024.
Pasan Dissanayake, Faisal Hamman, Barproda Halder, Ilia Sucholutsky, Qiuyi Zhang, and Sang- hamitra Dutta. Quantifying knowledge distillation using partial information decomposition. In International Conference on Artificial Intelligence and Statistics, pp. 4474-4482 PMLR, 2025.
David A Ehrlich, Andreas C Schneider, Viola Priesemann, Michael Wibral, and Abdullah Makkeh. A measure of the complexity of neural representations based on partial information decomposi- tion. arXiv preprint arXiv:2209. 10438, 2022.
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. ICL.R, 2024.
Gabriel Goh, Nick Cammarata 9. Chelsea Voss t. Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23911/diistll..0000
Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178, 2024.
John Hewitt and Christopher Manning. A structural probe for finding syntax word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 4129-4138. 2019.
12

---

Published as a conference paper at ICLR 2026
Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3543-355555 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Jingjing Jiang, Ziyi Liu, and Nanning Zheng. Correlation information bottleneck: Towards adapting pretrained multimodal models for robust visual question answering. International Journal of Computer Vision, 132(1):185-200 2024a.
Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. arXiv preprint arXiv:2410.0266,, 2024b.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Scong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.033211 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint vrXiv:2408.03322 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- vision-language models. arXiv preprint 2.Xi::3305.10355 2023b.
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Pro- cessing Systems, 36:27351-27392 2023a.
Paul Pu Liang. Yun Cheng, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multimodal fu- sion interactions: A study human and automatic quantification. Proceedings of the 25th International Conference Multimodal Interaction, pp. 425 435, 2023b.
Paul Pu Liang, Chun Kai Ling, Yun Cheng, Alexander Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Phillppi Morency, and Russ Salakhutdinov. Multimodal learning without labeled multimodal data: Guarantees and applications. In ICLR, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 34916, 2023b.
Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision-language models via latent space steering. In The Thirteenth International Conference Learning Representations, 2025.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 15:17359-17372 2022.
13

---

Published as a conference paper at ICLR 2026
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209. 2022.
Meta AL. Introducing laaa 3.1: Our most capable models date. https://ai. meta com/
blog/meta- 1ateett--llmaa---- 1-iiamma--1// July 2024. Meta Al blog post.
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpret- ing visual information processing in vision-language models. arXiv preprint arXiv:2410.07149, 2024.
Yaniv Nikankin, Dana Arad, Yossi Gandelsman, and Yonatan Belinkov. Same task, different circuits: Disentangling modality-specific mechanisms in vlms. arXiv preprint arXiv:2506609047. 2025.
nostalgebraist.
Interpreting gpt: The lens. http:////www.eessrrrnngoooo haspep/wppaittte//aaapppttrrrrro
August 2020. Less-
Wrong.
Changdae Oh, Zhen Fang. Shawn Im, Xuefeng Du, and Yixuan Li. Understanding multimodal Ilms under distribution shifts: An information-theoreti approach. arXiv preprint arXiv:2502.00577, 2025.
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL [RRLhhtps: /aa::///axiiii orgo abs /2fos/2444 2412 2.. 15115
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(14 1-67, 2020.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001.0033 2020.
Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pp. 2862-2867, 2023.
Xiaojun Shan, Qi Cao, Xing Han, Haofei Yu, and Paul Pu Liang. Mint: Multimodal instruction tuning with multimodal interaction grouping. arXiv preprint arXiv:2506.02308 2025.
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 2733):79-423, 1948.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786. 2025.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceed- ings the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601, 2019.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
14

---

Published as a conference paper at ICLR 2026
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang. Manoj Middepogu, Ziteng Wang, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems, 37:87310-87355 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.10 299,22 2023.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-220 212, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing aision-languag model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Ying Wang, Tim GJ Rudner, and Andrew G Wilson. Visual explanations of image-text representa- tions via multi-modal information bottleneck attribution. Advances in Neural Information Pro- cessing Systems, 36:16009 1609--1006,,, 2023.
Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2511 2010.
Qilong Wu, Yiyang Shao, Jun Wang, and Xiaobo Sun. Learning optimal multimodal information
bottleneck representations. arXiv preprint arXiv:2505. 19996, 2025.
Xiongye Xiao. Gengshuo Liu, Gaurav Gupta, Defu Cao, Shixuan Li, Yaxing Li, Tianqing Fang, Mingxi Cheng, and Paul Bogdan. Neuro- inspired information-theorets hierarchical perception for multimodal learning. arXiv preprint arXiv:2404. 094003 2024.
Shijia Yang, Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, and Chenfeng Xu. Law of vision representation in milms. arXiv preprint arXiv:2408. 16357, 2024.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305. 10415, 2023.
Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. Cross-modal information flow in multimodal large language models. Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pp. 19781-19791, 2025.
Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large lan- guage models. arXiv preprint arXiv:2408.098 09429, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Intermv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 2Xxv:2504.10477 2025a.
Zhiyu Zhu, Zhibo Jin, Jiayu Zhang, Nan Yang. Jiahao Huang, Jianlong Zhou, and Fang Chen. Nar- rowing information bottleneck theory for multimodal image-text representations interpretability. arXiv preprint arXiv:2502. 14889, 2025b.
15

---

Published as a conference paper at ICLR 2026
LLM USAGE
We used an LLM-based assistant only for writing support, including sentence paraphrasing, gram- mar/typ checking, and outline/flow refinement. The LLM was not used to design experiments, analyze data, generate results; all technical content (methods, equations, figures, and code) was produced and verified by the authors. LLM LLM-suggested text was reviewed and edited for accuracy, and references were inserted and checked manually.
A PARTIAL INFORMATION DECOMPOSITION AND ITS ESTIMATION
A.1 INTRODUCTION OF PARTIAL INFORMATION DECOMPOSITION
Classical mutual information (MI) quantifies the information a single variable provides about an- other: Y(X;Y) measures the reduction in uncertainty from H(Y H(Y X) (Shannon 1948). Extending this analysis to system with multiple source variables- for instance, two sources X1 and X₂ with state spaces X1 and X2, and target Y with state space V challenging, interaction information, [(X1; X2; Y), can be positive or negative. This sign ambiguity compli- cates its interpretation and motivates a decomposition of information into a set of well-behaved, non-negative quantities.
the standard
Partial information decomposition (PID), first proposed by Williams & Beer (010)) addresses this issue. We adopt the definition from Bertschinger et al.] (2014), which decomposes the total informa- tion into three conceptual components: redundancy, uniqueness, and synergy. These concepts are quantified by four non-negative atoms: redundant information (Rt), unique information from the first source (U1), unique information from the second source (U2), and synergistic information (S).
PID postulates the following consistency relations linking the four atoms to 4 classical mutual in- formation terms:
T T(X1, X2; = R+ U1 + +
(7)
1(X1;Y) R UU,
(8)
1(x2 = R+ U2,
(9)
1211; ((X;)))
((X, S.
(10)
Eqs. 1 establish the algebraic relationship between (R, U1, U2, and the usual information
measures, cleanly separating redundancy (R) from synergy (S) through the co-information identity
Eq. I0]
The definition for the PID atoms relies on optimization over the set of marginal- distribu- tions:
AP = (QE::: Q(I.. v) P(x,, Va, E N, y VED, iE {
(1I)
where A is the space of all possible joint distributions of X1, X₂ and Y. This set contains all distributions Q that preserve the marginal information of the true distribution P, while allowing other dependencies to vary. Let lQ(+) denote mutual information under distribution Q, the PID atoms are defined as:
Y,; R = max QEAr
(12)
U1 min QEAp
Iq(X;:Y Y (22)
(13)
U2 min Tq 1x(X); (x))
(14)
QEAr
= 1(X,, Qear
min
(x;
(15)
Intuitively, optimizing over Ap isolates each component of information. For instance, a distri- bution Q" Ap that minimizes the total mutual information Io;Xi,X;; does so by reduc- ing the higher-order dsynegisticcoompeeeetaay dependencies while preserving the source-target marginals. The resulting gap, used to define synergy in Eq. 15 therefore measures this emergent
16

---

Published as a conference paper at ICLR 2026
information. Similarly, the unique information components (Eqs. 13 and 14) represent the mini- mal necessary conditional information from each source, while redundancy (Eq. represents the maximal possible shared co-information.
Under this construction, the four atoms are non-negative and satisfy the axiomatic relations in Eqs.
7 10, A key insight from [Bertschinger et al. (014) is that the optimization problems defining
R. U1 22 and are equivalent, and is sufficient to solve one of them.
A.2 THE ESTIMATOR WE LEVERAGE: BATCH FOR CONTINUOUS REPRESENTATIONS
We adopt BATCH, scalable PID estimator for high-dimensional. continuous input representations X1 and X2 and large datasets (Liang et al. 2023a). The method amortizes the optimization problems
in Eqs. 12 15 by learning a parametric joint ( T2, y) that lies in the marginal-matchin set Ap
defined earlier. Specifically, Q is trained to approximately solve
min QEap
(16)
and, by the equivalence of the optimization characterizations, the remaining PID components obtained by evaluating the corresponding quantities under the same Q
Neural parameterization and projection. Given mini-batches (X1, 22, Y), two encoders fo(1) and S0(2) output features that define unnormalized joint via similarity matrix:
A = exp (Jec)) 11, y) S0(2) (X22)((221)) (x,.1) ))
(17)
where - X x2( for each y E y. To enforce the marginal constraints Q Ap, BATCH applies the Kinkhorn-Knoph algorithm (Cuturi (013) to iteratively normalize rows and columns of A the projected distribution matches the fixed pairwise marginals P(z1. y) and 9).
Training objective. Given matrix A representing @ 22, y), the objective can be written as:
( x Q(1 y) min E (z.gn)(ar log Eyerr ( Ii, 11 Q(v x Q((i)
(18)
QEAP =2~Q(1a()1.))
hence gradient descent leveraged to optimize this via updating the parameters fø(1) and fo(2)-
Estimating PID values via learned models. Upon convergence, estimate the required infor-
mation terms under the data distribution P and the learned Q. Using the consistency relations and the optimization equivalence, the PID components are obtained as
R Ia(Y; X2).
(19)
Ui Io(Y; X1, Ir(V;
(20)
U2 = X1, - 1r(Y;X1),
(21)
S = IP(Y; -
(22)
These quantities satisfy the PID consistency equations by construction and recover the optimization- defined components when Q is learned.
Rationale for adopting the BATCH estimator. The BATCH estimator provides a practical and scalable method for calculating these PID atoms from data. The core of this approach is to use neural networks to parameterize and learn approximate joint distribution, Q, that satisfies the required marginal-matching constraints. By optimizing an information-theereticc objective over mini-batches, the estimator can be effectively applied to large datasets. We chose to adapt this estimator for two primary reasons. First, it was explicitly designed for general multimodal learning contexts. Second, and most importantly, it operates high- digh-dimensional, continuous features. This latter property makes uniquely suited for analyzing modern LVLMs, as our framework can apply it directly to the rich vector embeddings these models produce quantify their internal information dynamics.
17

---

Published as a conference paper at ICLR 2026
B ABLATION STUDY
To validate our methodology, we examine sensitivity to two implementation choices: (i) feature summarization (mean pooling, last-hidden state, and max pooling) and (ii) the confidence threshold E {0.2, 0.3, 0.4]. We evaluate four representative LVLMs chosen to span families, scales, and strategy rypes: Qwen2.5. and 1ren2.55VLL-72. (fusion-centric, two scales of the same fam- ily), Gemma3-4B (language-centrr) centric), and Cambrian-23 -34 (language- with more parameters). Ablations are rum on the synergy-drive MMBench and the knowledge-drie
PMC-VQA datasets. Because S and U2 are the primary components in these regimes, we report two summary tables: synergy on MMBench (Table and language uniqueness U2 on PMC-VQL VQQ (Table
Table on MMBench for four chosen models under two ablations (feature summarization and confidence threshold).
Model
Feature summarization
Confidence threshold
Mean (ours)
Last-hidden
Max-pool
0.3 (ours)
0.2
0.4
Qwen2.5-VL-7B
1.112
1.112
1.112
1.112
1.112
1.112
Qwen2.5-VL-722
1.088
1.088
1.088
1.088
1.088
1.088
Gemma3-4B
0.167
0.172
0.173
0.167
0.167
0.167
Cambrian-34B
0,630
0.637
0,630
0,630
0,606
0.630
Table U2 on PMC-VQA for four chosen models under two ablations (feature summarization and confidence threshold).
Model
Feature summarization
Confidence threshold
Mean (ours)
Last-hidden
Max-pool
0.3 (ours)
0.2
0.4
Owen2.5-VI--7B
0.665
0.665
0.665
0.665
0.665
0.665
Qwen2.5-V1.-77.
0.893
0.893
0.893
0.893
0.893
0.893
Gemma3-4B
1.864
1.864
1.864
1.864
1.864
1.864
Cambrian-344 -34B
0.698
0.698
0.698
0,698
0.698
0,698
Input feature summarization. We compare mean pooling (used in the main experiments) with two common alternatives: last-hidden state and max pooling. On MMBench (Table 4 S is unchanged for the two Qwen2.5-VL models; Gemma3-4B varies slightly (0.167 + 0.173), and Cambrian-34B varies within 0.630-0.637. On PMC-VQA (Ta U2 is identical across all pool- ing choices for all models. Thus, feature summarization has negligible effect on the components most relevant to each regime.
Confidence threshold 7. We vary the regularization threshold around our default (7 - 0.3) 3) to r {0.2, 0.4). On MMBench (Table 4. is invariant for both Qwen2.5-VL models and for Gemma3- 4B; Cambrian 34B shows small dip at 0.2 (0.606 0.630 at 0.3/0.4). On PMC-VQA (Table[3), U2 is unchanged across т for all models. These results indicate stability of our conclusions with respect to the confidence-reguulariiatoon setting in the tested range.
Summary. Across both ablations, the regime-defining components (S on MMBench, U2 on PMC- VQA) remain effectively constant, supporting the robustness of our methodology.
18

---

Published as a conference paper at ICLR 2026
C DETAILED EXPERIMENTAL SETTINGS
All experiments reported in this paper, including model inference, PID estimation, and training reproduction, were conducted on servers equipped with NVIDIA A100 GPUs.
General inference details. For all multiple-choice VQA tasks, we use a standardized prompt that instructs the model to answer with only the letter of the correct option: "Please select the correct answer from the options above. You must answer with the letter of the correct option only." While most modern LVLMs adhere to this instruction, some earlier models (e.g., InstructBLIP, Fuyu-8b) tend to generate conversational, free-form text.
To handle these inconsistencies, our reported "accuracy" is not a standard logit-based metric. stead, perform strict string match the first generated token: prediction is correct only if the normalized token (lowercased, punctuation removed) exactly matches the ground-trut trtnhhhhrthhhu letter. This format-dependent evaluation means the performance floor is 0%, not the random-guess rate, models that fail to follow the required format will marked incorrect. For reproducible outputs, deterministic greedy decoding strategy (no sampling) for all models.
Hyperparrameters for BATCH estimator. Although the BATCH estimator is known to be robust to hyperparameter settings (Liang et al.) 2023a), we adhere to the original configuration for con- sistency and reproducibility. The key hyperparameters used for the estimator's 's neural networks are listed T
Table 6: Hyperparameters for the BATCH Estimator.
Hyperparameter
Value
Learning rate
lc-3
Optimizer
Adam
Number of epochs
Network architecture
3-layer MLP
Hidden dimension
32
Activation function
ReLU
Training batch size
256
Test batch size
256
Hyperparameters for reproducing LLaVA-1.5 training. We exactly follow the official two- stage recipe of Li al. (2023a) with no changes to data, model. or optimization hyperparameters. For analysis, we saved four equally spaced checkpoints from each stage and evaluated with greedy decoding (sampling disabled). For full hyperparameters, see Table of If et al. (2023ap.
19

---

Published as a conference paper at ICLR 2026
D CASE STUDIES
To provide a more qualitative understanding of our findings, we visualize the PID results for two representative examples. For each VQA pair, we show the outputs from four LVLMs that exemplity different strategies (fusion-centric vs. language-centrie) and scales. These cases provide concrete illustrations of how different models use information to solve tasks from the distinct regimes we identified.
Case 1: synergy driven task (MMBench). This task requires the model to identify the state of Massachusetts map. Success depends on correctly associating the visual shape of the state with its name in the text-a a classic fusion task where neither modality alone is sufficient.
Image:
Text:
Question: What is the name of the place shown? Vermont Connecticud Rhode Island Massachaetts Please select the correct answer from the options above. You must answer the of the correct option only.
Correct answer (label): D
Option:
PID value:
A
B
C
D
R
U1
U2
S
X()]
0.21
0.79
Gemma3-4B
P(V))) [0.25
0.25
0.25
0.25
[0
0.15
1.73
0]
P(V[X2) [0.93
0.07
P(VJX, X2) P(V(X,) [0.25
1.0
Llama-3.2-vision
0.25
[0.32
0
0.61
1.47]
P(V[X,) [0.05
0.72]
P(Y]x, x2) [O P(V[X,) [0.25
1.0
Qwen2.5-VL-7B
0.25
0.25
0.25
[O
0.47
0.57
1. 1.48)
P(Y(X2) [0.41
0.17
0.17
0.25
P(YIX, X2) [0.01
0.01
0.02
0.96]
Qwen2.5-VL-72B
P(У]X)) [0.25
0.25
0.25
0.25
[o
0.67 0.
13]
P(V[x2) [0.15
0.22
0.28
Figure 6: PID analysis for synergy-driven task. All models answer correctly, but the PID results reveal two distinct solution strategies: generating high synergy (Lllam-332-vision,, Qwen2.5-VL) versus correcting strong, incorrect language prior (Gemma3).
As shown in Figure all models arrive at the correct answer, but their methods differ dramatically. The fusion-centric models (Lllam-3.2-vviiion and the Qwen2.5-VL. series) solve the problem by gen- erating a large amount of synergy S, confirming that the answer emerges from the direct interaction of image and text. In stark contrast, the language-centric Gemma model also answers correctly, but does so by overcoming a strong, incorrect language prior (a high U2 favoring "Vermont"). Interest- ingly, the PID framework reveals this correction happens without generating synergy, showcasing a different path to success: the visual information acts to override mistaken language bias, rather than creating new information with it. This highlights the framework's ability to distinguish between models that truly fuse modalities to create new insight and those that arbitrute between conflicting unimodal beliefs. This latter "correction" mechanism, where visual evidence Un overrides a strong language bias U2. may represent efficient, non-synergistic strategy common in smaller more language-centric architectures.
20

---

Published as a conference paper at ICLR 2026
Case 2: A knowledge-driven task (Reefknot). This task asks about the spatial relationship "through", which requires a nuanced understanding of prepositions that primarily resides within the language model. The visual context is relatively simple, but the linguistic concept is complex.
Figure Sesure/illusen fillustrates how these different strategies adapt to a knowledge-driven task, presenting a clear contrast to Case 1. As expected, the language-centric Gemma model solves the problem almost exclusively with its language prior (high U2), generating no synergy. The fusion-centric models 3llvma-..2-vision and the Qwen2.5-VL family), however, tell also depend heavily on language uniqueness, they continue to generate significant synergy (e.g.. 0.95 for Qwen2.5-VL-7B)
interesting story: while they
This demonstrates persistent strategic difference between model families, Even when a task seems solvable with language alone, fusion-centrie models consistently attempt to integrate visual infor- mation to ground their linguistic understanding, whereas language-centric models default to their strong language priors.
Image:
Text:
Question: What is relation car window this photo?
through left of containing
select the correct onswer from the options above. You with the letter of the option only.
Correct answer (label): B
Option: B
PID value:
A
C
D
R
U1
U2
S
P0)] [0.24
0.67
0.09
0
Gemma3-4B
P(Y)x)) [0.25
0.25
0.25
0.25
[0
0.57
1.87
0]
P(V(x2) [0.79
0.18
0.03
0.
P(Y)X,,X,) [0.03
P(Y)x)) [0.25
0.25
0.25
0.25
[0.21
0.74
0.33]
P(Y)X,) [0.01
0.77
0.1
0.12]
P(YIX,;X,) [0.01
0.03
0.01]
Qwen2.5-VL-7B
P(Ч(X,) [0.25
0.25
0.25
0.25
[O
0.60
0. 68
95
P(Y(X2) [0.18
0.18
0.26
P(YIX, X2) [0.02
0.07
Qwen2.5-VL-72B
P(V[X,) [0.25
[0 0.07
1.11 0.41]
P(Y(X2) [0.01 0.07 0.17
0.75
Figure 7: PID) analysis for knowledge-driven task. While all models rely heavily on language uniqueness U2. fusion-centrie models like L.ama-3.2-vision and Qwen2.5-VL also generate non- trivial synergy, unlike the language-centric Gemma.
These case studies provide qualitative evidence for our quantitative findings, visually demonstrating how models with different core strategies can reach the same correct answer via entirely different information-processing pathways.
21

---

Published as a conference paper at ICLR 2026
E FULL RESULTS
E.1 FULL INFORMATION SPECTRA ON FOUR DATASETS
This section provides the full information spectra, including all four PID components (R, U1, U2, S) and the corresponding accuracy for all 26 models across the four evaluated datasets. These fig- ures supplement our main analysis Section 4 where we focused on the two most discriminative components, synergy S and language uniqueness U2.
2.5
1.5
Qvenz/vL
Gamera
Interii3
Cambrian Una-2.5
Figure 8: Full information spectrum and accuracy on the MMBench dataset. This figure provides detailed evidence for the synergy-drive regime. It visually confirms the existence of two primary model strategies: fusion-centric families (e.g.. LLaVA-ov, Qwen2.5-VL, InternVL3) show a large share of synergy S, while language-centric families (e.g., Gemma3) are dominated by language uniqueness U2. Furthermore, the plot illustrates that scaling models tends to increase synergy within many families.
2.8
2.5
Owind
Qwved.5
Cambeinn
Figure 9: Full information spectrum and accuracy on the POPE dataset. As synergy- task, POPE shows that synergy S is key component for many high- high-performiig models. Uniquely, this dataset also elicits significant redundancy (green bars). This is likely because the task uses simple binary questions about common objects (from COCO (Lin et al. 2014)). In this context, both the visual modality (seeing the object) and the language modality (understanding the object's name) can independently confirm the object's presence, leading to high informational overlap.
22

---

Published as a conference paper at ICLR 2026
2.8
6.5
Figure 10: Full information spectrum and accuracy on Reefknot. This plot exemplifies the knowledge-driven regime. Language uniqueness U2 is the overwhelmingly dominant information component for nearly all models, demonstrating that performance constrained by language-side priors. Even in this environment, the fundamental strategies of model families persist: fusion-centric models (e.g., Llama-3.2-v) still generate noticeably more synergy S than their language-centric counterparts (e.g.. Gemma3 and Cambrian).
2.5
3:5
72
122
Uamu:322
Unve3.3 eoSuCUBLP
Full information spectrum and accuracy on PMC-VQA. As a specialized, domain- specific task, this dataset clearly exemplifies the knowledge-driven regime. Similar Reefknot, performance is heavily dominated by language uniqueness U2. confirming that models must rely on internal, language-basse based knowledge for specialized topics. This result strongly supports our finding that for Nomain-sspecine tasks, performance is fundamentally limited by a model's 's internal, language-basse knowledge. The negligible synergy S in many top models highlights the challenge of multimodal fusion when highly specific domain knowledge is required.
Figure 11:
23

---

Published as a conference paper at ICLR 2026
E.2 FULL LAYER- WISE RESULTS ON MMBENCH AND PMC-VQL
This section provides the full, layer-by-layer information spectra for the representative models dis- cussed in our main analysis. These plots show the values for all four PID components (R, U1; U2, S) across transformer blocks for each model on both MMBench and PMC-VQA. They provide the de- tailed evidence for the three-phase layer-wise dynamics shared among different LVLMs.
InternVL3-28
Qwen7.5-VL.35
2.0
MMBBnnc
Vilue
1.0
14
21
20
10
27
36
2.0
1.5-
PMC-VIA
Value
1.0
21
28
Layer
Layer
Figure 12: Layer-wise PID dynamics for InternVL3-2B and Qwen2.5-VL-3B. The plot for Qwen2.5-VL-3B clearly illustrates the standard three-phase reasoning process: after information emerges, the later layers show a phase of representation building (rising U2), which culminates in decisive fusion event at the final layer (a spike in S and drop in U2). In contrast, lacks the final fusion event; its language uniqueness U2 continues to rise into the final layer without the characteristic drop. This may be due to its relatively smaller and shallower LLM, which might not have the capacity for the distinct final-layer fusion seen in other models.
VL.3-BB
1.5-7
LL3VA-1.5-13B
KRBench
Vaiue
0.5
0.0
PMC-VQA
16
Layer
Layer
Figure 13: Layer-wise PID dynamics for LLaVA-115578 and LLaVA-15--33B The LLaVA-1.5 fam- ily also exhibits the three- phose reasoning process, though with slightly different characteristics. In- formation emerges in the middle layers of the network. The representation building phase is distinct, with language uniqueness U2 rising to a high plateau while synergy forms a noticeable "hump." suggesting ongoing fusion process prior the final layer. The process concludes with the char- acteristic fusion event, marked by drop in U/2 and a final spike in S at the output layer. Unlike the other models, 1. shows some minor, non-zero activity for redundancy and vision uniqueness Un in its later layers.
24

---

Published as a conference paper at ICLR 2026
E.3 FULL LEARNING- RESULTS ON MMBENCH AND PMC-VQA
This section provides the full information spectra traced across the eight training checkpoints of 1L. (7B and 13B). These plots show the values for all four PID components (R, U1, U2 S) on both MMBench and PMC-VQA. They provide the detailed evidence for the learning dynamics and scale-dependent effects identified in Finding 6.
MMBench
PMC
1.-
0.8
0.6
0.4
0.2
0.0
1/6
2/4
3/4
104
3/4
404
4/4
Pre-training
Rinetuniing
Pre-training
Fine-tuning
Figure 14: Learning dynamics LLaVA-1.577B, This figure provides evidence for the first part of Finding 6, showing how smaller models develop fusion. During Stage I (Pre-training), all PID components are negligible. Upon commencing Stage (Visual Instruction Tuning). there is dra- matic and sustained increase in synergy S, which becomes the dominant information component by the end of training. Language uniqueness U2 also increases but to a much lesser extent, confirming that the 7B model primarily prioritizes developing synergistic inference.
1.0
U2
0.8
0.6
2/4
3/4
Pro-tralning
Pre-training
Fine-tuning
Figure 15: Learning dynamics of LLaVA-I.5-13B. In contrast the 7B model, this figure illustrates the second part of Finding 6. While PID values are also flat during Stage 1, the larger 13B model exhibits massive and continuous increase in language uniqueness U2 during Stage 2, becoming by far the dominant information component. Although synergy S also grows, its increase is less pronounced than that of U2, demonstrating that larger models prioritize enhancing their language- side priors during visual instruction tuning.
25


Reviews:
[{"review_id": "GbhESYPSCo", "reviewer": "Reviewer_Das5", "summary": "This paper proposes an analysis framework based on Partial Information Decomposition (PID), using a BATCH estimator to quantify the redundancy (R), visual uniqueness (U1), linguistic uniqueness (U2), and synergy (S) of decision-related information in LVLMs. The authors conduct a three-dimensional analysis across models/tasks, within-layer information flow, and training stages on 26 open-source LVLMs across four multiple-choice VQA datasets. The main findings are: the models exhibit two information usage patterns, “synergy-driven” and “knowledge-driven”; the model families display two stable strategies, “fusion-centric” and “language-centric”; a three-stage information pattern appears within typical model layers; multimodal synergy primarily forms during the visual instruction tuning stage.", "questions": "1. What effects would appropriate interventions have on cases with high S? \n\n2. I am more concerned about whether the authors have conducted more fine-grained experiments — for example, in multimodal scenarios where answering the question necessarily requires image information, such as geometry problems.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novelty: It is the first systematic application of a scalable PID estimator to large-scale LVLMs, providing a multidimensional quantitative analysis.  \n\n2. Scale and Coverage: The workload is substantial, and the experimental matrix is quite comprehensive, supporting family-level induction.  \n\n3. Discoveries with explanatory value: Quantitative insights into task attributes, model strategies, within-layer flow, and training stages provide references for future architectural design/evaluation.  \n\n4. Method engineering: No model modifications are necessary; inference alone suffices, accompanied by confidence gating and soft aggregation to reduce estimation noise.", "weaknesses": "1. Task limitations: All experiments are confined to “multiple-choice VQA,” with a very small answer space. The authors acknowledge that BATCH requires discrete Y but should still discuss the feasibility of extending the framework to open generative tasks (e.g., discretization strategies, sample efficiency).  \n\n2. The authors do not conduct interventions to high S (e.g., occluding images or rearranging text) for validation. Adversarial ablation experiments could be added to support causal conclusions.  \n\n3. This paper is merely an exploratory study of existing conclusions and does not propose any new algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UbTA7CLXuW", "reviewer": "Reviewer_dfu9", "summary": "This paper proposes a framework based on Partial Information Decomposition (PID) to analyze the internal decision-making processes of Large Vision-Language Models (LVLMs). The authors adapt a scalable PID estimator (BATCH) to decompose model predictions into four components: redundancy ($R$), vision uniqueness ($U_1$), language uniqueness ($U_2$), and synergy ($S$). They conduct analysis across 26 LVLMs, four datasets, and three dimensions: cross-model/task comparison, layer-wise information dynamics, and training-stage evolution. The paper argues that this PID-based framework provides a more principled understanding of LVLM behavior beyond accuracy-based evaluation.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Overall, this paper is easy to follow.\n2. Introducing PID into MLLMs might be new.", "weaknesses": "1. The analysis is restricted to multiple-choice VQA tasks with a small set of predefined answers. This narrow focus limits the applicability of the findings to open-ended generation, captioning, or reasoning tasks, which are central to LVLM capabilities. The use of a finite output space is a methodological convenience that may not reflect real-world multimodal understanding, where outputs are often continuous and compositional, typically in recent reasoning models.\n2. The PID framework is correlational, not causal. It quantifies statistical associations but does not explain how or why certain layers or training stages lead to synergy.\n3. The authors claim that model families exhibit “stable, opposing strategies,” but this is only demonstrated on four datasets.\n4. The theoretical contribution is modest: while PID is elegantly applied, the core framework is not novel, and the findings are largely empirical and descriptive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oAc4f3QgTe", "reviewer": "Reviewer_mp9C", "summary": "In this paper, the authors propose using partial information decomposition (PID) to analyze VLMs (such as Qwen and InternVL). Based on this PID analysis, the authors draw several conclusions. The findings include: (1) Some datasets focus more on knowledge, while others focus more on synergy. (2)Fusion-centric model families consistently prioritize cross-modal reasoning, while language-centric families rely more heavily on language priors. (3) Larger models benefit from combining inputs more effectively rather than relying heavily on language priors. (4) Information emerges in the middle-to-late layers, shifting from language-based representation building in the later layers to a decisive, synergistic fusion of modalities in the final layer. (5) Multimodal fusion is unlocked during visual instruction tuning.", "questions": "My biggest question is that since the Y/X1 is usually a uniform distribution and the metrics calculated are based on the difference between Y/X2 and Y/(X1,X2). Why not just use the accuracy delta to measure the model performance and draw the conclusion. This paper seems to introduce another layer of abstraction and makes it harder to find the real insight from the data.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The authors get many findings based on the analysis tool.\n- The authors evaluate the performance of several different models.\n- The motivation of this paper is clear.", "weaknesses": "- I noticed that Y/X1 is estimated with a uniform distribution. I suspect this is because, when using only the vision tokens, the probability for all four options is very low (perhaps the EOS token has a high probability instead). However, does this imply that the visual tokens provide no information? Intuitively, this seems incorrect, but the resulting metric might suggest it. I think a good alternative would be to block the question text and only provide the \"A, B,C, D\" choices to the model. This approach might provide a better estimation of the information derived purely from the visual tokens.\n\n- The authors state that the Partial Information Decomposition (PID) framework and the BATCH estimator are existing, well-established methods, not a novel contribution of this paper . While the goal of moving \"beyond accuracy-only evaluation\" is commendable, the paper does not sufficiently justify why this complex, information-theoretic approach is superior to simpler, more direct ablation metrics. Intuitively, one could measure fusion by calculating the delta between the full-model accuracy and the accuracy of language-only input. The authors should provide a clearer argument or a baseline comparison demonstrating what novel insights PID provides that cannot be captured by such simpler, accuracy-based metric.\n\n- Some conclusions may not be very reliable.\n\n(1) Findings 1 & 2: The paper's first findings—that VLM behavior is governed by \"synergy-driven vs. knowledge-driven\" regimes —appears to be an overly complex description of a basic dataset property. A \"knowledge-driven\" task simply seems to indicate that many questions in that benchmark can be answered correctly using only the text, without the image. This feels less like a deep insight and more like a dataset deficit or design. Furthermore, the conclusion in Finding 2 that \"model performance is dictated by S\" on synergy-driven tasks  is somewhat tautological. A \"synergy-driven\" task is defined by its need for fusion, so it is expected that a metric measuring fusion (S) would correlate with accuracy.\n\n(2) Finding 4: The analysis of scaling effects in Finding 4 and Section 4.1.2 seems to overlook a critical confounding variable: model distillation. The analysis assumes that models of different sizes (e.g., Qwen2.5-VL-3B vs. 72B) are trained independently. However, for many state-of-the-art families like Qwen2.5-VL and InternVL3, it is common practice for smaller models to be distilled from larger, more capable ones. If this is the case, the interpretation is reversed. The paper frames the results as \"gains\" in S when \"scaling\" up. But if the smaller models are distilled, the data in Table 3 would represent information \"lost\" during distillation, not \"gained\" during scaling. This confound is not addressed and could fundamentally change the interpretation of how model scale affects multimodal fusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M5hdFyKFck", "reviewer": "Reviewer_99um", "summary": "This paper presents a novel analysis of Large Vision Language Models through the lens of PID, offering a quantitative framework to examine how different modalities contribute to model predictions. The authors systematically evaluate 26 models across multiple datasets, investigating cross-model and cross-task behaviors, layer-wise information dynamics, and training trajectories. Key findings include the identification of distinct model families characterized by either fusion-centric or language-centric strategies, as well as task regimes ranging from synergy-driven to knowledge-driven scenarios. These results reveal how architectural and training choices shape multimodal integration.", "questions": "See above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Conceptual Novelty:** The adaptation of PID to analyze LVLMs is both innovative and timely. This approach moves beyond performance-centric evaluation by providing a principled methodology to quantify the composition of information used in model decisions. It specifically separates redundant, unique, and synergistic contributions, creating a valuable diagnostic tool for understanding multimodal fusion.\n\n2. **Extensive and Systematic Evaluation:** The scale and scope of the experimental analysis represent a significant strength. The study encompasses 26 models of varying scales and families, four diverse datasets, and multidimensional evaluations including cross-model, cross-task, layer-wise, and training dynamics analyses. This comprehensive approach enables the authors to draw robust conclusions about model strategies and behavioral patterns.\n\n3. **Actionable Insights:** The paper delivers valuable empirical findings with practical implications. The consistent distinction between fusion-centric and language-centric model families, maintained across tasks and scales, provides a new dimension for model comparison and design. Furthermore, identifying visual instruction tuning as the critical phase for emergent synergy offers concrete guidance for future training strategies.", "weaknesses": "1.  **Validity of Unimodal Approximation:** The use of Gaussian noise to mask a modality is a significant methodological approximation. This creates a distribution shift between the training data distribution `P_train(X₁, X₂, Y)` and the evaluation distribution `P_test = P(X₁, Y) × N(μ, σ²)`. The model's behavior under this out-of-distribution `P_test` may not faithfully represent true unimodal reasoning and could introduce artifacts into the PID estimates.\n\n2.  **Limited Task Scope:** The PID framework's requirement for a discrete, finite output variable `Y` restricts the analysis to multiple-choice VQA. This is a severe limitation, as it prevents the study from probing the information-use strategies of LVLMs in their core capabilities of open-ended generation. The findings may not generalize beyond discriminative tasks.\n\n3.  **Black-Box Analysis of Reasoning:** The PID analysis provides an end-to-end, input-output decomposition (`X → Y`) but offers no insight into the *internal reasoning process*. It quantifies the information in the mapping `P(Y|X)` but ignores the latent multi-step reasoning `X → Z₁ → Z₂ → ... → Y` that may occur within the model's layers, leaving the \"chain of thought\" as a black box.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lTt86ma1HE", "reviewer": "Reviewer_LJBz", "summary": "This paper proposes a novel framework utilizing Partial Information Decomposition (PID) to quantitatively analyze the internal decision-making strategies of Large Vision-Language Models by breaking down their decision-relevant information into redundant, unique, and synergistic components. The comprehensive analysis reveals two major task regimes (synergy-driven vs. knowledge-driven) and two opposing family-level strategies (fusion-centric vs. language-centric), providing key insights into how multimodal fusion evolves during training", "questions": "## Questions\n\n1.  **Q1: Method Generalizability to Generative Tasks (Core Limitation).**\n    The paper concedes that the PID framework is restricted to multi-choice VQA tasks. Given that the primary function and value of modern LVLMs lie in **generative** and open-ended tasks (e.g., complex dialogue, zero-shot captioning), can the authors provide strong evidence or a theoretical argument that the observed information-flow strategies (e.g., the distinction between \"synergy-driven\" and \"knowledge-driven\") **reliably generalize** to these generative tasks? If the framework cannot analyze the most common usage of LVLMs, what is its practical prescriptive utility?\n\n2.  **Q2: Quantitative Rigor of the Unimodal Approximation (OOD Concern).**\n    The estimation of unimodal components ($U_V, U_L$) relies on replacing the \"missing\" modality with noise embeddings. This is an Out-of-Distribution (OOD) input to the model's language backbone. Has this OOD approximation been rigorously validated to ensure it faithfully reflects the model's behavior when a modality is truly absent? Since the values of $U_V$ and $U_L$ are contingent on this approximation, doesn't this fundamentally compromise the quantitative reliability of the derived synergy ($S$) component, which is the core subject of the analysis?\n\n3.  **Q3: Prescriptive Utility via Intervention (Guidance for Design).**\n    The paper identifies descriptive strategies like \"fusion-centric\" and \"language-centric.\" For this work to be truly impactful, it must be prescriptive. Suppose a model is diagnosed as \"language-centric.\" Can the authors propose a **specific training intervention** (e.g., modifying the loss function or cross-attention mechanism) directly based on their PID findings, and empirically demonstrate this intervention successfully **shifts the model strategy toward \"fusion-centric\"** while maintaining or improving performance? If the metric is not actionable, its value is limited to post-hoc explanation.\n\n4.  **Q4: Interpretation of Low Redundancy ($R$) across Models.**\n    In many experiments, the Redundancy ($R$) component—which measures the mutual overlap of knowledge between the two modalities—remains consistently at a low level. Does this low $R$ value imply: a) that the **knowledge overlap between modalities is genuinely negligible** even in highly-trained large models; or b) that the PID estimator (BATCH) has an **inherent systematic bias to underestimate redundancy** when dealing with high-dimensional feature representations? How do the authors reconcile and explain the persistently low $R$ values?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "## Strengths\n\n1.  **S1: Rigorous Information-Theoretic Framework for Quantifying Multimodality.**\n    The paper introduces Partial Information Decomposition (PID)—a sophisticated information-theoretic tool—to the domain of Large Vision-Language Model (LVLM) interpretability.\n    * This represents a significant step beyond common interpretability methods (like attention map visualization or simple ablation) by offering a **rigorous, quantitative measure** of multimodal fusion (Synergy, $S$) and unimodal reliance (Uniqueness, $U$).\n    * The PID framework provides a conceptually elegant method to precisely measure the **attribution gap** (i.e., whether success stems from fusion or unimodal priors), addressing a fundamental problem in LVLM research.\n\n2.  **S2: Comprehensive Empirical Scope and Detailed Learning Dynamics Analysis.**\n    The paper executes a highly comprehensive empirical study, analyzing **26 diverse LVLMs** across various families and capabilities (e.g., LLaVA, MiniGPT-4, BLIP-2).\n    * The scale of the analyzed models and tasks is commendable, lending strong statistical weight to the observed findings regarding task regimes (synergy-driven vs. knowledge-driven) and model strategies (fusion-centric vs. language-centric).\n    * Crucially, the analysis of **learning dynamics** (Tracing PID components across training checkpoints, e.g., LLaVA-1.5 7B vs 13B) provides novel, quantitative insights into how fusion and language priors evolve during specific training stages (Pre-training vs. Visual Instruction Tuning). This is a valuable contribution to understanding the VLM development pipeline.", "weaknesses": "### Weaknesses\n\n1.  **W1: Crippling Restriction to Multiple-Choice VQA (Severe Lack of Generalizability).**\n    The framework's core reliance on the BATCH PID estimator restricts the analysis \\textbf{exclusively} to tasks where the output $\\mathcal{Y}$ is discrete and finite (i.e., multiple-choice VQA).\n    * This is a critical flaw because modern LVLMs are primarily valued for their \\textbf{generative, open-ended capabilities} (e.g., free-form captioning, complex dialogue).\n    * A \"comprehensive analysis\" that cannot be applied to the dominant and most challenging LVLM tasks has severely limited scope and impact on the field. The analysis is confined to only a small subset of the model's true functionality.\n\n2.  **W2: Unreliable Approximation of Unimodal Information via Noise Masking.**\n    To isolate unimodal components ($U_V, U_L$), the authors employ a \\textbf{modality masking technique}, replacing one modality's embeddings with a statistically calibrated noise sequence.\n    * This is a strong approximation that \\textbf{fundamentally alters the input distribution} to the cross-modal layers. The model's behavior under this unnatural, perturbed input may not accurately reflect its true unimodal capability on natural inputs.\n    * This raises serious doubts about the reliability and quantitative validity of the derived unique ($U$) and synergistic ($S$) information measures, which form the basis of the paper's conclusions.\n\n3.  **W3: Core Findings Lack Substantial Novelty.**\n    The paper's main conclusions largely serve as a \\textbf{quantitative validation of widely established domain knowledge}, rather than offering fundamental new insights.\n    * The distinction between *Synergy-Driven* tasks (high $S$) and *Knowledge-Driven* tasks (relying on LLM priors, high $U_L$) is an intuitive and expected outcome based on task design.\n    * The finding that Visual Instruction Tuning is the critical stage for synergy ($S$) growth merely \\textbf{quantifies the intended effect} of this alignment stage. The framework successfully describes \\textbf{what} is happening but fails to provide novel insight into \\textbf{why} it is happening.\n\n4.  **W4: Lack of Actionable Guidance and Mechanistic Explanation.**\n    The analysis is primarily \\textbf{descriptive} (e.g., identifying \"fusion-centric\" vs. \"language-centric\" model families) but fails to provide \\textbf{mechanistic explanations} for these differences (e.g., how specific architectural choices or pre-training data compositions cause this strategy). Crucially:\n    * The paper offers \\textbf{no intervention experiments} to demonstrate that the PID metrics can be used prescriptively (e.g., in a loss function) to guide or improve model design.\n    * The work remains at the \"post-hoc explanation\" level, limiting its value as a tool for engineering next-generation LVLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)