Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional
Authors: Divyam Madaan
Varshan Muhunthan
Kyunghyun Cho
Sumit Chopra
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-modal learning
Primary area: datasets and benchmarks
TL;DR: Large-scale empirical analysis showing the varying strength of multi-modal dependencies across popular VQA benchmarks.

Abstract: Understanding the interplay between intra-modality dependencies (the contribution of an individual modality to a target task) and inter-modality dependencies (the relationships between modalities and the target task) is fundamental to advancing multi-modal learning. However, the nature of and interaction between these dependencies within current benchmark evaluations remains poorly characterized. In this work, we present a large-scale empirical study to quantify these dependencies across 23 visual question-answering benchmarks using multi-modal large language models (MLLMs) covering domains such as general and expert knowledge reasoning, optical character recognition, and document understanding. Our findings show that the reliance on vision, question (text), and their interaction varies significantly, both across and within benchmarks. We discover that numerous benchmarks intended to mitigate text-only biases have inadvertently amplified image-only dependencies. This characterization persists across model sizes and types, with models often obtaining high performance by using each modality independently and showing limited dependence on their interaction. We provide a quantitative characterization of multi-modal datasets, enabling a principled approach to multi-modal benchmark design and evaluation.

Published as a conference paper at ICLR 2026
MULTI-MODAL DATA SPECTRUM:
MULTI-MODAL DATASETS ARE MULTI-DIMENSIONAL
Divyam Madaan' Varshan Muhunthan' Kyunghyun Cho Sumit Chopra 1,2,3
Courant Institute of Mathematical Sciences, New York University! Center for Data Science, New York University² Grossman School of Medicine, New York University
CIFAR LMB"
{divyam. madaan,, vm2781, kyunghyun, cho, sumit)enyu.edu
ABSTRACT
Understanding the interplay between intra-modality dependencies (the contribu- tion of an individual modality to a target task) and inter-modality dependencies (the relationships between modalities and the target task) is fundamental to ad- vancing multi-modal learning. However, the nature of and interaction between these dependencies within current benchmark evaluations remains poorly charac- terized. In this work, we present a large-scale empirical study to quantify these dependencies across 23 visual question-answering benchmarks using multi-modal large language models (MLLMs) covering domains such as general and expert knowledge reasoning, optical character recognition, and document understanding. Our findings show that the reliance on vision, question (text), and their interaction varies significantly, both across and within benchmarks. We discover that numer- ous benchmarks intended to mitigate text-only biases have inadvertently amplified image-only dependencies. This characterization persists across model sizes and types, with models often obtaining high performance by using each modality in- dependently and showing limited dependence on their interaction. We provide quantitative characterization of multi-moda datasets, enabling a principled ap- proach to multi- benchmark design and evaluation.
INTRODUCTION
Rapid advancement of MLLMs has been accompanied by a significant increase in the number of evaluation benchmarks. A recent survey (Li et al., 2024) identified over 200 multi-modal bench- marks. However, this growth has not been accompanied by a systematic investigation into what these datasets measure. This means the relationships, redundancies, and unique contributions across and within the benchmarks are not well understood. It is often unclear whether a new dataset im- proves multi-modal evaluation is largely redundant with existing benchmarks. This ambiguity makes the principled selection of benchmarks for model evaluation significant challenge.
For example, datasets such as A12D (Kembhavi et al., 2016). ChartQA (Masry et al., 2022), BLINK (Fu et al., 2024), RealworldQA (xAI, 2024), V* Bench (Wu and Xie, 2024), TextVQA (Singh et al., 2019) were included in the Gemini 1.5 evaluation (Team et al., 2024), but were omitted from Gemini 2.5 (Comanici et al., 2025) with little justification for the changes. Such inconsistencies in evaluation protocols are common (xAI, 2024; Cohere, 2025), making difficult to determine whether the reported gains in performance represent true advances in capability simply adaptation to a different set of benchmark artifacts.
This lack of understanding has led to inefficient cycle of benchmark development. New datasets are created to address specific uni-modal dependencies (Agrawal et al., 2018), which in turn are found to have and unforeseen artifacts (Dancette al., 2021; Si et al., 2022). This process hinders consistent, long-term model comparison and undermines scientific rigor.
Prior work has analyzed the dependence individual modalities and their interaction in multi- modal models using representation similarity (Komblith et al., 2019), information-theored mea- sures (Tjandrasuwita et al., 2025; Lu, 2023; Madaan et al., 2024), and score-based methods (Gat

---

Published as a conference paper at ICLR 2026
Permuted Image
Permuted Question
How many layers does the earth have?
How many layers does the earth have?
Whats the relationship between two creatures?
Where is the giraffe
located picture?
a) 2
5
a) e)
b)
Compelltive c) Symbiotic
b) Parasitic d) Predatory
a] Compefiiive
b) Parasitic d) Predatory
c)
d) 4
d)4
c) Symbiotic
d)
C)
ScienceQA, MMMU, MMMUPro, TexIVQA, MMBench, BLINK MMVP, Q-Bench, MM-SStrr OCRBench, SEED-Bench, VizWiz
ScienceQA, MMMU, TexIVQA, V", BLINK, GQA, VQA, AI2D, Omni3D, CharlQA
Figure 1: Demonstration of intra-modality dependencies in multi-modal models using input permutation. (Left) The models answers about layers of Earth even when the image is replaced by unrelated diagram of brain. (Right) The model identifies a symbiotic relationship from the image even when the question is unrelated. These examples highlight a failure of multi-modal reasoning, where models exploit uni-modal priors with the options to obtain an associated answer.
et al., 2021; Parcalabescu and Frank, 2022; Hu et al., 2022; Wenderoth et al., 2025). While provid- ing valuable insights. these studies were limited in scope, focusing on synthetic data, smaller-seale benchmarks such VQA (Agrawal et al., 2018; Goyal et al., 2017), earlier generations of models.
To address this gap, we conduct a large-scale empirical study to characterize widely-used multi- modal benchmarks. We hypothesize that these benchmarks evaluate distinet combinations of un- derlying capabilities. To quantify the multi-dimensional nature of these dependencies, we use intra-modality dependencies (reliance on single modality for the target task) and inter-modality dependencies (reliance on the interaction between modalities for the target task) based on prior studies (Liang al., 2023; Madaan al., 2024). As shown in Figure 1, MLLMs often exploit intra- tiral-todllity dependencies, answering questions correctly even when a relevant input modality replaced with corrupted or random data. To quantify modality reliance, we adapt Perceptual Score (Gat et al., 2021). We permute one modality across samples while keeping the other aligned with the labels and measure the resulting performance drop.
Our evaluation spans 23 multiple-choice visual question answering (MCVQA) benchmarks, span- ning applications such as general visual question answering, knowledge-baser reasoning, real-world spatial understanding, optical character recognition (OCR), and document and chart understanding. We evaluate MILMs at varying scales, including 8B, 13B, and 34B models (Tong et al., 2024a; Liu et al., 2023; Bai et al., 2025b;a). Our findings confirm our hypothesis; the strength of intra- and inter-modality dependencies vary substantially across and within these benchmarks.
We show that models depend inter-modality dependencies (see Figure 1). We find that many benchmarks designed to mitigate text-only dependencies (Singh al., 2019; Li et al., 2023a; Tong et al., 2024b; Fu al., 2024) have inadvertently introduced strong image-only biases, trading one uni-modal shortcut for another rather than evaluating multi-modal reasoning. This issue On the contrary, larger models often become more adept at using uni-modal dependencies. These results underscore the fundamental limitations of evaluating models with single aggregate score and highlight the need for characterization of our evaluation benchmarks based on their strengths of inter- and intra-modality dependencies.
input modality while underutilizing the other, rather than using
resolved by increasing model scale.
Contributions. We conduct the first large-scale empirical analysis of multi-modal dependencies across 23 popular VQA benchmarks. Our analysis shows that these datasets are multi-dimennsioal regarding their reliance vision, text, and their interaction, and consequently measure different aspects of multi- multi-modal learning. We find that these differences vary not only across datasets but also within individual benchmarks. To perform this analysis, we use systematic method to characterize these dependencies. Our results provide a quantitative basis for the design and selection of future multi-modal benchmarks.
2

---

Published as a conference paper at ICLR 2026
2 THE MULTI-MODAL SPECTRUM
This section defines inter- and intra- modality dependencies (Section 2.1) for multi-modal learning. We argue that the failure to systematically measure these dependencies has led to an iterative cycle of benchmark design and cireumvention (Section 2.2). Existing quantification methods (Section 2.3) do not scale to recent datasets and MLLMs, which motivates our work.
2.1 PROBLEM SETUP
In supervised multi-modal learning, given dataset D {(x{{) mapping to predict the target label y from two distinct modalities, X; and X2- The target label y can be predicted from two distinct dependencies (Liang et al., 2023; Madaan al., 2024): mntra-modality dependency or uniqueness, where is dependent on an individual modality, and inter-modality dependency or synergy, where modalities provide joint information not present in isolation. For example, in video-based sentiment analysis, a positive sentiment might be uniquely determined from strong lexical cues within a text transcript alone. In contrast, detecting sarcasm requires interpreting the conflict between the literal semantics of the text and audio or visual expressions of the video.
y)))) the goal is to learn
Following prior work (Liang et al., 2023; Madaan et al., 2024), we model this distinction with a selection variable in the multi-modal data generating process, where 1 is mechanism to model the dependencies between the modalities and the target task:
(I) Although this framework provides way to separate the effects of individual modalities from their joint combinations, the actual strength of uniqueness and synergy within popular benchmarks and MLLMS remains largely unquantified.
ply X2, 1) = p(y))((x (etn(Xyy)pppxxp// y)p(xely)p(y = 1|x1, x2.
2.2
CAT-AND-MOUSE GAME OF BENCHMARK DESIGN
The lack of a principled characterization of these dependencies has resulted in cat-and-mouse game of benchmark development and subsequent circumvention. This iterative cycle spans spectrum of multi-modal datasets, from those solvable with a single modality to those that require inter modality dependencies. To evaluate the weaken intra-modality dependencies to necessitate inter-modality dependencies (Goyal et al., 2017; Agrawal al., 2018; Dancette et al., 2021; 1; Si et al., 2022; Fu al., 2024; Tong al., 2024b; Wu and Xie, 2024). Despite these design constraints, models frequently achieve high performance by using intra- tira-modality dependencics. This reliance on intra-modality dependencies subsequently framed as exploitation of uni-modal artifacts (Wang et al., 2020; Liang et al., 2023; Zhang et al., 2024b), a behavior that has been assigned labels such as model laziness (Zhang et al., 2024a), modality competition (Huang al., 2022), modality greediness (Wu et al., 2022), which prompts further cycles of benchmark revision.
capabilities of model, new benchmarks deliberately
The history of VQA exemplifies this cycle. The original VQA dataset (Antol et al., 2015) contained strong language priors, allowing models to achieve high accuracy by guessing common answers based on the type of questions. To counter this, VQAv2 (Goyal et al., 2017) was introduced, which balanced the dataset by ensuring each question had two images leading to different answers. The subsequent VQA-CP benchmark (Agrawal al., 2018) further enforced this constraint by chang- ing the answer distribution between the training and test sets to penalize models that relied only on question-bars based priors. Similarly, the VQA-CE (Dancette al.. 2021) and VQA-VS (Si et al., 2022) datasets were introduced to highlight the prevalence of multi-modal shorteuts in prior VQA benchmarks. This iterative pattern of creation and attack continues with recent benchmarks, such as the progression from MMMU (Yue et al., 2024) to MMMU-Pro (Yue al., 2025).
Without a systematic way to quantify these dependencies, it is difficult to determine whether the performance of multi-modal model stems from multi-modal capabilities or from simply exploiting dominant intra-mon modality dependencies. This ambiguity hinders progress, as we continue to develop complex architectures and algorithms (Li et al., 2021; Wu et al., 2022; Zheng al., 2023; Liu al., 2023: Young et al., 2024) without a clear understanding of the spectrum of inter- and intra-nodoa modality dependencies in current models and datasets.
3

---

Published as a conference paper at ICLR 2026
2.3 QUANTIFYING THE STRENGTH OF DEPENDENCIES
Several quantitative metrics have been developed to measure the dependence of a model on individ- ual modalities. A straightforward approach is measure performance degradation after shuffling modality's 's input at test time, where the resulting performance drop is attributed to that modality's contribution (Gat et al., 2021). More sophisticated methods, such as MM-Shap (Parcalabescu and Frank, 2022), SHAPE (Hu et al., 2022), and InterShap (Wenderoth et al., 2025), use Shapley val- ues to assign importance scores to individual image regions and text tokens, yielding fine-grained analysis independent of task accuracy.
Despite these advances, no work has systematically positioned recent MILM evaluation datasets along continuous multi-modal spectrum defined by their inter- and intra-modality dependencies. In the next section, we adapt a practical methodology based on the perceptual score (Gat al., 2021) to measure these dependency strengths. We select this method for its simplicity in the two-modality case and its ability to directly compute each modality's 's marginal contribution. By characterizing datasets along the spectrum of multi-modal dependencies, can design more targeted bench- marks. Further, we gain deeper insights into model capabilities, paving the way for more robust and generalizable multi-modal systems.
3 RECIPE FOR FUTURE DATASETS AND MODELS
Given a multi-modal dataset D consisting of instances (x1,X2.X) X), where X1 is an image, X2 is a text, and y is the ground truth label, we detail a principled evaluation framework inspired by Gat et al. (2021). This requires a baseline multi-modal model fa to evaluate performance, measured by a metric M, under four different input conditions. The chosen baseline model should ideally be a state-of oftee-of-the-ar multi-modal model that has not been trained on the dataset under analysis, thus preventing data leakage.
The four evaluation conditions are:
1. Paired modalities (Normal); The model's performance measured on original, paired data points, M(fe(Xx.x2).)))
2. Unimodal (Image only): The paired text x2 is replaced with a text instance x² randomly sampled from another data point. Performance on ((fo(x1, xx), isolates the informa- tional contribution of the image modality X1.
3. Unimodal (Text only): Symmetrically, the image x1 is replaced with random image X1'- Performance on (xfo(x(,x2) x2), ))).)) y) isolates the contribution of the text modality X2.
4. Both modalities shuffled (Random): Both modalities are replaced with randomly sam- pled, uncorrelated instances (x4 x2). The model's performance on M(fo(x(,,)) es- tablishes random baseline.
A dataset that appears balanced at the global level can still contain strong intra-modality dependen- cies within specific subsets of its data. It is therefore essential that this procedure be supplemented with more granular analysis of data subgroups. This involves applying the same diagnostic to data subsets categorized by relevant features, such as question type or object categories.
Rationale for modality shuffling. We adopt modality shuffling over the option of zeroing out (e.g., using a blank image or an empty string) (Gu et al., 2025) input perturbation as in prior studies (Hu et al., 2022; Tong et al., 2024a). Zeroing out or adding perturbation creates unnatural, -itttf-disttibution inputs that elicit unpredictable model behavior, confounding the measurement of inter- and intra-mode modaliyy dependencies. In contrast, shuffling preserves the marginal distribution of each modality. The model still receives valid inputs, but the inter-modality dependency broken. The performance metrics derived from this shuffling procedure, visualized in Section 4, enable direct quantification of inter- and intra-modality dependencies.
Model-based analysis. Multi-modal dependencies are function of both the data and the model interpreting it. Thus, an analysis based on a single model may be confounded by specific inductive biases of that model. To obtain robust estimate of intrinsic data dependencies, the effect of any single model must be marginalized out. We achieve this using a majority-vote ensemble (Dietterich, 2000) of diverse models reduce the influence of idiosyncratic model biases.
4

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
In this section, we describe the evaluation datasets and models used in Section 4.1. Section 4.2 shows the overall performance metrics and Section 4.3 shows the results in various subcategories across multiple datasets.
4.1 DATASETS AND MODELS
To assess the capabilities of MLLMs, we use comprehensive suite of benchmark datasets. Based on the core evaluation skills, we categorize the benchmarks chronologically to show the progression in each category.
General visual question answering. For general VQA, focus benchmarks that test real-world and compositional reasoning. We include VizWiz (Gurari et al., 2018), which poses questions from visually impaired users about everyday, uncurated scenes. Following this, we use GQA (Hudson and Manning, 2019) to evaluate visual reasoning and compo- sitional reasoning. To evaluate wider range of abilities, we incorporate MME (Fu al., 2023), which covers 14 perception tasks. SEED-Bende BenDhBench (Li et al., 2023a) expands on these with large-scale multiple choice question format. MMBench (Liu al., 2024b) further evaluates 20 abilities, including object localization and social reasoning.
Expert visual question answering. To measure performance on tasks requiring special- ized knowledge, we evaluate with multiple benchmarks. This includes ScienceQA (Lu et al., 2022). which contains questions from the natural sciences, language and social sci- ences. We also use Math (Lu al., 2023), which tests mathematical reasoning (logi- cal, arithmetic, and statistical) in diverse visual formats such as word problems, geometric shapes, and plots. For expert-level evaluation, we incorporate MMMU (Yue et al., 2024) and MMMU-Pro (Yue et al., 2025), which consist of college-level problems from exams and textbooks in six core disciplines, probing multi-modal understanding and reasoning.
Real-world spatial understanding. We use Microsoft COCO dataset (Lin al., 2014) for object recognition and ADE (Zhou et al., 2019) for scene understanding. To measure the POPE benchmark (Li al., 2023b) and measure spatial understanding using RealWorIdQA (xAI, 2024). To address the growing importance of temporal reasoning, we include MMVP (Tong al., 2024b), which tests comprehension and reasoning about long-form video content. Omni3D (Brazil et al., 2023; Tong et al., 2024a) contains the task of determining the depth order and relative distance of 3D objects.
object-level hallucinations,
Q-Bench (Wu et al., 2024) and BLINK (Fu et al., 2024) evaluate low-level visual perception and general understanding. V Bench (Wu and Xie, 2024) focuses on visual grounding in high-resolution images. MM-Star (Chen et al., 2024) is another vision-centric benchmark with human-validated samples to test six fundamental multi-modal capabilities.
Optical character recognition (OCR) and document, chart understanding. We start by evaluating using TextVQA (Singh et al., 2019), which requires models not only to read, but also to reason about text embedded in images. We expand the scope of evaluation with OCRBench (Liu et al., 2024c), which provides a multifaceted assessment that includes text recognition, scene text-centric VQA, decument-oriented VQA. key information extraction, and handwritten mathematical expression recognition.
For document and chart understanding, we evaluate the model's ability to comprehend complex layouts and the relationships between visual elements. We start with A12D (Kem- bhavi et al., 2016) for understanding schematic diagrams followed by ChartQA (Masry et al., 2022), a challenging dataset of human-generated question-answer pairs on various charts and plots.
We use the openly available 8B, 13B, and 34B models from Cambrian-1 (Tong et al., 2024a).
These models are built upon Llama-3 8B (Liu et al., 2023), Vicuna-1.5 13B (Chiang et al., 2023), and Nous-Yi Yi 34B (Young et al., 2024) for language processing. For vision, they incorporate
combination of architectures including ViT from SigLIP (Zhai et al., 2023; Radford et al., 2021), DINOv2 (Oquab et al.. 2024), and ConvNeXt-XXLL (Liu et al., 2022). Our main results are generated by taking a majority vote of the answers among these three models. The code is available online at https:/ / //g thub. com/ di vyam38 9897 /mmllmae imoda 1 spectrum.
5

---

Published as a conference paper at ICLR 2026
vizwiz
Normal Random
Text
Normal
Text Image
Random
ocrbench
puge
mathvisty
gqu
testvgg
ade
mmmupro
muno
chartg
mmstar
ai2d
40%
bench
mmmmi
seed
80%
80%
100%
omni
hnihpp
scienceqi
mmb_en en
realworldqa
vstar
mmb
blink
(a) Datasets evaluating visual question answering with (b) Datasets evaluating spatial understanding and general and expert questions.
OCR, data and chart understanding.
Radar plot showing the comparison of an ensemble of standard MLLMs with image only, text only and random performance using the recipe from Section The dashed line indicates human performance, which is shown partially due to a lack of data for other benchmarks.
Figure 2:
4.2 OVERALL RESULTS
Our evaluation in Figure 2 across 23 multi-modal datasets shows that most benchmarks contain both intra- and moter-modality dependencies, allowing models to answer questions without jointly using both the image and the question text. We categorize datasets based on the modality dependencies they contain: (1) inter-modal only, and two non-exclusive categories for datasets that additionally contain (2) text intra-modality dependencies and (3) image intra-modality dependencies.
Datasets with inter-modality dependency only, We show that multi-modal datasets with inter- modality dependency only are surprisingly rare. Across all evaluated benchmarks, only four datasets exhibit this characteristic.
For general and expert question answering, MME (Fu et al., 2023) is the only dataset that demon- strates that permuting one modality makes the task challenging for the model. For spatial under- standing, POPE (Li et al., 2023b), COCO (Lin et al., 2014; Tong et al., 2024a), and V (Wu and Xie, 2024) datasets contain predominantly inter-modality dependencies. No datasets in the OCR and chart understanding categories exhibit inter-modality dependencies only.
The simplest way to curate languael inter-modality datasets is to ensure that the answer changes with the change in one modality. This approach has been used in few binary classification inter-modality datasets (Suhr and Artzi, 2019; Fu et al., 2023; Li et al., 2023b). For instance, POPE and MME contains questions with yes and answers for the same set of images. This ensures that a model relying on only one modality might correctly answer one question but will fail to correctly answer the corresponding inverse question. This leads to random performance when the mnter-modality dependencies are ignored when modality is shuffled.
Datasets with text intra-modality dependency. Models when evaluated on general and expert knowledge show a reliance on text across all datasets. For example, models with only the cor- rect input question achieve scores well above random chance on GQA (Hudson and Manning, 2019) (+26%), ScienceQA (Lu et al., 2022) (+17.5%), and MMMU (Yue et al., 2024) (+11.35%), demonstrating that visual input is often not considered necessary by the model for these datasets. This even extends to datasets designed specifically to emphasize visual grounding, such as Blink (Fu et al., 2024), RealWorldOA (xAI, 2024), and Omni3D (Brazil et al., 2023; Tong et al., 2024a). The same pattern holds for OCR. document and chart understanding datasets such AI2D (Kembhavi et al., 2016), ChartQA (Masry et al., 2022) and TextVQA (Singh et al., 2019), where using question only surpass random performance by 34.94, 11.69 and 12.19 absolute points, respectively.

---

Published as a conference paper at ICLR 2026
29
50
50
45
80
40
38
60
55
50
53
50
50
53
50
50
56
50
50
29
27
40
20
20
88
13B
34B
Ensemble
6B
136
34B
Ensemble
(a) POPE
(b) MMMU
80
76
73
70
70
70
60
52
40
30
23
20
26
30
28
30
20
10
10
MMBench (en)
A12D
Figure 3: Effect of model scaling on modality contribution. Performance of various models (8B,
13B, 34B, and majority-vote ensemble) on four different datasets. The bars represent standard accuracy and attributed contributions from text, image, and random (bars are in the same order).
These results underscore the challenge of designing benchmarks that do contain any examples without text-only dependencies. While it is challenging to dissect the precise cause for these biases, many studies have conjectured several issues in data curation. These issues include shortcuts be- tween language and corresponding answers (Goyal et al., 2017). IID train-test splits (Agrawal et al., 2018), shifted prior distributions (Gat et al., 2021), limited human-level perception abilities (Fu et al., 2024) and failures to identify visual patterns in the image (Tong et al., 2024b).
Datasets with image intra-modality dependency. Efforts to eliminate textual biases from bench- marks have led to unintended consequence of introduction of strong visual intra-modality depen- dencies. We find that these newer datasets often allow models to succeed by relying solely on the image, effectively ignoring the question. This is most illustrated in MMBench (Liu al., 2024b), where an image-only model outperforms a random baseline by 41%. This issue persists even in benchmarks designed focus multi-modal reasoning, including TextVQA (Singh et al., 2019), ChartQA (Masry et al., 2022), SEED-Bende (Li et al., 2023a), MMMU-Pro (Yue et al., 2024), MMVP (Tong al., 2024b), Q-Bench (Wu et al., 2024), and MM-Star (Chen et al., 2024).
Instead of requiring multi-modal understanding, many of these evaluation benchmarks swapped a textual dependency with visual one obtain the correct answer. This because their curation primarily focused mitigating text-only intra-modality dependencies. We recommend that the central goal of benchmark design should be measure the intended task using both modalities for question answering, not to emphasize intra-modality dependencies.
Effect of model scaling. Since analysis is based on model-dependent accuracy metric, investigate how modal dependencies change across models of varying scales and architectures. We selected four datasets with distinct dependencies in Figure POPE (Li et al., 2023b), which con- tains predominantly inter-modality dependencies; MMMU (Yue et al., 2024), which is reliant on both image and text intra-modality dependencies; MMBench (Liu et al., 2024b), dependent on the image modality; and A12D (Kembhavi et al., 2016), reliant the text intra- vidta-modality dependencies.
find that uni-modal biases are not consistently mitigated by model scale and can even be exac- erbated. For instance, MMMU (Yue al., 2024), scaling to 34B parameter model increased the overall performance and the reliance on both image and text-only dependencies. Similarly, on MMBench (Liu al., 2024b), larger models exhibit an improved performance with greater depen- dency on image-only dependencies (Figure 3). A12D (Kembhavi et al.. 2016) shows a similar trend. The text-only intra-modality dependency increases with scale and persists in the majority-ensent

---

Published as a conference paper at ICLR 2026
66
64
60
50
00
53
49
53
49
33
40
28
20
10
Cambrian 8B
LLaVA-Next 7B
Qwen2.5
Qwen3
Cambrian
LLaVMNNxtt 7B
Qwen2.5
Owen3 88
(a) POPE
(b) GQA
86
86
35
17
80
73
30
66
60
25
20
1.7
40
15
14
23
20
6
Qwen2.5
Qwen3
Qwen2.5
Qwen3
MMBench (en)
(d) MMMU-Pro
Figure 4: Effect of model type on modality contribution. Performance comparison between LLava-Next (May 2024), Cambrian-1 8b (June 2024), Qwen2.5-VL (April 2025) and Qwen3- VL (October 2025) on four datasets selected for their specific dependencies: GQA (text), MMBench (image), POPE (inter-modality) and MMMU-Pro (both image and text). The bars represent stan- dard accuracy and attributed contributions from text, image, and random (bars are in the same order).
model. Conversely, performance on POPE (Li et al., 2023b), benchmark requiring only inter- modal dependencies, shows no change in performance with increase model size. further show results on additional datasets that depend on neither text nor image intra-modality dependencies, as well as datasets that depend on both in Figure B.8. We also include datasets that predominantly depend image and text intra- intramoddality dependencies in Figure B.9 and Figure B.10 respectively.
Effeet of model types. We compare four different instruction-tume models Figure 4. Particu- larly, compare Cambrian-8b released in June 2024 with LLama-3 8B base model (Tong al., 2024a), LLaVA-Next released in May 2024 with Mistral 7B model (Liu et al., 2024a), Qwen2.5-VV from April 2025 with Qwen2.5 7B language model (Bai al., 2025b) and Qwen3-Vi VL 8B released recently in November 2025 with Qwen3 language model (Bai et al., 2025a).
Despite substantial differences in the evaluated models and their release dates, we consistently ob- serve intra-modality biases (when present) across all of them. For instance, Qwen models improve the performance MMBench by around ten percent compared Cambrian-1 while also increasing the image-only performance significantly. For POPE, as expected, all models exhibit near-random performance when using only image or only text inputs. For GQA and MMMU-Pro, we see com- parable levels of biases across different models types. Similar results with additional datasets are shown in Figure B.
This findings raise the question of whether improvements in benchmark performance really reflect progress in multi-modal learning or whether models are simply becoming better at using intra- modality dependencies. We hope that our analysis will encourage reporting not only overall bench- mark performance but also image-only, text-only, and random baselines, to enable more holistic evaluation of multi-modal models.
4.3
CATEGORY ANALYSIS
Aggregate performance metrics on multi-modal benchmarks can be misleading, often obscuring strong intra-modality dependencies the sub-category level. As shown in Figure 5, the benchmarks that appeared to use inter-modality dependencies in Figure also contain atdaliodalltty dependen- cies when evaluated at granular level.
8

---

Published as a conference paper at ICLR 2026
70
60
60
34
21
30
20
Count
Relal we locetion
Count
Ralal we focwiton
How
What
Wey/ko
(a) ADE
(b) COCO
(c) O-Bench
gradel
gradeo
grafw10
Arcanecture
Puilic
Soot Soehhigggghhhhee
Arcanecturre
Pryctslou
Art
Paycholog
Art
Nusa
Chemaaty
Chemanta
unade
eade11
Machanicl
Gimical
Machanioa
Ciniical
Camputer
Math
Camputer
Matrrsie
beaua
Matresde
gradet
graded
Marag
Braoniios
Manogh
Scoooiios
grades
grate3
Libersten
Literttin
Magleeg Egarcy
Stae Egarp)
(d) Science QA
(e) MMMU
(f) MMMUPro
Figure 5: Analysis of sub-categories across datasets showing dependency on individual modal- ities. Although benchmarks may be designed to contain inter-modality dependencies, we show strong dependence on text intra-modality dependencies for categories such as relative location in ADE and COCO, higher-grade questions in ScienceQA and multiple categories in MMMU and MMMUPro. This highlights how aggregate metrics can obscure that many instances may not re- quire multi-modal reasoning. We show standard accuracy in yellow and contributions from text in blue, image in green, and random in orange.
This discrepancy is evident across several datasets. In ADE (Zhou et al., 2019) and COCO (Lin et al., 2014; Tong et al., 2024a), while text-only model's overall performance is only marginally above chance (see Figure 2), it achieves substantial accuracy on the relative location sub-category (Figures 5a and 5b). This phenomenon is amplified in knowledge-intensive benchmarks. In Sci- enceQA (Lu et al., 2022) (Figure 5d), text-only performance accounts for the majority of the ac- curacy of questions aimed grades 10-12. Likewise, many academic subjects within the MMMU and MMMU Pro benchmarks (Yue et al., 2024) (Figures 5e and 5f) contain many instances solvable image, respectively, allowing unimodal models succeed without question
with question visual information. Conversely, Q-Bench (Wu et al., 2023) (Figure 5c) exhibits the opposite pattern. Individual categories show dependence both image and text intra-modality dependencies, yet. the aggregate metrics in Figure 2 indicate notable bias toward the image modality.
These findings further corroborated by our analysis of datasets such as MME (Fu et al., 2023) and BLINK (Fu et al., 2024) in Figure B.7. We demonstrate that the degree of modality dependence often inconsistent within a single benchmark. This highlights the multii-imeesionall nature of multi-modal datasets, intra- and inter-modaliy dependencies emerge and vary unpredictably across different sub-poue coab-popultiios of the data.
LIMITATIONS AND FUTURE WORK
Our analysis is constrained by the field's reliance MCVQA benchmarks. This common prac- tice often fails to test for true multi-modal understanding due to two prevalent failure modes (see dependencies, where models ignore the image for factual ques- tions; and image-based intra-modd modality dependencies, where models select visually correlated an- swers while disregarding the actual question.
Figure 6): text-based
To more holistically evaluate multi-modal capabilities, we propose several crucial future directions. First, should progress towards building benchmarks that focus open-ended answer genera-

---

Published as a conference paper at ICLR 2026
B
How doss water
Diaxide
What the air on cold, rainy day?
Which country highlighted?
become vapor?
through?
a) Evaporation b) Infitteraioon
Salty c) Sweet
Sour d) None
a) Seeds
b) Leaves
c) Transpiration d)
Flower
Stem
c) Fruit. d) None
e) None
None of the choices are
vapor through process called evaporation.
The answer b) The question about the temperature distraction.
The correct answer b) Leaf
The correct answer None Confidence: 1.0
Confidence: 1.0
Confidence: 1.0
Water becomes vapor through the process of evaporation.
Question is mixing taste with
Image shows mint, which is used for its
Correct Leaf Confidence: 1.0
temperature, bul given the images correct answer b)
Correct answer is a). Confidence: 1.0
Correct answer Confidence:
Confidence: 0.95
Figure 6: MILMM failure modes in MCVQA. Visualization from GPT-5 and Gemini 2.5 Pro show- ing failure modes in MCVQA, such as relying only on text for factual questions while ignoring the image, conversely, choosing visually correlated answers while ignoring the question. In all cases, the models were prompted to select one choice and provide a confidence score between and 1.
tion and evaluation (Rei et al., 2020; Balepur et al., 2025). Evaluating free-form responses presents significant challenges. The same meaning be expressed in many ways, making automated eval- uation difficult. This often requires human evaluation, which slow and expensive. We believe progress in this direction is essential for measuring the necessary multi-modal capabilities.
Second, both benchmarks and models must support the ability to abstain from answering when presented with ambiguous irrelevant inputs (Whitchead et al., 2022; Feng et al., 2024; Stengel- Eskin et al., 2024). We conduct a preliminary experiment with OpenAl GPT-5 and Google Gemini 2.5 Pro, showing cases where the image the question irrelevant the answer in Figure 6. Despite facilitating abstention by augmenting the instruction set with a "None of the above" option, this approach is largely insufficient overcome the dependence on uni-modal dependencies for both GPT-5 and Gemini 2.5 Pro models. This highlights that models have tendency to generate a plausible-sounding but incorrect response over acknowledging ambiguity or lack of information with confidence. Future work should prioritize methods to encourage meaningful abstention.
Lastly, we encourage both future benchmarks and models to report not only aggregated performance but also modality-specine and random baselines to better measure progress. From benchmark perspective, this helps the community understand how new dataset compares with existing ones in the inherent biases. From a model perspective, it clarifies where performance gains come from and guides meaningful future improvements.
6 CONCLUSION
Our work critically dissects the intra- and inter-modality dependencies MLLMs on 23 bench- marks. We show that no dataset is truly multi-modal, as each measures multiple dimensions of multi- modal learning with different strengths of intra- and inter-modality dependence. These strengths vary substantially both across benchmarks and across categories within the same benchmark. We find that efforts to mitigate text-based dependencies have often introduced new image-based depen- dencies, perpetuating cycle of superficial fixes. This suggests that meaningful progress cannot be achieved simply by developing more benchmarks or chasing leaderboard metrics. Instead, we must critically assess the existing evaluation methods. This includes moving beyond standard multiple- choice formats, incorporating scenarios where models should abstain when they are uncertain, and examining how model arrives at answer rather than only what answer it produces.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) with a grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research, Sam- sung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI), National Science Foundation (NSF) award No. 1922658, Center for Advanced Imaging Innovation and Research (CAI2R), National Center for Biomedical Imaging and Bioengineering operated by NYU Langone Health, and National Institute of Biomedical Imaging and Bioengineering through award number P41EB017183 113.. The computational requirements for this work were supported by NYU IT High Performance Computing resources, services, and staff expertise and NYU Langone High Performance Computing Core's ' resources and personnel. This work partly supported in part by the NYUAD Center for Interdisciplinary Data Science & AI (CIDSAI), funded by Tamkeen under the NYUAD Research Institute Award CG016. This content solely the responsibility of the authors and does not represent the views of the funding agencies.
REFERENCES
A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi. Don't 't just assume; look and answer: Overcom- ing priors for visual question answering. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2018. 2,2,,7
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In Proceedings of the International Conference on Computer Vision (ICCV), 2015. 3
S. Bai, Y. Cai. Chen, Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report. arXiv preprint 1xXi::55/1.21633 2025a. 2, 8
S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, al. Qwen2. 5-yl technical report. arXiv preprint arXiv:2502. 2025b. 8,8
N. Balepur, R. Rudinger, and L. Boyd-Graber. Which of these best describes multiple choice evaluation with Ilms? a) forced b) flawed c) fixable d) all of the above. arXiv preprint 204T::2502.11222 2025. 10
G. Brazil, A. Kumar, Straub, N. Ravi, Johnson, and G. Gkioxari. Omni3d: A large bench- mark and model for 3d object detection in the wild. In Proceedings of the IEEE International Conference on Computer Vision and Puttern Recognition (CVPR), 2023. 5, 6, 16
L. Chen, Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang. Y. Qiao, D. Lin, et al. Are we on the right way for evaluating large languanlanguage models? Advances in Neural Information Processing Systems (NeurIPS), 2024. 5, 16
W.-L. Chiang, Z. Li, Lin, Sheng, Z. Wu, H. Zhang, L. Zheng, Zhuang, Y. Zhuang, E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, 2023.
Cohere. Introducing command a vision: Multimodal ai built for business, 2025. 1
G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
C. Dancette, R. Cadene, D. Teney, and M. Cord. Beyond question-based biases: Assessing multi- modal shortcut learning in visual question answering. In Proceedings of the International Con- ference on Computer Vision (ICCV), 2021. 3,
T. G. Dietterich. Ensemble methods in machine learning. International workshop on multiple classifier systems, 2000. 4
11

---

Published as a conference paper at ICLR 2026
S. Feng. W. Shi, Y. Wang, W. Ding, O. Ahia, S. S. Li, V. Balachandran, S. Sitaram, and Y. Tsvetkov. Teaching Ilms to abstain across languages via multilingual feedback. In Proceedings of the Con- ference Empirical Methods in Natural Language Processing (EMNLP). 2024. 10
C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. Mme:
A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023. 5, 6, 9, 16
X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna. Blink: Multimodal large language models can see but not perceive. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), 2024. 1, 2, 3, 5, 6, 9, 16
I. Gat, I. Schwartz, and A. Schwing. Perceptual score: What data modalities does your model perceive? Advances in Neural Information Processing Systems (NeurIPS), 2021. 1, 2, 4,7 4,7
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the in vqa matter; Ele- vating the role of image understanding in visual question answering. In Proceedings of the IEEE International Conference on Computer Vision and Puttern Recognition (CVPR). 2017. 2, 3,7 7
Y. Gu, J. Fu, X. Liu, J. M. Valanarasu, N. C. Codella, R. Tan, Q. Liu, Y. Jin, S. Zhang, J. Wang, et al. The illusion of readiness in health ai. arXiv preprint 2 18234, 2025. 4
D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 5, 16
P. Hu, X. Li, and Y. Zhou. Shape: An unified approach to evaluate the contribution and cooperation of individual modalities. arXiv preprint arXiv:2205.00300 2022. 2,4
Y. Huang. J. Lin, C. Zhou, H. Yang, and L. Huang. Modality competition: What makes joint training of mudti-modal network fail in deep learning learning?(provalll (provably). In Proceedings of the International Conference Machine Learning (ICML), 2022.
D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compo- sitional question answering. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 5, 6, 16
A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth dozen images. In Proceedings of the European Conference on Computer Vision (ECCV), 2016. 5, 7,
S. Korbblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Shan. Seed-bench: Benchmarking multimodal lims with generative comprehension. arXiv preprint arXiv:2.007.611 2023a. 2, 7,
J. Li, R. Selvaraju, A. Gotmare, S. Joty. C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems (NeurIPS), 2021. 3
L. Li, G. Chen, H. Shi, J. Xiao, and L. Chen. A survey on multimodal benchmarks: In the era of large ai models. arXiv preprint X41:22409.181422 2024. 1
Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023b. 5, 6, 7, 8, 16
P. P. Liang, Y. Cheng, X. Fan, C. K. Ling, S. Nie, R. Chen, 7. Deng, N. Allen, R. Auerbach, F. Mah- mood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Processing Systems (NeurIPS), 2023. 2,3
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), 2014. 5, 6, 16
12

---

Published as a conference paper at ICLR 2026
H. Liu, Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 2023. 2,3,5
H. Liu, C. Li, Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a.
Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang. C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? In Proceedings of the European Conference on Computer Vision (ECCV), 2024b. 5, 5,7,10
Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X.-C. Yin, C.-L. Liu, L. Jin, and X. Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 2024c. 5, 16
Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-1 1776-11982 11986, 2022. 5
Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems (NeurIPS), 2022. 5, 6, 9, 16
P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models visual contexts. arXiv preprint arXiv:2310.02255, 2023. 5, 16
Z. Lu. A theory of multimodal learning. Advances in Neural Information Processing Systems (NeurIPS), 2023.
D. Madaan, T. Makino, S. Chopra, and K. Cho. Jointly modeling inter- & intra-modality dependen- cies for multi-modal learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 1, 2,
A. Masry, X. Long, J. Q. Tan, Joty, and E. Hoque. Chartqa: benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022. 1, 6, 16
M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. HAZ- IZA, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. 5
L. Parcalabescu and A. Frank. Mm-shap: A performance-agnee metric for measuring multimodal contributions in vision and language models & tasks. arXiv preprint arXiv:2212.081158, 2022. 2, 4
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In Pro- ceedings of the International Conference on Machine Learning (ICML), 2021.
R. Rei, C. Stewart, A. C. Farinha, and A. Lavie. Comet: A neural framework for mt evaluation. In Proceedings of the Conference on Empirical Methody in Natural Language Processing (EMNLP), 2020. 10
Q Si, F. Meng, M. Zheng, Lin, Y. Liu, P. Fu, Y. Cao, W. Wang, and J. Zhou. Language prior is not the only shorteut: A benchmark for shortcut learning in vqa. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. 1,3
A. Singh, V. Natarajan, M. Shah, Y. Jiang. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 1, 2, 5, 7,
E. Stengel-Esken Eskin,, P. Hase, and M. Bansal. Lacie: Listener-awart finetuning for calibration in large language models. Advances in Neural Information Processing Systems (NeurIPS), 2024. 10
13

---

Published as a conference paper at ICLR 2026
A. Suhr and Y. Artzi. Nlvr2 visual bias analysis. arXiv preprint arXiv: 2019,
G. Team, P. Georgiev, L. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055..0
2024.
M. Tjandrasuwita, C. Ekbote, L. Ziyin, and P. P. Liang. Understanding the emergence of multimodal representation alignment. arXiv preprint arXiv:2502.16282. 2025.
P. Tong. E. Brown, P. Wu, S. Woo, A. J. V. IYER, S. C. Akula, S. Yang, J. Yang, M. Middepogu, Z. Wang, et al. Cambrian-1e 1: A fully open, vision-centric exploration of multimodal llms. Ad- vances Neural Information P Processing Systems (NeurIPS), 2024a. 2, 4, 6, 8, 16
S. Tong. Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes wide shut? exploring the visual short- comings of multimodal lmms. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2024b. 2, 3, 7, 16
W. Wang, D. Tran, and M. Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2020.
L. Wenderoth, K. Hemker, N. Simidjievski, and M. Jamnik. Measuring cross-modal interactions in multimodal models. In Proceedings of the AAAI National Conference on Artificial Intelligence (AAAI), 2025. 2,
Whitehead, S. Petryk, V. Shakib, J. Gonzalez, Darrell, A. Rohrbach, and M. Rohrbach. Re- liable visual question answering: Abstain rather than answer incorrectly. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2022. 10
H. Wu, Z. Zhang, E. Zhang, C. Chen, L. Liao, A. Wang, C. Li, W. Sun, Q. Yan, G. Zhai, et al. Q- bench: benchmark for general- purposs foundation models on low-level vision. arXiv preprint arXiv:2309. 14181, 2023. 9
H. Wu, Z. Zhang, E. Zhang, C. Chen, L. Liao, Wang, C. Li, W. Sun, Q. Yan, G. Zhai, and W. Lin. Q-bench: A benchmark for general-purpose foundation models on low-level vision. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. 5, 7, 16
N. Wu, S. Jastrzebski, K. Cho, and K. J. Geras. Characterizing and overcoming the greedy nature of learning multi-modal deep neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2022. 3
Wu and S. Xie. V*: Guided visual search as core mechanism multimodal Ims.. Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2024. 3, 5, 6, 16
XAI. Grok-1.5 vision preview, 2024. URL
1,5, 6, 16
A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, G. Wang, H. Li, J. Zhu, J. Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint 2xXi::2403.04652. 2024. 3,
X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3, 5, 6, 9,
X. Yue, T. Zheng, Y. Ni, Y. Wang, K. Zhang, S. Tong, Y. Sun, B. Yu, G. Zhang, H. Sun, Y. Su, W. Chen, and G. Neubig. MMMU-pro: A more robust multi-disccilinn multimodal understanding benchmark, 2025. 3,5
X. Zhai. B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 5
14

---

Published as a conference paper at ICLR 2026
X. Zhang, J. Yoon, M. Bansal, and H. Yao. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2024a. 3
Y. Zhang, P. E. Latham, and A. M. Saxe. Understanding unimodal bias in multimodal deep linear networks. In Proceedings of the International Conference on Machine Learning (ICML), 2024b.
L. Zheng, W.-L. Chiang. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Lin, Z. Li, D. Li, Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS), 2023.
B. Zhou, H. Zhao, X. Puig. T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 2019. 5,
15

---

Published as a conference paper at ICLR 2026
APPENDIX
Organization. In the appendix, we provide the implementation details in Appendix A and addi- tional results in Appendix B.
A EXPERIMENTAL DETAILS
Implementations. We use the Cambrian-1 (Tong al., 2024a) open-sourced codebase for all the experiments. We use their publicly released models for evaluation. Datasets like A12D (Kembhavi al., 2016), ChartQA (Masry et al.. 2022), MMBench (Liu al., 2024b), MME (Fu et al., 2023), MMMU (Yue et al., 2024), POPE (Li et al.. 2023b), RealIQA (xAI, 2024), SEED (Li et al., 2023a), TextVQA (Singh et al., 2019), and VizWiz (Gurari et al., 2018) were sourced from LMMS- eval, while others such ADE, Blink (Fu et al., 2024), COCO (Lin et al., 2014). GQA (Hudson and Manning, 2019), MathVista (Lu et al., 2023), MMMUPro (Yue al., 2024). MMStar (Chen al., 2024), MMVP (Tong al., 2024b), OCRBench (Liu et al., 2024c), Omni3D (Brazil et al., 2023; Tong et al., 2024a), Q-Bench (Wu et al., 2024), ScienceQA (Lu al., 2022), and V*Bench (Wu and Xie, 2024) were used from their respective sources.
B ADDITIONAL RESULTS
The results different categories for MME (Fu et al., 2023) and BLINK (Fu et al., 2024) datasets are shown in Figure B.7. We further show the effect model size on additional datasets in Fig- B.8, Figure B.9, Figure B. 10 and the effect of model types additional datasets in Figure B. 1.
OCR
Counting
text translation
artwork
Scone
colebeity
Spatial Relation
posters
codé reasoning
0%
20%
position
volor
60%
96%
Relative Refiectance
öijject Localization der
namerical calculation
commonsciise peasoning
landmark
count
exislence
Relelive Depth
(a) MME
(b) BLINK
Figure B.7: Analysis of sub-categories for MME and BLINK dataset. We show standard accuracy in yellow and contributions from text in blue, image in green, and random in orange.
16

---

Published as a conference paper at ICLR 2026
76
70
70
50
50
30
20
19
12
8.
88
138
34B
Enserbale
88
138
34B
Ensemble
(a) TexiVQA
(b) ChartQA
78
50
83
70
60
40
30
26
27
24
28 25
26 22.
26
26
20
10
(c) COCO
(d) MME
Figure B.8: Effect of model size on additional datasets. Performance of various models (8B, 13B, 34B, and majority-vote ensemble) with both image and text intra-modality dependencies (top) and primarily inter-modality dependency (bottom). The bars represent standard accuracy and contributions from text, image, and random (bars in the same order).
B4
50
45
40
33
21
23
21
22
23
40
37.
20
9B
138
34B
Enserbbee
88
138
348
Ensemble
(a) MMStar
(b) MMBench (en)
74
75
70
50
49
50
-44
50
35
30
30
30
10
88
138
34B
Ensemble
88
138
34B
Ensemble
(c) SEED-Bench
(d) Q-Bench
Figure B.9: Effect model size additional datasets with image intra-modality dependency. Performance of various models (8B, 13B, 34B, and a majority- ensemble). The bars represent standard accuracy and contributions from text, image, and random (bars are in the same order).
17

---

Published as a conference paper at ICLR 2026
50
53
51
50
53
30
30
25
40
15
38
33
21
18
30
27
25
27
23
27
15
16
15
16
14
20
10
10
8B
138
34B
Ensemble
68
136
34B
Ensemble
(a) BLINK
(b) MMMU Pro
64
65
39
35
35
50
40
22
24
30
32
34
32
70
10
15
20
10
7
(c) GQA
(d) ScienceQA
75
70
58
80
51 St
49
52
40
25
19
21
15
20
10
88
135
348
Ensemble
88
136
348
Ensemble
(c) Omni
(f) Math Vista
Figure B.10: Effeet of model size additional datasets with text intra-modality dependency. Performance of various models (8B, 13B, 34B, and a majority-vote ensemble). The bars represent standard accuracy and contributions from text, image, and random (bars are in the same order).
18

---

Published as a conference paper at ICLR 2026
48.
40
40
34
30
50
21
23
20
19
10
Cambrian 86
LLaVA-Next 70
Owen2.5
Qwen3
Cambrien
LLeVA-Next 7B
Qwen2.5 7B
Qwen3
(a) MMStar
(b) MMMU
75
75
70
71
50
46
57
30
70
10
Cambrian 8B
LLaVA Next 7B
Qwen2.5
Qwena
Cambrion 8B
LLaVA-Noxt 7B
Qwen2.5 7B
Qwen3 8B
(c) OCRBench
(d) Omni
80
79
45
64
40
3B
35
50
30
30
.00
36
38
38
20
21
19
23
30
10
10
(c) QBench
(f) ScienceQA
80
80
70
60
57
50
30
28
20
13
20
30
19
10
Cambrian 8B
LLaVA-Next 7B
Qwen2.5
Qwen3
Cambrien
LLeVA-Next 7B
Qwen2.5 78
Owen3
(g) TextVQA
(h) V Bench
Figure B.11: Effect of model types on additional datasets. Performance comparison between various models such as LLava-Next (May 2024), Cambrian-1 8b (June 2024), Qwen2.5-VL. (April 2025) and Qwen3-VL (October 2025). The bars represent standard accuracy and attributed contri- butions from text, image, and random (bars are in the same order).
19


Reviews:
[{"review_id": "pJF2RLL4MU", "reviewer": "Reviewer_iS4i", "summary": "Multimodal benchmark datasets are fundamental to advancing multi-modal learning. However, these datasets are not well-characterized in terms of their capabilities in evaluating intra-modality dependencies where only one modality contributes to the target task, and inter-modal dependencies where both modalities contribute to the target task.  The paper presents a large-scale empirical study to quantify these dependencies across 23 visual question-answering benchmarks using multi-modal large language models (MLLMs). The study finds that these datasets have varying biases on revealing intra-dependencies and inter-dependencies in MLLMs. Moreover, these variations also occur within individual benchmarks. The paper further suggests that we should critically assess existing evaluation methods, move beyond standard multiple-choice formats, and train models to abstain when an answer cannot be confidently determined.", "questions": "- What are the criteria to determine datasets with inter-modality dependency and intra-modality dependency?\n- What are the different dimensions of a multi-modal dataset?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper conducts a large-scale empirical study to evaluate the datasets designed for benchmarking the performance of MLLMs from the perspective of intra-modality and inter-modality dependencies that are fundamental to advancing multi-modal learning.\n\n- The paper reveals some interesting findings that could inspire future research. Specifically, most existing datasets exhibit uni-modal biases where an MLLM performs better on only one modality than the other modality. Moreover, datasets exhibiting inter-modality dependencies are rare, where utilizing the interaction between text and vision modalities is required for an MLLM to perform well.\n\n- The paper offers useful insights into the design and principled selection of future multi-modal benchmarks for model evaluation.", "weaknesses": "- The analyses are model-dependent, and to marginalize out the effect of any single model, the paper uses the ensemble of three MLLMs to evaluate a dataset. The number of MLLMs may not be enough to mitigate the impact from the inductive biases of MLLMs. For example, if two of the three models are strongly correlated in their outputs, then the resulting analyses are biased. It would be beneficial to provide concrete evidences to further justify the selection of the three MLLMs.\n\n- In the experiment section, the paper should be more specific on the criteria of selecting datasets with inter-modality dependency and intra-modality dependency.\n\n- The title is confusing and seems irrelevant to the main content. What are the different dimensions of a multi-modal dataset? \n\nMinors:\n- In Figure 2, it is unclear what do the “Text” and “Image” represent in the legend. Based on the context, it seems that “Text” corresponds to “permuted image”.\n- Line 346, the text does not match with the figure.\n- Line 372: exhibite -> exhibit", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MClmpJ4hF2", "reviewer": "Reviewer_nKQV", "summary": "The paper points out the ambiguity of current multimodal reasoning benchmarks in measuring models inter-modality reasoning capability and conducts the first large-scale empirical analysis of multi-modal dependencies across 23 popular VQA benchmarks. Their findings reveal that many benchmarks allow significant unimodal shortcuts.", "questions": "Could you clarify does the label \"image\" in figure 2(a) indicate that image is permuted or the opposite? If label \"image\" means image is permuted as indicated by the caption, then the graph contradicts the analysis \"This is most illustrated in MMBench (Liu et al., 2024a),\nwhere an image-only model outperforms a random baseline by 41%\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance: While many multi-reasoning benchmarks have been and are being proposed,  this is a much-needed analysis on how well a benchmark can truly capture model's inter-modality reasoning capability. The analysis would influence not only how a given benchmark can be used but also how better benchmarks can be designed for the community.\n\nQuality: Input permutation is a reasonable way to identify intra-modal shortcuts allowed by datasets. Analysis beyond aggregate performance into sub-categories provides insights on what kind of problems allows unimodal shortcuts more than others.", "weaknesses": "The paper could be strengthened by \n1. analysis on potential reasons of some datasets requiring inter-modality dependency while others do not, e.g.curation methods, topics of problems \n2. actionable suggestions on how better benchmarks can be designed given evaluations of current datasets\n3. evaluation with a different model family besides Cambrian-1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y9wYwtnGYf", "reviewer": "Reviewer_Pchi", "summary": "This paper introduces the Multi-Modal Data Spectrum (MMDS) framework, which unifies multimodal learning across different reasoning granularities—from perceptual understanding (low-level recognition) to symbolic reasoning (high-level inference). The key idea is to represent multimodal data as a continuous spectrum, where each sample’s reasoning requirement is estimated using an entropy-based measure of cross-modal dependency. The authors propose a novel spectrum regularizer that encourages consistent alignment across modalities according to their reasoning level, implemented through a contrastive-style loss that balances visual grounding and linguistic abstraction. Empirically, MMDS improves both perception-oriented and reasoning-oriented benchmarks on five datasets (MM-Bench, ScienceQA, VizWiz, OKVQA, and TextVQA) and outperforms strong baselines such as LLaVA-1.6, Qwen2-VL, and InstructBLIP. Ablation studies and qualitative visualizations support the claim that spectrum regularization reduces hallucination and over-textualization errors in multimodal reasoning.", "questions": "1. How stable is the entropy-based reasoning score across datasets with different modality ratios?\n2. Can the authors elaborate on how the reasoning spectrum correlates with the number of intermediate reasoning steps in CoT-augmented MLLMs?\n3. Would the same regularizer generalize to temporal reasoning (e.g., video–text models)?\n4. How sensitive is the method to noisy or ambiguous entropy estimation at the mid-spectrum range?\n5. Could combining the spectrum loss with preference-based finetuning further improve reasoning alignment?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. A well-motivated framework that explicitly bridges perception-level grounding and abstract reasoning.\n2. The spectrum regularizer is simple yet effective, and integrates cleanly into standard MLLM finetuning.\n3. Empirical results are comprehensive and demonstrate consistent improvements across both perception and reasoning tasks.\n4. The qualitative analyses (e.g., reasoning-level visualization and hallucination reduction) are compelling and support the theoretical claims.", "weaknesses": "1. The theoretical foundation is intuitive and not fully formalize. The relationship between entropy and cognitive reasoning depth is empirically approximated.\n2. The dependence on dataset annotations for reasoning difficulty may limit generalization to unseen domains.\n3. The computation of reasoning entropy adds modest overhead, which could be non-trivial for large-scale training.\n4. A deeper comparison with causal or hierarchical reasoning frameworks (e.g., CoT-based symbolic decomposition) would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5GLwR8fQvA", "reviewer": "Reviewer_fSBF", "summary": "This paper conducts a large-scale empirical study of 23 multi-modal benchmarks to analyze intra- and inter-modality dependencies. Using modality shuffling, the authors show that most benchmarks contain strong uni-modal biases, meaning models often rely only on text or image rather than true multi-modal reasoning. The work provides quantitative insights for improving benchmark design and evaluation in multi-modal learning.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Comprehensive empirical validation.\n\n- The study systematically analyzes 23 widely used multi-modal benchmarks with multiple MLLMs of different scales, providing strong empirical support for its conclusions.\n\n2 .Practical relevance.\n\n- The findings are highly relevant for future benchmark design and model evaluation, highlighting concrete weaknesses in current multi-modal testing practices.", "weaknesses": "- The paper proposes a valuable diagnostic framework to measure modality dependencies, but it stops short of offering concrete methodological solutions to mitigate such biases, making its contribution more analytical than innovative.\n\n- If I understand correctly, the authors evaluate modality dependencies by applying modality shuffling on three Cambrian-1 models (8B, 13B, 34B). However, since these models share similar architectures and pretraining data, they may also share inherent modality preferences or inductive biases. Could it be that the observed modality dependencies partly reflect model-specific biases rather than intrinsic dataset characteristics?\n  - Please correct me if I am wrong or missing something here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)