Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
Authors: Shunki Uebayashi
Kento Masui
Kyohei Atarashi
Han Bao
Hisashi Kashima
Naoto Inoue
Mayu Otani
Koh Takeuchi
Venue: ICLR 2026 Poster (2026)
Keywords: VLM
Evaluation
IRT
Primary area: datasets and benchmarks
TL;DR: M3IRT decomposes IRT into single-modal and cross-modal components to identify and filter out shortcut questions in MLLM benchmarks, enabling compact and efficient evaluation of the multimodal reasoning skill of MLLMs.

Abstract: Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities.
Benchmarks for MLLMs should measure their ability for cross‑modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only single modality, and thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image‑only, text‑only, and cross‑modal components. M3IRT estimates cross‑modal ability of MLLMs and each question’s cross‑modal difficulty, enabling compact, high‑quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross‑modal questions over shortcuts and preserves ranking fidelity even when 50\% of items are artificially generated low‑quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross‑modal reasoning and refining multimodal benchmarks.

Published as a conference paper at ICLR 2026
EVALUATING CROSS-MODAL REASONING ABILITY AND PROBLEM CHARACTERISTICS WITH MULTIMODAL ITEM RESPONSE THEORY
Shunki Uebayashi', Kento Masui", Kyohei Atarashi', Han Bao Hisashi Kashima', Naoto Inoue", Mayu Otani", Koh Takeuchi* Kyoto University "CyberAgent. 'The Institute of Statistical Mathematics "Tohoku University
ABSTRACT
Multimodal Large Language Models (MLLMs) have recently emerged as gen- eral architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only single modality, and thereby yielding unreliable rankings. For example, in vision- language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational quirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's 's cross- modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reduc- ing evaluation cost while improving reliability. M3IRT thus offers practical tool for assessing cross- cross-modal modal reasoning and refining multimodal benchmarks.
INTRODUCTION
Multimodal Large Language Models (MLLMs) (Yin et al. 2024) have recently emerged as general architectures capable of reasoning over diverse modalities. A prominent subclass, Visual-Language Models (VLMs). jointly process images and text and are expected to support downstream tasks that require cross-modal reasoning (Jiang & Ye 2023), such medical image diagnosis and industrial inspection (Zhagg et al.) 2024). Consequently. rigorous and trustworthy multimodal benchmarks are essential for practitioners to choose appropriate models (Chen et al., 2024 Yue et al.) 2025).
Benchmarks for MILMS should measure their ability for cross-modal integration. However, current benchmarks are often filled with shortcut questions that can be solved using only single modality (e.g., ans answerable from text alone or image alone). For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmark and yields unreliable rankings Yuue) al.) 2025). ) As the pool of candidate models grows, evaluating thousands of mixed-quee mixed-quality questions model becomes increasingly costly, while single-modality shortcuts further obstacle evaluating the cross-modal reasoning ability.
Item Response Theory (IRT) is principled framework for assessing subject ability and item diffi- culty [Fan 1998). Without knowing the questions and answers, IRT estimates the ability and difficulty as parameters to predict the records of success or failure of subject
item. These parameters allow us to construct a compact subset of items tailored to each subject using Computerized Adaptive Testing (CAT) (Weiss& & Kingsbury 1984 Han 2018). Recent work on LLM has leveraged IRT, where they considered LLM as subject and questions as items, to construct compact and essential subsets of text questions from benchmarks (Polo al. 2024). However, classical IRT is agnostic

---

Published as a conference paper at ICLR 2026
to the modality of inputs and thus contains only single latent ability difficulty parameter. IRT cannot determine whether success on a multimodal item reflects true cross-modal reasoning or others.
To address the limitations, introduce MultiModal and Multidimensional Item Response Theory (M3IRT), and its variant called M2IRT. Our proposed methods simply extend classical IRT by decomposing both model ability and item difficulty into three latent components: image-only, text-only, and cross-modal integration. This decomposition allows us to (i) estimate each VLM's cross-modal ability and (ii) quantify each question's 's cross-modal difficulty. Using these estimates, our proposed methods identifies genuinely cross-modal items and enables compact, high-quality benchmark subsets that better reflect multimodal reasoning while reducing evaluation cost.
We conduct extensive experiments with 24 VLMs across three benchmarks. We construct semi- synthetic benchmarks by generating simple low-quality questions through the swapping of image or text from the original questions to introduce artificial shortcut or unsolvable questions. We obtain the answers of VLMs and make datasets indicating successes and false. We employ M3IRT, M2IRT, and other methods including IRT to refine our semi-synthetic benchmarks. First, we qualitatively observe that M3IRT prioritizes truly cross-modal items over shortcuts and preserves ranking fidelity even when 50% of the items are replaced with artificially generated low-quality questions. Representative highly and lower cross-modal difficulty items identified by M3IRT are shown in Figure [
Second, we conducted experiments to extract subsets of questions from the dataset high-quality problem-discovery task. We quantitatively evaluate the degree of ranking reconstruction for VLMs obtained from small number of subsets of varying sizes, well as the proportion of simple low- quality questions included in these small subsets. The former enables high performance with fewer items. The results show that our proposed framework nearly reconstructs the original ranking using only a 10% subset across all datasets, while also reducing the proportion of low-quality questions to less than half that of existing methods.
Our contributions are threefold:
1. We propose M3IRT, which explicitly models mopelity-specific (image-only, text-only) and cross-modal components of both item difficulty and model ability for multimodal evaluation.
2. We show that M3IRT yields compact, high-quality subsets that emphasize cross-modal reasoning and maintain reliable model rankings substantially reduced computational cost.
3. Through experiments with 24 VLMM across three benchmarks, we demonstrate that M3IRT is robust to large fractions of low-quality items (up to 50%) and provides interpretable characterizations of both benchmarks and models.
2 RELATED WORK
Recent VLM evaluation has relied on large, static benchmarks such as MMMU (uue et al. [024).
MathVista (Lu et al.) 2024), SEED-Bench (Li et al.) 2024a). EMMA (Hao et al. 2025) and
CCHall (Zhang al., 2025). These efforts shift the center of evaluation toward integration it-
self rather than isolated unimodal skills. Static expansions such as MMBench (Liu et al. 2024)
broaden ability coverage but still exposed to low-quality question contamination and leakage. Several
dynamic or live evaluation approaches have emerged such as VLB/FLEX (Yang et al. (025) proposes to automatically generate both image and text. MAC (Jiang et al. 2025b) and LiveXiv (Shabtay et al. 2025) automatically constructs VQA from current news and papers. While valuable, these benchmarks still exposed to the risk of contaminating oow-quality questions, such shortcuts.
Existing methods for single-modal benchmarks can be categorized into Non-IRT-based and IRT-based approaches. First, Non-IRT-based approaches include question clustering that selects representative questions from clustering results, such as active testing with multi-stage sampling (Huang et al. 2024). tailored benchmark creation (Yuan et al. 2025), LLM predictability exploration (Ye et al. (023). and
anchor points ( (Vivek et al. 20244). Adaptive sampling dynamically selects questions based on current assessments of a model's performance, including SubLIME (Xu et al.) 2024), Dele (Saranathan et al.
2024), and methods that model inter-example dependencies (Li et al. 2024b). FlashEval (Zhao et al.
'Ou code and data available at /t/ptt/:///tttttb..om/////
2

---

Published as a conference paper at ICLR 2026
Quedaa nostiail Bchiiseey
accured Seteato de falveicg
con
Be paint
Science artdds read
Quaisen: How may Gas-ro proor, in
dcingee
allmod cue und ugr
Quandien Kimbbiy'ss
Frems
Lo and whut serech mo 25%
Vilra
cany sclanc arbils they Oe nuntbes?
Divecedac hase
R Ore
A.Too
trom bevveh
C. Clllee
Di Fouz
Drad
1300
Lawoos
(a) MMMU (Highly quality)
(b) Mathvista (High quality)
(c) SEED-Bench (High quality)
Baadd dinope mon Hennittt Moroes on forartud Practs Baren and Lacas Proud Whith fomale arts dal the alt
Quntiion
Chaestonn could
that would mocmas the nomberof kill?
stading in the image?
fanill
tomoose phytoplenktre
Ct nod cuppet On cht woeden deck
Dishkh Fint
C. Balass Sepnort
lncresee this (D) tecmsee in binks
Di Georgaa Baddl
(d) MMMU (Low quality)
(e) Mathvista (Low quality)
(l) BEED-Bench (Low quality)
Figure Questions with the highest or lowest cross-modal difficulty Igross detected by M3IRT. Questions with high cross moos- difficulty require both modalities to find the correct answer. However, those with low difficulty allow us to solve using only the image or text.
2024) was recently proposed, offering novel evolutionary algorithm for text-to-image generation. However, they have not considered whether question demand the cross-modal integration not.
Item Response Theory (IRT) (Lord 1980), originating in psychometrics, provides simultancous modeling of subject (model) ability and item (question) parameters (c.g., difficulty, discrimination). The application of IRT has expanded to NLP (Lalor et al. (016)) dialogue (Hirai et al. 2023), ) and recommendation systems (Liu et al. 2023) In the LLM domain, IRT has been leveraged to reduce benchmark volumes; i. c. MetaBench (Kipnis et al. 2025) distills a sparse benchmark from several benchmarks, and TenhBanh (Polo et al. (2224) provides efficient cluster-based sampling method. IRT has also been employed for adaptive sampling/testing of LLMs; for example, dynamic test adjustment based on model performance (Zhuang et al.) (0236)) CAT-based cognitive ability measurement (Zhuang et al.) 2023a). human chatbot evaluation. training of callbbrtee question generators (Jiang et al. 2025a). and automated model evaluation (Guinet et al. 2024).
3 BACKGROUND
Consider a collection of MLLMs, treated as subjects and indexed by M = {1, (1...., m). and multimodal benchmark with questions treated as items and indexed by N = {1..... n). For each subject-questinn
pair (i,j), let Fi,y {0, 1} indicate whether subject answers question correctly (ri.j = 1) or not (ri.j = 0). We denote the resulting response matrix by R = (ri.) (..))Mxx-- Our objective is to assess the cross- abilities of the MILMS and the difficulty of the questions, and
to identify compact subset Ñ N consisting of items that demand strong cross-modal reasoning.
Item Response Theory (IRT) is a family of latent variable models that jointly infer subject ability and item characteristics from observed response data (Fan 1998). Given only the pattern of correct or incorrect responses, IRT estimates ability and difficulty parameters and predicts the probability that a subject will answer given item correctly. We use the two- parameter logistic (2PL) model, which can be viewed as a logistic regression with item-specific slope and threshold:
Pr(ris = 0,, ag. by) a(a (0, - -b)),
where (Ix) = 1/(1 + exp(-2)) is the sigmoid function. For each subject we define an ability parameter 0g E IR; higher values indicate a greater propensity to answer difficult items correctly. For each item j. define discrimination parameter a, > 0 and difficulty parameter by E R. Larger aj means the probability of a correct response is more sensitive to changes in ability, whereas smaller aj implies weaker sensitivity. As the difficulty by increases, greater ability is required to achieve
3

---

Published as a conference paper at ICLR 2026
Multimodal Benchmark
MMIRT
Modality- based
Cross-moue moddll Difficulty & Ability
Low
High
Image Text Cross
Image Text Cross
Image
Decompo sition
Text
Cross
Adaptive Question Selection & Rank
MLLMs Cal
Answers
CAT
New MLLM
AF C
Answer
Figure 2: M2IRT investigates the modality-specifit and cross-modal difficulties of questions that enables to contract tailored, compact, and high-quality subset for evaluating new MLLM.
high probability of correct response. IRT has been applied CAT (Weiss & Kingsbury 1984) to select test questions from an item pool to estimate a subject ability. Namely, we randomly initialize student ability, select question with the maximum Fisher information for a current ability, get an answer, and update subject ability. We repeat this procedure.
dulti-dimmnnsooal IRT (MIRT) method that extends IRT consider the relationship between models and questions in a more complex manner (Reckase) 2009). This method supposes d- dimensional latent parameter space. The ability vector for subject is 0, E and the difficulty and discriminative vectors for question providing correct answer for a pair of (1, and finds maximum likelihood estimator:
aj b; E Rd, MIRT parametrizes the probability for
P(r, - (a by), P(ris 0) -Prr =
4 PROPOSED METHOD
To assess mppelity- and cross-modal properties of MLLMs and multimodal benchmarks, introduce the decomposition of the standard IKT parameters into latent components. Building on this decomposition, introduce MultiModal Item Response Theory (M2IRT) and Multidimensional MultiModal Item Response Theory (M3IRT) as extensions of classical IRT and MIRT. We also develop procedure for selecting compact subset of benchmark items tailored these models. Figure 2 illustrrates the overall framework. Although the method is applicable to arbitrary modalities (e.g., action, audio), this paper primarily focuses on vision and language.
4.1 MODALITY-BASHR DECOMPOSITION OF IRT PARAMETERS
We assume that an MILLM has modallitysspeciff abilities as well as an ability to integrate information across modalities. Likewise, each multimodal question exhibits modality-peeiffi and cross-modal characteristics that can determine whether a subject can provide the correct answer.
In the vision-language setting, we define binary indicators gimage stext E {0, 1} to represent the
modalities present in question: s simage = 1 if an image is provided and gtext = 1 if text is provided; otherwise, the indicator format of representing a question. When (mimae g g5ext) (0, 0), the stimulus are withheld and the subject answers using only guess from introductions or the multiple-choice options.
Let (magge text)
{(0, 0). (0, 1), (1, 0), (1. 1)} denote
We assume each subject has base reasoning ability that, depending the input format combines with image-specific, text-specific, and cross-modal integration abilities. For subject i, denote the base, image, text, and cross cross-modaa modal abilities by gbase, gimage gtext grroes E [0, q]. respectively, where q 2 0 is shared upper bound that balances their scales. Given question and its modality indicator 8, we define the ability of a subject j as follows:
(s(s) =
gbsse image gimage + text glettt + image text gexeeeroes.
The second and third terms contribute when an image or text is present, respectively; the fourth term contributes only when both are present. This construction naturally extends to additional modalities.
4

---

Published as a conference paper at ICLR 2026
We view answering as exploiting hints provided by the item. For item j, let bluas, bjj b bj byrse [0.g] be the base, image, text, and cross- diffieulties, respectively, using the same upper bound q 2 O. We define the difficulty (y(s) of question j given the indicator
by(s) bl gimage J
image & teack Brross
(4)
Similarly, let alase E [0. g] be the base discrimination, let image next afrces E [0, q] capture the contributions from image, text, and pross-modke integration. The discrimination becomes
ag 0j
base
jinaee image + text text + jmmgee texes arross
dj
8 (1j
In general setting, we define indicators to represent the all modalities in a benchmark, and extend parameters applicable to represent combinations of modalities.
4.2 MULTIMODAL ITEM RESPONSE THEORY (M2IRT)
To capture cross-modal behavior, we control which modalities are provided, thus each subject answers each item under the four input formats corresponding to all 8 E S. For each subject-question formatt combination (i, j, s), let Te,ja E {0, 1} indicate whether subject answers question j given the format indicator correctly (ri,j, 1) not (ri,j, = 0). We denote full response set the resulting response tensor by R' = [ xXXS)
M2IRT extends the logistic IRT model in Equation Given discrimination ay (s), difficulty bs)s)
and ability 0, (x), we define zi.j.= = (0, by(8)) and introduce M2IRT as follows:
P(ri... = 1) = (z..j.) and P(rej. = 0) = P(ra... = 1).
(6)
This parameterization captures the modality-aware behavior of subject on item j.
4.3 MULTIMODAL MULTI-DIMENSIONAL ITEM RESPONSE THEORY (M3IRT)
M3IRT extends the logistic MIRT model in Equation with the modality-based decomposition. We modify the decomposed components into vectors. For subject i, define the ability vector 0 (0base gimage Otext (cross] For item j. define the diserimination and difficulty vectors aj
labace imaaee text croes T by = =bbbae b iiage bjoxt berass For convenience, we introduce
format indicator vector = [1, -ginage - gtext imagge & gtext]T where the negative signs align with
the subtractive role of the modality terms in Equation 4 and with the decomposition Equation 5 From these vectors, we define z4,j,- a diag(s)0;
bj. We propose M3IRT as follows:
P(ra,,e
o (2/.g..) and P(rej,s = 0)
P(ra.j. 1).
Here, diag(s) 9) is the diagonal matrix whose diagonal elements are 8. The probabilistic model equa- tion 6 is a variant of multi-dimensional IRT with the parametrization ži.j.s. This parametrization takes in the modality-awar nature of subject when answering multimodal question j.
4.4 LEARNING M3IRT USING STOCHASTIC GRADIENT DESCENT
Instead of the EM algorithm commonly used in IRT, we estimate M3IRT parameters with stochastic gradient descent (SGD). Let training dataset as R" C R'. Given R" and the Bernoulli model in Equation 6 the negative log- ligeeiill the negative log likelihood is
(()) =
Σ ((.),))ER"
(ruj.o log P(rij, = 1) + (1 (ig.) log P(riji -(rij,s=)) =0)
(8)
where the parameters set is θ = (HaseN, (by beN {0.)eM) We minimize L()) busing mini- bach SGD, ê argming L(e). We can estimate M2IRT in a similar manner. Note that our approach does not require a dense response matrix: M2IRT and MBIRT can be learned from partially observed data like tensor completion, reducing the cost of evaluating ML.LMs and benchmarks.
5

---

Published as a conference paper at ICLR 2026
4.5 COMPUTER ADAPTIVE TEST WITH M2IRT AND M3IRT
We integrate M2IRT and M3IRT with classical Computerized Adaptive Testing (CAT) (Weiss & Kingsbury 1984) to adaptively select an informative subset of items N S. N, guided by Fisher information. For M2IRT model, the Fisher information of item j for subject under format is
Lsj P(r.,, P) = (s))
(9)
where P(ras.s 1) is given by Equation 6 information matrix for item j at ability 0 is
For the multidimensional M3IRT model, the Fisher
J5 = P(ri... P)a,,, ()(dia((s)) (din)))))))))))))
(10)
We adopt the D-optimality criterion (Mulder & Linden 2009) to minimize estimation uncertainty by maximizing the determinant of the cumulative information. Let Ug S N be the set of items not yet
answered by subject i. At stage L, given the cumulative information matrix (((11) we select the next item and update:
argmax del (1 + I1j) LT)) jEU;
+ +
(11)
Iterating this rule yields subset that is maximally informative for estimating the subject's ' ability.
5 EXPERIMENT
5.1 DATASETS AND BASELINES
We employed three benchmarks for VLMs in this experiment. MMMU (Yue et al. 2024) is designed to evaluate the reasoning capabilities of VLM through undergraduate-Ievel questions in diverse disciplines such
and design, business, and science. We used 900 questions in the validation set. MATH VISTA (Lu et al. (022) evaluates mathematical reasoning capabilities through questions involving visual context including puzzle figures and graphs. We used 1000 questions of the test-min set. SEE.D-BENCH (Li et al. 2024a) is a large-scale benchmark designed to comprehensively evaluate the multimodal abilities. We used 1000 questions from L1 and L,2 sets.
To simulate the presence of questionable samples in real-world datasets, we constructed a synthetically contaminated benchmark. We made semi- syntheittettic benchmarks by generating simple low- low-quality questions through the swapping of image or text from the original questions. This process introduces artificial shorteut or unsolvable questions. We compile a benchmark contaminated with 50% low- quality questions. We provide a detailed description of data generation process in Appendix [A] To create more realistic low-quality questions, methods such as modifying text and options using LLM adding noise to images could be considered. Since such methods make the experiment overly complex, we excluded them. Note that our method learns ability and problem characteristics from whether VLMs answer questions correctly, even if there are different types of low-quality questions, the estimation results are unlikely to change.
We collected responses from 24 VLMs, including the GPT-4.1 series, Gemini-2.0 series, and Claude- 3.7 series, as well as open-source models such as Qwen-2.5-vl -l (Bai et al. 2025), Llama-3.2 (Metal 2024). and Pixtral (Agrawal et al. 2024). On unavailable at the start of the experiments on SEED-Bench, the experiments on SEED-Bench were conducted with 23 models other than Claude-sonnet-3.
since Claude-sonnet-3 became
We use four baseline methods in our experiments. Random selects subset questions at random. IRT uses Fisher information- based subset selection estimated by IRT (Reckase) 2009). MIRT uses Fisher information -wt--mm-aateeeeeeeee subset selection estimated by MIRT (Reckase 2009). TinyBench- marks (Polo et al.. 2024) is an IRT-based problem selection method for benchmark refinement in LLM. FlashEval (Zhao al. 2024) is a SOTA select prompts for image generation. We extended FlashEval to deal with VLM benchmarks by regarding questions as prompts.
We implemented our proposed method with PyTorch (Paszke 2019), and used Adam opti- mizer (Kingma & Ba 2014) whose learning rate was 0.01. We used a grid search to select hy- perparameter from 2, 4, 8, 16. We selected the optimal hyperparamters based on the highest AUC in predicting the correctness of the VLMs' responses on the validation dataset. We provide the detailed explanation of the experimental setting in Appendix

---

Published as a conference paper at ICLR 2026
Spage
Opraee
(Alinng)
Buo
8 (Aliling
Ouw
Gosine 1. Bhodee Cladee E-ennod
3 do
RC Srrmn
Toe 2200 li aee
Chooti comndd a
Home Cleak Meol
teves yausi
L
(a) MMMU
(b) MATHVISTA
Figure 3: Distributions of / estimated by M3IRT sorted in descending order.
1.00
0.5
0.5
NZIRT
00
KZIKT NOIKT
0.2
TayBeedan
0.0
Racder
04
Radees
Racker
10
301
10
30
10
20
40
Scbuet (M1
Schsit (KV))) [S]
Scbset (KVY)M)
(a) MMMU
(b) MATHVISTA
(c) SEED BEED-BENCH
Figure 4: The average and standard deviation of Spearman's 's rank correlations between model rankings on the original benchmark and those estimated on extracted question subsets with different sizes.
5.2 MULTIMODAT. DIFFICULTY AND ABILITY DECOMPOSITION
M3IRT estimates the extent to which a question requires cross-modie reasoning, represented by difficulty 1gross This facilitates the identification of questions that truly benefit model's cross-modal capability assessment. Figure shows examples of questions with high and low Jrress 'he questions with low beross are judged that they can be solved only with images text. For example, the bottom one in MMMU can be answered based on knowledge of artists without looking into the image. On the other hand, the questions with high beross cannot be solved if either the image the text is missing. For example, the one in MATH VISTA requires reading the numerical values in the table that cannot be confirmed only by the question text. Similarly, if only images are provided, is not clear what is being asked about in the table. We provide more examples in Appendix C
M3IRT also estimates the extent to which the reasoning ability for each modality contributes to the VLM performance. Figure 3 shows the decomposed reasoning abilities of VLMs. The top- performing model on MMMU exhibits high (O ), suggesting strong cross-modal reasoning capabilities. On the other hand, the second and third best-performing models demonstrate high textual reasoning ability (8text) but limited cross modsl reasoning capability. This analysis suggests that these latter VLMs rely heavily text understanding when solving the MMMU benchmark, rather than effectively integrating visual information. In MATHVISTA, most VLMs have high 0toxt This may reflect MATHVISTA 's emphasis text understanding. Most VLMs also exhibit moderate gimage suggesting that they also leverage the visual ability process diagrams and graphs. The result for SEED-BENCI is shown Appendix C
5.3 BENCHMARK REFINEMENT
We investigate whether a method can extract a compact subset of questions that enables us to evaluate the performance of unseen VLMs. We randomly select VLM from collection of VLMs and construct a subset of the responses of remaining VLMs. For a method, we select a subset of questions, estimate the performance of the VLM from its responses to the subset, and obtain an estimated ranking of VLMs. We compare the difference between rankings on the original benchmark for all models. We also investigate how much the artificial low-quality questions included in the subset.
7

---

Published as a conference paper at ICLR 2026
0.6
KOIRT
0.5
NOIKE
IRT
0.33
MZIRT
0.4
NIKT
NGIKT
NDIXT
02
NIKT
IRT
02
XIXI
IRT
0.1
0.1
Hinker
300
Subout size (N1
Scboct size (M
Scbout six (M0)/
(a) MMMU
(b) MATHVISTA
(c) SEED-EEEREE BENCH
Figure 5: The average and standard deviation of the proportions of the low-quality questions in extracted question subsets ? with different sizes.
We use two measures to assess the quality of subset N C N selected by method. First, we assess how much a method avoid the low-quality questions in the estimation of model rankings. We compute the Spearman's 's rank correlation between model rankings on the original benchmark and the extracted subset. Second, we evaluate how a method can distinguish between the original and low-quality questions. We measure the proportion of low-quality questions in the extracted subset as
1iq E Ñ is low-quality question}
|N|
We varied the subset size from 1% 50% of the whole benchmark in 1% increments. We employed CAT with M2IRT using the maximum Fisher information in Sec. 4.5 and M3IRT using D-Optimaliyy, We obtained the average and standard deviation from twenty four independent experiments.
Figure 4 shows the Spearman's 's rank correlations between the model rankings on the original bench- mark and on different sizes of subsets. Figure S shows the proportion y with varying size of subsets.
As shown in Figure 4 our methods accurately estimate model rankings from contaminated bench- marks, even with small subsets. MMMU, M2IRT achieves a rank correlation of 0.9 using only 3% of the benchmark subset, and M3IRT suprizingly achieves rank correlation of 0.8 using the only 1% subset. FlashEval, which SOTA but does not account for the presence of low-quality questions, performs similarly to Random. In MATIIVISTA, M3IRT achieves rank correlation of 0.84 with subset fraction of only 2%, requiring 30% to achieve a rank correlation of 0.9. In BEND-BENCII, M2IRT achieves rank correlation of 0.9 using only 3% of the benchmark subset, while MBIRT achieves the same rank correlation using only 1% the benchmark subset.
From Figure5] we confirmed that the proportion of artificial low-quality questions included in the subset selected by the proposed method is significantly smaller compared to existing methods. In MMMU, even with extraction subset size of 50%, all proposed methods keep the proportion of low-quality questions notably low at 24%. In contrast, the baseline methods choose substantially more low-quality questions than ours, which skew the estimated model rankings. When extracting 30% of MATH VISTA, the rank correlation between M3IRT and Random is about the same, but is smaller for M3IRT. We observed similar trends in the results of
5.4 ROBUSTNESS AGAINST LOW-QUALITY QUESTIONS
We have evaluated the performance of the proposed method using subset of questions. Here, we assess its performance as latent variable method for predicting missing responses from observed ones. First, from the set of questions N, we randomly select 100 or 10% questions each for validation and testing, using the remainder training data. Next, perform parameter estimation using the training data for both the proposed method and IRT. Finally, we evaluate the prediction performance on the test data using the estimated parameters with ROC-AUC. We measured ROC-AUC by varying the proportion low- -ualltyl problems introduced in Sec. 3.1 We used IRT as a baseline this experiment. We obtained the average and standard deviation from ten independent experiments.
We show the results in Figure 6 Our proposed methods achieved performance comparable to the standard IRT on ROC-AUC. M2TRT was slightly better than IRT MMMU, and comparable IRT
8

---

Published as a conference paper at ICLR 2026
MBIRT M2IRT IRT
0.83
0.89
0.80
AUC
AUC
AUC
MBIRT
0.82
MHIR
0.88
M2IRT
0.78
IRT
IRT
0.81
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Low-quality Size
Low-quality Size
Low-quality Size
(a) MMMU
(b) MATHVISTA
(c) SEED-BERE -BECHH
Figure 6: ROC-AUC on predicting answers of the noisy benchmarks containing the different size of the low-quality questions inserted into the original benchmark.
on MATHVISTA and SEED-BENCI, M3IRT was slightly lower than IRT on MATHVISTA but the difference is small. Even when low-quality questions are mixed in, the proposed method and IRT achieve ROC-AUC values around 0.8, suggesting that they effectively capture both the abilites of VLMM and the characteristics of the questions.
CONCLUSION
We addressed the challenge of assessing cross-modal reasoning characteristics in MLLMs and multimodal benchmarks while reducing evaluation cost. We introduced M3IRT and its variant M2IRT, which decompose both model ability and item difficulty of IRT into image-only, text-only, and cross-modal components. This decomposition enables the identification of highly cross-modal items that require tross-modal reasoning and supports lightweight assessment with far fewer items.
Across three benchmarks and 24 VLMs, we qualitatively evaluated that M3IRT can estimate the degree to which an item requires modal reasoning, and assigns higher cross-modal difficulty to genuinely cross-modal items than to single-modalee contaminated benchmarks confirmed that M3IRT and M2IRT yields evaluations aligned with the original benchmarks, demonstrating robustness to low-quality contamination.
shortcut. Moreover, analyses with synthetically
Limitations and future work. Our study focuses on multiple-choice, which typical form of closed-ended questions. Extending the framework to open-ended settings with open-ended questions a natural next step. enabling the discovery items that demand stronger cross-modal reasoning and the evaluation of MLLMs under generative outputs. Beyond vision-language applying the approach to additional modalities (e.g., audio, actions) and developing question-generation methods that control cross-modal difficulty are promising directions.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. All datasets used in this paper (MMMU, Math Vista, SEED BEED-Bench) are publicly released benchmarks, and strictly followed their respective licenses.
ACKNOWLEDGMENTS
This work was supported by JST FOREST Program Grant Number JPMJFR232S. JST CREST Grant Number JPMICR2IDI, JST BOOST Grant Number JPMJBY24E8.
REFERENCES
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, and Robin Lutz. Pixtral 12b. arXiv preprint arXiv:2410.07073. 2024.
Anthropic. Introducing the next generation claude. http://wwwaattnnpicccnneew/ddd family, 2024a.

---

Published as a conference paper at ICLR 2026
Anthropic. Claude 3.5 sonnet. http://www...aatinicccc/m/////de
2024b.
Anthropic. Claude 3.7 sonnet and claude code. https://www.aaatnpoccccoococ//cs 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
Chongyan Chen, Samreen Anjum, and Danna Gurari. Vqa therapy: Exploring answer differences by visually grounding answers. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Xitao Fan. Item response theory and classical test theory: An empirical comparison of their item/person statistics. Educational and Psychological Measurement, 58(3):357-381, 1998. doi:
10.11770013164498006800
Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Gauthier Guinet, Behrooz Omidvar- Omirvar-Tehrani, Anoop Deoras, and Laurent Callot. Automated evalua- tion of retrieval-augmmented language models with task-specific exam generation. In Proceedings of the s1 International Conference on Machine Learning (ICML), 2024.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 8.
Kyung Chris T Han. Components of item selection algorithm in computerized adaptive testing. Journal of Educational Evaluation for Health Professions, 15:7, 03 2018.
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can MILMs reason in multimodality? EMMA: An enhanced multimodal reasoning benchmark. In Forty-second International Conference on Machine Learning (ICML), 2025.
Ryu Hirai, Ao Guo, and Ryuichiro Higashinaka. Applying item response theory to task ess--oreeted dialogue systems for accurately determining user's task success ability. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and SIGDOA))) pp. 421 421-427, 2023.
Yuheng Huang, Jiayang Song, Qiang Hu, Felix Juefei-Xu, and Lei Ma. Active testing of large language model via multi-stage sampling, 2024.
Amazon Artificial General Intelligence. The amazon nova family of models: Technical report and model card. https//www..mmaznn.ccr www.:/wwawaa...ii.cc.c.c. http:///epeeccccciioieemmmmmmmna
technical-report-ann and-port-add-ooone -wp-lpoo---d-d-mo---arrrrdd 2024.
Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Han Jiang, Xiaoyuan Yi, Zhihua Wei, Ziang Xiao, Shu Wang, and Xing Xie. Raising the bar: Investigating the values of large language models via generative evolving testing. arXiv preprint arXiv:2406. 242.:3000..42330 14230,
2025a.
Mohan Jiang, Jin Gao, Jiahao Zhan, and Dequan Wang. MAC: A live benchmark for multimodal large language models in scientific understanding. In Second Conference on Language Modeling (COLM), 2025b.
10

---

Published as a conference paper at ICLR 2026
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representatioss//LLR) 2014.
Alex Kipnis, Konstantinos Voudouris, Luca M Schulze Buschoff, and Eric Schulz. metabench sparse benchmark of reasoning and knowledge in large language models. In International Conference on Learning Representations//CRR. 2025.
John P Lalor, Hao Wu, and Hong Yu. Building an evaluation scale using item response theory. In Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing( EMNLP), pp. 648-657, 2016.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024a.
Yang Li, Jie Ma, Miguel Ballesteros, Yassine Benajiba, and Graham Horwood. Active evaluation acquisition for efficient Ilm benchmarking, 2024b.
Yang Liu, Alan Medlar, and Dorota Glowacka. What we evaluate when we evaluate recommender systems: Understanding recommender systems' performance using item response theory. In Proceedings of the 17th ACM Conference on Recommender Systems (RecSys), 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision (ECCV), 2024.
Frederic M Lord. Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates, 1980.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Meta.
Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. wtom//lo//llamm-33--ommeee
edge-mobll-ddeiicss. 2024.
Joris Mulder and Wim Linden. Multidimensional adaptive testing with optimal design criteria for item selection. Psychomelrika, 74:273-296, 2009.
OpenAl. Hello gpt-4o. http:///wonnii.ccommmmmxxxxbllep 2024a.
OpenAl. Gpt-4o mini: Advancing cost-efficient intelligence. http:///ppmmi.coom//deee/tttoooo poapoingg-ootttfffffiee-inllliii
2024b.
OpenAl. Introducing gpt-4.1 1 in the api. http:///oeemi.com//nnxxx//s--/ 2025.
A Paszke. Pytorch: An imperative style, high-performance deep learning library, 2019.
Sundar Pichai. Introducing gemini 2.0: Our new ai model for the agentic cra. https//blog.googleeechhoooo htwpa///wyooo.ooooooooooooooeeea message, 2024.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: Evaluating Iims with fewer examples, 2024.
M. D. Reckase. Multidimenssonal Item Response Theory. Springer Science & Business Media, 2009.
Gayathri Saranathan, Mahammad Parwez Alam, James Lim, Suparna Bhattacharya, Soon Yee Wong, Martin Foltin, and Cong Xu. DELE: Data efficient LLM evaluation. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models(ICLR Workshop), 2024.
Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh, Wei Lin, Muhammad Jehanzeb Mirza, Leshem Choshen, Mikhail Yurochkin. Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, and Raja Giryes. Livexiv a multi-modal live benchmark based on arxiv papers content. In The Thirteenth Interna fional Conference on Learning Representations (ICLR), 2025.
11

---

Published as a conference paper at ICLR 2026
Google Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of
context. wowp:/oooragggoooggpg.....o.oo commddeppinnddmcccalgggmimmmmmme mediaa/gemin//geminn v1 _i_______________ 5_report.ppfff 2024.
MiniMax Team. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08//3, 2025.
Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. In Proceedings of the 18th European Chapter of the Association for Computational Linguistics (EACL), pp. 1576-1601, 2024.
David Weiss and G Gage Kingsbury. Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21 21(4):361-3755 1984.
XAL. Grok-2 beta release. https://wai/neeww/rrkk22 2024.
Cong Xu, Gayathri Saranathan, Mahammad Parwez Alam, Arpit Shah, James Lim, Soon Yee Wong, Martin Foltin, and Suparna Bhattacharya. Data efficient evaluation of large language models and text-to-image models via adaptive sampling, 2024.
Yue Yang, Shuibo Zhang, Kaipeng Zhang. Yi Bin, Yu Wang, Ping Luo, and Wenqi Shao. Dy- namic multimodal evaluation with flexible complexity by vision-language bootstrapping. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Qinyuan Ye, Harvey Yiyun Fu, Xiang Ren, and Robin Jia. How predictable are large language model capabilities? case study on big-bench, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12):wwae033 2024.
Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, and Kan Li. Beyond one-size-fits-all: Tailored benchmarks for efficient evaluation, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Xiang Yuc, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU-pro: A more robust multi-disciiline multimodal understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5625-566 5644, 2024.
Yongheng Zhang, Xu Liu, Ruoxi Zhou, Qiguang Chen, Hao Fei, Wenpeng Lu, and Libo Qin. CCHall: A novel benchmark for joint cross- lingulin and cross-modal hallucinations detection in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025.
Lin Zhao, Tianchen Zhao, Zinan Lin, Xuefci Ning, Guohao Dai, Huazhong Yang, and Yu Wang. Flasheval: Towards fast and accurate evaluation of text-to-image diffusion generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao Zheng Zhang, Qingyang Mao, Shijin Wang, and Enhong Chen. Efficiently measuring the cognitive ability of Ilm: An adaptive testing perspective, 2023a.
Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Zachary A Pardos, Patrick C Kyllonen, Jiyun Zu, Qingyang Mao, Rui Lv. Zhenya Huang, and Enhong Chen. From static benchmarks to adaptive testing: Psychometrics in ai evaluation, 2023b.
12

---

Published as a conference paper at ICLR 2026
A DETAILS OF LOW-QUALITY QUESTION GENERATION
For MMMU, we made three types of low-quality questions: (A) 300 questions consisting of the image, text, and multiple-choice selected from all different questions, (B) 300 questions where the image was replaced with that from different questions; and (C) 300 questions where the text was replaced with that from different questions. For MATIVVISTA, we made 333 questions each for (A), (B), and (C). For SEED- -BNCHH we made 333 questions each for (A), (B), and (C).
B ALIGNMENT WITH HUMAN ANSWERING PATTERNS
To investigate whether the cross moosl difficulty bfross estimated by M3IRT corresponds to human answering patterns, we conducted a crowdsoureing experiment. We randomly sampled 200 questions per benchmark. Nine crowd workers answered each question under three ipput formats: Image+Text, Image-only, and Text-only. For each question j, we computed the accuracy under each format and defined two measures of cross-modal synergy:
AcCmm + ACCuex.
Declinemean.j =
ACCinageHexr.
(12)
2
Declinemax.j =
ACCimggo+txx.. max ACCimageui AcCrext.s).
(13)
Declinemenn measures how much the joint presentation improves the average single-modality performance, while Declinenux measures the improvement over the best single modality. We report Spearman's rank correlation between beross and each measure. anter-annotator agreement assessed by Fleiss' K, where values above 0.6 are generally considered acceptable.
Table D summarizes the results. On SEED-BENCH, we observed weak positive correlations between 02 Beross and both Declinemean (0.35) and Declinemax (0.35), with moderate inter-annotator agreement (x 0.44) and crowd worker accuracy of 0.62 (compared to 0.57 for VLMs). On MMMU and MATHVISTA, the correlations were weak negative (Declinemeun! 0.077 and 10.16; Declinemox' 0.090 and -0.15, respectively), with low inter-annotator agreement (x = 0.25 and 0.28). Crowd worker accuracy on these benchmarks was notably low (ACCinage-text = 0.41 and 0.23), whereas VLMs achieved 0.52 on both. These results indicate that VLMs and humans exhibit different cross- modal answering patterns: the questions that M3IRT identifies requiring strong cross-modal reasoning for VLMs do not necessarily pose the same cross- modal demand for human solvers. This difference was observed across all three benchmarks, though varying degrees, and larger MMMU and MATHVISTA, where crowd worker accuracy was notably low.
For the human evaluation, collected responses from nine crowd workers via Amazon Mechanical Turk (MTurk)). The tasks consisted solely of answering multiple-choice questions drawn from the aforementioned public benchmarks. No personally identifiable sensitive information was collected.
Table I: Relationships between VLMs and human answer patterns
MMMU
MathVisha
SEED-Bench
Declinemar
0.032
-0.23
0.18
Corr. between Declineman and b
boross
0.090
-0.15
0.35
Declinemean
0.14
-0.11
0.31
Corr. between Declinemean and Jyroos
0.077
-0.16
0.35
Fleiss'
0.25
0.28
0.44
Accuracy of Crowd workers
0.41
0.23
0.62
Accuracy of VLMs
0.52
0.52
0.57
C OMITTED RESULTS OF MULTIMODAL DIFFICULTY AND ABILITY DECOMPOSITION
Figure 7 shows the decomposed reasoning abilities of VLMs estimated by M2IRT. Figure Ta Fig 7b and Fig. 7c show the decomposed reasoning abilities of VLMS for MMMU, MATHVISTA, and
13

---

Published as a conference paper at ICLR 2026
7.5
oasse
(Ability)
magg
5.0
Gtoxt
Orooss
2.5
0,0
Gemini-2 Demini-2.0-flass 0-flashh
GPT-4. GPT-4.1-mmin 4.1-min
Gemini-1 Cemini-1.5-pra 1-po
GPT-4.1
Qwen-2 Qwen-2.5-VLL 5--27---
Claude 1lude-3.5-50m1 3.55-onneet
GPT-40
Gemini-1 1..mini-1.5-flas. 15-ssh
Claude-3. 3.7-5onnet
OPT-
Nova-Pro
Grok-2-V/000
Gemini-1.5 1.5-flash-BB
GPT-4o. 111--ommiin
Liama-3.2-900 3.22-33--900
Nova-Lite
pixtral-125
Llama- 1-ma--32-118
3-30nnee
MiniMax-01
0wen-225.VL.
MMMU
15
base
8 (Ability)
mmage
10
2zext
Beooss
Grok Grok-2-Viston --isson Gemini-2 0-0-ffshh
Gemini-1 Geminni-155ppm 1.roprro
Claude-3.7-55 7.7--onnnet
Pixtral-Laa Laagge
Gemini-1 5-faash
Large
Nova-Pro
Claude-3 5-soonnet
GPT-4o
GPT-4- -1
Gemini- Cominn-11.5-laas. 5-ffash--80
GPT GPT-40- 40--mminni
turboo
Nova-Lite
Llama-3.2-90 3.ma-3.2-905 33.2-908
Pixtral-128
Claude Claude-3-50m 3-sonnet
GPT-4.1
Claude Claude-3-tane 3-haakk
GPT-4. 1.1minni
Owen- -.en--....... 5-V-222
2.5--1--b
GPT-4.1-11 4-1--nn
1amm--3.2-11.
(b) MATHVISTA
15.
Susse
e (Ability)
mmmage
10-
Ocress
5
GPT-4.1
GPT-4.77 4.1-min
Gemini-2.0 f-0llahhh
Qwen-2.5-Vi. 5-VL-72
GP
Gemini-1 5-pro
Gemini-1 1.5-fi-----flash 1.5-fishh
MiniMax-01
GPT-40-mini
Nova-Lite
Grak-2-Vislon
Plxtral-Large
GPT-4.1 -nn--oo-an
Pixtral-12b
Claude-3 3.5..3555.00 5-sonnet
Nova-Pro
Gemini-1 1.5-ffassh-hB 8b
Qwen-2.55 Qwen-2.5-VL-77 7b
Claude-3 Mouud-3.7-somme 3.7-sonnet
GPT-4-turia
Claude-3 Baude-3-haiku 3-haiku
(c) SEED-BENCH
Figure 7: Distributions of 9 estimated by M2IRT sorted in descending order.
SEED-BENCH. Figure shows the decomposed reasoning abilities of VLMS for SEED-BENCIL In SEED-BAL BEDCH, most VLMs have high gimage This result corresponds to the fact that SEED-BENCH contains problems which require images strongly to solve.
14

---

Published as a conference paper at ICLR 2026
Baase
Ommage
0 (Ability)
4
Ocross
2
GPT-411
GPT-4o
GPT-4.1 PPT-41-mibn I-mini
Gemini-2 Gemini-2.0-fp 0.0-flashh
Gemimi-155 1.5
Qwen-2.5-VV 1-L 72b
Gemini- S-fllab
Claude S.adde335--am 3.5.smne
GPT-4o miii
GPT-4.1 I-namo
Cladd 3.nde-3.7--onn 7.7maonme
Nova-Lin Lite
MiniMax 01
Grok- Grok-2-W 2-4----/siinn Vision
Pixtral-1 Large
Pixtral-122 12b
Gomini- Gomini-115-fpp 1.5.----aaahh 35-4lash-88
Nove Nova-Pro
Qwen-2 2.o.225.VVL. 7b
GPT-4-turho
Claude 3-hakku
BEED-BENCH
Figure 8: Distributions of f estimated by M3IRT sorted in descending order.
C.I QUESTIONS WITH HIGH OR Low CROSS-MODAL DIFFICULTY
We show the questions detected by M2IRT that require the high or low cross-modal reasoning ability from MMMU in Fig. 10 and Fig. 9 from MATHVISTA Fig. and Fig. and from
SEED-BENCH in Fig. 14 and Fig. respectively.
As shown in Fig, Fig. and Fig.
13 questions which require the high cross-modal reasoning and Fig. 14 can be solved by using single-modality
ability, whereas questions in Fig. T0
Fig 12
only. For example, the question shown in Fig. 10c presents an image of cholera bacteria, where the correct answer (A) can be identified solely from the image and answer choices,
without the text. The question shown in Fig. c2b be solved correctly without the image if one knows the number of veins for each plant. For the question shown in Fig. [4c]. since the question in Fig. [4c this problem can be solved correctly simply by answering the characters shown in the image. On the other hand, the question shown in Fig. requirrsss both the image. which provides velocity information, and the text, which specifies the particular conditions to identify within the figure. Consequently, the problem cannot be solved correctly if either the image or text component missing. The question shown in Fig. 11c The question in Fig. 13c requires both the image, which provides there is one person who wears black clothes, and the text, which specifies specifying what to count within the figure. Consequently, the problem cannot be solved correctly if either the image or text component missing. Thus, the geross successfully distinguishes between questions suitable for evaluating moosl ability where essential information is distributed across both image and text, requiring an examination of both to obtain the necessary information and those that do not effectively evaluate cross-modal ability.
15

---

Published as a conference paper at ICLR 2026
Tompeattu E
Bood
Doypan satuationn (at nal)
92%
Daygen snureton (watin)
perh on mom
Quadiio:: An 22-yeerood worsan presents the oftice with year histocy worseiing oough and shoctness of treath. She has a 15 pock hustory of smiking 15
Aiaage
<Image
Quasdion: <iuage 1> Hicás Products produces and set palio funtlare through nalional desierchip
ciparble smoking
<inaaee ECG findings nomal. I/FV 65%s predicted, most approsiiate iahaled medication for this patient works by blooking which the following receptons?
<image >>iiven the graph of the velocity vs. time duck flying due south the winter At what point did the duck stop its
Question:
network. They gurchese materials from viriety of supeliers all nanufocturing assembly wark plact oulside Cleveland, your anding
performed
motion?
They reconded these Docember 31, 201 W total meceue?
SWbeta adrenetiic mocpioes glucooortoood receptics
A:A
histamin receptors
B:
A: 133100.000 $2,616,000
D: inlknttiine ieceptons
C:
$2,474,000
E: muscammic mooptors
D:
D: $494,0000
(a) validation Pharmacy 2
(b) validation Physics 26
(c) validation Accounting 12
Figure 9: MMMU: Questions with the high cross-modal difficulties byross
<imago 1>
<iaage 1>
<Imsaee
Question: Refer the figure <image 1>; which term refers to lines that give the impression and tranquility, those sven in the ocean and open prairies?
Question: Name the written- out omament, which is marked with brackst. <image 1>
The cirvalor of muscle that are at the eatrance and exit the stomach are called, Choosing matching <image 1>
Question:
A: aociacratura appoggiat
A: Diagonal Line
Jower morden upper tums
B: Emulsiicction
B: Horizontal
C: nthrrax
Vertical line
D: Pertstalsis
D: List spacing
(a) validation Literature 6
(b) validation Music 11
(c) validation Agriculture 12
Figure 10: MMMU: Questions with the low cross-modal difficulties bgross.
D
B
M
A
Qustionnn zAOBB <BOC 20OD
2:3:4. AtsOM. ON, ZAB
5cCOD, XzMON ][lAAOB
Question: Acmos all years, what the maximum rabing of statstical capacity in Maldives
Question Is Dark Orchid the maximum?
0
(A) yes
(A) 28
(B) no
(C) 32'
(D)
(a) 784
(b) 618
(c) 998
Figure II: MATHVISTA: Questions with the high cross-modal difficulties beross, Each caption the ID of the problem in MATHVISTA
16

---

Published as a conference paper at ICLR 2026
10
2
8
4
Questtion: What time is shown? A nswer by typing time word, number: five.
Question: If you wanted the leaf with the main veins, you choose?
diagrom
congreent
angle angic congruent to angle mao the length of A.B to the length ED is 5:1, the area of the izangle BED S"a"2 + +10 what is the triasgle
(A)
(A) half
(B) pinnate
(B) quarter
(C) reticulate
[A] 5ha 10
(C) o'clock
(D) palmate
(R) 25"" +05
(C) 23"a 100
(D) quarter
a2 (2) runnot be determined
(E) quarter
(a) S31
(b) 514
(c) 315
Figure 12: MATHVISTA: Questions with the low cross-modie difficulties b5 Each caption is the ID of the problem MATHVISTA
Question
How many buildings the
Question: Which object closest bottom of the image?
Question How many people the image wearing black jackets?
image?
A:
A: Tree
A: One
C:
C: Fog
C: Four
D.
Three
(a) 234357012
(b) VizWiz train 00018501
(c) 1907002
Figure 13: SEED-BENCH Questions with the high cross-modal difficulties bgross
SIZE
Question: What the relation between the people standing on the beuch und the occan?
Question: What color are the hats worn by the two men?
What is the text in the image?
Question:
and green
None of the above
A: ocean behind them their left
green
B:
Red and white
SINE
C: The ocean is front them
Blue and yellow
SIZE
D: The their right
(a) VizWiz train 00003082
(b) 354261000
(c) 279787000
Figure 14: VQAAT: Questions with the low cross-modal difficulties Boross
17

---

Published as a conference paper at ICLR 2026
Table Estimated 0 MATHVISTA
model
Dasee
0 imaage
Ovext
Crooss
total
Gemini-2.0-flash
0,00
0.93
0.99
4.0
5.9
Soude-3.7-sonnee
0.00
0.63
2.0
3.1
5.7
Nova-Pro Pro
0.10
0.36
4.0
0.56
5.0
Grok-2-Vision
0.10
0.89
3.4
0.57
5.0
Gemini-1.5 p-o
0.00
0.97
1.4
2.6
4.9
Gemini- 1.5-flash
0.00
0.60
0.84
2.7
4.1
Nova-I lite
0.10
0.00
4.0
0.00
4.1
Qwen-2.5-VI--726
0.10
0.00
0.00
4.0
4.1
Pixtral-Large
0.10
0.70
2,4
0.25
3.4
GPT-4o
0.00
0.32
2.5
0.57
3.4
GPT-4o-mini
0.00
0.00
288
0.41
3.2
Llama-3.2-90B
0.10
0.00
3.0
0.00
3.1
Claude-3.5-somnet
0.00
0.38
1.7
0.61
2.7
GPT-4-turbo
0.00
0.46
1.1
0.51
2.1
0.00
0.36
0.60
0.78
1.7
Pixtral-12h
0.00
0.41
1.1
0.00
1.5
GPT-4.1
0.10
0.30
0.00
1.1
1.5
Qwen-2.5-VL-7b
0.10
0.00
0.00
1.2
1.3
GPT-4.1-mini
0.10
0.00
0.00
1.2
1.3
Claude-3-sonnc
0.00
0.00
0.98
0.00
0.98
MiniMax-01
0.00
0.64
0.02
0.31
Hlaude3-haiie
0.00
0.00
0.71
0.00
0.71
Alama-3.2-111
0.10
0.00
0.40
0.00
0.50
GPT-4.1-name
0.10
0.00
0.00
0.15
0.25
C.2 DETAILED RESULT
Table
Table
and Table show the values of 0 predicted in Fig. and Fig.
Table 2: Estimated on MMMU
model
Abasc
i Oinage
Oxext
Beroes
Gemini-1 1.5-ffsh--h
0.00
1.2
0..0
4.0
5.2
Claude-3.7-sonae
0.03
0.78
4.0
0.00
4.8
Claude-3.5-sonme
0.06
0.68
0.00
4.7
Gemini-2.0- 2.0-fi-
0.09
1.4
0.68
4.3
Gemini- 1.55ni-1..5-pr
0.10
1.0
0.29
2.8
4.2
BPT--41-minn
0.09
1.8
0.70
1.4
3.9
Pixtrl-large
0.03
0.90
0.37
2.5
3.8
GPT-4.1
0.10
1.4
0.57
3.8
GPT-4o
0.10
0.66
0.71
3.7
Gemini-1.5 1.5--lash flash
0.06
0.75
0.38
2.4
3.6
Qwen2.5-VL-720
0.07
1.2
0.51
1.7
3.5
Pixtral-12b
0.00
0.81
0.00
2.6
3.4
Nova-Pro
0.08
0.62
0.68
1.7
3.0
Llama-3.2-90B
0.09
0.08
0.20
GPT-4o-mini
0.07
0.43
0.41
2.0
3.0
GPT-4-turbo
0.10
0.54
0.94
1.1
2.7
Grok-2-Vision
0.10
0.38
1.0
1.0
2.5
Llama-3.2-11B
0.01
0.65
0.00
1.6
2.3
Qwen2.5-VL-7B
0.03
0.32
0.79
Nova-Lite
0.10
0.22
0.24
1.6
2.1
GPT-4.1-nano
0.10
0.13
0.19
1.4
1.9
Claude-3-haile
0.10
0.00
0.63
0.09
0.82
Soonnet
0.10
0.07
0.50
0.00
0.67
MiniMax-01
0.00
0.00
0.00
0.00
0.00
18

---

Published as a conference paper at ICLR 2026
Table 4: Estimated ! on SEEDBENCH
model
Dbsse
0 imaage
Ocext
Crooss
total
GPT-4.1
0.10
2.0
1.9
2.0
6.0
GPT-4o
0.00
2.0
0,51
2.0
4.5
GPT-4.1-mini
0.10
2.0
2.0
0.00
4.1
Gemini-2.0-flash
0.10
2.0
2.0
0.00
4.1
Gemini-1.5-pro
0.10
2.0
2.0
0.00
4.1
Qwen-2.5-VL-726
0.10
2.0
2.0
0.00
4.1
Gemini-1.5-flas
0.10
1.7
2.0
0.00
3.8
Claude-3.5ssoneet
0.00
0.80
0.00
2.0
2.8
GPT-4o-mini
0.00
2.0
0.58
0.07
2.6
GPT-4.1-namo
0.00
0.98
1.6
0.00
2.6
Blaude-3.7-somee
0.10
0.35
0.00
2.0
2.4
Nova-Lite
0.00
1.9
0.00
0.54
2.4
MiniMax-01
0.00
2.0
0.39
0.00
2.4
Grok-2-Vision
0.10
1,4
0.23
0.43
2.2
Pixtral-Large
0.00
1.3
0.45
0.02
1.8
Pixtral-12h
0.00
1.2
0.30
0.00
Nova-Pro
0.00
1.1
0.00
0.00
1.1
Gemini-1.5 1.5--h---8
0.00
0.96
0.00
0.00
0.96
Qwen-2.5-VI--7b
0.00
0.63
0.09
0.00
0.73
GPT-4-turbo
0.00
0.00
0.00
0.00
0.00
Claude-3-haie
0.00
0.00
0.00
0.00
Benchmark Refinement
Multimodal Adaptive Test
High- figh-quality Benchmark
Ranking
New MLLM
New Benchmark
New Ranking
Nigg
x(N
IV
R(M
Benchmark
New Benchmark
N
N
Figure 15: An illustration of our benchmark refinement and testing processes.
D OMITTED RESULTS OF MULTIMODAL BENCHMARK REFINEMENT
First, illustrate our problem setting for benchmark refinement in Fig. To investigate how the estimated parameters of the original questions and low-quality questions vary, we show the distribution of the estimated difficulty, discrimination, and the Fisher information of the original questions and the low-quality questions in Fig. [6 and Fig. T7 We also investigated whether there is a significant difference between the two distributions with the Mann- Mann-Whitney Whitney U test. Asterisks mark
**0
60
Original Low-qualit
Original
Fisher Information
50
Lew-quality
40
brut ditticuit))
(diserimination
10
Origiinal
Sautiled
20
10
MMMUU
Mudivista
SEED Bench
MMMU
Mathvista
SEED-Bench
MMMU MATHVISTA
SEED
(a) Difficulty
(b) Discrimination
(c) Fisher information
Figure 16: Comparisons of parameters estimated by M2IRT between the original and artificial questions.
19

---

Published as a conference paper at ICLR 2026
15
Origital quality
40
Original
Ficher Infommaionn Detemizznn
Origiaal
Lew-quiitt
Low-quality
10
(discrimintion)
(diflicalt))
30
0.3
20
0.2
10
MMMU
Matavista
SEED-Beneh
MMMU
Mathvista
SEBD-Bennch
0.0
MMMU
MATHViSTA
SEED
(a) Difficulty
(b) Discrimination
(c) Fisher information
Figure 17: Comparisons of parameters estimated by M3IRT between the original and artificial questions.
the Mann- WhinnWy U test results comparing the original questions with the low-quality questions. We confirmed significant differences between the groups of original and low-quality questions.
We additionally examined the Wasserstein distance between original and low-quality questions, and found that the Wasserstein distance for MMMU, MATHVISTA, and SEED- BBNCH were 0.20, 0.14, and 0.051, respectively.
D.1 DETAILED RESULTS
Detailed results of the experiments depicted in Fig. are reported in Table Detailed results of the experiments depicted in Fig. 5 are reported in Table
Table 5: The average and standard deviation of Spearman's 's rank correlations between model rankings on the original benchmark and those estimated on extracted question subsets with 10%, 20%, 30%, 40%, and 50% of whole dataset.
Benchmark
Method
10%
20%
30%
40%
50%
M2IRT
0.96 0.061
0.96 = 0.057
0.96 0.047
0,97 = 0.042
0.96 + 0.035
M3IRT
0.89 1 0.046
0.92 1 0.030
0.92 1 0.033
0,93 0.023
0.93 1 0.023
IRT
0.50 0.21
0.36 1: 0.25
0.38 0.20
0.23
0.12 0.28
MMMU
MIRT
0.16 0.089
0.33 0.091
0.47 + 0.11
0.61
0.69 0.055
Tiny Benchmarks
0.43 0.13
0.50
0.61
0.082
FlashEval
0.79 0.029
0.79 0.025
0.79 0.912.0002
0.00
0.80 1801100111
Random
0.77 0.062
0.80 0.027
0.82 + 0.028
0.82 0.029
0.81 0.024
M2IRT M3IRT
0.81 0.036
0.84 0.032
0.842
0,91 0.034
0.93 0.017
0.92 0.028
0.94 + 0.018
0.93
093+0.018
0.93 + 0.010
IRT
0.81 0.047
0.88 + 0.040
0.023
0,93 0.017
0.94 4 0.014
Math Mast
Vista
MIRT
0.58 0.049
0.67 = 0.038
0.76 0.029
0.81 0.022
Tiny Benchmarks
0.79 0.038
0.84 0.020
0.018
0.88 0.017
0.88 0.011
FlashEval
0.020
0.91
0.91
0.91 0.014
0.91 0.008
Random
0.89 = 0.040
0.91 0.029
0.01
0.93 0.015
0.93 0.013
0.94 0.008
0.006
0.97 0.0080
0.97 1 0.005
0.95 0.005
M3IRT
0.94 0.019
0.011
0,95 0.018
0.95 0.017
IRT
0.86 = 0.13
0.83
0.15
0.81
0.79 = 0.17
SEED-Bench
MIRT
0.42 + 0.120
0.59 1:9-0.00 0.074
0.702
0.77
0.82 18013.003
Tiny Benchmarss
0.69 = 0.057
0.75 0.050
0.77 10.06
0.80 0.049
0.81 0004
FlashEval
0.89 0.021
0.90
0.90 0.010
0.90
Random
0.046
0.032
0.0277
0.91 1
D.2 VQAAT
AOA-ANSWETTHERAPY (VQAAT) (Chen et al. 2023) consists of VizWiz Dataset GGrari et al.
2018
which is visual questions asked by visually impaired people, and VQA v2.0 Goyal
201 7
We randomly sample 1000 questions from the train and validation sets of Single Answer
S Significance follows: "p < 0.05, "'p 0.01, "" ap---p < 0.001.
20

---

Published as a conference paper at ICLR 2026
Table 6: The average and standard deviation of the proportions of the low-quality questions in extracted question subsets with 10%, 20%, 30%, 40%, and 50% of whole dataset. "TB" means Binchhaass,,,s, and "FE" means FlashEval.
Benchmark
Method
10 %
20% %
30%
40%
50%
M2IRT
0.038 1 0.015
0.090
0.014
0.12 1 0.014
0.16 1: 0.014
0.19 0.013
M3IRT
0.091 + 0.017
0.11 0.013
0.14 0
0.16 0.011
0.19 = 0.0000
IRT
0.32 1 0.040
0.34
0.36 0.029
0.37 1: 0.023
0.38 1: 0.020
MIRT
0.016
1.012
0.37
0.36
0.38 0.023
0.09
0.38 +.88+0.00 0.022
0
0.36 0.020
FB
0.28 + 0.014
0.29:1
0.29 1 0.021
0.27 1: 0.017
0.29 0.010
Random
0.31 1 0.031
0.30
0.30 1 0.010
0.30 0.0081
0,30 1 0.0000
M2TRT
0.011 + 0.0060
0.014 0.037
0.068 0.021
0.11 0.021
0.15 0.016
M3IRT
0.054 1 0.013
0.014
0.15 1 0.019
0.17 1.07100011
0.20 1 0.007
IRT
0.27 = 0.055
0.037
0.24 = 0.028
0.020
0.22 0.018
Vist
MIRT
0.50 0.024
0.323
0.39 + 0.018
0.36 0.00+
0.33 + 0.0083
0.027
0.019
0.015
0.010
0.28 - 0.0065
0.27
0.088
0.30 0.016
0
0,30 0.014
Random
0.29 + 0.031
0.29 + 0.017
0.29 + 0.0086
0.29 + 0.0074
0.29 + 0.0070
M2IRT
0.045 0.016
0.011
0.13 0.0074
0.17 0.0062
0.22 0.0037
M3IRT
0.01 0.018
0.18
0.21 0.0091
0.24 1 0.0080
IRT
0.31 13140.005 0.048
0.33 0.041
0
0.35 + 0.038
0.37 0.031
SBED-Bench
MIRT
0.43
0.11
0.39 0.012
0.38 + 0.0083
0.37 = 0.0087
TB
0.36
0.003
0,34 0.012
0.34 0.0081
0,30 0.019
0.020
0.01
0.34 1 0.015
0.34 0.016
Random
0.34
0.22
0.04
0.34 0.0084
0.34 = 0.0084
1.0
0.3
0.9
Spoeman's rank con.
0.8
0.2
MZIRT MEIKI
0.7
MIIKI
IRT
IRT
0.6
MIRT
TinySenchmarks
Thay chnmarks
FlachEval
0.5
FlashEvvl
Random
0.4
Random
0.0
10
20
30
40
50
10
30
Subset size ((NI/IN) [96]
Subset sizze (//)/NN 196]
(b) The proportions of the low-quality questions 7
(a) Spearman's 's rank correlations VQAAT
on VQAAT
Figure 18: The average and standard deviation Spearman's rank correlations on extracted question subsets of VQAAT and the proportions of the low-quality questions in extracted question subsets with different sizes.
Grounding Challenge. This dataset presents images, questions, and multiple annotators' responses those questions to the VLM, asking whether the annotators' answers are based on the same part of the image. Therefore, this dataset consists solely of binary-choice questions.
We conducted an additional experiment on VQAAT under the same condition as Section 5.3 Figure shows the Spearman's 's rank correlations between the model rankings on the original benchmark and on an extracted subset and proportions of the low-quality questions. In contrast to experiment in Section 5.3 on VQAAT, M2IRT and M3IRT are worse than baselines terms of the Spearman's rank correlation. Interestingly, while proposed methods extracts fewer low-quality questions than the baselines. this filtering does not translate to improved accuracy in model ranking. We hypothesize that the discrepancy arises because VQAAT itself contains numerous low- low-quality questions, significantly influencing its "ground truth" ranking. As our method filters such low-quality questions, the resulting ranking deviates from the original benchmark.
21

---

Published as a conference paper at ICLR 2026
10c
Spenteat wwt...oo E13
0.
MEXT
MCST
07
64-
Masth Trae (Nippen
Mostth Troe (t5ppmn
0.4
Tee
Pinnbre
0.2
Subedt wic.eiieieeii. (NV))) (9)
Subuct (NV])) 1961
Sabaus die (NV()D (4)
(a) MMMU
(b) MATHVISTA
(c) SEED-BENCH BENCH
The average and standard deviation of Spearman's ' rank correlations between model rankings on the original benchmark and those estimated on extracted question subsets with different M2IRT and M3IRT are trained with sparse-response matrix.
Figure 19:
0,4
0A
MISKT
MEMI 100
MUSRT
Malii Feea
Miltf/itaa in goo)
SUSET Tica pan
Trafios leenis
0.3
Teafion leatii
Tlaktral
Ploafed
0.1
5.1
3.
3.0
3.0
Sabod size (
Salec six (M1
Subout size (0))))
(a) MMMU
(b) MATHVISTA
(c) SEED BENCH
Figure 20: The average and standard deviation of the proportions of the low-quality questions in extracted question subsets y with different sizes. M2IRT and M3IRT are trained with sparse-response matrix.
D.3 FOR SPARSE RESPONSE MATRIX
we explain that M2IRT and M3IRT don't require all models to respond to all questions. To demonstrate this, conducted additional experiment identical to Section 5.3, except that used only 10% of all (model, question) pairs on MMMU for training M2IRT and M3IRT. We then selected informative questions for evaluating a new model. These models trained with sparse dataset are compared with original and strong baselines.
In Section 4.4
Figure 19 shows the Spearman's rank correlations between the model rankings on the original benchmark and on different sizes of subsets. Figure 20 shows the proportion y with varying size of subsets.
Remarkably, as shown in Fig. 19a using just 3% of the total questions, M3IRT trained only 10% of all (model, question) pairs achieved a Spearman rank correlation exceeding 0.84 with the original full- rankings--equiivleen
dataset rankings equivalent to the baseline performance that requires 50% of the dataset. Beyond this point, M2IRT consistently maintained higher ranking consistency than all baselines. Moreover, the proportion of low-quality questions selected remained below 23%, showing that M2IRT is not only efficient but also discriminative identifying high-quality items.
This setting allows model comparison at only 13% of the inference cost required for full evaluation across all models and questions, demonstrating that M2IRT offers substantial cost savings while preserving evaluation reliability.
D.4 STATISTICAL SIGNIFICANCE TESTS
We conducted a one-sided Wilcoxon signed-rank test to evaluate the performance difference between M2IRT and MalshEvall.MIRR and TinyBenchmarks, M3IRT and FlashEval.and M3IRT and Tiny- Benchmarks. Table 7 Table & and Table the results for 5%, 10%, 30%, and 50% of Fig. 4 and Fig. with confidence level of 1%. MMMU shows significant differences from the baseline
22

---

Published as a conference paper at ICLR 2026
method. MATHVISTA shows significant differences in all conditions except for Spearman's 's rank conr at 5% against FlashEval's score. USED-BENCH shows significant differences from the baseline method.
Table 7: Wilcoxon signed-rank test on MMMU comparing FlashEval and Tiny Bench against M3IRT.
Comparison
5% subset
10% subset
30% subset
50% subset
p-value
W
p-value
W
p-value
W
p-value
W
FlshhEval (Rank corr.)
< 0.0001
0.0
< 0.0001
0.0
< 0,0001
0.0
0.0001
0.0
vs TinyBench (Rank corr)
< 0.0001
0.0
< 0.0001
0.0
< 0.0001
0.0
0.00001
0.0
vs FlashFval (Shuffle ratin)
< 0.0001
0.0
< 0.0001
<. 0.0001
0.0
0.0001
0.0
vs TinyBench (Shuffle ratio)
0.0001
0.0
< 0.0001
0.0001
0.0
0.0001
0.0
Table Wilcoxon signed-rank test on MATIVVISTA comparing FlashEval and TinyBench against M3IRT.
Comparison
5% subset
10% subset
30% subset
50% subset
p-value
W
p-value
p-value
p-value
FlashEval Rank Corr)
0.0197
222.0
0.0004
262.0
0.0019
251.5
0.0000
293.0
TinyBecch(Rank Com:)
0.0001
300.0
0.0001
300.0
0.0001
300.0
0.00001
300.0
FlashEvvl(Shhffle ratio)
0.0001
0.0
0.0001
0.0
0.0001
0.0
0.0001
0.0
TinyBench(Shutiee Shuftle ratio)
< 0.0001
0.0
< 0.0001
0.0
< 0.0001
0.0
< 0,0001
0.0
Table 9: Wilcoxon signed-rank test on SEEDBench comparing FlashEval and TinyBench against M3IRT.
Comparison
5% subset
10% subset
30% subset
50% subsel
p-value
W
p-value
W
p-value
W
p-valuc
W
vs FlashEval (Rank corr.)
0.0001
231.0
0.0001
231.0
0,0001
231.0
< 0.0001
231.0
vs TinyBench (Rank corr.)
0.0001
231.0
0.0001
231.0
< 0.0001
231.0
< 0.0001
231.0
vs FlashEval (Shuffic ratio)
0.0001
0.0
< 0.0001
0.0
< 0.0001
0.0
< 0.0001
0.0
vs TinyBench (Shuffle ratio)
0.0001
0.0
0.0001
0.0
< 0.0001
0.0
< 0.0001
0.0
E DETAILS OF EXPERIMENTAL SETTINGS
E.1 COMPUTATIONAL RESOURCES
The computational resources utilized in this study presented in Table 10 The experiments in
Section 5.3 require hours per dataset, and those in Section 5.4 necessitate hours per dataset.
E.2 DATASETS
A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert
AGI (MMMU) (Yue et
2024)
The license for this dataset "Apache License 2.0".
MATHVISTA eL al
2024) The license for this dataset is "Creative Commons Attribution Share
Alike 4.0 International"
VQA-ANSWERTHERNNN (VQAAT) (Chnn et al. 2023 The license for this dataset is "Creative
Commons Attribution 4.0 International License".
SEED-BENCH (Li et al. 2024a): The license for this dataset is "Creative Commons Attribution Non Commercial 4.0".
E.3 VLMS
We use 24 commonly used VLMs listed in Table [T] Table for our experiments. We access open-source models and subset of closed models through Openrouter.
23

---

Published as a conference paper at ICLR 2026
Table 10: Computer Specifications Used for Experiments
Component
Specification Ubuntu 20.04 LTS
Operating System
CPU
AMD EPYC Milan 7763 DP/UP (64C/128T, 2.45GHz) 2
Memory
2048GB
python version
3.12.9
torch version
2.6.0
Table 11: Overview of Al Models Used
Model Name
Type
License or Terms
GPI-4-turbo
Closed
OpenAl Terms of Use
GPT-4o (OpenAl 2024a)
Closed
OpenAl Terms of Use
GPT-4o-min (OpenA
2024b)
Closed
OpenAl Terms of Use
GPT-4.1 (OpenAl) 2025
Closed
OpenAl Terms of Use
GPT-4.I-minI OpenAA
2025
Closed
OpenAl Terms of Use
GPT-4.1-nane (OpenAl
2025
Closed
OpenAl Terms of Use
Gemini-1.5-flash Team
2024
Closed
Gemini API Additional Terms of Service
Gemini-1.5-ffash--5 [Team]
2024
Closed
Gemmni API Additional Terms Service
Gemini-1..-pro (eam 2024
Closed
Gemin API Additional Terms of Service
Gemini-2.0-flassh (Pichai 20244)
Closed
Gemmn API Additional Terms of Service
Claude-3 3-haiku (Anthropic 2024a)
Closed
Anthropic Consumer Terms of Service
(Anthropic 2024a
Closed
Anthropic Consumer Terms Service
Claude-3.5-sonnec (Anthropic
2024b)
Closed
Anthropic Consumer TTerm Service
Claude-3.7 3.7-onnet Anthropic
2025
Closed
Anthropic Consumer Terms of Service
Grok-2 (xAI (2044)
Closed
XAI Terms Service
Nova-pro (Inteligence
2024
Closed
AWS Terms of Service
Nova-lite (Inteligence) 2024
Closed
AWS Terms Service
Qwen-2.5-vl-7b (Bai et al. (0255)
Open
Apache 2.0
Qwen-2.5-vl-72h (Bai et 2025
Open
Apache 2.0
1lama-3.2-11b--1
(Meta
2024
Open
Llama 3.2 Community License
1lama-3.2-90b-instmu (Meta
2024
Open
Llama 3.2 Community License
Pixtral-12b (Agrawal et al. 20
Open
Apache 2.0
Pixtral-large (Agrawal et al.
2024
Open
Apache 2.0
Minimax-01 Team 2025
Open
MIT License
24


Reviews:
[{"review_id": "n9SgRRXCBz", "reviewer": "Reviewer_5cL5", "summary": "In this paper, the authors proposed M^3-IRT, a multi-modal and multidimensional item response theory framework to evaluate cross-modal reasoning ability of Multimodal Large Models (MLLMs). Multiple experiments were conducted with several VLMs to show that M^3-IRT could reduce evaluation cost while improving reliability.", "questions": "Please refer to the weakness section and provide more justifications accordingly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Both the motivation of decomposing model ability and item difficulty and the proposed M^3-IRT framework make sense and are technically sound to me. Even though it builds upon the classical IRT framework, the extension to MLLMs makes lots of sense and could provide more nuanced evaluation and improved reliability with marginal cost.\n2. Extensive experiments were conducted to justify the effectiveness and efficiency of the proposed framework.\n3. Writing is good and easy to follow.", "weaknesses": "1. Multiple hyper-parameters were introduced by the proposed model. It would be better to provide more discussion on: a. how accurate the estimation of these parameters based on the method in section 4.4? b. sensitivity of the hyper-parameters; c. how many data are needed? d. cost of the estimation.\n2. The proposed framework mainly filters the items to be tested rather than help curate new dataset. While evaluation might be costly, we only need to run it once for every new model. With the lowering  trends of inference cost of LLMs, I am not sure whether the gain on reducing some eval examples would outweigh the cost of train a separate M^3-IRT model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O1DhVGo957", "reviewer": "Reviewer_akTA", "summary": "To address unreliable rankings and high costs in MLLM evaluation caused by shortcut questions answerable from a single modality, this paper introduces the M³-IRT and M²-IRT frameworks. These extend classical Item Response Theory (IRT) by decomposing both model abilities and question difficulties into image-only, text-only, and cross-modal components. This allows for quantifying a model's cross-modal reasoning and an item's cross-modal demand. Experiments show the framework effectively prioritizes genuine cross-modal questions, faithfully reproducing full-benchmark rankings with as little as a 10% subset. It remains robust even when 50% of the benchmark is contaminated with low-quality items, significantly reducing evaluation costs while enhancing reliability.", "questions": "Have you tested any non-linear variants? If not, how sensitive are your conclusions to the linearity assumption?\n\nIn addition, beyond swapping-based corruption, does your filter still identify low-quality items under subtler artifacts?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper makes a contribution by addressing the critical challenge of \"shortcut questions\" in multimodal benchmarks. It provides a systematic solution that enhances the reliability of evaluations while simultaneously reducing computational costs.\n2. The paper effectively targets two major pain points in current multimodal evaluation: unreliable model rankings caused by shortcut question contamination, and the high computational cost associated with large-scale benchmarks. The proposed M³-IRT framework offers an efficient and effective solution to both issues.\n3. By decomposing model abilities into image-only, text-only, and cross-modal components, it offers valuable and interpretable insights into the specific strengths and weaknesses of different MLLMs, moving beyond a single, monolithic accuracy score.", "weaknesses": "1. The model's core assumption of linear decomposition for abilities and difficulties might oversimplify the complex, potentially non-linear interactions that occur during cross-modal reasoning. \n2. The interpretability of the estimated parameters such as cross-modal difficulty is derived purely from model performance patterns and lacks external validation against human cognitive judgments of what constitutes a cross-modal task.\n3. It is noted that the paper generates artificial low-quality questions through full swapping of images or text, which effectively simulates extreme scenarios involving complete modal mismatch. That said, it may be worth considering whether this approach fully covers the more prevalent and subtle shortcut features commonly observed in real-world multimodal benchmarks. In practical contexts, shortcuts often exhibit characteristics of concealment and diversity. Given that the framework’s ability to filter such subtle shortcuts has not yet been evaluated, it might be valuable to further explore its generalizability when applied to real-world benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lPjchDn8PS", "reviewer": "Reviewer_Wauf", "summary": "This paper discusses the assessment of Multimodal Large Language Models (MLLMs). The authors observe that some multimodal problems in conventional datasets or benchmarks can be solved using only one modality. This makes current evaluation efforts of Multimodal Large Language Models less efficient and less reliable. The authors then propose their method, M3-IRT, which estimates the cross‑modal ability of MLLMs and each question’s cross‑modal difficulty, enabling compact, high‑quality subsets that better reflect multimodal reasoning.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper studies a very important problem in Multimodal Large Language Models, and the authors' observations about the weaknesses of existing evaluation efforts are reasonable.\n2. This paper provides detailed descriptions of their evaluation framework. There are a lot of figures to visualize the results of evaluation.", "weaknesses": "1. The problem that this paper has pointed out (prior evaluation efforts use multimodal problems that can be solved with only one of the modalities) has been studied previously. This paper did not provide proper reference to these prior works. For example, MMEvalPro [1] is a recently proposed dataset with manually labeled questions to mitigate the problem.\n2. The results are mostly displayed in the figures, while I expect more accurate numbers to be displayed in tables. While figures are effective for visualizing high-level trends and making qualitative comparisons, they are insufficient for a rigorous and reproducible scientific paper. It is difficult, and in some cases impossible, for the reader to extract the precise performance metrics, which is essential for understanding the exact magnitude of the reported improvements and for future comparative analysis by other researchers.\n3. This paper should include a more comprehensive discussion of related works in MLLM evaluation, including [2-3].\n\n[1] MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation\n[2] Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness\n[3] Revisiting multi-modal llm evaluation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3cZErFbvC", "reviewer": "Reviewer_jMkP", "summary": "This paper addresses a key deficiency in current benchmarks for Multimodal Large Language Models (MLLMs): the prevalence of \"shortcut questions\" that can be answered using only a single modality, such as text or image alone. This flaw leads to unreliable model rankings and unnecessarily increases evaluation costs due to large benchmark sizes. To resolve this, the authors propose a novel framework called Multimodal and Multidimensional Item Response Theory (M³-IRT). This approach extends classical Item Response Theory (IRT) by decomposing both the model's ability and the question's difficulty into three latent components: image-only, text-only, and cross-modal integration.\n\nThis decomposition allows the framework to precisely identify high-quality questions that genuinely require cross-modal reasoning while filtering out low-quality shortcuts. The method was validated through experiments on 24 Vision-Language Models (VLMs) across three benchmarks: MMMU, MATHVISTA, and SEED-BENCH. The results demonstrate that M³-IRT is robust against contamination, maintaining ranking fidelity even when benchmarks contain up to 50% artificially generated low-quality questions. Furthermore, the framework enables the creation of compact, high-quality benchmark subsets that can accurately reproduce the full ranking using only a small fraction of the questions, thereby significantly reducing computational overhead while improving evaluation reliability.", "questions": "1. The $\\theta_{base}$ in Supp. Table 2: Mathvista shows that GPT-4o's base is 0.00 and Llama-3.2-11B is 0.10, which is weird. Can authors explain this phenomenon?\n\n2. Given that the evaluation scores are relative to the tested models, how do you address the potential instability of these scores as the model evolves?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper's primary strength lies in its novel framework, M³-IRT, which moves beyond simple accuracy scores to offer a more granular and interpretable evaluation.\n\n2. The framework enables the creation of compact, high-quality benchmark subsets that significantly reduce the computational cost of evaluation.\n\n3. The paper effectively demonstrates the method's resilience to the low-quality and \"shortcut\" questions that are common in existing benchmarks.", "weaknesses": "1.  The decomposition of a model's ability, theta ($\\theta$), is a heuristic and oversimplified mathematical model. It operates on the assumption that the distinct abilities (e.g., image, text, cross-modal) are independent and linearly additive. However, a model's actual reasoning ability is complex. For instance, a model might leverage textual concepts to guide its attention within an image, creating a synergistic effect where cross-modal ability mutually enhances the image ability. Such complex, non-linear interactions cannot be captured by the current linear additive framework.\n\n2.  Given that numerous parameters are optimized via SGD, the stability of $\\theta$ across benchmarks of varying scales has not been adequately validated. A critical question remains unanswered: if the number of questions were to substantially increase or decrease, would a model's ability score exhibit significant fluctuations? This lack of demonstrated robustness raises concerns about the reliability of the scores when the evaluation context changes.\n\n3.  The model's ability ($\\theta$), question difficulty ($b$), and discrimination ($a$) are all optimized simultaneously. Consequently, the difficulty of a question is inherently influenced by the abilities of the models in the test cohort. This renders all metrics as **relative scores** within the context of a specific group (24 VLMs in this paper), rather than absolute measures. If the composition or average capability of the VLM cohort were to change, the perceived difficulty of the questions would also shift, raising concerns about the potential for instability in the evaluation framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)