PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies
Authors: Lukas Selch
Yufang Hou
Muhammad Jehanzeb Mirza
Sivan Doveh
James R. Glass
Rogerio Feris
Wei Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Large Multimodal Models
Scentific document understanding
evaluation benchmark
Primary area: datasets and benchmarks
TL;DR: PRISMM-Bench is the first benchmark of real reviewer-flagged multimodal inconsistencies in scientific papers, revealing that even state-of-the-art LMMs struggle to detect and resolve them.

Abstract: Large Multimodal Models (LMMs) are increasingly applied to scientific research, yet it remains unclear whether they can reliably understand and reason over the multimodal complexity of papers. A central challenge lies in detecting and resolving inconsistencies across text, figures, tables, and equations, issues that are often subtle, domain-specific, and ultimately undermine clarity, reproducibility, and trust. Existing benchmarks overlook this issue, either isolating single modalities or relying on synthetic errors that fail to capture real-world complexity. We introduce PRISMM-Bench (Peer-Review-sourced Inconsistency Set for Multimodal Models), the first benchmark grounded in real reviewer-flagged inconsistencies in scientific papers. Through a multi-stage pipeline of review mining, LLM-assisted filtering and human verification, we curate 384 inconsistencies from 353 papers. Based on this set, we design three tasks, namely inconsistency identification, remedy and pair matching, which assess a model's capacity to detect, correct, and reason over inconsistencies across different modalities. Furthermore, to address the notorious problem of choice-only shortcuts in multiple-choice evaluation, where models exploit answer patterns without truly understanding the question, we further introduce structured JSON-based answer representations that minimize linguistic biases by reducing reliance on superficial stylistic cues. We benchmark 21 leading LMMs, including large open-weight models (GLM-4.5V 106B, InternVL3 78B) and proprietary models (Gemini 2.5 Pro, GPT-5 with high reasoning). Results reveal strikingly low performance (27.8-53.9%), underscoring the challenge of multimodal scientific reasoning and motivating progress towards trustworthy scientific assistants.

Published as a conference paper at ICLR 2026
PRISMM-BENCH: A BENCHMARK OF PEER-REVIEW GROUNDED MULTIMODAL INCONSISTENCIES
Lukas Selch'
Yufang Hou
M. Jehanzeb Mirza
Sivan Doveh
James Glass"
Rogerio Feris
Wei Lin'
JJoannees Kepler University Linz *Interdisciplinary Transformation University Austria
3 MIT CSAIL "Stanford University 5 MIT-IBM Watson AI Lab
ht https: / //da//daaa// -aggasssss....
, github we.iii/iiiie pww.snnnenncch/////
ABSTRACT
Large Multimodal Models (LMMs) are increasingly applied to scientific research, yet it remains unclear whether they can reliably understand and reason over the multimodal complexity of papers. A central challenge lies in detecting and resolv- ing inconsistencies across text, figures, tables, and equations, issues that are of- ten subtle, domain-specific, and ultimately undermine clarity, reproducibility. and trust. Existing benchmarks overlook this issue, either isolating single modalities relying on synthetic errors that fail to capture real-world complexity. We intro- duce PRISMM-
Inconsistency Set for Multimodal Models), the first benchmark grounded in real reviewer flagged inconsistencies in scientific papers. Through a multi-stage pipeline of review mining, L-Mssiiiteed filtering and human verification, we curate 384 inconsistencies from 353 papers. Based on this set, we design three tasks, namely inconsistency identification, rem- edy and pair matching, which assess a model's capacity to detect, correct, and rea- son over inconsistencies across different modalities. Furthermore, to address the notorious problem of choice-only shortcuts multiple-choiee evaluation, where models exploit answer patterns without truly understanding the question, we fur- ther introduce structured JSON-based answer representations that minimize lin- guistic biases by reducing reliance on superficial stylistic cues. We benchmark 21 leading LMMs, including large open-weight models (GLM-4.5V 106B, In- Ve3 78B) and proprictary models (Gemini 2.5 Pro, GPT-5 with high rea- soning). Results reveal strikingly low performance (27.8-53.9%) underscoring the challenge of multimodal scientific reasoning and motivating progress towards trustworthy scientific assistants.
(Peer-Reviewsoouree
reviewer-faggeed
Revieewer s186
Question:
In Pigure roward is deftned as. 1/ppi wherros 10 ppl in the reward
Leaning Agen
PLPYN
What is the incomiatency those of reseatiiic paper?
funetion...
Answer Options
Layer:
The Sllstrate the rowand funetion stiffrently than the definition in the teet.
Pio
ceffleitee is see in 1 haard on LTA benchmarks, the figune shows coofiictent
function's
[) The text statee the cofffielen of the woanrd the figian <hows finetion not instend
K Searched Sparsity
information about the reward functioon.
Sho poplaaiy colladd be WisTber wiliónn oeeee fai
appureas from the cuption ur toxt what they vecooeent.
We collect reviewer-flageed inconsistencies in scientific papers and transform them into QA tasks that probe detection, correction, and reasoning over multimodal inconsistencies.
Figure 1:

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in Large Multimodal Models (LMMs) have sparked growing interest in their po- tential to serve as intelligent assistants for scientific research (Eger et al., 2025), supporting tasks such as figure and chart interpretation (Roberts et al.] 2024 Tanasá & Opreal 2025 Wu et al., 2024),
paper summarization (Tan et al.
2025
Saxena et al. 2025
Yu et al. 2025), and error detection Yet, central open question remains:
(Yang et al.] [0256]] Miyai et al.
2024
Alsaif et al.] 2024).
can LMMs truly reason over the complex multimodal structure of scientific documents?
A central challenge in this setting is detecting and resolving inconsistencies between text, figures, tables, or equations in scientific papers. These issues are often subtle, arising from copy-paste mistakes, outdated results, inconsistent notation, and require domain knowledge to detect. Fig. illustrates such a case, where the reward function is defined differently in the figure and the in- line text. Our analysis of ICLR 2025 submissions reveals that 17.0% contained least one such inconsistency flagged by reviewers. These discrepancies undermine clarity, reproducibility, and ultimately scientific trust.
Existing benchmarks, however, fall short of exposing this. Document QA datasets (Mathew et al., 2021 2022, Zhu et al. 2022) or standalone scientific visual element tasks focusing on diagrams, charts, or tables (Kafle et al. 2018) Masry et al.) 2022] Cheng et al. (022) miss the multimodal dependencies of scholarly works. Synthetic datasets (Yan et al., 2025) generate artificial errors, but these are often obvious and unrepresentative of real-world complexity. However, constructing such a benchmark of authentic inconsistencies is challenging as these cases are rare, scattered, and labor-intensive to verify, often requiring domain expertise to identify and validate. To address this, we leverage a valuable but underutilized resource open peer reviews, focusing on instances where reviewers flag mismatches across different modalities, such as between text, figures, tables, and equations, thereby providing a natural source of real-world, human-identified inconsistencies.
In this paper, Multimodal Models. Unlike synthetic datasets, PRISMM-Bench captures inconsistencies explic- itly flagged by human reviewers in scientific papers on OpenReview. Through multi-stage pipeline combining large-scale review mining, LLM-assisted filtering. and rigorous human verification, we curate a dataset of 384 inconsistencies across 353 papers submitted ICLR 2024 & 2025, spanning 15 categories of visual-textual and inter-visual mismatches. PRISMM-Bench provides a principled resource for evaluating and improving LMMs, grounded in the real challenges of understanding and verifying scientific papers. Grounded on these recent reviews, we minimize data contamination risk and demonstrate a pipeline with the potential to construct a continuously updated live bench- mark. Building on this inconsistency set, we design a benchmark suite of three multiple-choice question (MCQ) tasks: 1) Inconsistency Identification detect what the inconsistency is; 2) Incon- sistency Remedy determine how to fix the inconsistency: and 3) Inconsistency Pair Match identify which two elements conflict. Together, these tasks form a tiered framework that evaluates not only model's ability to spot inconsistencies but also its capacity propose remedy, and reason over relationships between different modality components.
introduce PRISMM-Bench,
Peer- sorde-eeee-- Inconsistency Set for
In MCQ evaluation, a key challenge is models' tendency to rely on linguistic biases in answer choices. Prior work has shown that LLMs often exploit choice-only shortcuts, achieving non-trivial accuracy without reading the question (Chandak al., 2025; Turner & Kurzejal 2025 Balepur
et al. 2024 Chizhov et al. 2025). and similar effects appear in multimodal MCQs CChnndak et al.] 2025 L To address this, we propose novel structured JSON-based answer representation that de- emphasizes stylistic cues and minimizes spurious correlations. Inspired by Das et al. (2014) and
Banareseu et al. (2013), our design converts free-form natural language into uniform structured representations that reduce model sensitivity to surface-level patterns. Our user study confirms that this approach suppresses linguistic shortcuts and better aligns models with human reasoning.
We benchmark 21 state-of-the-art LMMs, spanning large ween-w models such GLM-4.5V 106B (Hong et al. 2025) and InternVL3 78B (Zhu et al. 2025), as well as leading proprietary models including Gemini 2.5 Pro (Comanici al. (025) and GPT-5 (OpenAl (025). Results show that while large ppenhteight models achieve around 40% accuracy, even the strongest proprictary models reach just 53. 53.9%, underscoring difficulty of the benchmark and limitations of current L.MSs.
2

---

Published as a conference paper at ICLR 2026
Our contributions are fourfold: (1) We propose reviewer-soureed dataset of real multimodal in- consistencies in scientific papers, spanning diverse categories and grounded in peer review. (2) We construct a benchmark suite of three tasks probing detection, correction, and relational reason- ing over these inconsistencies.
(3) We are the first to propose JSON-based debiasing method for MCQ, converting free-form responses into uniform structured representations. (4) We evaluate 21 state-of-the-art LMMs, exposing their current limitations in detecting, understanding and correcting inconsistencies in scientific papers.
2 RELATED WORK
Large Multimodal Models (LMMs). Large Multimodal Models (LMMs) pair vision encoder with a large language model, enabling open-ended reasoning across tasks such as image caption- ing, VQA, document understanding. and chart interpretation. Early approaches like BLIP-2 (Li
jet al. 2023) and InstructBLIP (Dai et al. 2023) introduced instruction tuning on pre-trained vision-
language models, while the LLa VA series (Liu et al. 2023b al Li et al. (224a) advanced perception
and reasoning via large-scale visual instruction tuning. Several recent studies (Doveh et al. 2024
Gavrikov al. 22244] Lin al.] 2024 Huang et al. (024 Mirza et al.
2025 Hansen et al. 2025
Mei et al. (225) have advanced these models by introducing improved training and adaptation
strategies. Recent models extend these capabilities: Qwen-2.5 VL (Bai al. (025) offers precise
object localization, dynamic resolution, and agentic tool execution; Intern VL3 (Zhu et al. 2025)
improves perception and reasoning through domain-specific pretraining on 3D scenes, GUIs, and
video; Gemma (Team et al. 2025). Ovis 2 (Lu et al. (025). and GLM 4.5V (Hong et al., 2025) demonstrate strong performance across diverse multimodal benchmarks. High-resolution variants such as InternI .M XComposer 2.5 (Zhang et al. (024a) and VILA HD 4K (Shi et al.) 2025), enable detailed perception and document processing. Proprictary models like GPT-5 (OpenAl 2025) and Gemini 2.5 Pro (Comanici et al. 2025) set the state-of-the-an complex multimodal tasks through large-scale training and enhanced reasoning.
These LMMs form the foundation for evaluating multimodal reasoning over scientific documents. In FRUSMM-Bennche we benchmark 21 top- pep-performin models to detect, understand, and correct real-world inconsistencies in peer-reviewed papers, exposing both their strengths and limitations.
Multimodal Benchmark on Scientific Paper Understanding. Prior benchmarks often focus on isolated scientific elements such as diagrams (Katle et al. 2018: Chaudhry et al., [220 Kahou
al. 2018), charts (Masry et al., 2022; Methani et al.] [2020) or tables (Cheng et al.) 2022, Nan
et al. 2022). Recent datasets like Math Vista (Lu et al. 2024). Math (Zhang et al. 2024b),
and ArXivQA (Li et al.) 2024b) integrate multiple modalities, but still treat figures and equations in isolation rather than in full-paper context. Whole-paper QA resources such as PubMedQA (Tin) ctt al. 2019), BioASQ (Krithara et al., (023) and QASPER (Dasigi et al. 2021 provide human- written questions, yet these are mostly abstract-based and limited to yes/no or short-span answers.
Closer to our setting, QASA (Lee et al. 2023) provides 1.8K exiert- questions on ML papers, but remains text-only and does not require reasoning over figures or tables. SPIQA (Pramanick ct al. 2024 introduces multimodal scientific QA, yet the questions are LLM-generated or human- annotated with emphasis on information seeking, not grounded expert reviews that often aim to critically evaluate scientific findings. SciDQA (Singh et al.) 2024) is sourced from reviewer-author QA pairs, but it remains text-only LLM benchmark without involving visual elements. In contrast, PRISMM-Bench is the first benchmark grounded in flaiewer-flagged multimodal inconsistencies in scientific papers. Unlike prior work that isolates figures, tables, or text, our benchmark integrates visual and textual reasoning within the natural context of full research papers, while grounding tasks in authentic peer review feedback rather than synthetic or abstract-level annotations.
Understanding of Inconsistencies. Research on inconsistencies in language models spans predic- tion variance across paraphrased queries (Ravichander et al. 2020 Elazar et al.) 2021 to factual inconsistency in summarization and long-form QA. To address the latter, prior work has introduced Qa-based benchmarks (e.g., WikiContradict (Hou et al., 22024),, evaluation metrics (e.g., QAFactE- val (Fabbri et al., 2022)). ) and detection methods based on QA (Wang et al., 2020), natural language inference (Lattimer al. 2023), probabilistic reasoning (Marinescu al. 2025
3

---

Published as a conference paper at ICLR 2026
Closest to setting, MMIR Yan et al. 2025 evaluates multimodal models artificially injected inconsistencies in materials such as slides and posters. In contrast, PRISMM- Benchh introduces real- world prriewer-flagge inconsistencies in scientific papers. Rather than synthetic perturbations, benchmark captures authentic challenges faced during scholarly review, spanning textual, visual, and cross-modal errors, and extends evaluation beyond detection to proposing remedies.
Language Biases in Evaluation Benchmarks. Multiple-choice evaluation is prone to linguistic biases, where models exploit surface-level patterns in answer options rather than reasoning over content. Prior studies show L.LMs can achieve high accuracy even without the question, such as in TruthfulQA (Turner & Kurzeja 2025), HellaSwag (Zellers et al., 2019), and ARC (Balepur et al.) 2024). For example, Chandak et al. (2025) report 83% accuracy on TruthfulQA v2 using answer choices alone, with shortcut rates above 70% on HellaSwag, The recent trend of generating distrac- tors with LLMs (e.g.. in MMLU-Pro; Wang et al. (2024a)) can even exacerbate these artifacts.
To mitigate such biases, structured representations offer promising direction. Analogous to thorship obfuscation in stylometry (Chinchor] 1998). structured formats remove stylistic and surface cues while retaining semanties. Drawing inspiration from FrameNet- -ased semantic parsing (Das) ett al. 2014) and MUC slot filling (Uchendu et al.) 2023), PRISMM- -Bnhhhh introduces JSON-based answer representations that encode key elements for capturing inconsistencies in scientifie papers. This design reduces artifacts in answer choices and compels models to engage with multimodal content rather than exploiting linguistic shortcuts.
PRISMM-BENCH
FEISMM-Bende is built through a six-stage pipeline (Fig. 2 (1) review sourcing, (2) LLM-based
review filtering, (3) manual annotation of reviewer-flagged inconsistencies (Sec. 3.I). (4) LMM-
based task generation, (5) manual verification to finalize benchmark tasks (Sec. (.2). and (6) LLM-
based debiasing to reduce language biases (Sec. 3.3 The evaluation step introduced in Sec. 3.4
3.1 COLLECTION OF REVIEWER- FLAGGED INCONSISTENCIES
To build benchmark of realistic and authentic inconsistencies, we sourced cases flagged by review- ers on OpenReview (ope b). where comments often highlight discrepancies between textual content and visual or mathematical components, including figures, tables, and equations.
Review Soureing Strategy. We collected reviews from ICLR 2024 2025 submissions via the OpenReview API v2 (ope c). To maximize the likelihood that flagged inconsistencies persisted in the final public PDFs, we restricted to rejected or withdrawn submissions without rebuttals! [ This yielded 18,009 reviews (details in App. D.I.
LLM Review Filtering. As manual screening for all reviews infeasible, we employed LLM for review filtering. Specifically, we used Mistral Nemo 2407 (Mistral 2024) with low temperature settings summarize reviews and identify potential inconsistency mentions, resulting curated set of 6,056 potential inconsistencies spanning 2,458 reviews (prompt details in App. F.I..
Manual Verification. We performed a manual annotation pass using custom web-based annotation tool. The tool presented the annotator with one reviewer-flagged inconsistency at a time, alongside the corresponding paper in embedded PDF viewer. Annotators (I) verified whether reviewer comment described factual and identifiable inconsistency, and (2) annotated the relevant textual and/or visual parts of the paper. For visual elements, the annotator selected and cropped regions from the PDF. For textual elements, they specified the page, line, and content. addition, each inconsistency assigned category and brief description in the annotator's own words. The tool logged annotations together with the original reviewer's comment and automatically collected metadata such as the crop bounding boxes in structured format. Full details of the annotator background, annotation tool, captured metadata, annotation criteria and schema are provided in App.
Our carlier exploration of review sourcing revealed that most reviewer-flaggee flagged inconsistencies were re- solved during rebuttal and did persist in the final versions, motivating the current refined sourcing strategy.
4

---

Published as a conference paper at ICLR 2026
Step Review Sourcing OpenReview
Step Review Filtering
Step 3 Manual Amnotation via Anmotation
www
Step 4 LMM Task Generation
Benchmark Evaltuation
Step B
Step 6
LLM Debiasing using JSON Format
Manual Task Sample Verifcationn
Filtered Reviews
Manual Annotation
Question
Natural Languge
(Iucomiitcncyy Mdentiicttion Taskk
Reviewer YaKmm
Visual elemsont:
incoodices ditasst rigure 2. the botson left swo polots thas represented edge tognlogy, the
What Lhe inmmideruy in lles purs of scimilic papee?
Calegoeyy Egune-tenc
3 Answer Options Natural Language:
A] The visualization in Figure shows missing edgms betwers nodes, the mutches the actual road network structure.
roed netwock
Incoonlatedd
9 Location
conccudicts text's piefectly
Openeeciew Papse
(Pagu Figure 2)
Bj The blue noder Pigune unevenly rood
making
Lion a02)
C) The visulitation in Figure slows catroncous edgu betwer nodie, conrraliet the the matches the actual read mevork stracture.
tepaiagr
maulbariia
piefectly
Textual element:
results demonstaate generated algorithm perfectly
udjeesncy
D) Pigure 2 depicts only a discomeeted portion necwork, implying algeeithmn complete
muiches actual
Question with Answers JSON Format (Inconsistency IdenciBcation Task)
Queetion What the inconsistemyy in these ports of wienttffe papee?
Options in I8ON Format
A)
B)
CJ
D)
":tter":
"Atterr: "attrihutat: "nodow"
"etter": "C*,
"lester" "D'
tattribura': Fedtger",
"attributa": "ndgee",
"scteliete": "notwork".
*slaimm:
"elaim":
"clalm": {
"chim":
"anurre "taett.
"wuumee "repectation.
"amurnan "stolement"" "perSeetly mutches"
"Ammrra "espuctation",
"tolement" "pesteetly matches"
"stotetncul" Fevenlyy distributed"
"statemenu"" "complete stzucture*
evidenee";
evidemee" "
ovidemee" "
eeideneee
fanurcef: 2 Pigure 2",
faouroe"; Figuue 2",
"amurna:: Pieeae 2",
"aomrre " Tigure 2",
statement "miieing udges
Istatement". unmmmll distributed"
"statement" Textrameou udges*
"statement" Milcooneettd portion"
Interpentation Ald Band nnewee A
"vdgew
"olgrs perfectly matei"
Pigure
"ycleing riles"
Figure 2: Pipeline of PRISMM-Bench. The top row illustrates the six main steps: (1) review sourc- ing, (2) LLM-based review filtering, and (3) manual annotation of metadata for syviewer-flagged inconsistencies (Sec. 3.1. (4) LMM-based task generation, (5) manual verification to construct benchmark tasks (Sec. 3.2 and (6) LLM-based debiasing to mitigate language biases (Sec. (.3).
The bottom row shows representative outputs each stage: filtered reviews after step inconsis- tency annotation after step 3, an example multiple-choice question in natural language after step 4, and its debiased JSON-format counterpart after step 6.
This process produced 384 validated inconsistencies across 353 ICLR submissions. We identified 15 categories of inconsistencies based on the elements involved (distribution shown in Fig. 9h. The most common cases were figurfigure inconsistencies (23.7%) and figure-text mismatches (21.9%).
3.2 GENERATION OF BENCHMARK TASKS
From the verified inconsistencies, we constructed three multiple-choice tasks with four options (one correct, three distractors), following the evaluation choice of most recent frontier model re- leases (Yang et al. 2025a [ Liu et al. 2025 Comanici al. 2025 Team et al. 2024) and benchmark-
ing efforts (Wang et al.) 2024b) Zhhan et al. 2025 Shabtay et al. 2025). We design the following three multiple-choice tasks.
Inconsistency Identification (Ident). The first task evaluates a model's ability to recognize incon- sistencies within the given paper context, framed by the question: "What is the inconsistency in
5

---

Published as a conference paper at ICLR 2026
these parts of scientific paper?" We adopt this generic question style because our preliminary study showed that sample-specific questions (e.g. "What inconsistency observed between Fig- ure 2 and the accompanying text regarding the generated road network?") reveal the inconsistency content and oversimplify the task (see App. D.4.2 for details).
Candidate answers were generated using Gemini 2.5 Flash based inconsistency descriptions and corresponding multimodal context. The answers were manually refined to ensure (1) the correct choice captured the inconsistency precisely and (2) the distractors are contextually relevant and plausible, but incorrect. We show an example of the Ident task in Fig. 2
Inconsistency Remedy (Remedy). This task extends beyond simple detection by requiring models to how to fix the inconsistency by asking the question "What action needs be taken resolve the inconsistency in these parts of scientific paper?" Gemini 2.5 Flash was employed to reformulate the inconsistency statements from Ident into specific, actionable remedy formulations. This task evaluates whether models can propose plausible solutions, rather than merely spotting inconsisten- cies.
Inconsistency Pair Match (Match). This task is built on a subset of inconsistencies that involve two distinct visual elements within a paper (192 samples). Given one element context, the model must select its inconsistency counterpart from four options. By restricting the task to visual-visur mismatches, we specifically assess a model's ability to detect representation errors without relying on textual cues, simulating the common peer-review challenge of cross-checking figures and tables for consistency.
More details about the task generation process and a validation of the multiple-choice format against open-ended evaluation are provided App. D.4 A.4 and App. C.3 respectively. We provide qualitative
examples of the three tasks in App. A and a dataset viewing tool in the supplementary materials.
3.3 ALLEVIATION OF LINGUISTIC BIASES
During pilot experiments, we observed that models achieved well above random accuracy even when the visual context was withheld. For example, Gemini 2.5 Flash reached 57.6% accuracy on the Ident task without context (vs. 25% random chance). This indicated that models exploit linguistic priors and surface patterns in the answer options rather than reasoning over the actual content. Further analysis showed that factors such as answer length, relative position, and phrasing contributed to this bias, echoing known challenges in multiple-choice design (Gierl al. (017).
To combat this bias, we first tried refining the distractors with text manipulation, which proved insufficient. Therefore, cues. We designed the Eividence-Clain JSON format for the Ident task and the Targer-Action JSON format for the Remedy task. Converting answers into these structured formats using LLM reduced Gemini 2.5 Flash's 's no-context accuracy on the Ident task to 34.0%. We manually verified a 20% subset of the inconsistencies example of the Evidence-Claite JSON format for the Ident task is provided in Fig. 2 Full details on our debiasing procedure and the structured formats are provided in App. D.4.2]
introduced structured representations that minimize natural language
ensure the semantic fidelity of the JSON-formatted answers. An
This design choice is further supported by our user study (Sec. 4.4), which reveals that humans rely minimally on linguistic priors. contrast, models evaluated on natural language options maintain high accuracy without context, exposing fundamental mismatch in evaluation fidelity. By adopting structured ISON representations, we align model evaluation conditions more closely with human cognitive constraints, suppressing surface-level shorteuts and enabling fairer assessment of true multimodal reasoning.
3.4 CONTEXTUAL GRANULARITY IN EVALUATION
We evaluate model performance under three levels of contextual granularity, reflecting different real-world reading conditions and reasoning demands.
Focused Context (Focused). The model is presented only with the minimal necessary components extracted visual element (e.g., cropped figure table) and/or the precise text passage (e.g..

---

Published as a conference paper at ICLR 2026
sentence or paragraph) involved in the inconsistency, annotated. This setting isolates the key content, testing the model's ability to detect inconsistencies with minimal noise.
Page Context (Page). The model receives 144 DPI rasterized image of the entire page(s) where the inconsistency occurs. Visual elements are not pre-cropped, requiring the model to locate and interpret relevant content within the full page layout. This simulates realistic reading conditions where inconsistencies must be identified without prior localization.
Document Context (Document). The model receives the entire scientific paper as a sequence of page images. To accommodate architectural constraints, we follow MMLongBench-Doo (Ma et al. 2024) and segment the document into collages: total of images are fed to the model, each containing npages/55 pages arranged in 3-column grid. This setting evaluates the model's capac- ity for long-range, cross-page page reasoning and document-level grounding. For models with high- resolution processing constraints, such L.LVV Onevision (7B, 728), we reduce input to images with mpages/ /3 pages each to avoid exceeding the context window.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Model Selection. We evaluate 21 LMMs spanning a diverse range of architectures: 16 open- open-weight models of varying scales, two specialized high- righ-resouution models, and three proprietary models. Selection was guided by performance on the Open VLM Leaderboard (ope a) and the availability of the latest model versions.
Inference Details. To ensure consistent scoring, we enforced a strict answer format. Prompts at both the system and user level instructed models to output only the letter corresponding to their chosen option. For reasoning-enabled models, answers were extracted separately from reasoning traces, enclosed in <think></thh ink> tags, before postprocessing against the ground truth.
All ween-weight models were grouped into three parameter count categories and evaluated with vLLM v0.10.1 (vll). Experiments were conducted on 4XNVIDIA A100 64GB GPUs with greedy decoding, except for InternVL3.5 (8B, 38B) which required temperature of 0.6 for stable reason- ing. Proprietary models were accessed via their official APIS with greedy decoding except GPT-5 (minimal, high) which has fixed temperature of 1.0.
Each model was evaluated on all three tasks (Ident, Remedy, Match), and across three contextual granularity levels (Sec. 3.4). For Match, only Focused Context was used. This design yields seven evaluation configurations per model, enabling fine-grained analysis of how model architecture, scale and input context affect inconsistency detection performance.
4.2
MAIN RESULTS
Table summarizes the aggregate performance of all evaluated models. Our results reveal clear trends in how LMMs handle inconsistency detection and correction.
Performance Landscape. Proprietary models substantially outperform their open-weight counter- parts. GPT-5 (high reasoning) reached the highest average performance of 53.9%. By contrast, the best open-weight model GLM 4.5V 106B achieves 42.5%. matching GPT-5 (minimal reasoning) but trailing its high-reasoning variant by 11. points. These results underscore the difficulty of the benchmark: even the pestoreefo models remain far from the reliability required of automated scientific assistants.
Impact of Context and Task Formulation. Performance drops consistently context scope pands. Models achieve their best accuracy in the Focused setting but often degrade toward random chance under Page and Document inputs, reflecting persistent challenges with distraction and long- range grounding in dense, multi-page inputs. To rule out input quality effects, we performed an ablation study on rasterization resolution, confirming our 144 DPI baseline (cf. App. C.I.
Task formulation also plays a critical role. Remedy scores are consistently lower than Ident, show- ing that proposing corrections requires deeper reasoning than only detection. Performance on Match
7

---

Published as a conference paper at ICLR 2026
Accuracy (%) of 21 diverse LMMs across three tasks (Ident. Remedy, Match) and three levels of contextual granularity (Sec. B.4p. For Match, results are reported only under the Focused setting. Best result per task bolded, second best underlined. denotes reasoning models.
Table 1:
Focused
Page
Document
Model
Params [B]
Ident (384)
Remedy (384)
Match (192)
Ident (384)
Remedy (384)
Ident (384)
Remedy (384)
Average (960)
Small Open- Weight Models (<9B)
Gemma 3B
4,0
27.9
29.9
39.6
25.0
24.7
26,6
27.1
27.8
LLaVA OV 7B
7.0
30,5
28.4
29.7
32.0
28.4
28.1
27.9
29.2
Ovis2 8B
8.0
35.4
29.4
22.4
34.4
27.3
31.8
28.1
30.4
Qwen 2.5 VL 7B
7.0
32.8
31.3
58.9
29.9
29.7
26.8
27.1
31.9
Interv 8B
8.0
36.5
29.4
56.3
28.6
27.6
30.7
31.8
32.7
Intern VL3.5 8B*
8.0
49.5
35.9
45.8
38.3
30.5
36.7
31.0
37.7
Medium Open- Weight Models (9B 38B)
Gemma 3 27B
27.0
36.2
32.8
$9.9
30.7
28.6
31.0
27.3
33.3
Gemma 3 12B
12.0
33.9
30.5
63.5
30.7
25.8
30.7
30.5
32,9
Qwen 2.5 VL 32B
32.0
42.4
37.0
45.8
37.2
34.6
38.3
27.9
37.0
Ovis2 34B
34.0
50.0
41.1
37.0
40.6
36.2
33.3
31.8
38.7
InterVL3 38B
38.0
46.6
38.5
56.8
40.6
35.7
37.0
32.0
39.8
Inte5VLL355 38B"
38.0
54.4
43.5
50.9
40.9
31.3
33,9
31.5
39.8
Large Open
Weight Models (2-38B)
LLaVA OV 72B
72.0
35.4
30.5
28.1
32.3
28.4
31.5
26.0
30.5
Qwen 2.5 VL. 72B
72.0
49.7
37.2
32.8
44.0
33.3
35.4
25.3
37.1
IntervVL3 78B
78.0
49.5
39.3
45.3
39.3
33.9
35,9
30.5
38.6
GLM 4.5V 1
106.0
51.8
43.2
52.1
45.8
35.9
40.9
33.1
42.6
Specialized High-Resotunie Models
InternLM XC 2.5 7B
7.0
28.4
25.3
27.6
29.9
27.1
29.9
28.6
28.2
VILA HD 4K 8B
8.0
31.0
30.7
25.5
30.2
29.2
28.6
28.4
29.4
Proprictary Models
GPT-5 (minimal)*
53.6
43.5
63.0
47.1
36.5
40.9
32.8
44.0
Gemini 2.5 Pro GPT-5 (high)*
65.9
61.2
66.7
54.7
51.8
39.8
36.7
52.8
54.4
70.3
58.1
46.9
41.1
53.9
varies widely across models: Gemma 3 12B achieves 63.5%, rivaling proprietary models, whereas much larger models such Intern VL3 78B lag behind at 45.3%. These results suggest that archi- tectural design, not just scale, critical for relational reasoning.
Model Characteristics. Reasoning-enabled models show benefit. For example, Intern VL3.5 8B outperforms its non-reasoning predecessor InternVI.3 8B by over 5 percentage points, achieving accuracy comparable to models with nearly nine times more parameters. Turning off chain-of- thought reduces accuracy by up 14 points (cf. App. C.2 directly confirming the contribution of reasoning. In contrast, high-resolution specialists (VILA HD 4K 8B, InternLM XC 2.5 7B) show little advantage in extended-context settings. More broadly, our results challenge the "bigger is better" paradigm: scaling up parameter counts alone does not guarantee higher performance, with diminishing returns observed from medium- large-scale models.
Overall, these findings highlight the current limitations of LMMs for scientific document analysis. Future progress will require advances in reasoning architectures to move beyond error detection toward correction, as well more robust mechanisms for grounding over long, distractive contexts.
4.3 ANALYSIS OF CHAIN-OF-THOUGHT REASONING
Reasoning variants perform better than non non-reasoning counterparts and reach results comparable to much larger models. For instance, VL3.5 8B achieves average of 37.7%, rivaling large open-weight models and surpassing several 72B non-reasoning models. We therefore study how reasoning-enabled models leverage chain-of-thought (CoT) improve performance PRISMM-
8

---

Published as a conference paper at ICLR 2026
Bench. To do re-evaluated selection of reasoning models the Ident task with Focused Context with reasoning turned off and compared the performance.
Ablation Results. Disabling reasoning reveals the critical role of CoT in detecting subtle incon- sistencies. For example, GLM 4.5V drops from 51.8% to 43.2% (-16.6%), Intern Intern SB from 49.5% to 40.6% (-18.0%), and InternVL3.5 38B suffers the largest decline, from 54.4% to 40,4% (-25.7%).
Why Reasoning Helps. To understand these performance differences, we focused on inconsis- tencies where Intern 38B succeeded with reasoning but failed without. We identified three consistent patterns: (1) Structured input handling: Reasoning-enabled models interpret the ISON- formatted options in natural language, clarifying subtle distinctions without exploring linguistic biases (cf. low without-context performance for reasoning models Sec. 4.4). (2) Cross-modal grounding: CoT traces show models explicitly reasoning plex information into smaller units and reusing them later in the reasoning chain. (3) Concept
both text and visuals, breaking com-
linking: Reasoning enables models connect fine-grained context with domain knowledge and ab- stract concepts, allowing stronger logical inference beyond surface pattern recognition. We provide detailed case study illustrating these effects in App.
4.4 USER STUDY AND LINGUISTIC BIAS ANALYSIS
To complement our benchmark, we conducted a user study to establish a human baseline and quan- tify visual reliance - the extent to which answers depend genuine multimodal reasoning rather than linguistic shortcuts. While our benchmark uses structured JSON outputs for models, our par- ticipants evaluated on natural language (NL) questions structured formats less practical without prior training. To enable direct comparison, we re-evaluated representative LMMs on the same Ident task subset using natural language answer options.
Setup. Eight non-author participants with at least PhD-level computer science research experience each answered ten randomly sampled Ident task questions: five in Focused context and five in Doc- ument context. We provide detailed analysis of the representativeness of this subset in App. E.2 For each question, participants first answered without context (question + answer options only), then with context. Focused context consisted of cropped images and/or text excerpts; Document context contains links to original PDFs. The survey was implemented via custom web interface (cf. App. E.T.
Table 2: User Study Results. For each context scope, we report Accuracy (%) for both Natural Language (NL) and JSON answer options. Human performance with NL is shown for reference.
Without Context
Focused Context
Whole Document Context
Model
NL
JSON
NL
JSON
NL
JSON
Human
27.5
77.5
65.0
Intern VL335 8B
49.3
28.4
76.3
47.4
56.8
35.1
Intern VL3.5 3
53.7
25.3
76.3
71.1
70.3
40.5
Qwen 2.5 72B
47.8
38.8
65.8
65.8
43.2
48.6
Gemini 2.5 Pro
70.1
37.3
81.6
65.8
83.8
37.8
Analysis. As shown in Table 2 top models like Gemini 2.5 Pro exceed humans under Focused Context (81.6% vs. 77.5%) and Whole Document Context (83.8% vs. 65.0%). However, a crucial difference emerges in the Without Context condition: LMMs maintain high accuracy (up to 70.1 %)), whereas human performance drops near chance (27.5%). This indicates that LMMs rely heavily on linguistic regularities that humans cannot exploit. Switching to JSON formatting neutralizes this ad- vantage. Without context, model performance collapses toward human levels (c.g., Intern VL3.5 38B drops from 53.7% to 25.3%). With context and -truccurd answer, LMMs no longer match human NL performance, confirming that linguistic shorteuts inflate perceived model capability.

---

Published as a conference paper at ICLR 2026
Table 3: Impact of answer representation without-context performance and visual reliance. Ac-
curacy is reported for the Ident task. R is computed according to Eg. [ the with-context baseline.
using Focused context as
Natural Language
JSON
Model
Accuracy
R
Accuracy
R
Intern VL.35 8B
45.6
17.1
28.6
29.3
InternVL3.5 38B
52.9
22.5
26.3
38.1
Qwen 2.5 VL 72B
49.7
16.1
36.5
20.8
Gemini 2.5 Pro
61.2
43.8
37.8
45.2
To quantify how much models and human rely on visual evidence versus linguistic priors, we com- pute the Visual Reliance Ratio R, adapted from the normalized Perceptual Score (Gat 2021
R ACCwith.coniext
ACCwilbout.comeex
1
ACCwitoon_contet conteet
Higher It indicates stronger dependence on visual context. Human participants achieve R = 69.0%, while the top model (Intern VL3.5 8B) achieves It 53.5%, confirming that humans rely more on visual grounding than current LMMs.
Probing Linguistic Bias. To confirm this effect generalizes beyond the user study subset, we re- evaluated the same four representative LMMs on the full Ident dataset under both Natural Language and JSON formats (Table 3). The same pattern holds: under natural language, models achieve in- flated accuracies without context (e.g. 61.2% for Gemini 2.5 Pro) but performance drops toward chance under JSON. Correspondingly, R. increases under JSON for all models, showing that struc- tured outputs suppress linguistic shortcuts and force models to rely more on visual evidence.
Insights. Two key conclusions emerge: (1) MCQs with long-form answers in natural language overstate LMM performance, as models can exploit linguistic regularities imperceptible or irrelevant to humans. (2) Structured ISON representations mitigate this bias, revealing that even the strongest LMMs still fall short of human-leve visual grounding and rely on surface cues when available.
CONCLUSIONS
We introduce PRISMM-Bent multimodal benchmark for evaluating LMMs on real-world scien- tific inconsistencies. We show that even top-perforning models struggle with cross-modal reason- ing and long-context grounding, while structured answer formats mitigate linguistic shortcuts. This work highlights limitations of LMMS scientific assistants and motivates future improvements in filtering pipelines, cross-domain datasets, and debiasing strategies for long-form MCQs evaluation.
Limitations. Our benchmark is limited in scope: it currently focuses on Al-domain papers from ICLR 2024 & 2025 and emphasizes rejected submissions capture unresolved errors. As result, both the scale and domain coverage are restricted. Future work should expand to other fields and venues, and explore inconsistencies that may persist in accepted papers, offering broader and more representative testbed.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work introduces PRISMM-BeIne a benchmark for evaluating multimodal large language mod- els (MLLMs) scientific document understanding. In developing the benchmark, we exclusively use publicly available research papers from ICLR 2024 & 2025, which are distributed under the Creative Commons Attribution 4.0 (CC-BY 4.0) license. This license explicitly permits redistribu- tion, remixing, and adaptation of the material with proper attribution, and we ensure that all source materials are used in full compliance with these terms.
Our study also includes a small-scale user study to establish a human baseline. All participants were experienced researchers, voluntarily consented to take part, and no personally identifying informa- tion was collected or reported. The study design posed no foresecable risks to participants and did involve vulnerable populations.
We recognize that benchmarks can influence the direction of future model development. While PRISMM- may highlight weaknesses in existing systems, is not intended facilitate mis- use, such as adversarial attacks on models, but rather to promote more robust and trustworthy scien- tific document analysis. We release the benchmark with the goal of supporting transparent, repro- ducible, and ethical research, in line with the ICLR Code of Ethics.
No conflicts of interest, sensitive data, or privacy concerns arise this work.
REFERENCES
Open vim leaderboard.
vlm leaderboard a.
Openreview.
b.
Openreview api v2. openapi- - definitionn C.
vllm. brtps://docs: wtt.://............/.....0......
Khalid M Alsaif, Aiiad A Albeshri, Maher Khemakhem, and Fathy E Eassa. Multimodal large language model-based fault detection and diagnosis in context of industry 4.0. Electronics, 13 (24):4912, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. 2025. URL htttp:://aaat
org/ abs 2502 13923
Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts abduction: How do Ilms answer multiple-choice questions without the question?, 2024. URL https:://aaaatt
org/abs/2400 /20aa/2242..2 12483
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. Abstract Meaning Represen- tation for sembanking. In Antonio Pareja-Lora, Maria Liakata, and Stefanie Dipper (eds.), Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pp. 178 186, Sofia, Bulgaria. August 2013. Association for Computational Linguistics. URL
Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. Answer matching outperforms multiple choice for language model evaluation, 2025. URL https:
/ //arxiv.orr/abb//25500 02856
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. Leaf-qa: Locate, encode & attend for tigure question answering. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3512-3521, 2020.
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia. Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. Hitab: A hicrarchical table dataset for question answering and natural language generation, 2022. URL _ httpp:::tp //a rxiv. www..orrgaaaoe a 02..2 06712
11

---

Published as a conference paper at ICLR 2026
Nancy A. Chinchor. Overview of MUC-7. In Seventh Message Understanding Conference (MUC- 7): Proceedings of a Conference Held in Fairfax. Virginia, April 29 May 1, 1998, 1998. URL
https:
Pavel Chizhov, Mattia Nee, Pierre-Carl Langlais, and Ivan P. Yamshchikov. What the hellaswag? on the validity of common-sense reasoning benchmarks, 2025. URL pectps://arxiin org/
abs/2500 07825
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang. Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. 2025.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. 2023.
Dipanjan Das, Desai Chen, André F. T. Martins, Nathan Schneider, and Noah A. Smith. Frame- semantic parsing. Computational Linguistics, 40(1):9-56, March 2014. doi: 10.1162/COLLa. 00163. URL /atttps://aclaathoooooo -1t.oo//////0110////20000
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seekking questions and answers anchored research papers, 2021. URL https:
//axxiv.org/abbt
03011
Sivan Doveh, Shaked Perek, M Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. Towards Multimodal In-Context Learning for Vision & Language Models. arXiv preprint 2et.::2233...7... 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsumori B. Hashimoto. Length-controlledd alpacaeval: A simple way to debias automatic evaluators, 2025. URL/htts::/arxiv..
Steffen Eger, Yong Cao, Jennifer D'Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, Chenghua Lin, Nafise Sadat Moosavi, Wei Zhao, and Tristan Miller. Transforming science with large language models: survey on ai-assisted scientific discovery, experimentation, content generation, and evaluation, 2025. URL
http://awwwv...////o//////oo/ 05151
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- els. Transactions of the Association for Computational Linguistics, 9:1012-1031, 2021.
Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. Qafacteval: Improved qa- based factual consistency evaluation for summarization, 2022. URL httts:://axiiiiiii org/
abs foss/1122 08542
Itai Gat, Idan Schwartz, and Alexander Schwing, Perceptual score: What data modalities does your model perceive?, 2021. URL http::ttt//aaat /ttp:s////a/vvvv/a org/ wttp:/iww.rraa/a//////a/ 14375
Paul Gavrikov, Jovita Lukasik, Steffen Jung, Robert Geirhos, Bianca Lamm, Muhammad Jehanzeb Mirza, Margret Keuper, and Janis Keuper. Are Vision Language Models Texture or Shape Biased and Can We Steer Them? arXiv preprint urXiv:2403.09199 2024.
Mark Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87:00334553117726559 08 2017. doi: 10.3102/003465433777269999
Jacob Hansen, Wei Lin, Junmo Kang, Muhammad Jehanzeb Mirza, Hongyin Luo, Rogerio Feris, Alan Ritter, James Glass, and Leonid Karlinsky. Instructify: Demystifying metadata to visual instruction tuning data conversion. arXiv preprint arXiv:2505. /8/ 15, 2025.
12

---

Published as a conference paper at ICLR 2026
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang. Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, arXiv e-prints, pp. arXiv-2507, 2025.
Yufang Hou, Alessandra Pascale, Javier Carnerero-Cano, Tigran Tchrakian, Radu Marinescu, Elizabeth Daly, Inkit Padhi, and Prasanna Sattigeri. Wikicontradict: A benchmark for eval- uating Ilms on real-world knowledge conflicts from wikipedia. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Infor- mation Processing Systems, volume 37, pp. 109701-109747. Curran Associates, Inc., 2024. neur ips ct/pppe.. fww.fiielf/ee... wtt.e////papee////////.... URL c5531977551ee32225577ffffaaaabb
Track.pas pdf
Irene Huang, Wei Lin, M Jehanzeb Mirza, Jacob A Hansen, Sivan Doveh, Victor lon Butoi, Roei Herzig, Assaf Arbelle, Hilde Kuhene, Trevor Darrel, et al. ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs. arXiv preprint arXiv:2406.08166 2024.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering, 2019. URL httts:://aattttt www.:orgaaba.....
1909. 06146
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan, Dvqa: Understanding data visual- izations via question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5648-5656, 2018.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning, 2018. URL https:
//axxii.or//abbat.......... http:ab//1111..0001 07300
Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10 (1):170, 2023.
Barrett Lattimer, Patrick CHen, Xinyuan Zhang, and Yi Yang. Fast and accurate factual incon- sistency detection over long documents. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 1691-1703. Association for Computational Lin- guisties, 2023. doi: 10.18655/w1/2223emmmffl
mai.npaaiinnaaaiemmmmmmm.a. 105. URL http:://xx doi www.dooo.........
18653/v1 emn wwwpppmmmmmmmmmmmmmmmmmmmmmm 105
Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jachyeon Kim, Hong-in Lee, and Moontae Lee. Qasa: advanced question answering on scientific articles. International Confer- ence on Machine Learning, pp. 19036-19052. PMLR, 2023.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a. URL htthttts:: httttp:://aaxiv.orrte
www.aiivvvo... wwtp:ii..oggaaaooo hbw.g//bbs/2450 2408 03326
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models, 2023. htRl:httt:///rrxt
org/abs/ 2301 12597
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multi- modal arxiv: A dataset for improving scientific comprehension of large vision-language models, 2024b. URL htttsss::///axx org. httpgg//bb////333//////// 00231
Wei Lin, Muhammad Jehanzeb Mirza, Sivan Doveh, Rogerio Feris, Raja Giryes, Sepp Hochreiter, and Leonid Karlinsky. Comparison Visual Instruction Tuning. arXiv preprint 24x::2406.0024000 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. 2025. URL
https:://ars www.vvorgg/ab.. 14437
19437
13

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruc- tion Tuning. arXin:2310.03744, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. 2023b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL http:///awwwww../aa/ooooo/
02255
Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, et al. Ovis2.5 technical report. 2025. URL
org/abs/
11737
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-dooc: Benchmarking long-context document under- standing with visualizations. Advances in Neural Information Processing Systems, 37:95963- 96010, 2024.
Radu Marinescu, Debarun Bhattachariya, Junkyu Lee, Tigran Tchrakian, Javier Camerero Cano, Yufang Hou, Elizabeth Daly, and Alessandra Pascale. Factreasoner: A probabilistic approach to long-form factuality assessment for large language models, 2025. URL https://axix.
org abs/ 2502. 18573
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. URL
https://arxiv. hatt:ss//aaaa...rrrraaa///222 10244
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievga. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Guofeng Mei, Wei Lin, Luigi Riz, Yujiao Wu, Fabio Poiesi, and Yiming Wang. Perla: Perceptive 3d language assistant. In Proceedings of the Computer Vision and Patlern Recognition Conference, pp. 14369-14379, 2025.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: scientific plots. In Proceedings of the ieeelcvf winter conference applications of computer vision, pp. 1527-1536, 2020.
Reasoning over
M Jehanzeb Mirza, Mengjie Zhao, Zhuoyuan Mao, Sivan Doveh Wei Lin, Paul Gavrikov, Michael Dorkenwald, Shiqi Yang, Saurav Jha, Hiromi Wakaki, et al. GLOV: Guided Large Language Models Implicit Optimizers for Vision Language Models. arXiv preprint 4rX0v:2410.01154, 2025.
Mistral. Nemo. huttpss/mmittrallll///es//tttss model.
nemo/ 2024. Large language
Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Evaluating trustworthiness of large multimodal models. 2024.
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryšcińsski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, et al. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35-49, 2022.
OpenAI. Gpt-5 https://chan opena /amee//aaiccm///// 2025. Large language model.
14

---

Published as a conference paper at ICLR 2026
Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. Spiqa: A dataset for mul- timodal question answering on scientific papers. Advances in Neural Information Processing Systems, 37:118807-118833 2024.
Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Che- ung. On the systematicity of probing contextualized word representations: The case of hypernymy in bert. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pp. 88-102, 2020.
Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie. Scifibench: Benchmarking large multimodal models for scientific figure interpretation. Advances in Neural Information Processing Systems, 37:18695-18777 2024.
Rohit Saxena, Pasquale Minervini, and Frank Keller. Postersum: A multimodal benchmark for scientific poster summarization, 2025.
17540]
Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh, Wei Lin, M. Jehanzeb Mirza, Leshem Chosen, Mikhail Yurochkin, Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, and Raja Giryes. Livexiv multi-modal live benchmark based on arxiv papers content, 2025. URL https://arxiv.
org/ bkg/abs/2en 2410. 10783
Baifeng Shi, Boyi Li, Han Cai, Yao Lu, Sifei Liu, Marco Pavone, Jan Kautz, Song Han, Trevor Darrell, Pavlo Molchanov, and Hongxu Yin. Scaling vision pre-training to 4k resolution, 2025. URL
org/ abs//503. 19903
Shruti Singh, Nandan Sarkar, and Arman Cohan. Scidga: A deep reading comprehension dataset over scientific papers, 2024. URL htttts://axxii.. org/ abs /etp//11l... 05338
Zusheng Tan, Xinyi Zhong, Jing-Yu Yu Ji, Wei Jiang, and Billy Chiu. Enhancing large language models for scientific multimodal summarization with multimodal output. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pp.
2025.
Mndreca-Maria Tanasä and Simona-Vasilica Oprea. Rethinking chart understanding using multi- modal large language models. Computers, Materials & Continua, 84(2), 2025.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. 2024. URL https:://arxitt abs/ abs/2408 /2002 00118
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. 2025. URL htt.h:tts///// //taxi//aiiiii org/ abs/2503. 19786
Alex Turner and Mark Kurzeja. Gaming truthfulqa: Simple heuristics exposed dataset weaknesses, 2025.
Adaku Uchendu, Thai Le, and Dongwon Lee. Attribution and obfuscation of neural text author- ship: data mining perspective. SIGKDD Explor Newsl., 25(1):1-18, July 2023. ISSN 1931- 0145. doi: 10.1145/36627443662276 3666276
URL
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries, 2020. UtL.ttts:://axtt wttp://w..rrra///////ooo/oo 200p://bs/00044.00 2004 04228
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro- pro: A more robust and challenging multi-tas task language understanding benchmark, 2024a. URL https: :tappss://rxivvv./ www.iiii.g.........ooooo.ooooooe abs 2405. 01574
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Jiang, et al. Mmlu-pro pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 11195266-955999 2024b.
15

---

Published as a conference paper at ICLR 2026
Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. Chartinsights: Eval- uating multimodal large language models for low-level chart question answering, 2024. URL
https://arxiv: org/ abs 2405. 07001
Qianqi Yan, Yue Fan, Hongquan Li, Shan Jiang, Yang Zhao, Xinze Guan, Ching-Chen Kuo, and Xin Eric Wang. Multimodal inconsistency reasoning (mmir): A new benchmark for multimodal reasoning models, 2025. http:ttsss//ai..oorrt.
abs 2200/55022.. 16033
Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. 2025a. URL https://
arxiv. htt.:/orgbbbb//0000 051.20988888088888 09388
Haiqi Yang, Jinzhe Li, Gengxu Li, Yi Chang, and Yuan Wu. Can large multimodal models actively recognize faulty inputs? systematic evaluation framework of their input scrutiny ability, 2025b.
URL wtt.tttss///cxxi.ooo hbtp://b////00/ 04017
Wenhui Yu, Gengshen Wu, and Jungong Han. Deep multimodal-interaciiv document summariza- tion network and cross-modal text-image retrieval application for future smart city information management systems. Smart Cities, 8(3):66, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? 2019. URL /tRtttst:://ett http///axvvvv/// wwwp:i.oogg/bb//10.....
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internin-xcompose-2.555 A versatile large vision lan- guage model supporting long-contextual input and output. 2024a. URL https://arxix:
204/abb/2407..0 03320
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186.
Springer, 2024b.
Yuhui Zhang, Yuchang Su, Yiming Liu, Xiaohan Wang, James Burgess, Elaine Sui, Chenyu Wang, Josiah Aklilu, Alejandro Lozano, Anjiang Wei, et al. Automated generation of challenging multiple-choice questions for vision language model evaluation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29580-29590, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URLhtttss//aaxxv abs / 22002236..20 05685
Fenghin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. To- wards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4857-4866, 2022.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. 2025. URL. [ http:: //aa////iiiiii v. org/ abs/ 2504. 10479
16

---

Published as a conference paper at ICLR 2026
APPENDIX
In the appendix, we first discuss illustrative qualitative examples (App- A including examples of text-table and figure-equation inconsistencies. We then provide comprehensive list of assets (App. [ detailing the data sources, licenses, and models used, including both open-source and proprietary ones. The ablations section (App. C explores the impact of rasterization resolution on model performance, extends the analysis of the effectiveness of Chain-of- soning with detailed case study, and validates our choice of the multiple-choice question format against open-ended evaluation. The dataset construction section (App. D.I) explains our refined methodology for sourcing, filtering, and annotating inconsistencies from scientific papers, including discussion of the annotation criteria, a custom -uiltt annotation tool, and dataset statistics. Next, detail the process of generating LLM-basee questions (App. D.4 for our benchmark tasks (Incon- sistency Identification, Remedy, and Pair-Match), including ISON representations. Finally, we provide details on the user study implementation and repre- sentativeness (App. [E.I]. the full LLM prompts (App. F used for various tasks, and screenshots (App. G and H) of our annotation and survey applications to provide a clear understanding of our methodology.
(CoT) rea-
debiasing strategies using structured
A QUALITATIVE EXAMPLES
We show qualitative examples of text-table inconsistency in Fig. andd figure-equation inconsis- tency in Fig. 4 together with their corresponding evaluation tasks.
B LIST OF ASSETS
Our images and annotations are sourced from publicly available datasets, and we distribute our data in compliance with the licensing terms the original sources.
The document and review data source can be found here:
ICLR 2025 on OpenReview http:///wwwuooooo/g///////////lt
ICLR 2024 on
All papers released under the CC BY 4.0 license.
All papers were released under the CC BY 4.0 license.
The list of source code and model weights can found here:
Qwen2.5-VL. http:///gghhaaaccmmm////eeMMMMee Released under the Apache-2.0 license.
LLaVA-NeXT http://gghaacccmmllaaavvvLLVVaAt Released under the Apa license.
Gemma 3 http://ggttb.coooooggggeeeemmmem Released under the Apache-2.0 license.
Ovis httt:://gthhbbcoommmA/DCCAAA//// Reluusd under the Apache-2.0 license.
Vn httpp////ghhiiiccoooooooocaaaaa/ Released under the MIT license.
GLM-V hutp::/giihhbbooomm/maoggggl../: Released under the Apache-2.0 license.
Relese
Mistral http://hggggghhooooooommmmmmmaaa license
Released under Apache-2.0
vLLM (fttss//giihhbb.ccom/////mmppgii Released under the Apache-2.0 license. MinerU https//ggihbbboomm//ppmaaaabbabs Released under the AGPL-3.0 license.
The list of proprietary models used can be found here:
Google Gemini htap:p/mmmn..ooooloooooeggo////m Used in version Gemini Flash 2.5 and Gemini Pro 2.5, released June 17, 2025.
17

---

Published as a conference paper at ICLR 2026
OpenAl GPT http:://dthabccmmm/LLavVVVVVVaa Used in version GPT-5, leased on August 7, 2025.
C
ABLATIONS
C.I IMPACT OF RASTERIZATION RESOLUTION
Scientific papers contain dense text and fine-grained visual elements such as axis labels, annotations, and subscripts, which are often crucial for detecting subtle inconsistencies. To test whether rasteri- zation resolution impacts detection performance, we varied the DPI used to extract images from the PDF and evaluated a representative set of strongest proprietary and open-weight models of different sizes on the Inconsistency Ident task with Focused Context, keeping all other settings fixed.
Table 4: Accuracy of LMMs under different rasterization resolutions. Results are reported for the Ident task with Focused Context. Percentage change calculated relative the 144 DPI baseline.
VILA HD 4K
Intern VL3,5
InternVL3.5
Ovis2 34B
Gemini 2.5 Pro
Default
Change
Default
Change
Default
% Change
Default
Change
Default
% Change
72
31.3
+1.3
42.7
-12.7
45.8
20.11
43.1
-13.1
67.2
-3.3
144
30.9
48.9
57.3
49.6
69.5
Low resolutions harm performance. Most models showed significant drops DPI, up to 20.1% for VL3.5 38B. Open-weight models were generally Gemini 2.5 Pro declined by -3.3%. Surprisingly, VILA HD, despite being trained for high-resolution inputs, showed slight accuracy gain at this lower setting.
more vulnerable, though even
Higher resolutions do not always improve accuracy. Increasing from 144 to 300 DPI yielded mixed outcomes. While Gemini 2.5 Pro and Ovis2 34B benefited slightly, InternVL models per- formed worse, and VILA HD again failed to leverage the higher fidelity despite its specialized train- ing. This suggests that additional detail can sometimes overwhelm global reasoning or misalign with training distributions.
Resolution sensitivity is architecture-speefffc Overall, the assumption that higher resolution im- proves inconsistency detection does not hold universally. Performance varies with model design and pretraining data, and high-resolution training does not guarantee an edge handling scientific in- consistencies. Careful DPI control is critical for fair evaluation. In our benchmark, 144 DPI provides a practical balance visual clarity, computational cost, and cross model comparability.
C.2 CASE STUDY: CHAIN-OF-THOUGHT REASONING
To illustrate the effect of th reasoning discussed in Section 4.3 we present a represen- tative example in Fig. 5 The figure reports results for Unique Successful Jailbreaks, count-based metric that is strictly non-negative. However, some error bars extend below the zero line on the y-axis-an inconsistency that invalidates the figure.
18

---

Published as a conference paper at ICLR 2026
Paper Context
Inconsistency Identification Task
Question: <Paper Context>> What is the inconsistency in these parts scientiãi papes?
Answer Options in Natural Language
A) The x-axis lahels are rotated and overlap in some instances, making clearly.
difficult
distinguish
B) The range of values the y-axia (from 0.0 to 20.0) appears narrow given signiflennt spread indieated by the error bars for categories like 'No Guardrail!
bars for categories extend below the 0.0 liue on the y-axis, which not possible for 'Unique
C) The
Gootool
Uloed
Successful Jailbreaks'.
The lack of clear legend makes difficuls understand what the different colored bars represent or what specific metries they track.
siccess one inorice bet alo lesils in much loser conthrr of jaloooks risconedd
Figure 5: Inconsistency example for case study. Right: Visual context. Left: Question and answer options for Ident task. Natural language options used for ease-of- www.oooooccccsiooooooooooo LMM was tasked using JSON.
Without reasoning, IL3.5 38B selected the distractor "x-axis labels overlap," (option A) jus- tifying it with generic but factually incorrect critique, as the labels were perfectly legible. The model defaulted a template- like response rather than verifying claims against visual evidence.
In contrast, the eenableeg-enabled model produced systematic Ohain-of-thought, (1) ruling out label overlap (by observing the labels were 'spaced out and readable'), ) (2) confirming the y-axis range was sufficient (noting all data was 'within the 0-20 range'), (3) dismissing legend critiques (since a legend 'isn' I necessary' when bars are directly labeled), and (4) crucially, identifying the logical error of еттоr bars that 'shouldn' go below zero if the metric [...] can't 't be negative'). This stepwise elimination and domain- -ware inference led to the correct answer. The full reasoning chain of Intern VL3.5 38B is available in Fig. 6
This case highlights two key strengths of reasoning: (1) systematic elimination distractors, and (2) integration of domain knowledge (e.g. non-negativity of counts) with visual grounding. Although reasoning increase output length (average 473 tokens per query in our case), it substantially improves multimodal consistency and robustness, making CoT key mechanism for handling subtle scientific document inconsistencies.
C.3 VALIDITY OF MULTIPLE- CHOICE VS. OPEN-ENDED EVALUATION
While open-ended generation is closer to real-world usage, the choice of multiple-choie ques- tion (MCQ) format for our benchmark was motivated by the need for reliable, reproducible, and bias-controlled evaluation protocol. To validate this choice, we conducted additional experiments comparing our MCQ format with an open-ended evaluation using an LLM-aas---uddee approach.
Setup. We tested representative subset of the identification task using open-ended responses from eight models. Their outputs were rated by three different judges Gemini 2.5 Pro, GPT-5 (high), and GLM 4.5V 106B-on Likert-5 scale measuring semantic alignment with the ground-truth inconsistency. Each judge was evaluated both with and without Focused Context.
Analysis of Open-Ended Evaluation. As shown in Table 5 our open-ended evaluation revealed several critical limitations that undermine its suitability for standardized benchmark. First, we observed significant inter-judge inconsistency, where LLMM judges produced diverging scores and rankings for identical model outputs. For instance, Gemini 2.5 Pro and GPT-5 frequently disagreed on absolute scores, leading rank shifts of two three positions for certain models depending the judge employed, Furthermore, GLM 4.5V 106B proved particularly unstable in its role as a judge, further complicating reproducibility.
second challenge relates to non-deterministic scoring in proprietary reasoning models. Since temperature of zero is not always available for models such as GPT-5, repeated evaluations of the same answer do not yield deterministic results. This introduces noise into the benchmarking process, which is especially problematic when performance differences between high-performing
19

---

Published as a conference paper at ICLR 2026
Table 5: L.LM-as-a-judge evaluation. Scores average Likert ratings (1-5); percentages indicate
scores >> 3. R denotes the model's 's rank per judge.
Gemini 2.5
GPT-5
GLM 4.5V 106B
Candidute Model
w/o context
context
w/o context
context
w/o context
context
Score
Score
Score
R
Score
Score
Score
Gemini Pro 2.5"
(44.0%%
(%))
2:71 (40.0%
(44.0%)
3.06
3.50 (62.0%)
GPT-5 (high)"
3.00 (54.0%)
2.70 (12.0%)
2.76 (52.5%)
1
2.94 (52.0%)
3.02 (51.0%)
3.32 (58.0%)
2
InternVI. 381*
(34.0%)
Qwen 2.5 VI. 72B
2.46 0%%
2.40 (38.0%)
2.30 (28. 5%%)
3
2.13 (26.75%)
2,42 (34.0%)
2.600 (40.0%)
GLM 4.5V 1068*
0%))
2:04 (22.0%%
(47.9%%
InternYL. 3.5 8B*
2.04 (28.0%)
1.94 (28.0%)
1.94 (18.0%
7
1.66 (18.0%)
2,60 (42.0%
2.78 (42.0%
5
Gemma
0%%
2:03
(18.4%)
Ovis
1.96 (20.0%)
1.86 (22.09)
1.83 (14.0%)
1.74 (16.0%)
2.18 (32.0%)
2.30 (32.0%)
models are small. Finally, we identified a clear judge-model similarity bias, where judges tended outputs from their model families favorably. This phenomenon, documented in prior evaluations such as MT Bench (Zheng et al. 2023) and AlpacaEval (Dubois et al., 2025 makes it difficult to disentangle true model capability from stylistic alignment with the judge.
Conclusion. In contrast to open-ended evaluation, the MCQ format remains more reproducible, providing deterministic scoring, controlled difficulty through distractors, and identical evaluation conditions without dependence on third-party proprietary judges. Combined with our JSON-based debiasing strategies, the MCQ format provides robust and reliable framework for BEEMM-BBenhh.
D DATASET CONSTRUCTION
D.1 REVIEW SOURCING
Initial Exploration with Regex Matching, Before finalizing the review sourcing strategy described in the main paper, we conducted an exploratory study to detect potential inconsistencies mentioned in reviews for ICLR 2024 using simple regular expression (regex) approach. Reviews were cessed via the ReenReview APE focusing on the "Weaknesses" and "Questions" sections, which were likely to contain critical feedback. Each sentence was parsed for co-occurrence of terms related to inconsistencies (e.g., "mismatch", "conflict") and references to visual elements (e.g.. "fig- ure", "table", "equation") The pseudocode for this procedure is shown below:
Algorithm Pseudocode for regex matching
1: DEsNNEATTERNiIcooiittnn patten
r' (inconsisten | mi nismatch doesn [t match not match conflict discrepanc) patterr, r' (figure fig.? | table graph plot | image diagram equation)")
2: DEFNEPATTERN(visuts PHANNPPATERNIisuullpattttttt
3: results < []
4: for each review in reviews do
5: sections < EXTRACTSSECTTONs(eeveee)
6: for each section in sections do
7: sentences + SPLLTINTOSENTENNESSSSttin
8: for each sentence in sentences do
9:
MArcIEs(sentence, eiconsistency_pattems and MATcHEss(entence, vi-
sual pattern) then
10:
APPEND(results, sentence)
end if 12: end for
end for
14:
end for
15: return results
20

---

Published as a conference paper at ICLR 2026
Manual inspection confirmed that reviews indeed contained valuable references visual-textual mismatches. However, two limitations emerged: (1) regex captured only strict keyword formula- tions, missing paraphrased or indirect mentions of inconsistencies, and (2) many inconsistencies referenced papers that had been updated after rebuttal, making impossible to locate the original emrors in the PDF versions available through OpenReview.
Refined Collection Strategy. To address these issues, we refined our strategy in two ways:
1. Conference selection: We shifted our focus papers without author rebuttals. This sured that flagged inconsistencies were more likely to remain in the available PDFs.
2. LLM filtering: Instead regex, we employed Mistral Nemo 2407 at a low temperature to summarize reviews and extract candidate inconsistency statements. This approach captured non-strict formulations (e.g.. "does not align with" instead of "mismatch") and produced structured outputs, making them easier present to annotators in the verification interface.
This refinement reduced our initial pool of 1 120,329 ICLR reviews to 18,009 reviews. The LLM outputs were stored in structured ISON format, with each paper ID associated with a list of flagged inconsistencies.
Example Output. Throughout the appendix, we going illustrate data preparation pipeline use the paper ID vXSCD3TOCS as an example. We illustrate the example out after the LLM- assisted filtering Fig.
This structured representation provided a natural starting point for the subsequent manual verifica- tion stage described in App. D.2
D.2 ANNOTATION PROCESS
Annotator Background. The annotation was conducted by the first author, who has an advanced background in Computer Science and Machine Learning. A consistent annotation standard was maintained throughout the project; any ambiguous or borderline cases were discussed with senior researchers until consensus was reached.
Annotation Criteria. During manual verification, the annotator judged each reviewer comment against the following criteria:
1. The comment reflects objective feedback rather than a subjective suggestion.
2. The comment describes an inconsistency involving two contradicting facts.
3. Both conflicting parts can located in the PDF.
4. The inconsistency can be identified without deep domain-specific expertise (focus on visual/document-levv inconsistencies)
5. The inconsistency is significant and factual, not minor typo or stylistic choice.
Annotation Interface. We implemented a custom web-based tool in Next js. The interface displayed the reviewer's comment (extracted by the LLM) alongside the corresponding paper em- bedded as PDF viewer (compare Fig. 18. Fig. 19 for screenshots of the app's interface). The annotator could:
Read the reviewer's comment and decide whether it fulfilled criteria (1) and (2). If not, the instance was skipped.
Search and inspect the relevant region of the embedded PDF.
Toggle between one-parl and (wo-parl annotation modes:
One-part: A single element (c.g., figure-caption inconsistency).
Two-part: Two separate elements (e.g., figure vs. text, figure vs. figure).
. Specify for each part whether it was textual or visual:
21

---

Published as a conference paper at ICLR 2026
Visual: Select the page, then draw bounding box rendered thumbnail version the PDF page.
Textual: Enter the page and line number, and copy the relevant text snippet from the PDF.
Assign inconsistency category via drop-down menu.
Provide short free-text description of the inconsistency in their own words.
Recorded Metadata. Each annotation combined automatically and manually collected fields:
Automatically recorded: element type (text or image), bounding box (relative coordi- nates) for visual selections, internal image identifier, reviewer's original comment.
Manually entered: page and line numbers, copied textual content, inconsistency category, and a short description by the annotator.
Example Output. Annotations were stored in JSON format, combining visual/textual parts, re- viewer comment, category, and description. We illustrate the example annotation output in JSON format in Fig.
D.3 STATISTICS OF INCONSISTENCY COLLECTION.
The annotation resulted in 384 inconsistencies from 353 ICLR papers. The average page count of each PDF was 16 pages. A total of 29 papers (7.6%) had more than one inconsistency. The paper subjects were equally distributed across the range of topics for ICLR4 with (1) representa- tion learning (26.0%), (2) transfer learning (10.4%), (3) generative models (9.6%) and datasets and benchmarks (7.8%) being the most frequent topics.
We identified 15 categories of inconsistencies based on the elements involved, with the distribution shown in Fig. The most common cases were figure- text mismatches and intra-figur inconsisten- cies.
D.4 LLM-BASED QUESTION GENERATION
D.4.1 INCONSISTENCY IDENTIFICATION (IDENT)
The Inconsistency Identification (Ident) task was the first benchmark task we designed. For each annotated inconsistency, we instructed Gemini Flash to generate multiple-choice question (MCQ) with four options, one of which correctly describes the inconsistency.
Inputs. As input to the model, we provided:
The annotated context (visual and/or textual parts).
The annotator's free-text description of the inconsistency.
Prompt. After experimenting with several formulations, found that minimalist prompt yielded the most creative and plausible distractors. We provide the final version of the prompt in Fig. 10
Output Format. The model produced structured output containing the question, the correct an- swer, and three distractor answers.
Manual Verification. Each generated question underwent manual verification:
Correct answer: must (1) faithfully reflect the annotator's description and (2) directly connect to the annotated context.
Distractors: must (I) be grounded in the annotated inconsistency parts, (2) avoid obvious contradictions within the answer itself, (3) only mention elements present in the provided context, and (4) describe an inconsistency rather than confirming correct fact from the paper
22

---

Published as a conference paper at ICLR 2026
We also post-processed the text to remove stylistic artifacts often appended by the LLM, e.g. the parenthetical ", indicating an inconsistency."
Example Output. We illustrate an example of the generated multiple-choice question for inconsis- tency identification task in Fig. I1
D.4.2 DEBIASING THE INCONSISTENCY IDENTIFICATION TASK
Initial Observations. When first evaluating the Ident task with Gemini 2.5 Flash, we observed unexpectedly high accuracy:
84.4% with the original -LM-generated questions. (e.g.: "What inconsistency is observed between Figure and the accompanying text regarding the generated road network?") ")
79.4% after replacing the LLM- MiMMggeerated question with the generic formulation: "What is
the inconsistency in these parts of a scientific paper?"
Even in a sanity check where the model shown only the question and answer options (without the annotated context), performance remained at 57.6% accuracy, far above the random baseline of 25%. This indicated strong reliance on linguistic cues in the answer phrasing.
Mitigation Strategies. Moving forward, we solely employed the generic question formulation throughout all inconsistencies. For reducing the without context accuracy Aceue we systematically explored ways of reducing linguistic priors by rewriting the answer options:
Normalizing answer length: Aceue 48.1%.
Filtering for MCQs where the correct answer is shortest: Accae 46.2%.
Rephrasing distractors according best practices MCQ test design (Gierl et al. 2017)
Accue 41.6%.
Shortening all answer options into nominal style: Accae 38.2%.
While these interventions reduced bias, they did not remove it completely.
Structured Representation: Evidence-Claim JSON, As a more robust solution, we abandoned free-form natural language and introduced structured, human-readable JSON representation that removes stylistic cues while preserving the semantic contradiction. The schema is:
"letter": "A"
"B" | "C" "D",
aattribute"; str,
"clain":
"source"; "expectation" str, "statement"; str
Vevidence":
sooucce; str,
"statement"; str
Patterns. Two patterns of contradiction are covered:
Claim vs. Evidence: A claim from one paper element is contradicted by evidence from another.
Expectation vs. Evidence: claim contradicts common expectations of scientific correct- ness. In this case, the claim's 's source is always "expectation"
We prompted Gemini 2.5 Flash to convert the natural language MCQs into this structured format. The full prompt be inspected in App. F2 A 20% subset was manually validated for consistency.
23

---

Published as a conference paper at ICLR 2026
This representation further reduced the no-context accuracy to 34.0%. Given the full context, accuracy on the new JSON format decreased from 79.4% to 69.5%. How-
Effect Model Behavior.
the fraction of performance attributable to visual grounding (Eq. D increased from 51.4% to 53.8%. Thus, the structured format acts as a regularizer, forcing models to rely more strongly on the provided paper context.
Example Output. For the rumning example, we illustrate the example of debiased output in the evidence-clain JSON format for the inconsisteney identification task in Fig. 12
D.4.3 INCONSISTENCY REMEDY TASK
Task Design. The Inconsistency Remedy (Remedy) task extends beyond identifying an inconsis- tency determining how it can be resolved. To avoid linguistic artifacts, directly employed structured representation in ISON format. This representation adapts the Evidence-Clain schema more action-oriented form, the Target-Action JSON:
"letter: "A" "B" | "C" "D",
aatrribute": str,
"target": str, pother_involven str, "action": "modi "modify" | "remove" "add" Redit_statements str, "reason";
"reposition" "replace",
Here, attribute captures the element issue, target specifies where the change is applied, other_involved records additional parts if necessary, and the fields action, edit_statement, and reason summarize the correction.
LLM Conversion Process. We found that prompting LLM to directly convert the natural lan- guage MCQs from the Ident task into Target-Action JSON yielded the most reliable results in terms of readability and correctness. The prompt is depicted in Sec. F.3
Example Output. Four our example used throughout this appendix, the task looks as follows: We illustrate the example output in Target-Action JSON format for the inconsistency remedy task in
Fig. 13
D.4.4 INCONSISTENCY PAIR-MATCH TASK
Task Design. The Inconsistency Pair-Match (Match) task focuses the subset of inconsistencies that involve two distinct visual parts. The model is presented with one element (text or visual) as the question context and must identify the corresponding inconsistent visual element among four options.
Filtering of Eligible Cases. Not all inconsistency categories suitable for pair matching. Cat- egories where the contradiction is contained entirely within a single element (i.e., figure-caption, figure-only, table-only, table-captiog the 262 inconsistencies in the dataset.
Eigorithm-onily were excluded. This filtering left 135 out of
Distractor Construction. To ensure challenging and fair distractors, we extracted all figures, tables, and equations from the 242 papers in our dataset using Minert which produced image crops with unique IDs, modality labels, and page numbers. We then implemented a python script to sample distractors as follows:
Distractors were restricted to the same modality the correct answer.
Preference given elements appearing the same page adjacent pages the correct element, so that distractors were topically similar.
24

---

Published as a conference paper at ICLR 2026
Sampling was done within the same paper. Each paper contained enough visual elements of the same modality so we didn't 't have to fallback to using elements from other papers.
This procedure reduced trivial elimination strategies (e.g., selecting "the only figure among tables") and forced models to consider
inconsistencies.
Example Output. In the running example, the annotated inconsistency links an in-line text with figure. The text is fixed as the question context, and the answer options are image IDs referring to extracted figures, with one image ID being the correct image cropped in the annotations. We show the example of output used in the inconsistency pair-match task in Fig. 14
E USER STUDY IMPLEMENTATION & STATISTICS
E.1 IMPLEMENTATION OF THE USER STUDY
Setup. We conducted the user study in online form, using custom web app. The participants were greeted with an onboarding screen, where they entered the following information to assess their eligibility to be included in the user study: (1) email address, (2) academic field, (3) academic level and(4) Al exposure. Afterwards, they were shown instructions for the survey and an introduction into the question formats and different context modalities. For each participants, ten tasks were randomly sampled from our dataset. For the first five tasks, the participants were shown the Focused Context with the exact cropped images and/or text passages from the paper. For the last five tasks, the participants were instructed to open link to the original PDF and use the whole document to answer the question. this case, they were provided with the visual element they should focus on in the paper. Screenshots of the user interface are provided in appendix H
Upon submission, the following datapoints were saved automatically: (1) The task ID, (2) The chosen answer by the participants, (3) whether the task was correctly answered with/without context, (4) whether the question was accompanied by Focused Context or Full Document Context and (5) the time it took the participants to answer each question.
Statistics. Our eight participants all have background in either artificial intelligence, computer sci- ence or mathematics at an academic level of PhD or higher (7 PhD, postdoc). All stated to exhibit advanced exposure levels AI, which we defined being comfortable with reading, interpreting and critically evaluating AI scientific literature. The median answer time for questions without pro- vided visual context was 45s, with Focused Context 145s and with Whole Document Context 169s. In 65% of the cases, participants changed their answers once provided with the context. In total, participants processed 80 inconsistencies.
E.2 REPRESENTATIVENESS OF THE USER STUDY SUBSET
To ensure that the 40 samples used in the user study are representative of the full benchmark, we clarify that they were randomly drawn from the full dataset without any filtering or cherry-picking. As for some models, such as Intern 38B and Qwen2.5 VL 72B, accuracies on this subset were higher than their full-benchmark performance, we assessed whether this deviation was expected by running 100-trial simulation. In each trial, we repeatedly sampled 40 random instances and computed the resulting scores.
As shown in Table 6) the simulation results indicate that the user-study subset happened to be in the upper quartile of model difficulty for these two models-a comparatively "easier" slice of the benchmark for them. Such variation is expected with small sample sizes and does not reflect system- atic bias in the subset selection. Importantly, this strengthens our human-LMM comparison: even on subset where LMMs performed better than their typical accuracy, humans still outperformed the models by a clear margin under both focused and whole-document conditions. the subset had been closer to the mean model difficulty, the human-mode gap would have been even larger. Thus, our user study results likely understate the core conclusion that humans substantially outperform current LMMs at detecting multimodal scientific inconsistencies.
25

---

G SCREENSHOTS OF THE ANNOTATION APp Published as a conference paper at ICLR 2026
Table 6: Model Performance Comparison: User Study vs. Simulation
Model (Context)
User Study Subset (Reported)
Simulation Mean
Simulation Q3 (75th Percentile)
Human Perf.
(Reference)
Intern 3.5 38B (Focused)
71.1%
58.6%
65.0%
77.5%
InternVL. 3.5 38B (Whole Document)
40.5%
30.6%
35.0%
65.0%
Qwen 2.5 (Focused)
65.8%
50.5%
55.0%
77.5%
Qwen (Whole Document)
48.6%
35.9%
40.0%
65.0%
F
LLM PROMPTS
Here we provide the full prompts used to instruct the LLMs.
LLM PROMPT FOR REVIEW FILTERING
We provide the prompt for LLM-based review filtering in Fig. 15 Given the reviewer's s comment, the model is instructed in a chain-of- fiiin-of-hhouee thought prompting manner to systematically analyze each desired characteristic of visual inconsistency. Few-shot examples help clarify the output format.
F.2 LLM PROMPT FOR CONVERTING INTO EVIDENCE-CLAN CLAIM FORMAT
We provide the prompt for LLM-assisted conversion of natural language answers (for the inconsis- tency identification task) into evidence-claim JSON format in Fig. [6 The evidence-claim JSON format is used answer options in the inconsistency identification task. The structured JSON-based answer representation for mitigating the language biases in multiple-choice evaluation.
F.3 LLM PROMPT FOR CONVERTING INTO TARGET-ACTION FORMAT
We provide the prompt for L.LM-assssted conversion of natural language answers (for the inconsis- tency identification task) into target-action JSON format in Fig. 17 Based on the question-answer pairs in inconsistency identification task, we generate question with answers for the inconsistency remedy task. The target-action ISON format is used as answer options in the inconsistency remedy task.
We show some examples of the interface of the annotation tool Fig. 18 and Fig.
H SCREENSHOTS OF THE SURVEY APP
We show some examples of the interface of the survey web interface in Fig. 20 Fig. and Fig.
22
26

---

Published as a conference paper at ICLR 2026
Paper Context
element:
element::
Tasls ID: Upperporteee
" where y is hyperparameter controlling the strength of the gradient guldance, Y balances the trade-off between fitting the observed data and adhering the learned data distribution"
Phrandee
Stecks
MajaCa
Energr
DMLL
Alzation hoad
Arration bost dieendaa
16
34
Deoder
3
Bacch
138
64
Tinedggss Sanplieg.opp
500
500
1000
1000
1000
Pro-ruisal salsity deps
10000
14000
29000
28000
Inconsistnncy Identifteation
Question: < Paper Context> What the inconsistency in these parts of scientifi papet?
Auswer Options in JSON Format
"letter": "A",
"kttter; "C"
"sttribuit": "ebx and gamma hyperparammttrre "clalm":
buteilluna: "gradiont guidhnee and trade=of"" "claim": "wourne "expectation", "statement") "should
"asurcas" "statement": "present"
A)
evidenoe""
evidence": 1
"ourrce" TTbbl 10",
"ourree: "Tuble 10".
statements "miising"
"etatement": " "not in table?
"letter": "Attention haade", "sttribatt"" "clalm":
Nitter": 'D"
"attrilmta": "Alphs".
"claim": [
"auurca": "expectation"
B)
"statement" : "shoold be explained
"staeemmut" "should be gamma"
"ecddenee" (
"source":
"source"; TTble
explained
"stteement "letted Alpha"
answer There claimn made in the tost about the attribute 'eta and gummn lypesparanetess and that claim that they "prosent") 1, but there evideeoe Taile 10 that they "miising"
Interpretation
Incousistency Remeedy Task
Question: <PPupe Contectss sclentific paper?
inconsistency parts
Answer Options in JSON Format
A)
"attribute";
"Istuee
"trrast: "usked" "uld""
immitwed" "tiect,
"ayperpaaaettrr rea (t) and gamma (+)",
C)
"artion" lnvolsed:: "toxs".
"lotter" *C""
"atteibute": "grudient guidunce and teude-of""
"target"; "table 10",
"parametene"
*ddit statement" "vallas". "reaeen' : "miooding
"resson": "miiiin
"Iutter": "B".
Nlattrr: 'D",
"astribute") "attcution hoads" "target" "tuxt"
*attelbute", "olpba, o², "cargot": "table 10",
"other invalved': "table 10",
D)
"action?: "msplaln" "reasoa" : "missing"
"action": "replace".
tedls statement" "with gamma, Y"; "rcuson": "typo"
Interpretation Aid Readl answer nec The attribute "teperparsmeters eta (n) and gummo (y)* Table 10 nceds to edited action parumuttens with the reason that they une "missing" compared to the otaer involved elemons "Toat".
Incousistency Pair-Match Task
Question: une pencided
part acientiie paparr:
where y is a hyperparameter controlling the strengrh of the gradient guidance, balanoee the trude-off between fitting the observed data and adhering to the Jearmed duta distribution" The commiaaatiin with one of the other gor within the some paper moculi in au incomsstanngg Pirk the letber of the cocteet anerer optioe.
necwer Optiome
Tabl Foreemsen
PAMO
TAE
B)
Figure 3: A qualitative example a text-table inconsistency and its corresponding evaluation tasks of Ident, Remedy and Match.
27

---

Published as a conference paper at ICLR 2026
Paper Context
Visual element:
Visual element:
Grntsat Lyair Xohox doCina + fins
(8)
Inconsistency Identification Task
Question: <<Paper Context>> What is the incousistency in these parts of a scientific paper?
Options Format
"lester"s "A"
"letter"s "C",
*attribute": "regularization
"attribura": "optimization process",
A)
"target" "äguree 1",
oother involved "equation 8", "action": : "replace",
()
"target" ; "equation 8",
involved", "fgure_!!,,
"action") "modlfy"
"L [pair] with L [b_oe"" "reason": " "different"
*ddit statement":
Fedit statement "yeflect two-stage adjustmentt" reesonn: : "differenn""
"lotter": "D".
"letter": "D".
"attributa": "panudo-labaling process", "target" "equation 8",
"ateribute"; "logit adjustmmeutt,
B) "other involved""; "figure
D)
"other Invalved": 'fgure 1",
"target": "equation 8*,
"action";
"action": "modify"
statement":
"edit statement": "reflect. unlaheled data only". "resson": "different"
"reason": "missing"
Interpretation Aid Read answer C as:
There claim made in the Figure about the attribute "optimization process" and that claim is that is "too-stage process"; but there is evidenee in Equation & that shows is 'shmoltancots optimization"
Inconsistency Remedy Task
Questiou:
< <appe Context What action needs to be taken to resolve the inconsistency in these parts of scientific paper?
Answer Options in JSON Format
"ltteer : "C""
"lettur":
"asteibute": "optimization process",
"ottribute": 'regularization term". "target") "tigure 1",
"targtt" "equation 8",
A)
"other involved E. "uquation 8".
"other involved"; fggure_1" 1',
"nesson": "replace",
"action". "modify"",
"edit statement"; 'L. [pair] with L [b_coo'' "reowmm" "dliffooenr""
two-stage adjustment"
"remsii"" "dilfferen!"
"lotter"; : 'B", "attribute": "pando-laheling process",
"letter: "D"
"attribute": "logit adjustment",
B) "other involved": "figure
"target ; "equation 8",
D
"target"" "equation 8".
"other involved": Bgure
"action": "add".
"nction": "modify"
"edltt "missing"
"Inea
"edit statement "reflect unlabeled data only",
"renson: "diiffrrntt"
Interpretation Aid Road C as:
The attribute ggrup lobels" Pigure needs be edited action "modiiy updaling labels' with the
reason that jt "mismatches" text. The abtribute "optimization procese" Equation 8 needs to edited by action "moodif to reflect two-stage adjustmentt' with the reason that it is "different" in the other involved
element Figare
Inconsistency Pair-Match Task
Question: You are provided with part of sclentific papprt:
Optious:
A)
Lrvi Eyor do Aofisce Ereg
Rur -
Σ
m
B)
D)
iif]= urmar /63f/aa -2 Saiiiaal--iiieeeeeee ineBepy)
16
exp
( T
x
-fa
(4)
Figure 4: A qualitative example of a figure-equation inconsistency and its corresponding evaluation tasks of Ident, Remedy and Match.
28

---

Published as a conference paper at ICLR 2026
Full Reasoning Chain of InternVL 3.5 inconsistency 3MDmMMrrMPQ
Okay, let's take a look this question. The user asking about incousistencies in parts scientific paper based on the provided figure. The options are to D, each pointing out a different potential issue.
First, I need to check each option against the figure provided. Let's start with option A: x-axis labels overlapping. Looking at the x-axis in Figure 6, the labels are "No Guardrail," "IPE, "OpenAl Moderation, "Perspective API," "Prompt-Based Llama2," and "LlamaGuard22 They seem to be spaced out and readable. There's no overlap here; rach lahel is clearly visible. So isn't the right answer.
Option B mentions the y-axts range being too narrow. The y-nxti goes from to 20.0, and the data points are all below 15, with error bars extending up around 18. If the range too narrow; the error bars might get cut off. But in the figure, the error bars within the 0-20 range. The maximum error bar seems to reach just. under 20, so the range sufficient. B not correct.
Option C states that bars should be uon-uegative extend below 0. Looking the eroor bars, especially "IPE" and "OpenAl Moderation, the error bars go below the 0 line. For example, the "No Guardrail" bar has an error bar that dips below the y-axis. Since error bars represent variahility, they shouldn't go below if the metric (like jailbreaks] can't negative. This seems like valid inconsistency. C is the answer.
Option D claims the legend isn't clear. The ligure's legend uses different colors for each method, but the legend itself isn't t shown in the image. However, the user's question refers to the figure's elements. Without a legend in the figure, that's an issue, But looking at the figure, each bar is labeled directly with the method name, so a legend isn't 't necessar. Therefore, D is not the issue here.
So, the most likely answer is C, because error bars extending below zern when the metric can't be negative is an inconsistency. The other options don't hold upon closer inspection.
Figure 6: Raw reasoning output by Intern VL3.5 38B on 3MDmMOrMPO.
Example Output After the LLM-assisted Filtering
"yXSCDSTOCS": {
"has inconsistency"; truc,
"inconsisteucics":
"Table The performance improvement from using years of data (MAE 20.24) compared year (MAE 21.90) in the 3-day setting is marginal, contradicting the emphasis on the scale DynST as a major contribution."" "
"Figure The bottom left corner shows a road segment between two points that is not represented as an edge in the topology, which appears inconsistent with actual road network." ", "Figure 5: The description the figure suggests changes in sensors deployment, not actual road network dynamics, which contradicts the paper's 's claim considering dynamic road network topology."
Figure 7: Example output after the LLM-assisted filtering.
29

---

Published as a conference paper at ICLR 2026
Example Annotation Output in JSON Format
"inoonsistency parts":
"type": "image";
"page":
"image id"; "xXSCD3TCCS alcSadc6".
"bbox": { "x": 0.50, "y": 0.25, "widch": : 0.35, "height": : 0.31 }
"type": "text",
"page"; 5,
"line": 262,
"content": "The results demonstrate
rreview text":
"Figure 2: The bottom left corner shows ...",
cateeory"" : "figure-text",
"description"; "Missing edges between nodes compared claim." "
Figure 8: Example annotation output in JSON format.
Count
ladentedd
Gromudalee dable
Category
Figure 9: Distribution of inconsistency types. We identified 15 categories of inconsistencies based on the elements involved. The most common cases are figure-text mismatches and intra-figure (figure-only) inconsistencies.
Prompt for Preparing MCQs in Inconsistency Identificatio Task
You are visual assistant that analyze image and text excerpts from scientific papers. You receive either image, two images, pair of image and text that contain visual inconsistency flagged by reviewers. Alongside the content, you also receive description of the inconsistency. Based these, generate multiple-choice question testing the model's ability to detect the inconsistency. Follow these strict rules:
TTee question should directly reference the provided content of the paper.
'hhre must be exactly 4 answer choices.
Only one answer should correctly describe the inconsistency.
The distractors must be plausible but incorrect. They should either be incorrect due omission or subtle misinterpretations
content.
Do not invent details beyond what is provided.
Clearly label correct answer.
Figure 10: Prompt for preparing multiple-choice questions in the inconsistency identification task.
30

---

Published as a conference paper at ICLR 2026
Example of Generated MCQ Inconsistency Identification Task
"meq":
default": {
"question": "What inconsistency is observed between Figure 2 and the accompanying text regarding the generated road network?".
"correct": "The visualization in Figure 2 shows missing edges between nodes, which contradicts
the text's claim that the generated network perfectly matches the actual road network structure." "incorrect":
"Thhe visualization in Figure shows extraneous edges between nodes, which contradicts the
text's claim that the generated network perfectly matches the actual road network structure." "Figure 2 depicts only disconnected portion of the road network, implying the algorithm did not generate the complete structure.",
"The blue nodes in Figure 2 are unevenly distributed, making it difficult to determine the precise road paths."
letters": "D", "A" "B", "C"
Figure 11: Example of generated multiple-choice question for the inconsistency identification task.
Example of Debiased Output in Pridence-Claim JSON Format
"meq":
"letter": "D".
"default":
"attribute": "network",
"question": "What is the inconsistency in
"claim": {
these parts of scientific paper?",
"source": "expectation", "statement": "complete structure"
"correct":
"letter"; "A",
"evidence": {
"attribute"; "edges",
"source": "Figure 2". "statement": "disconnected portion"
"claim":
"source": "text".
"statement"; "perfeetly matches"
evidence":
"letter": "B" "attribute": "nodes", "claim": { "source"; "cxpectation", "statement": "eveuly distributed"
"source":
"Figure 2"
"statement"" ; "missing edges"
}.
"incorrect":
"evidence": {
sourcee: "Figure 2". "statement": "unevenly distributed"
"letter": "C"
"attribute": "edges",
"claim":
"source"; "text", "statemmett: "perfectly matches"
"letters::
"A" "C". "D", "B"
"evidence": {
"source": "Figure 2",
"statement"; "extrancons edges"
b
Figure 12: Example of debiased output in evidence-claim JSON format for the inconsistency iden- tification task.
31

---

Published as a conference paper at ICLR 2026
Example of Output in Target-Action JSON Format
"meq":
"letter": "D"
"edit":
"attribute": "road network", "target": ffiuure 2",
"question": "What action needs to be
taken to resolve the inconsistency in these parts of scientific paper?"
"other involved"; "algorithm", "action": "modify", "edit statement"; "disconnected portion", "reason": "incomplete structure"
ccorrect::
"letter": "A"
"attribute": "edges".
"target": " "figure 2"
"other involved"; "text",
"letter": "B",
"action"; "add".
"attribute": "blue nodes".
"odit statement":
"missing edges",
"target": "figure 2" "other involved":
"reason": : "contradicts claim"
"action": "modify", "edit statement": "distribute nodes evenly", "reason": "unclear paths"
"incorrect":
"letter": "C",
"attribute": "edges". "target": "figure 2"
letters": ":
A "C", "D", "B"
"other involved": "text",
"action": "remove".
"edit statement": "extraneous edges", "contradicts claim"
"reason":
Example of output target-action JSON format for the inconsistency remedy task, directly converted from the natural language MCQs from the inconsistency identification task.
Figure 13:
Example of Output the Inconsistency Pair-Match Task
"mog": {
"part pair": {
"question": "The results demonstrate that the adjacency matrix generated by our algorithm perfectly matches the actual road network structure.",
"correct": "rXSCD3TOCS 5 aleSa4c6",
"incorrect":
"xXCCD3TCCS 5 image figure3".
"xXSCD3TcCS 5 image figure11.
"VXSCD316CS 6 image figure5"
letters::
"D", "A", "C", "B"
Figure 14: Example output for the inconsistency pair-match task.
32

---

Published as a conference paper at ICLR 2026
Prompt for LLM-based Review Filtering
You are an ussistant specialized in analyzing academic paper reviews. Your task is to identify incousistencies between visual elements (such ligures and tables) and their corresponding descriptions the original paper being reviewed. These inconsistencies should be explicitly mentioned or highlighted by the reviewer in their review.
Here peper review you to analyze:
<review>>
{prompt)
</review>>
Instructions:
Carefully read through the entire review.
Focus exclusively on identifying ing instances where the reviewer mentions inconsistencies in original paper between visual elements (figures, tables, graphs, etc.) and their corresponding descriptions.
For each identified inconsistency:
Determine the type of mismatch (e.g., ligure logend vs. content, text results figure data, table values vs. text mentions)
b. Note the specific location reference in the original paper (e.g., ligure number, table number, page number available)
Brielly describe the nature of the incousistency mentioned by the reviewer
Disregard any general inconsistencies that are not related vision-text mismatches original paper.
Before providing your final response, analyze the review in <review analysisc> tags:
List mentions of visual elements the review.
For each visual element, note whether reviewer mentions any inconsistencies with the text.
7. For identified inconsistencies, write down the specific quote from the review that mentions it.
thorough examination of the review and prevent misinterpretation inconsistencies within the review itsolf versus those in the original paper.
This analysis will help
After your analysis, present your findings JSON format. Each identified inconsistency should object array, with the following structure:
"has inconsistency":: boolean, "inconsistencies".
"string (brief explanation the inconsistency, always including place the original paper where located and as close to the reviewer's text possible)", Additional inconsistenciees..
no vision text inconsistencies in the original paper are mentioned by the reviewer, return:
inconsistency false, "inconsistenciie""
Example of desired output structure (purely for format, not content)):
"has inconsistency":: true, "inconsistencies":
'Table 1: The performsnce for model A is 69.74 percent but the text mentions 65.47 percent." "The refers Group and Group 0, but Figure labels groups as Group Group
Remember to focus solely on vision text mismatches in the original paper mentioned by the reviewer. Provide easy for researchers to locate and verily the incousistencies in the original
clear, concise descriptions that make it paper based the review's is comments..
Figure 15: Prompt for LLM- review filtering.
33

---

Published as a conference paper at ICLR 2026
Prompt for Converting Natural Language Answer into Evidence-Clain JSON Format
You system that converts multiple choice question answers into Svidence-Clainn JSON format.
Evidence-Claim JSON format:
** json
"letter": "A" "B" "C" "D",
"uttribute": str,
"claim":
"source". : "expectation" str, "statement", str
eevidence"; {
"source": str,
"statemeut" :
There two patterns of options:
Pattern 1: One part the answer makes a claim that is contradicted by evidence in another part
Example: "json
"letter": "C", // The letter the answer option "attribute": "optimal trade-off", // The attribute the center of answer option (e.g. parameter, complexity, name, etc.)
"claim": {
"source": "caption", // The source the claim about the attribute is based on (e.g., caption, text, figure etc.)
"statement": "at 128 tokens" brief words description
"evidence"; {
"source": "plot", // The source the evidence about the attribute contradicting the claim is based on (c.g. plot, table, equation etc.) "statenint"" ; "not visible 128 tokens" brief 2-3 words description
Pattern One part the answer makes claim that contradicts common expectations scientific correctness
Example: ""json
"letter": "A"
"attribute": "legend".
"claim": {
"source": "expoctation", // In that case, the source for claim always "expectation" "statemeut": "shouldn't occlude plot"
"vvidence::
"source": "figure 8".
"statement": : "occludes plot"
Given:
The question
The answer options with letters (A, B, D)
The correct answer letter
The visual elements relevant to the inconsistency
Convert each multiple choice question answer (A, C, into the Target-Action JSON format. Ensure that the answer letters remain consistent with the imput. Keep the JSON output concise. Do not use adjectives or any other descriptive language. The goal is to remove linguistic cues and focus on the core content of each answer option.
Figure 16: Prompt for LLM-assisted conversion of natural language answers of the inconsistency identification task into evidence-claim JSON format. The evidence-clain JSON format used as answer options in the inconsistency identification task.
34

---

Published as a conference paper at ICLR 2026
Prompl for Converting Natural Language Answer into Tatiet-Action JSON Format
You are system that converts multiple choice question answers about inconsistencies in scientific papers into Parget-Action JSON format. The goal is to identify what needs to be changed in the paper to resolve the inconsistency.
Target-Action JSON format:
.** json
"lotter": "A" "B" | "C" "D",
"attribute": str, // the core clement at issue (e.g. logend, mothods evaluated, F1 scores)
"target": str, // where the edit is applied (e.g., caption, figure 4b, table 5, equation 2) "other involved": str // (optional) other elements involved in the inconsistency, comma-separated "action":
"modify" "remove" "add" "reposition" "replace", "edit statement": str, // short 2-3 words description of the needed change (exclude word from action)
"reason": str // why the change is needed in 2-3 words
Example: "json
"letter": "C"
"attribute": "windows".
"target": "figure 1b".
"other involved": "figure_ la", "modify",
"action":
"edit statement""
"align door position",
"reason": : "different"
Given:
The question
The answer options with letters (A, B, C)
The correct answer letter
The visual elements relevant to the inconsistency
Couvert each multiple choice question
(A, B, C, D) into the Target-Action JSON format. letters remain consistent with the input. Keep the JSON output concise. Do adjectives or any other descriptive language. Most important is linguistic cues and focus on the content of each answer option.
Ensure that the
Figure 17: Prompt for LLM-assisted conversion of natural language answers of the inconsistency identification task into target-action ISON format. The target-action JSON is used as answer options in the inconsistency remedy task.
35

---

Published as a conference paper at ICLR 2026
Paper Inconsistency Annotation
Export Batch (1)
Export All
Completed
Valid Inconsistencies
Skipped 1/1
0/1 (0.0%)
(39,2%)
(100.0%)
Skip This Inconsislency
Progress
2 of
complete
Current Inconsistency
Paper ID: xXSCD3TeCS
Figure comer road segment between points not represented an edge the topalogy, which appears Inconsistant with the actual road network.
PDF Document
DYNST: RGE-Sc LE TI L-TEMPOR SET FOR TR NSFER BLE TX FRI FOREC STING WITH Dy: NETWORKS
LITE
so thihorm caeds cediii cur
wode day Ton tose liyel
bavdd
budf mevenst challl
Ou 2ith del. 111 2ilk bivil 2tile
soreett Eloelags brodleg one reploit Ba boakd badl
10
rower aold crneinaaii tn aelon wrrvod eace
loodly Seyiit doal incoped Alberd
Erable inconsistency peri
Figure 18: First part of annotation app showing an overview over the annotation progress and em- bedded original PDF file.
36

---

Published as a conference paper at ICLR 2026
P First Inconsistency Part
Image Inconsistency
Page Number
Click and select the inconsislency the image
t Crap Selection
Reset
T Second Inconsistency Part
Inconsistency
Page
Text Content
The results demonsirate the adjacency metric generated by our algorithm perfectly metches actual network structure.
Line Number
262
Annotation Metadata
Inconsistancy Category
figure text
Description
can missing edges belween nodes perfectly road structure claimed the text the
Save Annotation & Coriinue
Figure 19: Second part of annotation app for drawing bounding boxes, entering text and further details about the inconsistency.
37

---

Published as a conference paper at ICLR 2026
Research Survey
Question of 10 Phase
Question (Phase Initial Guess)
Inconsistancy parts papor?
The Original Question' defines as 'Complattly Dissatiefled Completely the Converted Statemente redefine the tower numbers 'very numbers (9,10) dissstisteed.
but the Conveerte Statements" only group responses ranges.
higher
'Driginal Unstructured Survey Questions; while the Conwerted Statements' expressing statements, Indicating data tуpo.
The Inconsistency lies the "LMd' Predictions' section,
defined
percentages 186%
Submit Initial Guess
Figure 20: First part survey interface showing a question with no context provided.
38

---

Published as a conference paper at ICLR 2026
Research Survey
Question of 10 Phase
Context
Original Servey
Predecton::
yoer what Boow Aupst
Touar mover veleeren you
defici livil
povar
Opplens
law A.l
Genbions
bathice pepgle
Banh ond
tede Colche
Soore right
Staterr
vodlen oor fine more
Figure VALUEVALUECA ATALOG, transformed from World Value Survey (W VS),) contains state- menis expressing insiniddal adlatisiic human values and preferencee from 94K real bursans workdwidk. With this resouree, study L.Ms ability resson about individual haman values.
Question (Phase 2: With Context)
What is the inconsistency in these parts of a scientific paper?
OOiggnal discrete ranges.
only group responses
The Original Question' defines Commletely Dissatisfied 10 Completely Satisfied', whereas the 'very satiefled' numbers 'very
dissatisfled':
The OOiggial Question' allowe for open-ended Unstrrcttree Survey Queetione', while the 'Converted Statements' limited defined value- expressing Indicating
The inconsistency lies in the 'LMs' Predictions' section, where Person and Person exhibit different accuracy percentages (56% 67%).
Next Question
Figure 21: Second part of survey interface showing question with Focused Context.
39

---

Published as a conference paper at ICLR 2026
Research Survey
Question of 10 Phase
Context
PDF Document
For this question, you can scrall the whole paper PDF to answer.
& Open PDF Document
Pay attention following parts of paper:
Text parts: From Line 371
Visual parts: Figure
Question (Phase With Context)
the inconsistency parts scientific paper?
The text claims that 'our model exhibits a more concentrated peak near Qdifference O', but Figure shows 'normal_ ours' Lours' having a main peak that is clearly more negative than CQL main peak.
Figure illustrates that models their primary differencee concentrations around positive values, contradicts statement both display peak negative Q_dilferrece values.
The text states that COL shows more spread in the positive Q_ difference direction, indicating more frequent overestimations, but Figure clearly depicts that 'normal_ oure' proposed method) has extensive region compared
pronounced
The text mentions that both models have 'long tail extending toward positive values' but Figure indicates that neither hoormalooo"" nor normal_ours' show any data points in the positive O_difference range.
Figure 22: Third part of survey interface showing question with Full Document Context.
40


Reviews:
[{"review_id": "IzOnSfKVXz", "reviewer": "Reviewer_stE4", "summary": "This paper introduces PRISMM-Bench, the first benchmark derived from real peer-reviewed inconsistencies in multimodal scientific papers. It systematically mines OpenReview comments to construct tasks that assess LMMs on detecting and correcting inconsistencies across text, figures, tables, and equations. The work fills a meaningful gap in multimodal reasoning evaluation in terms of inconsistency detection in scientific papers.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The topic of instructing LMMs to discover the inconsistencies is interesting. This is a very difficult task, even for reasoning models, and it evaluates various abilities, including reasoning, scientific graph understanding, and long multimodal context understanding.\n\n2. The idea of using reviews from the OpenReview website is clever, which reflects the real-world scenarios and leverages the implicit human-labeled data.\n\n3. The data collection pipeline is reasonable, and the authors have done extensive experiments to provide insightful conclusions.", "weaknesses": "1. The questions are all in multiple-choice form. Although this is easy to evaluate, it is not representative of how LMMs will be used in real-world scenarios, which may weaken the results and conclusions of the paper. \n\n2. The scope of the benchmark is another concern. The benchmark only contains 262 inconsistencies, which may be too small to derive a solid conclusion. Besides, the paper is only from ICLR 2025, which mainly consists of AI fields. The LMMs' ability to discern inconsistencies in other fields has not been explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qq6rO6LUhp", "reviewer": "Reviewer_CmSk", "summary": "This paper presents PRISMM-Bench, a benchmark for evaluating large multimodal models (LMMs) on identifying and fixing inconsistencies in research papers. The inconsistencies are sourced from those explicitly mentioned by reviewers in OpenReview reviews of ICLR 2025 papers. The benchmark consists of 3 tasks in multiple-choice format for 1) identifying the inconsistency (Ident),  proposing solution to fix inconsistency (Remedy), and matching elements of inconsistency (Match). Observing that LMMs tends to leverage text priors as shortcuts that heavily biased the results, the authors propose a debiasing method that converts the natural language options into JSON format which has shown to be effective. Experiment results of 21 LMMs on the benchmark reveals that even the best performing models struggles in these tasks especially in more realisitic long-context setting, showcasing the challenges of applying these LMMs as scholar assistants for tasks like identifying and fixing inconsistency with fine-grained grounding in multimodal context.", "questions": "1. In addition to the weaknesses mentioned above, can the authors explain why the yield rate in terms of  # of human validated inconsistencies / # LLM filtered inconsistencies is so low? Judging from the numbers in the paper, only ~5% of the LLM recognized potential inconsistencies are validated, which significantly constraint the overall size of the benchmark.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presented task is both novel and of potential practical value. Testing LMMs capability in identifying and fixing inconsistency in papers could be a good indicator of their capabilities in fine-grained multimodal grounding and understanding, while also foster the development of tools based on these models to help authors check their papers.\n2. The observation & analysis of choice-only shortcuts is insightful, and the proposed JSON-based debiasing method largely increase the utility of the benchmark and also provide a potentially useful debiasing approach for future MCQ-based benchmark construction.", "weaknesses": "1. For both the identification & remedy tasks, I am not fully convinced that the MCQ setup is the optimal way of presenting this task. While the MCQ task does help probe LMMs capabilities, they are both indirect and less realistic - in real world scenarios we would expect the LMMs to identify and remedy these inconsistencies from scratch instead of selecting from a set of options. Plus all the shortcut biases the authors have observed for MCQ. So I would consider formulate them as generation tasks that directly ask LMMs to output what are the inconsistencies & solutions. It seems to me the evaluation should also be easy and less biased - I would expect LLM-as-a-judge can determine if the generated inconsistency & solution are the same with human annotated ground-truth at a pretty high accuracy.\n2. The subset results for the user study might not faithfully reflects the human-LMMs gap. In table 2, the JSON performance of InternVL3.5 38B R & Qwen 2.5 VL 72B on both focused & Document are significantly higher than their corresponding performance on the full set as shown in table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uguD7fpdg3", "reviewer": "Reviewer_KPqE", "summary": "This paper introduces PRISMM-Bench, a benchmark for evaluating LMMs on multimodal inconsistencies in scientific papers. Its primary contribution is sourcing 262 inconsistencies from *real ICLR 2025 peer reviews* rather than synthetic data. The benchmark includes three tasks (Identification, Remedy, Pair Match) and proposes a novel JSON-based answer format to mitigate linguistic bias in MCQ evaluation. Experiments demonstrate that even state-of-the-art LMMs (e.g., Gemini 2.5 Pro) struggle (max 54.2% accuracy), proving the benchmark's difficulty.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* **High Originality and Authenticity:** The paper addresses the critical problem of multimodal inconsistency using real-world data sourced directly from expert peer reviews. This is a significant improvement over benchmarks based on synthetic errors.\n* **Novel Debiasing Method:** The structured JSON answer format is an innovative and effective method to combat linguistic shortcuts in MCQ evaluations. The user study provides strong evidence for its necessity and utility.", "weaknesses": "* **Limited Scale and Domain:** The benchmark's primary flaw is its small scale (262 instances) and narrow scope (only AI papers from ICLR 2025). This severely limits its statistical power and generality as a benchmark.\n* **Methodological Gaps:** The paper fails to report Inter-Annotator Agreement (IAA) for its human-verified dataset. This is a crucial omission that makes it difficult to assess the objectivity and quality of the annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xDbIv9L2iU", "reviewer": "Reviewer_gBzJ", "summary": "This paper presents PRISMM-Bench, a new benchmark designed to evaluate large multimodal models (LMMs) on their capacity to detect, remedy, and reason about inconsistencies between multimodal content (text, figures, tables, and equations) in scientific papers. The authors construct the benchmark by mining real reviewer-flagged inconsistencies from ICLR submissions, refining this through LLM-based filtering and human annotation, resulting in 262 validated inconsistency cases across 242 papers. The benchmark includes three multiple-choice tasks (inconsistency identification, remedy, and pair match) and introduces a structured JSON-based answer format to minimize shortcut exploitation in model evaluation. Results from evaluating 21 LMMs demonstrate that the tasks remain highly challenging, even for state-of-the-art proprietary models.", "questions": "1. Dataset Scale/Future Expansion: Are there specific plans or pilots for expanding PRISMM-Bench to domains beyond machine learning (e.g., other scientific fields) to address the current limitations in scope and domain diversity?\n2. Inconsistency Existence Judgment: Most scientific papers may have no detectable inconsistencies, yet models first need to judge inconsistency existence before analysis. Currently, PRISMM-Bench only uses samples from papers with reviewer-flagged inconsistencies, and all three tasks’ answer options assume \"inconsistencies definitely exist\". Has the team considered supplementing it with samples from inconsistency-free papers and adding a \"This section has no inconsistencies\" option to multiple-choice tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Authentic Benchmark Construction: A key strength is the grounding of all inconsistency cases in real reviewer feedback from ICLR 2025 submissions, ensuring ecological validity. The paper moves away from synthetic or artificially injected errors, avoiding pitfalls of prior work.\n2. Thorough Pipeline and Documentation: The multi-stage pipeline involving LLM filtering, human verification, and metadata annotation is robust and clearly described, supporting reproducibility.\n3. Bias Mitigation: The use of structured JSON answer formats is a methodologically novel move to counteract answer-choice linguistic shortcuts.\n4. Comprehensive Empirical Analysis: The experimental results cover 21 models, consider various model sizes and architectures, different input granularities, and ablation on answer format and context.", "weaknesses": "1. Limited Benchmark Scale and Generalizability: Despite claims of diversity, the dataset is relatively small (262 inconsistencies), and coverage is restricted to rejected or withdrawn ICLR 2025 submissions, all presumably from the AI/ML domain.\n2. Potential for Shortcut Leakage Remains: The proposed JSON-based format mitigates, but does not fully obviate, the risk of answer-pattern exploitation. As seen in Table 2/3, model accuracy without context remains above chance. More analysis on possible residual shortcuts and the robustness of the JSON representation to adversarial answer construction would strengthen the remedy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)