Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning
Authors: Rongjin Li
Zichen Tang
Xianghe Wang
Xinyi Hu
Zhengyu Wang
Zhengyu Lu
Yiling Huang
Jiayuan Chen
Weisheng Tan
Jiacheng Liu
Zhongjun Yang
Haihong E
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models; Academic Paper Reasoning; Scan-Oriented Reasoning
Primary area: datasets and benchmarks
TL;DR: We present ScholScan, a scan-oriented benchmark for full-paper scholarly reasoning that requires models to build a paper-level evidence view; spanning 1,800 questions from 715 papers, which exposes MLLM gaps and shows RAG ineffective.

Abstract: With the rapid progress of multimodal large language models (MLLMs), AI already performs well at literature retrieval and certain reasoning tasks, serving as a capable assistant to human researchers, yet it remains far from autonomous research. The fundamental reason is that current work on academic paper reasoning is largely confined to a search-oriented paradigm centered on pre-specified targets, with reasoning grounded in relevance retrieval, which struggles to support researcher-style full-document understanding, reasoning, and verification. To bridge this gap, we propose **ScholScan**, a new benchmark for academic paper reasoning. ScholScan introduces a scan-oriented task setting that asks models to read and cross-check entire papers like human researchers, scanning the document to identify consistency issues. The benchmark comprises 1,800 carefully annotated questions drawn from nine error categories across 13 natural-science domains and 715 papers, and provides detailed annotations for evidence localization and reasoning traces, together with a unified evaluation protocol. We assessed 15 models across 24 input configurations and conducted a fine-grained analysis of MLLM capabilities for all error categories. Across the board, retrieval-augmented generation (RAG) methods yield no significant improvements, revealing systematic deficiencies of current MLLMs on scan-oriented tasks and underscoring the challenge posed by ScholScan. We expect ScholScan to be the leading and representative work of the scan-oriented task paradigm.

Published as a conference paper at ICLR 2026
NOT SEARCH, BUT SCAN: BENCHMARKING MLLMS ON SCAN- ORIENTED ACADEMIC PAPER REASONING
Rongjin Li, Zichen Tang, Xianghe Wang, Xinyi Hu, Zhengyu Wang, Zhengyu Lu, Viling Huang, Jiayuan Chen, Weisheng Tan, Jiacheng Liu, Zhongjun Yang, Haihong E* Beijing University of Posts and Télecommunications
S
Scan
ABSTRACT
With the rapid progress of multimodal large language models (MLLMs), AI al- ready performs well literature retrieval and certain reasoning tasks, serving a capable assistant to human researchers, yet it remains far from autonomous re- search. The fundamental reason is that current work academic paper reason- ing is largely confined to search-oriente paradigm centered on pre-specified targets, with reasoning grounded relevance retrieval, which struggles to sup- port researcher-style full-document understanding, reasoning, and verification. To bridge this gap, we propose ScholScan, a new benchmark for academic paper rea- soning. ScholScan introduces scan-oriented task setting that asks models to read and cross-check entire papers like human researchers, scanning the document to identify consistency issues. The benchmark comprises 1,800 carefully annotated questions drawn from nine error categories across 13 natural-science domains and 715 papers, and provides detailed annotations for evidence localization and rea- soning traces, together with a unified evaluation protocol. We assessed 15 models across 24 input configurations and conducted fine-grained analysis of MILMM capabilities for all error categories. Across the board, retrieval-augmented genera- tion (RAG) methods yield no significant improvements, revealing systematic defi- ciencies of current MLLMs on scan-oriented tasks and underscoring the challenge posed by ScholScan.
We expect ScholScan be the leading and representative work of the scan-oriented task paradigm.
INTRODUCTION
Enabling multimodal large language models (MLLMs) (OpenAl, 2025a Anthropic 2025
ByteDance Seed Team 2025 Meta 2025 XAI 2025) to conduct comprehensive understanding and generation based on academic literature is the ultimate goal of Deep Research (Comanici al. 2025). and a critical milestone on the path toward artificial general intelligence (AGI) (Ge et al. 2026). With rapid advances. MLLMs are increasingly capable of supporting academic workflows through retrieval, reading, and writing. For example. PaSa (He) al. 2025) can invoke a series of tools to answer complex academic queries with high-quality results, while Google Deep Research (Comanici et al. 2025) is capable of producing human-level research reports based on specific queries.
2023 Morris al. 2024 Phan
However, most of the existing work still follows a search-oriented paradigm, where models retrieve a few relevant passages and reason over local evidence based on pre-specified targets (Gao et al.) 2024 Lou et al. 2025). Such methods are effective for tasks with clearly pre-defined targets, but struggle with researcher-style full-document reasoning and verification (Zhou et al. 2024). To func- tion researchers, models must move beyond reactive question answering and toward proactive discovery of implicit problems.
To fill this gap, shown in Figure 1 introduce scan-oriented paradigm, where models ad- dress queries with absent targets, requiring them to exhaustively scan papers, actively construct
"Corresponding author.

---

Published as a conference paper at ICLR 2026
- c
S can-O
Target Pre-specified: What methodological issues arise from short-interval calcein labeling?
Target Absent:
Assess the Methods section for Measurement & Operationalization issues.
Retrieval
Unonswerable
Retrieval
Page dynamic analysis using inorr-interval calcein labelingindicated Uhat the bone formation rale (BFR), mineral apposition rale (MA R)...
PUO
Page3:... using short- interval calocin labeling...the hone formation rate (BFR))...
one-day calcein labeling interval cannot validly measure MAR or BFR, yet these values are still reported, creating a disconnect between method and
Al
A short-interval calcein labeling protocol does not allow valid measurement of MAR or BFR.
Pages:... analysis of the BFR/BS, MAR. and MS/BS
A comparison between search-oriented and ccan-oriented task paradigms. Unlike the former, the scan-oriented paradigm provides no pre-specified targets, requiring the model actively scan the entire paper, construct document-level evidence view.
Figure 1:
a document-level evidence view, and perform evidence-based reasoning, In contrast to search- oriented tasks that assess the model's ability identify and reason over relevant fragments, scan- oriented tasks emphasize consistency. Instead of relying on pre-specified targets or hints, models must derive all necessary concepts and inferences solely from given documents.
We instantiate this setting via scientific error detection, as it naturally demands discovering non- obvious flaws without target cues, and present ScholScan, a new multimodal benchmark for aca- demic reasoning. ScholScan features the following key highlights:
Scan-Oriented Task Paradigm. In ScholScan, models receive or complete academic papers with target-absent queries, undergoing rigorous saddence-basee reasoning evaluation. The benchmark comprises 715 papers spanning 13 natural science disciplines.
Comprehensive Error Taxonomy. ScholScan covers nine categories of scientific errors through- out the research workflow, including citation and reference errors, rigorously evaluating models' cross-source reasoning abilities.
Process-Aware Evaluation Framework. ScholScan provides fine- grained annotations for both evidence location and reasoning steps, enabling a comprehensive evaluation framework that as- sesses model performance in terms of both process and outcome.
evaluated 15 models across input configurations and retrieval- aurreval-augmennee generation (RAG) frameworks. All models exhibited limited performance, and none of the RAG methods delivered significant improvements. These results highlight the inadequacy of search-oriented frameworks when applied scan-oriented tasks and underscore both the challenges and the potential of enabling MLLMS to perform reliable document-level reasoning over full academic papers.
2
RELATED WORK
2.1
MULTIMODAL LARGE LANGUAGE MODELS
With the rapid progress of MLLMs, the models have evolved beyond perception tasks (c.g.. image recognition and explanation) Liu et al. 2024) toward deep understanding of structured multimodal long documents. Their strengths lie the ability to integrate cross modsl information and perform multi-hop reasoning extended contexts. These capabilities not only valuable for specific
2

---

Published as a conference paper at ICLR 2026
question answering instruction-followiin tasks (uue al. 2024) but particularly well suited to simulating human thought processes and generating explainable reasoning trajectories (Zheng et al. 2023). Consequently, achieving comprchensive understanding of entire documents has emerged as core challenge that MI.LMs are inherently equipped to address.
2.2 DOCUMENT UNDERSTANDING BENCHMARK
Document understanding tasks challenge models to identify the relevant context and perform ac- curate reasoning grounded in that information. Progress in document understanding benchmarks has followed two main axes. Across the input dimension, it has evolved from short to long con- tents, from everyday to specialized domains, and from plain text multimodal formats (Chen al., 2021 Yang et al.) 2018, Mathew et al., 202 Deng et al. 2025 Across the output dimension, has shifted from limited-output formats to more open-ended responses (Pramanick et al., 2024). DocMMath-Evalccmpp..o EvalcompLong (Zhao et al., (224) evaluates numerical reasoning on long specialized docu- ments, while MMMLonnneechh-De (Ma et 2024) builds a multimodal benchmark with layout-rich documents. The recently proposed FinMMDocR ("aan et al., (025) pioneers the integration of sce- nario awareness, document understanding, and multi-step reasoning in financial scenarios. However, comprehensive benchmark that integrates all the above challenges has yet to be introduced.
2.3 ACADEMIC PAPER UNDERSTANDING BENCHMARK
Compared with general documents, academic papers distinguished by their rich domain knowl- edge and logical rigor. Reasoning over papers has emerged as a major challenge in recent research. Some studies request local elements such charts snippets, leveraging their internal complexity, but neglect the need for cross- source integration and domain-specific interpretation within the full document (Wang et al., 2024 Li et al., 2024). Recent studies extends to document-level inputs using image-based formats for real-world reading scenarios (Auer et al. 2023 Yan et al. 2025). However, benchmarks based the QA paradigm face inherent limitations, as they typically pre- suppose the existence of answers and embed explicit cues in the question itself, reducing the need for comprehensive understanding and information organization. Moreover, mainstream evaluation protocols focus on the final outcome, with limited assessment of whether intermediate reasoning is evidentially grounded and logically valid. More examples and analysis are shown Appendix B
3 SCHOL.SCAN BENCHMARK
Benchmark
Mod.
Para.
Eval. Dom.
Document Understanding
DocMath Evalcomptoog
DocMatt-EEalccom.os
T+TD Search
N/A
FinMMDocR
T+MD Search
0
N/A
Othera
MMLLnnBench-Doo
T+MD Search
0
N/A
Physies
LongDoeURI.
T+MD Search
0
N/A
Computer
Science
SlideVQA
T+MD
Search
O
N/A
Balowy 112
Academic Paper Understanding
CharXiv
T+I
Search
0
8
Micattinn
ArXivQA
T+I
Search O
10
MMCR
T+MD Search
O
CS
AAAR-1.0
T+MD
Search
O
CS
ScholScan (ours)
T+MD
Sean
P+O
13
Figure 2: Left: Overview of ScholScan. Right:
Comparison related benchmarks. Mod.: Modal-
ities; Para.: Task Paradigm; Eval.: Evaluation Focus; T: Text; I: Image; TD: Text Document; MD:
Multimodal Document; P: Process; O: Outcome; Dom.: Academic Domain Coverage.
3.1 OVERVIEW OF SCHOLSCAN
We introduce ScholScan, a benchmark designed to comprehensively evaluate MLLMs' ability to de- tect scientific flaws in academic papers under scan-oriented task settings. As illustrated in Figure
3

---

Published as a conference paper at ICLR 2026
3DC
Research Question Definitions
Design & Identifiability
Sampling & Generalizability
Explanation: The design is described as probing both and long-range interactions, yet the paper still claims unique large-q selectivity, creating disconnect.
Explanation: The experiments wee nurrow diabetic muuse substrain, yet the paper generalizes findings to all patients, creating involid popplllto-poplaationn inference.
Explanation: The definition of "actionable variants" shifts acrass vections (LOE 1-5 in Abstract, LOE 1-3 Results), causing ombiguity.
iMeasurement & Operationaliiaation
966
Data Handling Preprocessing
Computation Formulae
Explanation: Feature selection for NSCLC and HCC models was done the full dataset before splitting, causing data leakage. while Discussion falsely claims unbiased validation.
Explanation: The Methods claim 200- fold concentration but 200 subssample is incorrectly tlyyy said to
Explanation: First-harmonts
demodulation is dominated by far-field
back background and cannot produce reported high-quality near-field images.
represent -00 mL instead creating twofold calculation error.
mL.
Inference Conclusions
Referential & Citation Alignment
Language & Expression
Explanation: The data show PGKI
Explanation: Figure! report un LPS dose of 1.5 mg/ks, /g. but Figure 5 reports 15 mg/kg, creating a tenfold discrepancy that makes the actual experimental dose unclear.
Explanation: The paper swaps
promotes EGFR degradation, yet the Discussion claims inhibiting PGK1 as therapy, directly contradicting the result.
C. elegans gene and protein nomenclature (e.g. 'ncc45 vs. 'UNC- 45'), creating technically mislouding references.
Figure 3: Sampled ScholScan examples with 9 error categories, covering the whole process of scien- tific research, each requiring the model to perform thorough cross- evidence- sidencce-based reasoning.
ScholScan spans 13 disciplines in the natural sciences, including physics, chemistry, and computer science, and encompasses
100 subfields such immunology, total synthesis, and machine learning. The benchmark comprises 1,800 questions derived from 715 real academic papers and covers nine major error categories commonly observed real-world research scenarios (see exam- ples in Figure E more examples in Appendix D). These include issues in numerical and formulaic computation, experimental design, inference and conclusion, and citation misuse, among others. Figure SaareeZalso also compares ScholScan with existing document and paper understanding benchmarks.
3.2 DATA CURATION AND QUESTION GENERATION
We curated papers from ICLR 2024 and 2025 as well as Nature Communications collecting public reviews for the former. Questions constructed based two dimensions, where the source is either generated or sampled, and the context is either within-paper or cross-paper.
http:////eaaa........../o///aa/s https:/www.naturr

---

Published as a conference paper at ICLR 2026
Generation. On high-quality accepted papers, prompt Gemini 2.5 Pro to perform coordinated sentence-level edits spanning multiple sections or pages. It then synthesizes composite errors and generates the corresponding question along with an explanation grounded in the edited context.
Sampling. From rejected ICLR submissions and their public reviews, we prompt Gemini 2.5 Pro to extract explicit, falsifiable scientific errors and convert them into questions with initial explanations. Subjective remarks on novelty or writing quality are excluded.
Within- Paper. This setting focuses verifiable facts and internal consistency within single paper, and supports both Generation and Sampling.
Cross-Paper. This setting examines citation consistency across papers. For each instance, Gemini 2.5 Pro receives an accepted paper and one its cited sources, then edits the accepted paper to introduce paraphrases or reasoning errors about the citation. As public reviews mainly address nonfalsifiable aspects, such as appropriateness, all cross-paper instances are constructed exclusively using the generation method.
The detailed prompt templates and specific instructions used are provided in Appendix A
3.3 QUALITY CONTROL AND ANNOTATION
Despite explicit instructions, initial outputs exhibited substantial hallucinations, logical inconsisten- cies. and low-quality questions. To ensure quality, 10 domain experts conducted rigorous annota- tion process. Each instance underwent independent dual review, and disagreements were resolved by a third expert. Among the 3,500 initial candidates, 1,700 were discarded, and 1,541 of the remain- ing were revised, including 535 question rewrites, 1,207 explanation edits, and 1,141 corrections to error categories or metadata. Appendix popendix[ddetails data sourcing and validation protocols.
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Models. benchmark a total of input configurations by feeding academic papers as images or
-Cs-b text using the Tesseract engine (Smith) 2007), covering 15 mainstream models (Yang)
2025 Bai et al. 2025 DeepSeek-Al et al. 2025 Guo al. ct al.
2025
OpenAl et al. 2025)
Evaluation Protocol. Inspired by MMLongBench-Dou (Ma et al.
2024). prompt the models to
generate the necessary reasoning chains from evidence to detected anomalies without constraining the format. format. This design assesses wvidence-based reasoning ability rather than basic instruction
following. For open-ended responses, we use GPT-4.1 (OpenAl (025b) to extract cited evidence and reasoning steps, and quantify alignment with annotated explanations. Human evaluation con- firms high agreement between our pipeline and expert annotations. Human validation results and evaluator robustness checks are detailed in Appendix
Metrics. We define a structured evaluation framework by parsing the model response into tuple:
(a) + (lexist, 1contain: ε, R, n))
(I)
Here, lexist and 1contain are binary indicators for whether output contains any error and includes the annotated target error; ε, R. and & RR are the predicted and gold evidence sets and reasoning chains;
wrefix_match/ R, R*) counts matched reasoning steps; E N is the number of unre- lated errors. 1 contain is if the output contains any predicted error, and otherwise. Based on Y (a), define an end-to-end score S(a)= [0, 1] that combines all aspects of prediction quality:
(i) Error Detection Score. We consider the error detected only if the model identifies the target error:
Stetoction
Lexist
Lpontain
(2)
(ii) Evidence Location Score. Even when the target emror is identified, the cited evidence may be incomplete or noisy. We compute Dice score with squared penalty for over-reporting
2|
+
[ | -0]
E Ex
S Siocation max
-0.8
(3)
max 3 + 1-
max 18
5

---

Published as a conference paper at ICLR 2026
(iii) Reasoning Process Score. Even if the target emror is detected, the reasoning may diverge from the gold chain. We use prefix match to assess reasoning completeness:
9
Sreasoning = R = 0) R 0}
(4)
[
(iv) Unrelated-Enne Error Penalty. Models may list unrelated items to inflate recall at the cost of precision. We penalize this with a rapidly increasing function of unrelated eπror count:
Punrelated.cr (n) 0.9 min(x.2) exp( -0.6 max((n 2,0) 1.5
(5)
(v) Overall Score. The final score for response integrates detection accuracy, evidence quality, and reasoning faithfulness:
S(a) = Sdetection
Slocation
Sreasoning Punrelated.ert
Table 1: Model performance across 9 error categories (scaled by 100). ROD: Research Question & Definitions: DI: Design Identifiability; SG: Sampling & Generalizability; MO: Measurement
& Operationalization: DHP: Data Handling & Preprocessing; CF: Computation & Formulae: IC:
Inference Conclusions; RCA: Referential Citation Alignment; LE: Language & Expression.
Models
Avg.
RQD
DI
SG
MO
DHP
CF
IC
RCA
LE
MILM (Image Input)
Proprietary MLLMs GPT-5
19.2
10.1
9.7
28.2
14.6
26.6
13.8
25.3
25.3
6.9
Gemini 2.5 Pro
15.6
11.9
12.6
35.7
12.3
27.0
4.6
14.7
15.2
7.4
Doubao-Secc -ereo--eee--------11 16-thinking
3.5
22.3
10.2
12.2
10.9
Doubao-Seed-1.5
9.9
3.0
4.4
29.2
4.9
15.0
6.3
17.9
8.0
3.9
Grok 4
4.0
0.0
1.9
16.7
3.2
7.4
0.7
1.9
3.6
0.0
Open-Source LLMs
Llama Maverick
7.3
6.7
8.8
Mistral Small
3.3
2.0
2.0
1.5
1.0
2.2
8.6
Gemma 3 27B Qwen2.5-VL-722
1.7
0.5
2.7
2.3
1.7
1.0
1.0
1.3
2.6
0.0
0.1
0.0
0.7
0.0
0.0
0.0
0.0
0.0
0.2
0.0
OCR LLM (Text Input)
Proprietary LLMs Gemini 2.5 Pro
30.3
21.5
34.2
44.3
27.6
56.6
10.3
28.8
35.6
8.1
GPT-5 Grok
22.5
16.1
21.4
26.0
20.3
36.7
4.7
29.8
30.0
2.6
20.8
7.7
37.4
12.3
34.4
20.0
31.2
Doubao- Seed-1.6-1te 16-tthiniing
15.3
8.2
10.1
24.3
10.1
24.2
6.4
19.2
21.0
4.2
Doubao-Seed-1.6
13.9
5.4
6.9
26.4
10.3
23.6
6.3
20.1
17.5
2.3
Claude Sonnet 4
5.7
3.7
2.5
10.8
4.3
10.3
1.4
8.4
6.6
3.5
Open-Source I.MMs
Qwen3 A22B (Thinking)
16.2
31.9
15.1
23.7
22.3
21.1
DeepSeek-R1
11.4
5.1
11.9
25.4
8.7
22.5
4.7
16.3
9.8
3.5
gpt-oss-1200
7.3
6.3
5.7
18.3
4.9
14.5
1.6
12.5
5.5
0.0
Mistral Small 3.1
6.9
3.0
2.7
5.5
7.0
2.0
8.5
4.0
12.2
3.0
Llama 4 Maverick
2.3
1.5
2.0
4.8
3.0
3.6
5.8
1.6
0.2
Gemma 3 27B
1.6
3.0
2.7
0.2
0.7
7.7
1.0
Qwen3 A22B (Instruct)
1.7
1.2
0.0
2.7
0.4
1.0
0.1
4.3
2.5
1.1
DeepSeek-V3. V3.1 Dwen2.5-VL-772B
1.7
1.2
2.0
1.7
1.0
5.8
0.5
2.2
2.1
0.0
0.2
0.0
0.7
0.0
0.0
0.0
0.0
0.0
0.6
0.0
4.2
MAIN RESULTS
Table
presents evaluation results. Our main findings summarized follows:
Overall performance remains unsatisfactory. GPT-5 achieves the highest average score in the image input group (19.2), while Gemini 2.5 Pro, the best-perforning model in the text input setting, still fails to surpass the 60-point threshold in any enror category. Even the SG category, which yields the best overall performance, nearly half of the models receive single-digit scores. Most

---

Published as a conference paper at ICLR 2026
models perform poorly under the scan-oriented task paradigm and fail to detect any issues in many papers. This challenge is particularly pronounced for open-source models.
Reasoning-enhanced models demonstrate clear advantages. Across both input configurations, beasoning-enhancee variants consistently achieve higher scores. Almost all best-performing mod- els, measured by metrics for both specific error categories and overall performance, fall into this category. In particular, than 10% average scores, with substantial gains observed across all error categories. These results indicate that reasoning-enhance models are better able to simulate the iterative process of extraction followed by reasoning, which is essential for effectively handling scan-oriented tasks and producing higher-qualinn responses.
and Deepseek-R1 outperform their base versions by more
MLLMs face significant bottlenecks in handling long multimodal inputs. Across most error categories, text inputs outperform image inputs. Among the nine MLLMs tested, the average per- formance gap between text and image inputs reaches 4.81 points, highlighting visual processing as key limitation in current MLLM capabilities.
Although overall performance generally weaker, multimodal input remains indispensable. In certain categories such as CF, where OCR-based text extraction leads to substantial loss of formu- laic or tabular content, image inputs outperform their text counterparts. This highlights the essential role of multimodal reasoning and the irreplaceable value of visual information in addressing specific error categories.
4.3 FINE-GRAINEL ANALYSTS
Capability Dimensions. We compute pairwise Spearman correlations between eтror categories across two input configurations (text and image) for the eight evaluated MILMS excluding Qwen2.5-VI.- 72B, as shown in Figure 4 insights:
RQD
pI
MO
DHP
IC
RCA
1.0
ROD
We derive the following
(i) With image input, CF exhibits consistently low cor-
Mo
relations with other error categories, suggesting that the skills required for mathematical reasoning are rel- atively distinct. In contrast, with text input, CF shows moderate correlation with LE, indicating that OCR- flattened formulas lose their structural specificity and interpreted by models in manner natural language. Combined with the overall poor performance on CF tasks, this underscores the unique challenges of this category and the need for targeted improvements.
DIP
CF
RCA
akin
LE
0.5
Figure- 4: Spearman correlation matrix among the 9 error categories.
(ii) Although DI is also related to experimental set- tings, it does not exhibit strong correlations with SG,
MO, or DHP. This indicates that DI primarily emphasizes causal framing and variable identifiability, rather than the procedural understanding of experimental operations.
(iii) OCR severely degrades structured content such as figures and formulas, making questions that depend on multimodal information unanswerable. This diminishes the expression of multimodal reasoning capabilities and artificially inflates inter-category correlations under text input.
Based on the above analysis, we consolidate the original nine error categories, each defined by its objective target. into five core latent skill dimensions evaluated by ScholScan under image input. While each dimension emphasizes the primary competence of its corresponding eиrоr categories, they not mutually exclusive, many questions involve overlapping reasoning abilities.
RQD and DI correspond to research concept comprehension, which requires models to identify the scope and definition of research objectives by integrating contextual cues and prior knowledge. SG, MO and DHP fall under experimental process modeling, which tests a model's ability to re- construct procedural workflows such as sampling, measurement, and data handling. CF captures

---

Published as a conference paper at ICLR 2026
Omission
Hallucination
Evidence Location
Reasoning Step
995
18.13
1000
17:17
16
15-73
16.03
800
790
675
12-
11.32
11.24
600
8.73
400
368
367
290
245
3.48
103
97
2.11
49
36
Gemiai (Image)
Gemini (Teet)
Liama 4 (Image)
Liama (Text)
Qweng-Instruct Instruct
Qwenn Quveng- Thinking
GFT-3 (Image)
GPT-5 (Teat)
Gemini (Image)
Gemial (Test)
Gold Beference
Qweng-
Figure 5: Left: Distribution of omission and hallucination emoos. Right: Average reasoning steps
and evidence locations involved the answer generation, compared against the gold reference.
formal reasoning and symbolic computation, focusing syntactic parsing and numerical logic. IC evaluates causal inference, where models must synthesize dispersed causal evidence to reach sound conclusions. RCA and LE reflect referential alignment and linguistic consistency, which assess the ability to verify citations and maintain coherent expression throughout the document.
Hidden Complexity in Gean-Oriented Tasks. We analyze the reasoning traces of GPT-5 and Gem- ini 2.5 Pro across both imput configurations, focusing on the number of evidence pieces scanned and the reasoning steps performed. As illustrated in Figure even the most advanced models often scan up to times more evidence and execute 3.5 times more reasoning steps than the reference answers, merely approximate correct response, yet they still frequently fail. This highlights the substan- tial hidden complexity inherent in scan-oriented tasks, which significantly amplifies the challenge of successful task completion.
4.4 ERROR ANALYSIS
Omission and Hallucination. Most zero-score cases fall into two categories: either the model fails to detect any errors the actual errors present in the reference answer. We analyze the number of zero-score questions and the proportion of these two failure modes across models, as shown in Figure Stronger models tend to have fewer zero-score cases overall, but are more prone overconfident ballucinations.
paper, or it becomes overwhelmed by hallucinations and entirely overlooks
Fragile Reasoning under Complex Evidence. F how best best tesefrerforing models behave under different numbers of reasoning steps and evidence locations. As reasoning steps increase, both reasoning and overall scores steadily decline, revealing a clear bottleneck in MLLMs' ability to con- struct long causal chains. contrast, variation evidence locations has weaker and less consistent impact. However, this does not imply that multi-evidence questions pose only marginal difficulty. Since the evaluation metric allows partial evidence omissions, more evidence items do not necessar- ily incur large score penalties. Still, heavier evidence loads often require longer reasoning chains, which substantially affect the coherence and completeness of inferred logic. These results highlight the persistent challenge for MIL.Ms in integrating evidence and maintaining logical structure as task complexity increases.
4.5
RAG ANALYSIS
We evaluated 8 RAG methods across both input configurations (Robertson et al. 1994 Chen al.)
2024 Lee al. 2025 Faysse et al.) 20255] Yu et al. 2025; Wang al. 2025 Izacard et al. 2022). Key findings presented below, with detailed results shown in Tables and
The Oracle condition yields significant accuracy gains. Providing gold images alleviates the scanning burden in long-context inputs, increasing the chances of generating correct answers. Al- though overall performance improves, the gains are limited for CF errors and minimal for LE errors. For CF, the sparse formulaic content means gold images offer limited assistance. For LE, the dense text distribution makes even direct access to target regions insufficient for current models to reduce complexity.
8

---

Published as a conference paper at ICLR 2026
GPT-5 (Image)
GPT-5 (Text)
Gamini (tmage)
Gemini (Text)
50
70
50
Reasoning Process Score
60
Overall Score
Overall Score
40
50
30
30
40
30
20
20
20
10
10
25
24
2.
24
Reasoning Steps
Evidence Locations
Figure 6: Model performance trends across reasoning steps and evidence locations (scaled by 100).
Table 2: Overall scores of RAG methods across the 9 error categories (scaled by 100).
Models
Avg.
RQD
DI
SG
MO
DHP
CF
IC
RCA
LE
Text Input (Base Model:
Sive3-TThnkingg
Baseline
17.4
8.9
16.2
31.9
15.1
23.7
5.6
22.3
21.1
2.3
Oracle
24.5
20.6
27.9
43.6
21.3
40.8
7.4
26.9
26.0
1.9
BM25
16.7
9.7
13.7
33,0
17.3
23.8
6.8
25,4
16.5
3.0
Contriever
16.6
18.2
33.7
10.7
20.8
18.5
BGE-M3 M3
11.3
8.6
7.5
24.8
9.1
15.4
5.3
15.6
11.4
1.0
NV-Embed-v2
6.8
4.0
4.0
9.4
6.1
4.9
5.5
5.7
10.0
2.0
Image Input (Base Model:
Llama Maverick)
Baseline
7.0
7.0
7.3
9.4
4.5
4.0
6.5
6.7
8.8
3,0
Oracle
6.5
3.0
4.5
15.6
8.2
9.4
4.9
10.0
4.4
1.4
VRAG-RL
10.9
9.8
11.6
17.8
8.2
11.0
6.8
13.1
10.8
8.1
Co1Qwen2.5
1.2
0.7
0.5
0.0
1.2
0.2
2.7
VisRAG
1.0
2.0
0.0
1.0
0.0
1.0
1.6
1.3
1.2
0.0
ColPali-v1.3
0.8
1.5
0.0
0.5
0.0
0.9
0.5
1.3
1.4
0.0
In consistency-centre scan-oriented tasks, most retrieval-based enhancement methods show mini- mal effectiveness. All embedding models exhibit poor retrieval accuracy. None achieves recall of 50% within the top-5 retrieved items. More critically, performance deteriorates after retrieval, especially for multimodal embedding models, where post-retrieval responses are almost entirely incorrect and scores approach 0.
Table 3: Retrieval performance of RAG methods.
Models
MRR@5
Recall @5
Text Input (Base Model: Qwen3-Thinking)
BM25
0.41
0.48
Contriever
0.31
0.39
NV-Embed-v2
0.30
0.38
Complex embedding model architectures do not yield better performance. Under the text-input set- ting, BM25 achieves the highest retrieval metrics, out- performing Contriever and NV-Embed-v2. Under the image-input setting, although VisRAG shows certain advantages in retrieval performance, its overall score remains comparably low and converges with methods such ColPali-v1.3, Under such cireum- stances, comparisons between retrieval metrics lose their substantive significance. The underlying reason lies the fact that existing embedding models are primarily designed to enhance retrieval performance at the level of semantic relevance. They already struggle with traditional multi-hop reasoning tasks, alone scan-oriented tasks with target suppression.
BGE-M3
0.16
0.21
Image Input (Base Model: Llama 4 Maverick)
VisRAG
0.46
Co1Qwen2.5
0.30
0.35
ColPali-v1.3
0.26
0.31
Reinforcement learning frameworks with visual focus have emerged as leading approaches. Despite being built on a compact 7B model, VRAG-RL consistently delivers improved performance and is the only method that achieves gains under image input following RL optimization. Its en- hanced retrieval sharpens evidence selection, while strong reasoning provides effective guidance during document scanning, The retrieval and reasoning components interleaved in the design, with each stage informing the other in an iterative loop. This tightly coupled interaction contributes the method's superior performance potential.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we introduce ScholScan, a benchmark designed to evaluate the performance of MLLMs scan-oriented tasks that require the detection of scientific errors across entire academic papers. We conduct a comprehensive evaluation and in-depth analysis of mainstream MLLMs and RAG methods. The results demonstrate that current MLLMs remain far from capable of reliably addressing such tasks and that existing RAG methods provide little to no improvement. This high- lights the complexity, integrative demands, and originality of the ScholScan benchmark. Looking ahead, our goal is to develop cannorriented task paradigms suited to diverse academic scenarios and explore new techniques improve model performance on target-suppressed inputs. These direc- tions support the larger goal of advancing MLLMS from passive assistants to active participants in scientific research.
ETHICS STATEMENT
Data Provenance. All data used in this paper constructed by the authors and do not include ternal public or proprietary datasets. The academic papers and author names referenced are publicly available through arXiv and OpenReview.
Annotation Process. A team of 10 domain experts was assembled to thoroughly review all tasks initially generated by Gemini 2.5 Pro. All annotators provided informed consent to participate. To ensure accuracy and neutrality of both model-generater and human-verified content, we employed rigorous multi-stage validation process involving cross-review and third-party adjudication.
Model Evaluation. Evaluation of 15 mainstream models in 24 input configurations was carried out using legally authorized API access through VolcEngine, Alibaba Cloud's LLM services, and OpenRouter.
Dissemination. ScholScan is open source and freely available for academic and non-commereial search. All personally identifiable information has been removed from the dataset and its collection and release comply with the ethical and legal requirements in place at the time of data acquisition.
REPRODUCIBILITY STATEMENT
All results presented in this paper are fully reproducible. To facilitate verification and extension, we provide the complete dataset on Hugging Face, source code and detailed documentation on GitHub. The GitHuub repository includes step-by-step instructions and the exact hyperparameter configurations used in our experiments, ensuring full reproducibility. The retrieval components in all RAG experiments were executed on server equipped with NVIDIA A40 GPUs.
ACKNOWLEDGMENTS
This work is supported by the Beijing Natural Science Foundation (Grant No. QY25345), the Na- tional Natural Science Foundation of China (Grant Nos. 62473271, 62176026), and the Funda- mental Research Funds for the Beijing University of Posts and Telecomnunications (Grant No. 2025AI4S03). This work also supported by the Engineering Research Center Information Net- works, Ministry of Education, China. We would also like to thank the anonymous reviewers and area chairs for constructive discussions and feedback.
REFERENCES
Authropic. System Card: Claude Opus 4 & Claude Sonnet 4. Technical report, Anthropic, May 2025. URL https://wwet anthropic cwm wlw.ude--sysseer.t.... 2025-09-24.
Accessed:
Sören Auer, Dante A. C. Barone, Cassiano Bartz, Eduardo G. Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, Ivan Shilin, Markus Stocker, and Eleni Tsalapati. The sciqa scientific question answer- ing benchmark for scholarly knowledge. Scientific Reports, 13(1):7240, May 2023. ISSN 2045-2322. doi: 10.1038/s41598. -22-898-23-33---22 URL b
s4 021598-023-33600
10

---

Published as a conference paper at ICLR 2026
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu. Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL wtt.hhttps///rrrtt wwwp:/iwww...ovaoooooo.. attpabbss//2222 13923
ByteDance Seed Team. Seed1.6 Tech Introduction, June 2025.
URL https://seed.
bytedance www.coo//eeae..eee ceaemmee//eeee hindseedl__ 6
Accessed: 2025-06-25.
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi-linguality fulliiffucctionality. multi- granularity text embeddings through self- self-knowledge distillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Asso- ciation for Computational Linguisticx: ACL 2024, pp. 2335,, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. http5//ww/2224...nn-g---oo https: //acl antho ogy wrg...rg//o 2024. . f nd rgs- acl 1377
URL
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 3697-3711, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/0222 emnlp-main.300. URL https: http://aaaaaathhhoo antoologoyy........oooooooo org 2rgg/022 2021 emnlp ma mmain..30 3007
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities, 2025. URL htt.:htttts//aa.tt
org/ /tw./////// 06261
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, et al. Deepseek-v3 technical report, 2025, URL
hhttps://arxte org/ abs///bs//// 2412. 19437
Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, and Cheng-Lin Liu. LongDocURL: a comprehensive multimodal long docu- ment benchmark integrating understanding, reasoning, and locating. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1135- 1159, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 8-89 16-5156-5 doi: 10.18653/v1/222 1/202/ acl-long.57, URL httpp::/accaattholl. 2025. acl 1ot.-oong..777
org/
Manuel Faysse Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDE- LOT, and Pierre Colombo. Colpali: Efficient document retrieval with vision lan- guage models. In Y. Yue, A. Garg. N. Peng, F. Sha, and R. Yu (eds.), In- ternational Conference on Learning Representations, volume 2025, pp. 61424-61449, 2025. URL [9984444acc44400000336553-----00
9
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-angmmeted generation for large language models: A survey, 2024. (tRLphtts:///arxttt wwwprxiw...ogaaao. www./ogg..bo..o..o.../ /2312. 10997
Yingqiang Ge, Wenyue Hua, Kai Mei, jianchao ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang. Openagi: When IIm meets domain experts. In A. Oh, T. Naumann, A. Globerson, K. Saenko. M. Hardt, and S. Levine (eds.), Advances in Neural Infor- mation Processing Systems, volume 36, pp. 5539-5568. Curran Associates, Inc., 2023.
URL https://proceedins..eeee wwtp:pppppcppppppppppp facpieerr__eeeepppaeeeee /ttss//ppp////2/22/111 and Benchmarks.
pdf

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseck-rl incentivizes reasoning in Ilms through rein- forcement learning. Nature, 645(8081):633 638, Sep 2025. ISSN 1476-4687. doi: 10.1038 941586-025-09422-- URI. URR.https::/ddo http://i..orggg///// wrg/10. 000.1....003000 1 58 025-09422--22
Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, and Weinan F.. PaSa: An LLM agent for comprehensive academic paper search. In Wanxiang Che, Joyce Nahende, Eka- terina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1 11663-11679, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 10.18653/v1/2225..ae wtl.:255.cll-ooo.... URL ac1-long.572
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learn- ing. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URI. https:
/oopenzeview.net
sN1pxi7bb
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms gen- eralist embedding models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, pp. 79310-79333, iclr, www.. files/ https//aeeeseeeeeeeeeeeeaaeeeeee 1894 89888999119e6fee
2025. URL org.://22.r.r.ate.eeeeeeereeeeer
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multi- modal ArXiv: A dataset for improving scientific comprehension of large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14369-14387, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653v1ll 1o-4.acl-oong.711 URL https: ac antholooohgo org 2024 acl- 775/
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL httpp::/
llaaa-vll sava-vl.githuu. io ht/poolg/////a// 30- llava-neexe next/ Accessed: 2025-05-13.
Renze Lou, Hanzi Xu, Sijia Wang. Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang, Jihyun Janice Ahn, Hongchao Fang, Zhuoyang Zou, Wenchao Ma, Xi Li, Kai Zhang, Congying Xia, Lifu Huang, and Wenpeng Yin. AAAR-1.0: Assessing Al's potential to assist research. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd In- ternational Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 40361 40383. PMLR, 13-19 Jul 2025. URL https: httpss::/ppocceednnng///o mmlr. Dress/v267/ ftess/2267/oo25c.hhm
Yubo Ma. Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangmingg Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. Mmlongbench--doc: Benchmarking long-context document under- standing with visualizations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems. vol- ume 37, pp. 95963-960111 Curran Associates, Inc., 2024. doi: 10. 10.522021099017330441 neur wttppppppccppppppppppppppppppeee http::/aeepppppe.eeeeepee/////// wtt.ff/s/ppperr/222/11111
URL https::/proceee///
becee32990ffee0111444333toooooe0 Datase and Benchmarks Track.pd5
Datasets
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. In 2021 IEEE Winter Conference on Applications of Computer
Vision (WACV), pp. 2199 2208, 2021. doi: 00110099AAC44466022200.000000 https: /ttp:///pppcccccsscmm thecvf. com/ content http:///.....ttoeeee//////////// DocVQA BaccoA___Dataset....
URL
Dataset oot_.oooooooo.ecctettttt____tt Taaemmeet____aas_____________ WACV 2021 111021_apeerppee
12

---

Published as a conference paper at ICLR 2026
Meta. Llama 4 Model Cards and Prompt Formats, September 2025. URL https:
www./o/oo///oooooccoccooo..eeeee cessed: 2025-09-24.
7/www. ama com/ does fbossmodel-catee promoo wow.aoooooooooos///o/ooooo---oet www.eooooottaaaaaaaae
Ac-
Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Alek- sandra Faust, Clement Farabet, and Shane Legg. Position: Levels of AGI for operationalizing progress on the path to AGI. Forty-first International Conference on Machine Learning, 2024. URL htkl.httts:://peeneeeee httpe/wwwwee//oerrrn//e// (idormmm22dd00f2253333
net forum?
OpenAL GPT-5 System Card. Technical report, OpenAI, August 2025a. URL https://edn. Accessed: 2025-09-24.
openai Satpmaaa.ccoomptt----------
card. c
OpenAl. Introducing GPT-4.1 in the API. Technical report, OpenAl, April 2025b. URL https: Accessed: 2025-04-14,
OpenAl, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, et al. spt-oss-1209 120b & gpt-oss-20b model card, 2025. URL
pattps://arivy org/ abs /tsp/////... 2508 10925
Long Phan, Alice Gatti, Nathaniel Li, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausen- loy, Oliver Zhang, Mantas Mazeika, Dan Hendrycks, et al. A benchmark of expert-level aca- demic questions to assess ai capabilities. Nature, 649(8099): 139 1146, January 2026. ISSN 1476-4687. doi: 10.1038/s411866 086-0225-0992--4. URL http: dx. doi www..oogg........ s41586 025 09962-4
Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. Spiqa: A dataset for multi- modal question answering on scientific papers. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 118807-118888 18833. Curran Associates, Inc., 2024. doi: 10.52202/079911-33773
URL https: /ttppe///roo www.ainnggneuuursre wwtpppp..cppppppppppppppp. http://rriileepppppppeeeeee htt.aa/prr...aaepaa////////////
Track.pff
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulian and Mike
Gatford. Okapi at tree-3. In Text Retrieval Conference, 1994. URL https://api.
commantisccoolas :o
R. Smith. An overview of the tesseract ocr engine. In Ninth International Conference on Doc- ument Analysis and Recognition (ICDAR 2007), volume 2, pp. 629-633, 2007. doi: 10.1
URL httpss://eeexpporee leee. org/ sttappstttttttpp isp?
Ep wer-aanuume--ree 437669 4376991
Zichen Tang, Haihong E, Rongjin Li, Jiacheng Liu, Linwei Jia, Zhuodi Hao, Zhongjun Yang, Yuanze Li, Haolin Tian, Xinyi Hu, Peizhi Zhao, Yuan Liu, Zhengyu Wang, Xianghe Wang, Yiling Huang. Xueyuan Lin, Ruofei Bai, Zijian Xie. Qian Huang, Ruining Cao, and Haocheng Gao. Finmmdocr: Benchmarking financial multimodal reasoning with scenario awareness, document understanding. and multi-step computation, 2025. htt.:ttss//axx...ppppp//
24942
Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang. Pengjun Xie, Fei Huang, and Feng Zhao VRAG-RL. Empower cision-errceptio-beee based RAG for visually rich information understanding via iterative reasoning with reinforcement learning. In The Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025. URL https:// openreview net forum? id EeAHhNwXPV
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. Charxiv: Charting gaps in realistic chart understanding in multimodal 1lms. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Informa- tion Processing Systems, volume 37, pp. 113569 113569-113660 113697. Curran Associates, Inc., 2024. doi: files/
10.52202/0790017-3660
URL https: http.ppss///poeeeeese....oooooo.
and Sea__Bennhmarkss.cm Fra
13

---

Published as a conference paper at ICLR 2026
XAI. Grok 4 Fast Model Card. Technical report, xAI, September 2025. URL htttttt::/// //dtaa.////aaxxx data.x. ai/ -025-00--9------ -ost-----------eee------------e- card. pdf Accessed: 0925-09-19.
Dawei Yan, Yang Li, Qing-Guu Chen, Weihua Luo, Peng Wang. Haokui Zhang, and Chunhua Shen. Mmer: Advancing visual language model in multimodal multi-turn contextual reasoning, 2025.
URL httptt:/psaxxtttt org/ atsp//bs//2/000 18533
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL http:: /ttp://aaavvvv.... org/ 2505/2505.00 09388
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369- 2380, Brussels, Belgium, Oetober-Novemiln 2018. Association for Computational Linguistics. doi: N..18653/v1/D1S
Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. Visrag: Vision-based generation on multi-modality documents. In Y. Yue, A. Garg, N. Peng. F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, pp. 21074-21098, ngs iclr wttp:eccepppppppeeeeeeeeeeeeeee otg.aa/ceeeeeeteeeeee---rr---rae
aariivell-aumentte
2025. URL
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and rea- soning benchmark for expert agi. In 2024 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9556-9567, 2024. doi: 10.1009CVPR3223..222.000 thecvf com/ http:///ooaeeeee///////////eeeee
URL
MMMU A wilsssiveMuutii-oe
watee_Multmmmaaaa wawaaaannnnneettannn.ooooo Beasoniin_BBencharkk for _0PR__ Cew___222________2222222222 pdf
Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. DocMath-eval: Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 16103-16120, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1...8653/11/222.acllho 20t4g22224.aao
URL
8527|
Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct ofan--of-thought prompting for multimodal reasoning in language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 5168-5191. Curran Associates, Inc., 2023. URL http:://prooeeelng..errrio...... http://pppeeelleeeep////222/// faleee/0000cceeeeeee0oeeeeeeeeee Pww.-aaappr--ooreeeeeeeeeeeeee
Ruiyang Zhou, Lu Chen, and Kai Yu. Is I.M a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu- ation (LREC- 2024), pp. 9340 9351, Torino, Italia, May 2024. ELRA and ICCL. URL www.:111re....m........ 816/
14

---

Published as a conference paper at ICLR 2026
Appendix Contents
Prompt Templates
17
A.I
Within-Paper Generation Prompt
17
A.2
Within-Paper Sampling Prompt
20
A.3
Cross- Generation Prompt
22
Extractor Prompt
25
A.5
Evaluation System Prompt
27
Examples from Existing Datasets
28
B
DocMath-Evall
28
B.2
MMLongBench-DDe Doc
29
B 3
FinMMDocR
30
B.4 4
LongDocURL
B.5
SIi
32
B.6
DocVQA
33
B, 7 CharXiv
34
B.8 ArXivQA
35
B.9 SPIQA
36
B 10 MMCR
37
ll AAAR-1.0
38
Dataset Annotation and Construction
39
Data Sourcing and Quality Control
39
Annotation Statistics
Annotation Examples
40
C.3.1
Case Discard Directly
40
C.3.2
Case 2: Modify Question
41
C.3.3
Case 3: Modify Explanation
42
D Examples from ScholScan
43
D.1 RQD (Research Question and Definitions)
43
D.2 DI (Design and Identifiability]
44
D33 SG (Sampling and Generalizabilitt)
45
D.4
MO (Measurement and Operationalization)
46
D.5
DHP (Data Handling and Preprocessing)
47
D.6
CF (Computation and Formulac)
48
D.7
IC (Inference and Conclusions)
49
D.8
RCA (Referential and Citation Alignment)
50
15

---

Published as a conference paper at ICLR 2026
D.9 LE (Language and Expression)
51
Human-Machine Consistency Evaluation
52
Hyperparameter Sensitivity Analysis
53
Use of LLNS
54
16

---

Published as a conference paper at ICLR 2026
A PROMPT TEMPLATES
A.1 WITHIN-PAPER GENERATION PROMPT
Within-Paper Generation Prompt
You will receive high-quality, already accepted scientific paper as a PDF. Working only with the PDF itself (and any appendix embedded in the same PDF), edit specific textual spans to inject one or more errors chosen only from the taxonomy below, such that the errors hard yet clearly identifiable by a professional reviewer reading the PDF alone.
Error Type (
Research Question & Definitions Definition: The core construct/appothesss/vriis is insufficiently or inconsistentll defined (conceptual vs operational), leaving the estimand ambiguous.
Design & Identifiability
Definition: Given a clear estimand, the design violates structural identification conditions the effect is not identifiable even with infinite data and perfect measurement.
Sampling ing & neneralizabilic Definition: The sampling frame/pracessscoopsittoo cluster/power setup does not support valid or stable sample-toopulation claims.
Measurement operationalization.
Definition: Measures/mmmipulationn lack feasibility/ sellaaoiltty/valldttttiiiiin
observed variables systematically diverge from the intended construct/ treatment.
Data Handling Preprocessing
Definition: Pipeline choices in missing handling, joins/ keys, temporal splitting, feature construction, or partitioning introduce bias (inel. leakage unit/ scale conflicts).
Computation Formulae
Definition: aesthmeticc/aggebr///eeeee
errors (totals/ ratios, unit conversion, CI vs point estimate, p-value vs label, symbol reuse, undefined variables, dimension mismatch).
Inference & Conclusions
Definition: Interpretations or causal statements exceed what nethods/date support, contradict the shown staristiies/aables/catttooo Referential Citazion Alignment
Definition: Contradiction about the same quantity/term across text, tables, captions, appendix within the paper.
Language Expression
Definition:
ambiguities
that affect meaning or domain-critical term
consistency (not cosmetic typos)
17

---

Published as a conference paper at ICLR 2026
Within-Paper Generation Prompt (Continued)
Global constraints (must comply)
1. Each error must map to exactly one primary category in the taxonomy. Do not mix causes.
2. Each error must involve more than 2 micro-edits (each edit
< 20 English words) spread across distinct pages or paragraphs.
3. If an edit would create immediate contradiction in the you may add shadow patch (es) for the same error to keep the text natural (still counted as edit locations).
same
4. Independence across errors (per-copy generation) Generate each error on a separate copy of the original PDF Different errors must be logically and operationally independent:
No progression variant relations: stricter/lcoser version, superset/subset or minor wording variant another error.
must not be
No anchor reuse: do not target the same sentence/caption table cell or reuse the same old_str str (or a near- duplicate paraphrase) across different errors. Applying any single error isclation to the original PDF must still yield detectable, clearly categorizable error according to the taxonomy.
5. Every error must be supportable using text inside the PDF. Do not rely on external supplementary files or prior knowledge.
6. Design as difficult as possible but clean errors. Prefer edits that force cross- cress-checking between two spots (e.g., Methods vs Results). Avoid trivialities. Edits must remain locally plausible and not advertise themselves via obviously artificial phrases (e.g., avoid contrived tokens purely added to be detectable).
7. 'No cosmetic issues'' applies except for (Language Expression). For I, edits must affect meaning or domain- critical terminclogy (e.g., ambiguous phrasing, inconsistent technical terms). Pure typos, punctuation tweaks, layout nits are not allowed.
8. Do not edit titles, author lists, bibliography entries, equation numbering, figure images, add new figures/
9. Frame each question as a neutral imperative that asks for a decision about a specific condition, using (but not http://teemmmeeuuuggg////aaaeeee
limited to) whether.... Do not presuppose an outcome or use sucgestive intensifiers (e.g., clertly/obvooudl///keek/// suspicious as examples).
18

---

/* Add 1-2 more locations; each location 20 words changed. Shadow patches for local coherence count as locations. Published as a conference paper at ICLR 2026
Within-Paper Generation Prompt (Continued)
10. Output English-only and strictly follow the JSON schema below. Do not include any additional text outside the JSON:
"id": "1-based integer as string" modiify"::
"location"; "Page number + short unique nearby quote 315 tokens).",
"old_str" "Exact original text from the PDF (verbatim) .",
" "Edited text after your change."
1, */
"question"; ; "One neutral audit-style task (1-25 words) ", "
in 2-4 sentences why a reviewer can detect this error from the edited PDF alone.", "Type": "Name the primary category (e.g., Inference Conclusions)... "
/* More Errors */
19

---

Published as a conference paper at ICLR 2026
A.2 WITHIN-PAPER SAMPLING PROMPT
Within- Sampling Prompt
will receive paper and the weaknesses mentioned in its peer-review comments. Your task is, based only on the content of that PDF, to sample from the review comments and verify possible errors related to the categories below, and for each confirmed roef or highly plausible error, generate one question and one explaration.
Error Type (fixed)
Research Question & Definitions
Definition: The core construtt/uppohhsiiaavaibbbl is insufficiently or inconsistentlly defined (conceptual operational), leaving the estimand ambiguous.
Design & Identifiability
Definition: Given a clear estimand, the design violates structural identification conditions so the effect is not identifiable even with infinite data and perfect measurement.
Sampling & Generalizability Definition: The sampling trame/process/ooopsiiee cluster/power setup does not support valid or stable sample+toopulation claims.
Measurement & Operationalization
Definition Measurres/maniiplatiioo lack feasibility/ httlaallltt//alllitttttiiiiii so observed variables systematically diverge from the intended construct/ treatment.
Data Handling Preprocessing
Definition: Pipeline choices in missing handling, joins/ keys, temporal splitting, feature construction, or partitioning introduce bias (incl. leakage or unit/ scale conflicts).
Computation Formulae
Definition: httpl//eeii//lggbr//etaaeaee
errors (totals/ ratios, unit conversion, CI point estimate, p-value vs label, symbol reuse, undefined variables, dimension mismatch).
Inference & Conclusions
Definition: Interpretations or causal statements exceed what methods/date support, or contradict the shown statisstict/talles/ccpiioo
Re Referential & Citation Alignment
Definition: Contradictions about the same quantity/term across text, tables, captions, appendix within the paper.
Language Expression Definition:
ambiquities
that affect meaning or domain-critica term
consistency (not cosmetic typos)
20

---

Published as a conference paper at ICLR 2026
Within-Paper Sampling Prompt (Continued)
Global constraints (must comply)
1. Output only the specified categories; even if other error types appear in the reviews, do not output them.
2. Sample first, then verify: extract candidates from the review comments, then confirm them in the PDF. If you cannot locate supporting anchors in the PDF (page number plus phrase/label), do not output that candidate.
3. Questions must be reutral and nen-leading: use an "audit task decision" style, avoiding yes/no bias.
4. Independence: each question must target different figure different textual anchor; no minor variants of the same issue.
5. Evidence first: the explanation must cite locatable anchors in the PDF (page number + original phrase/caption You may mention key short phrase from the review a clue, but write the question and explanation in your own words
6. Language format: both question and explanation must be
in English; output JSON only, with no extra text.
7. Quantity: sort by evidence strength and output up to 5 items; none qualify, output an empty array [].
Output JSON schema:
"id": "1",
"question"; "Audit y-axis baselines and possible axis breaks in Figure 2; decide presence/absence and cite evidence.",
"explanation:: "The review flags possible exaggeration in Fig.2. In the baseline'), the y-axis starts at 0.85 with a break, magnifying small differences; panels use different: ranges."
(p.6, caption "Ferformance
"Type": "Visuaiiiiiiiiiiii Presentation Bias"
21

---

Published as a conference paper at ICLR 2026
A.3 CROSS-PAPER GENERATION PROMPT
Cross- Paper Generation Prompt
will receive two PDFs: "focus paper" and a cited evidence paper" S (exactly one pair per run). Edit only P with textual changes so that P's statements about become incompatible with or unsupported by 8 along one more dimensions (direction, scope, condition, metric, unit, version, protocol). Do not modi fy S. Each error must be detectable by a professional reviewer who reads both PDFs.
Error Types (fixed)
Research Question & Definitions (A)
Definition: The core construct/uppoohhsis//rriait is
insufficiently or inconsistently defined (conceptual vs operational), leaving the estimand anbiguous. Design & Identifiabilie (B)
Definition: Given a clear estimand, the design violates structural identification conditions so the effect is not identifiable even with infinite data and perfect measurement.
Sampling & Generalizability (C)
Definition: The sampling /
cluster/power setup does not support valid or stable sample-populationn claims.
Measurement Operationalization (D)
Definition: Deasures//maipplatte
lack feasibility/
veliadiitty/valldttttiiiiii
so observed variables
aystematically diverge from the intended construct/ treatment.
Data Handling Preprocessing (E)
Definition: Pipelire choices in missing handling, joins/ keys, temporal splitting, feature construction, or partitioning introduce bias (incl. leakage unit/ scale conflicts).
Computation Formulae (F)
Definition: Arithmetic/ haametticc/leebrr/nttttiio errors (totals/ ratios, unit conversion, CI point estimate, p-value vs label, symbol reuse, undefined variables, dimension mismatch).
Inference & Conclusions (G)
Definition: Interpretations or causal statements exceed
what nethods/date support, or contradict. the shown
statistiics/cablles/appiio
Referential Citation Alignment (H)
Definition: Contradictions about the same quantity/term
across text, tables, captions, appendix within the paper.
Language Expression (T)
Definition: Terminology/ httpll////appaalaaaiiooooo/////e that affect meaning doma comaiticria term
ambiguities
consistency (not cosmetic typos)
22

---

Published as a conference paper at ICLR 2026
Cross-Paper Generation Prompt (Continued)
Global constraints (must comply)
1. Edit P only, never S. Do not add/remov references, figures, or external links.
2. Micro-edits, not rewrites.
3. Cross-paper verifiability. For every error, your explanation must quote >=1 anchor from P and >=1 anchor from $ (page number
short verbatim span, 10-20 words) that, together, demonstrate the conflict or lack support.
4. Local coherence. The edited text in P must read naturally in context; the contradiction should emerge only when P is compared with S.
5. Independence across errors. You may output multiple errors but generate each on an independent copy of P. Errors must not reuse the stzeengthen//vaaee///nseeeeteete
P anchor be minor variants (no paraphrases).
6. PDF-only evidence. Rely solely on content present in the two PDFs. No outside knowledge or materials. Scope exclusions. Do not edit titles, author lists, bibliography entries, equation numbering, LaTeX commands, or figure pixels; do not add new figures/tables/ references.
8. Type must be a single letter in C,
9. Inter-paper note. "Inconsistency" describes the evidence relation (P vs S). The Type must reflect the primary upstream cause (e.g., B/C/D/P/C/I/A)
Quantity & ordering. Sort by evidence strength and output up to errors; if none qualify, output an empty array []. Language & format. Output English-only JSON exactly in the schema below; include no extra text.
10. Multiple mentions of S. The evidence paper $ may be cited multiple times in P. You may (1) introduce a single error spanning >=2 of those citation points, or (ii) generate multiple independent errors, each based on a different citation of S.
23

---

Published as a conference paper at ICLR 2026
Cross-Paper Generation Prompt (Continued)
output JSON schema:
"id": "1-based string",
"modify":
"location": "P: page short nearby quote (<=15 tokens) ",
"old_str": "Exact original text from (verbatim) .",
"new_str": "Edited text in P after your change."
/* could add 1-2 more locations +/
"question": "One neutral audit-style task (1-25 words) for checking P against S.", "explanation": "Diagnostic voice: state that the edited P now contains an error of Type <letter>, and substantiate it with anchors from both PDFs. Include page numbers and shert verbatim quotes from and s showing the mismatch (e.g., scope/condition/metii untt/veersion/protooo))) Do not describe how to create the error; assert and evidence the error present.", "Type": "AIBICIDIE/FIGIA"
24

---

3. Existence decision (binary correctness only): asserts an error exists, Published as a conference paper at ICLR 2026
A.4 EXTRACTOR PROMPT
Extractor Prompt
You will receive three inputs:
Q: the open endeeended question;
E: the gold explanation (describes exactly one error; extra details still belong to the same single error);
A: the model's answer to be evaluated.
Your job is to extract counts only and output single JsON object with the exact schema below. Do not compute any scores. Do not add fields.
Core selection rule (multiple errors in A)
1. Parse E into a single gold error (the "target error")
2. From identify how many distinct error claims are made. Cluster together mentions that support the same error multiple locations for one error are still one error)
Let the gold existence be else 0.
Let the predicted existence
if A asserts any error, else 0 (e.g., states error).
Set existance 1 if predicted existence equals gold existence; otherwise set existance 0.
4. If existance 0: set contains tatttainn_targe_errro location and reasoning counts to 0; and set
= 0; set all
unrelated_errors to the total number of distinct error claims in A. Then output the JSON.
5. If existance = 1:
the gold existence is determine whether contains the target error (match by the main error idea in E: category /intent/scope; treat E's subpoints as the same error). If yes, set contairs_target_erre and reasoning only for the target error. Count all other error claims in A as sarelaaed_errort If no, set reasoning counts to 0; set unrelated_errors to the total number of distinct error claims in A. If the gold existence is 0: set contains_target_erra set all location and reasoning counts to 0; set unrelated_errors
= 1 and compute location
= 0; set all location and
0; the total number of distinct error claims in A. (These negative items are for binary accuracy only; they are not used for detailed scoring.)
Matching guidance (A error + target error): match by the main error idea in E (ettagory/iteett/scope)
not by wording. Treat E's subpoints as part of the same single error. Prefer the best matchngg cluster in A; if ties, choose the one with stronger alignment to E's core claim.
25

---

Published as a conference paper at ICLR 2026
Extractor Prompt (Continued)
Counting rules
Location (for the target error only when existance=1 and containsstargetterroo-l] gold_steps: number of unique error locations described in after normalization and deduplication)
nit_steps: number of predicted locations in that match any gold location for the target error.
extra_steps, number of predicted locations target error that do not match any gold location.
for the
Reasoning (for the target error only when existance=1 and saraans_ttrget_errrri
Convert E into canonical set or ordered chain of reasoning steps for the target error. geld_steps: total number of such steps. reached_steps:
single-chain tasks: length of the longest valid prefix A along the gold chain;
multipathhpparalel tasks: size of the intersection between A's steps and the gold step set (or the maximum across gold paths if multiple are defined). mi missing_steps: gold_steps reached_steps (non-negative integer).
Unrelated errors:
unrelated_errors: number of distinct error claims in A that are not the target error (0 if none).
Output JSON schema (return exactly this JSON; integers only):
"existance"; 0, "contains tarpaeess__rgee_errrrrr 0,
"location";
steps=' 0, "hit_steps": 0, "extra_stepp":;
}, "reasoning": "gold_ steps": 0, Preached_stepst 0, "missing_steps";
"unrelatee_errors"""
26

---

Published as a conference paper at ICLR 2026
A.5 EVALUATION SYSTEM PROMPT
Evaluation System Prompt
You are a neutral, careful academic reviewer. You will
receive an open-ended question and the paper content. The paper may or may not have issues related to the question Do not assume there are errors. If the question is about citations, you will be given a citing paper and cited paper; evaluate only the citing paper for possible issues and use the cited paper only as the reference for comparison. Write in natural prose with no fixed template
Rules:
Speak only when sure. State an error only if you are confident it is real error (not mere weakness). Stay on scope. Discuss only what the question asks about. Evidence completeness. For every error you state, list all distinct evidence cues you are confident about from the PDF. Include plain identifiers (figuree/tablessectiinl) aquation/citation) or quotes. Avoid redundant repeats of the exact same instance; include all distinct locations needed to support the error.
Be clear and brief. Use short, direct sentences. No metaphors. No fancy wording.No guesses outside sources. Do not invent figures, tables, equations, citations, or results.
Report as many distinct, well well-supported errors as you can within scope. If none are clear, write exactly: "No clear issue relevant to the question." and nothing else.
27

---

Published as a conference paper at ICLR 2026
B EXAMPLES FROM EXISTING DATASETS
B.1 DOCMATH-EVAL.
DocMath-Eval
Question ID: complongg-esstiin-300
Question: What is percentage of total offering cost on total amount raised in the IPO if total offering cost is $14,528,328 and each unit sold is $10?
Context Modalities: Text Text Document
Offering costs consist of legal, accounting and other costs incurred through the balance sheet date that are directly related to the Initial Public Offering. Offering costs amounting to $14,528,328 were charged to sharcholders' equity upon the complction of the Initial Public Offering.
Pursuant the Initial Public Offering July 20, 2020, the Company sold 25,300,00 Units, which includes full exercise by the underwriter its option purchase an additional 3,300,000 Units, a purchase price of $ $10.00 per Unit. Each Unit consists of one Class A ordinary share and one-half of one redecmable warrant ("Public Warrant",, Each whole Public Warrant entitles the holder to purchase one Class A ordinary share exercise price of $11.50 per whole share (see Note 7).
Covered Areas:
Only Mathematics
Cross-Evidence Reasoning:
Limited
Task Paradigm:
Search-Oriented
28

---

Published as a conference paper at ICLR 2026
B.2 MMLONGBENCH-DE Doc
MMLLngBenchh- Doc
Document ID: afe62009beac86c11227788 ld396407.pdl
Question: How much higher was the proposed dividend paid (Rupees lacs) in 2002 compared to 2001?
Context Modalities: Text + Multimodal Document
SHARPHO DER SEFERENCER
Paymoum oriuad ter Coaetice troeer
raaiiig Varcul Praeeti memoed
foom LuId,
Connec to Concery soremre Contection unding
3016
144
88118 370
18:725
Avad 200
ingoul namaad way nee Moery bodioe
reporie ser dask alfiee
Covered Areas:
Limited (7 areas)
Cross-Evidence Reasoning:
Limited
Task Paradigm:
Search-Oriented
29

---

Published as a conference paper at ICLR 2026
B.3 FINMMDOCR
FinMMDocR
Question ID: test-41
Question: Based on the detailed financial forecast tables provided at the end of the report, analyze the company's working capital management efficiency related to its customer collections for the fiscal year 2025. Using the year-end balances of (Accounts Receivable) for 2024 and 2025 calculate the average balance for 2025, and the corresponding (Total Operating Revenue) for 2025, calculate implied average collection period for receivables during 2025 (assume 365 days a year, round to one decimal place, unit: days).
Context Modalities: Text + Multimodal Document
EXLL
22 E.SAPAREE
Covered Areas:
Limited (Finance Only)
Cross-Evidence Reasoning:
Full
Task Paradigm:
Search-Oriented
30

---

Published as a conference paper at ICLR 2026
B.4 LONGDOCURL
LongDocURL
Question ID; vrve-geminilS_pppe 1_ppro__066660 160166700-771
Question: What was the total fair value options that vested in 2016, 2015. 2014, millions Canadian dollars?
Context Modalities: Text + Multimodal Document
tallung aurredos aoti ophor
DIENC SOSS AOO DOR
01
Comalite henil
Covered Areas:
Limited
Cross-Evidence Reasoning:
Limited
Task Paradigm:
Search-Oriented
31

---

Published as a conference paper at ICLR 2026
B.5 SLIDEVQA
SlideVQA
Question ID:
Question: How much difference in INR is there between the average order value of CY2013 and that of CY2012?
Context Modalities: Text Multimodal Document
KEY FINDING
Executive summary
Key findings
Increating average
Average order values climbing up rapidly
INR 3,600
year there signfficant jump avarage arder vo be as there was a penecretion new catagories jewellery. home détor etc. Also, users are becoming more comforteble buying higher priced items online.
INR 1,000
CY2012
CY2013
CF2015P
Fashion Foetwear Accessories GMV
category doubled year
Last year was the rise of the fast lon category-
fashion e-commeence
doubled sirce 2012.
2. G ven the younn demogrrppicc wh ich is shopping for looks online - we extimate that this category will see and val electronics and mobile category in GMV.
and increasing choice 400% grovth the
100%
$276
Accel estrestes induarry sauoe
Covered Areas:
Limited
Cross-Evidence Reasoning:
None
Task Paradigm:
Search-Oriented
32

---

Published as a conference paper at ICLR 2026
B.6 DocVQA
Doc
Question ID; 24581
Question: What is name of university?
Context Modalities: Text + Multimodal Document
UNIVERSITY OF IIORRIA SAN
To
11..000 L.B.2 Time 2004
Datee
WHILE YOU WERE OUT
Walaaaed 45S 8056
Ms.
Iriieee Clirreo
From
(AT Peleghoned
gchone again
Please piaane Rudb
C ClCaee see
Will comn again
ls Orograme Cermmtttee tha It will biote le e weel & morel (1983) Ai then litter leeff Phomm nomed to ee el (lrmm))
MESSAGE
Taken by Miss
1463 iilpes HENNE inc cuiiie ths viaa eddutacc/66o882222
Covered Areas:
Limited
Cross-Evidence Reasoning:
None
Task Paradigm:
Search-Oriented
33

---

Published as a conference paper at ICLR 2026
B.7 CHARXIV
CharXiv
Question ID: 2004.10956 Question: Which model shows a greater decline in accuracy from Session to Session in the 5-way full-shot scenario?
Context Modalities: Text
Image
50
40
30
20
S-way
S-way
BRILY
Oan--Ai
Covered Areas:
Limited (8 areas)
Cross-Evidence Reasoning:
None
Task Paradigm:
Search-Oriented
34

---

Published as a conference paper at ICLR 2026
B.8 ARXIVQA
ArXivQA
Question ID: physics- 8049
Question: Based on the top-right graph, how would you describe the behavior of P(z) as z approaches zero?
Context Modalities: Text Image
0.2
0.15
log 1 [p(())]
-2
p(z)
0.1
0.05
-6
0
-40
-20
0
20
40
-40
-20
0
20
40
z= a
z=a
a
z.rms
z,rms
0.2
0.1
2
p(z)
0.5
-011
-0.2
-40
-40
-20
20
40
-20
0
20
40
Z=a
a
Z=a
a
2.rs
2,rms
Covered Areas:
Limited
Cross-Evidence Reasoning:
None
Task Paradigm:
Search-Oriented
35

---

Published as a conference paper at ICLR 2026
B.9 SPIQA
SPIQA
Question ID: 1611 104684v1
Question: What is the role of the knowledge gates in the KEHNN architecture?
Context Modalities: Text
Image
BIGRU
Pooting
M
BIGRU
CNN
MLP
Maerioes
Covered Areas:
Limited
Cross-Evidence Reasoning:
None
Task Paradigm:
Search-Oriented
36

---

Published as a conference paper at ICLR 2026
B.10 MMCR
MMCR
Question ID:
Question: Which module's weights are frozen?
Context Modalities: Text + Multimodal Document
He sino Leara and Renor:: Explertin the Craa medil Sumuntic Carrokatienn Langragr-geleed HOT delnctos
Vickes
Santhus comttolluayyye
Calo
Soathau
Sep 2023
II-NVv22011 11522y22 (csCV)
Abtract
1. Iamoelaciee
rquaee ahrecdl
Aomed
hai momu de are your ahooser me nou
putlin Dhu aathiee waeting
bortd to Ii Ber ccuppe 16 Dook coest
Anale youy my offe divermeee
seteed sage mncice dhe
del desoaa lonpeg loeiing dene
ane the cround the por
roloce sae perio
per fenure Fer mode
Covered Areas:
Limited
Cross-Evidence Reasoning:
Limited
Task Paradigm:
Search-Oriented
37

---

Published as a conference paper at ICLR 2026
B.11 AAAR-1.0
AAAR-100
Question ID: 1902.00751
Question: What experiments do you suggest doing? Why do you suggest these experiments?
Context Modalities: Text Multimodal Document
Layer Norm
Layer
O
O O
Transformrr
Layer
Adapter
2x Fewd-forward
layer
Nonlinarity
Layer
Feedforward down-projct
Fuod-forward
0 O
Muli-haaded
attortion
Covered Areas:
Limited
Cross-Evidence Reasoning:
Full
Task Paradigm:
Search-Oriented
38

---

Published as a conference paper at ICLR 2026
C DATASET ANNOTATION AND CONSTRUCTION
C.1 DATA SOURCING AND QUALITY CONTROL.
The defective academic papers in our dataset are curated from three primary sources:
We synthetically injected nine types of errors into papers accepted at ICLR and Nature Commu- nications.
For papers rejected by ICLR, identified the shortcomings based on reviewers' comments and categorize them into the same nine error types.
For accepted ICLR papers, we generated consistency-related erross by cross-referencing their content against the cited literature.
To ensure the quality of each error, all entries underwent rigorous multistage validation protocol executed by human annotators. For synthetically generated errors, annotators manually embedded them into the source papers following this protocol:
Credibility Validation. Each error must be logically sound and verifiable. For generated errors, annotators first confirm their logical coherence and unambiguity. Flawed error descriptions revised whenever possible; only irrepairable cases are discarded.
Evidence Verification. All evidence substantiating an error must be either directly traceable the source document or grounded in established domain-specific knowledge. Annotators required to meticulously verify the origin and accuracy of all supporting data and background information.
Category Classification. Each error must be accurately classified into of the nine predefined categories according to their formal definitions. Annotators verify the correctness of the assigned category and reclassify it if necessary.
Manuscript Editing. Upon successful validation, annotators embedded the generated error into the original manuscript by adding, deleting, or modifying relevant text segments as dictated by the error's specification.
This unified and standardized annotation protocol enables the creation of high-quality dataset of academic papers with curated errors, providing robust benchmark for evaluating the document scanning and error detection capabilities of MLLMs.
C.2 ANNOTATION STATISTICS
Initially, generated sampled pool of 3,500 academic paper instances containing potential errors. During the manual annotation phase, following the protocol described above, we discarded 1,700 instances ensure the logical rigor of the errors, the accuracy of the evidence, and balanced distribution of categories.
Of the remaining 1,800 instances, 1,541 (85.6%) underwent manual revision. The distribution of these modifications is as follows:
535 questions were rewritten eliminate ambiguity or increase their retrieval and reasoning difficulty.
1,207 explanations were revised to correct erroneous evidence references and resolve logical flaws.
1,141 instances underwent category reclassification manual paper p This process served fix classifications that were inconsistent with our definitions and, for errors generated, man- ually inject them into the source papers to create the flawed documents.
39

---

Published as a conference paper at ICLR 2026
C.3 ANNOTATION EXAMPLES
C.3.1 CASE I: DISCARD DIRECTLY
Example
pardle
comple
apper loot Bol abooiny
Question: Assess whether the conclusions drawn about the protein's 's functional state and therapeutic applicability are supported by the presented methods and results.
Explanation: Edits in the abstract and discussion claim the paper presents an active-state structure that reveals the activation mechanism and provides a roadmap for drug design. This overstates the findings, as the paper repeatedly describes solving the 'apo" (unbound) structure and explicitly states the activating ligand is unknown (p.6). To make the error subtle, contradictory sentence was added to the methods (p.9) claiming stabilizing agonist was used, but this is falsified by the numerous, unmodified mentions of the 'apo GPR179' structure throughout the results and methods. Error Type: IC (Inference Conclusions)
Decision: Discard
Analysis: Based on the modifications, the revised abstract and conclusion claim that the paper elucidates the protein's 's 'active-state' structure and provides roadmap for drug design. However, the original text repeatedly states (e.g.. pages 5 and 9) that it is the 'apo' (inactive) structure that was resolved, and crit- ically notes on page 6 that the "activating ligand is still unknown', This constitutes a clear RCA- -tye error. defined by the inconsistent description of a concept within the article. Yet, the large model misclassifies this as an IC-type (Inference & Conclusions) error, which is significant mistake. Considering that the in- consistency regarding the 'active-state' description is overly superficial and obvious, type of error almost never encountered in actual academic literature, lacks practical value. Even reclassifying as H-type question would be little significance. Therefore, we have decided to delete this instance.
40

---

Published as a conference paper at ICLR 2026
C.3.2 CASE 2: MODIFY QUESTION
Example
channg laace majur
Question: Judge whether the mathematical description of the scTab model architecture presents any dimensional mismatches.
Explanation: The Methods section on page 11 (and the Figure 1b caption) now states that the model's feature transformer creates an embedding of dimension n., However, the same paragraph then describes splitting this embedding into two components of dimension ny and na respectively. This is algebraically impossible for any non-zero na. and the hyperparameter table on page 12 confirms that na is set to 64, creating a fundamental dimensional mismatch the model's description. Error Type: CF (Computation & Formulae)
Before:
question: Judge whether the mathematical description of the scTabb model architecture presents any dimen- sional mismatches.
Decision:
Modify
After:
question:
Assess the Methods section for Computation & Formulac issues.
Analysis: Based on the error information and the text, modified model description states that vec- tor of dimension na is split into two parts: one of dimension naa and another of dimension na. This is algebraically impossible, as the total dimension (R.) cannot equal the dimension of one of its parts (na) plus another non-zero part (rt,, is set to 64). This constitutes clear dimensional mismatch, rendering the explicitly model's architectural description logically invalid. The original question was overly specific, contained prompted an assessment of whether the mathematical description of the scTaab model architecture 'any dimensional mismatches', This hint was too detailed, reducing the analytical difficulty for the model. To increase the difficulty, we have revised the question's 's phrasing ask only whether the mathematical description of the scTab model architecture presents any problems.
41

---

Published as a conference paper at ICLR 2026
C.3.3 CASE 3: MODIFY EXPLANATION
Example
bach dhe lepirents
sitty cower
All tort
and tih
Question: Evaluate if the composition of the SNAPemini construct is consistently defined through- out the paper.
Explanation: The results page 4 state that the assembled SNAPcmini construct includes the SNAPC2 subunit. However, the methods on page 12 describe the construction of SNAPemini using only SNAPC4, SNAPC3, and SNAPCI, with SNAPC2 explicitly removed from the cloning description. A third conflicting statement on page 6 implies SNAPC2 was expected to be part of the minimal core, creating conceptual and operational inconsistency regarding this key experimental complex.
Error Type: RQD (Research Question & Definitions)
Before:
Explanation: ..twith SNAPC2 explicitly removed from the cloning description. A third conflicting state- ment on page
Decision:
Modify
After:
Explanation:
with SNAPC2 explicitly removed from the cloning description.
This instance targets an inconsistency in the operational definition of the SNAPemini construct. Specifically, the results section states that the assembled SNAPemini complex includes the SNAPC2 sub- unit, while the methods section explicilly describes the construction SNAPemini using only SNAPC4, SNAPC3, and SNAPCI, with SNAPC2 removed from the cloning procedure. The original explanation ad- ditionally referenced a speculative statement regarding SNAPC2's s expected presence in the minimal core, which introduced unnecessary ambiguity and reduced the clarity of the definition-level inconsistency. By removing this auxiliary statement, the modified instance focuses on clear and realistic mismatch between experimental description and implementation, which is representative of RQD-type error commonly en- countered in academic writing.
Analysis:
42

---

Published as a conference paper at ICLR 2026
D EXAMPLES FROM SCHOLSCAN
D.1 RQD (RESEARCH QUESTION AND DEFINITIONS)
Example: 1001
cded
Question: Assess the Methods section for Research Question and Definitions issues.
Explanation: The definition of 'promoter region', a key analytical construct, is inconsistent across the paper, making the estimand ambiguous. The RNA-seq methods (page 12) define it as +/-lkb from the TSS, the ATAC-seq analysis methods (page 11) define it as +/-500bp from the TSS, and the Results section (page 4) defines it as +/-2kb from the TSS. These three conflicting operational definitions mean that analyses involving 'promoters' are not comparable and the construct is insufficiently defined.
Error Type: RQD (Research Question & Definitions)
Type: Within-Generate
43

---

Published as a conference paper at ICLR 2026
D.2 DI (DESIGN AND IDENTIHABBILTTY)
Example: 1006
Question: Assess the Experiments section for Design & Identifiability issues.
Explanation: The paper's core argument is that identifics a specific 'dynamic coupling' pathway as essential for RTP, distinct from 'static coupling' pathway. The edits state that the key experiment phoppation-poppphoreseeeee mapping) cannot distinguish between these phosphorescence shows spectral signatures of originating from both. This introduces structural identification problem: with two potential causal pathways leading to the same outcome and no way to isolate their effects, the claim that the dynamic pathway is the definitive mechanism is not identifiable from the data presented.
pathways, the final
Error Type: DI (Design & Identifiability) Type: Within-Generate
44

---

Published as a conference paper at ICLR 2026
D.3 SG (SAMPLING AND GENERALIZABILITY)
Example: 1014
Make
High-contrast in vivo fluorescence imaging exploiting wavelengths beyond 1880 ni
cciay Bie scrrect difirc bnlagt male on clund
Question: Assess the Methods section for Sampling Generalizability issues.
Explanation: The Methods section is edited to state that the experiments used "specific substrain of diabetic" mice, highly specialized sample. However, the Abstract and Discussion make broad, unsupported claims of generalizability "all patients" and the "general patient population." This constitutes an invalid pooplllto-poollation inference. Error Type: SG (Sampling & Generalizability) Type: Within- Generate
45

---

Published as a conference paper at ICLR 2026
D.4 MO (MEASUREMENT AND OPERATIONALIZATION)
Example: 1015
129 150- 160
1700 1507
TESS
Question: Assess the Figures/Table Tables section for Mcasurement & Operationalization issues. Explanation: The figure captions on pages 3 and have been edited to specify that the background for
Ratio (SBR) calculations was defined as the single minimum pixel intensity in the image. This is not valid reliable operationalization of the "background" construct, it's highly susceptible to single-point noise or detector artifacts. This flawed measurement procedure systematically undermines all conclusions based on the SBR metric. Error Type: MO (Measurement & Operationalization) Type: Within Generate
46

---

Published as a conference paper at ICLR 2026
D.5 DHP (DATA HANDLING AND PREPROCESSING)
Example: 528
Question: Assess the Methods section for Data Handling & Preprocessing issues.
Explanation: The reviewer correctly identifies that the authors tuned hyperparameters on the test set. The paper's "Implementation Details" section tuning, employed Bayesian optimization with the wandb sweep tool (Biewald, 2020), aiming minimize MPJPE for the and S11 dataset, following the convention of prior works." According to standard protocols for the H36M dataset, subjects S9 and S11 constitute the test set. Tuning hyperparameters directly on the test set introduces data leakage, leading generalization. This critical violation of machine learning best practices and fits the Data Handling & Preprocessing (E) category, as pipeline choice introduces bias. Error Type: DHP (Data Handling & Preprocessing) Type: Wathee-Sample
page 5 states: "For hyperparameter
H36M dataset and PA-MPIPE the S8
H3WB
optimistic bias in the reported results and invalidating claims
47

---

Published as a conference paper at ICLR 2026
D.6 CF (COMPUTATION AND FORMULAE)
Example: 1909
Question: Scan the Methods section for Computation & Formulac errors. Explanation: Algorithm 1 on page 5 uses the parameter T' in the loop dcfinition on line 5: for == 1 to T do. This parameter determines the number of iterations for the Randomized Global Initialization phase. However, the value of T' is never specified anywhere in the paper, including the "Hyper-parameten's section (Section 4.1 on page 7). An algorithm cannot be implemented or reproduced with an undefined critical parameter. This fits the Computation & Formulae category as an "undefined variable".
Error Type: CF (Computation & Formulae)
Type: Within Within-Sample
48

---

Published as a conference paper at ICLR 2026
D.7 IC (INFERENCE AND CONCLUSIONS)
Example: 875
Tear
Evaluate Abstract, Introduction and Experiment section for issucs in Inference & Conclusions.
Question:
supported by its evidence. The titlc and abstract introduce "FedGraph: New Paradigm for Federated Graph Learning" (page 1), suggesting broadly applicable framework. However, the methodology heavily tailored to, and the experiments are exclusively focused on, the single downstream task of anomaly detection. For example, a stated contribution is "Broad application." but this is immediately qualified with "the models are successfully transferred to FEDGRAPH framework in anomaly detection tasks" (page 2). Furthermore, Section "EXPERIMENTS" exclusively reports results on anomaly detection tasks. This discrepancy represents an issue Inference & Conclusions, as the broad conclusion having created new "paradigm" for FGL is an overstatement that exceeds what the narrow experimental results can support.
Explanation: The paper's claims of generality
Error Type: IC (Inference Conclusions)
Type: Within- Sample
49

---

Published as a conference paper at ICLR 2026
D.8 RCA (REFERENTIAL AND CITATION ALIGNMENT)
Example: 0
buddy
thu
Question: Scan the errors in cited reference Chen et al. (2021)
Explanation: The edited P contains a Type H стоо by misrepresenting the performance of the cited model. P (p. 8) claims that the NSTPP model from Chen al. (2021) 'reported performance comparable
standard Hawkes process baseline'. This contradicts the results in S, where the proposed models (i.e., NSTPP) consistently outperform
Hawkes process baseline, often by a large margin. For example, (p. 9, Table 1) shows on the BOLD5000 dataset that the 'Attentive CNF' model achieves temporal ligliiklldood of 5.842 + 0.005, which is substantially better than the Hawkes Process at 2.860 + 0.050.
Error Type: RCA (Referential Citation Alignment)
Type: Cross-Generate
50

---

Published as a conference paper at ICLR 2026
D.9 LE (LANGUAGE AND EXPRESSION)
Example: 1785
Question: Assess the Methods section for Language & Expression issues.
Explanation: The paper introduces key contribution, the moaddcompute-stor---oe finish' template, and its acronym 'LCSF. This error introduces inconsistencies in this critical term: it's defined as 'LCS-F' on page called 'LCFS' in figure title page 7, and written out in full in the conclusion page 10, while the original 'LCSF" acronym remains elsewhere. This terminological inconsistency for central, paper-defined concept creates ambiguity and undermines the paper's precision. Error Type: LE (Language & Expression)
Type: Within Generate
51

---

Published as a conference paper at ICLR 2026
E HUMAN-MACHINE CONSISTENCY EVALUATION
To evaluate whether GPT-4.1 accurately extracts detailed information from model responses, we conducted human-machine consistency evaluation. We first randomly sampled 200 questions from the dataset. Then, we invited human experts to analyze the corresponding model-generated responses for these questions and manually extract key information, including evidence sets, rea- soning chains, and the number of unrelated errors. The results are presented in Table
Table 4: Spearman's correlation coefficients among S, Stocation. Sreasoning. and Punrelated.err-
loccation
Sreasoning
P Punrelated.ore
Correlation Coefficients 0.841
0.806
0.842
0.954
In summary, GPT-4.1 can extract relevant evidence and reasoning steps with considerable accuracy, leading to precise evaluation scores.
In addition, we substituted GPT-4.1 with Qwen3-32B and Gemini 2.5 Flash to independently re- evaluate the same 200 samples (Tables and (6). The results further confirm that our evaluation framework is not dependent any particular LLM and exhibits strong robustness.
Table 5: Model performance under Qwen3-32B evaluation (scaled by 100).
Model
Avg.
RQD
SG
MO
DHP
CF
IC
RCA
LE
MLLM (Image Input)
GPT-5
21.2
12.6
11.8
33.5
15.4
26.6
16.0
27.1
27.0
6.4
Gemini 2.5
19.7
15.0
23.2
44.7
13.2
31.4
7.8
17.3
17.8
12.8
Doubao-Son -een---
12.0
4.8
5.9
36.2
7.5
13.3
5.7
19.9
9.9
5.5
Soubao-Seed 1.6 16-thinking
11.6
5.1
6.9
28.0
7.7
14.3
10.5
15.6
10.8
4.5
Grok 4
4.2
0.0
1.3
20.5
2.6
5.5
1.2
3.9
2.6
0.8
OCR
LLM
(Text Input)
Gemini 2.5 Pro
35.0
26.6
39.6
53.9
31.4
56.2
15.9
35.6
39.0
10.0
GPT-5
25.3
19.0
28.3
28.3
24.1
38.8
10.3
32.2
31.8
3.1
Grok 4
22.6
10.7
8.9
40.6
14.1
31.3
11.1
22.6
33.6
7.3
Doubao-Sced- Sced-1.6-thhnking
17.4
11.0
14.9
31.9
9.5
26.3
9.2
20.5
21.1
4.7
Doubao-Soed-4
15.3
9.8
31.7
10.8
22.4
8.5
21.9
17.8
2.4
Claude Sonnet 4
6.3
5.7
2.4
12.4
4.2
8.3
2.8
9.5
6.6
4.4
Table 6: Model performance under Gemini 2.5 Flash evaluation (scaled by 100).
Model
Avg.
RQD
DI
SG
MO
DHP
CF
IC
RCA
LE
MILLM (Image Input)
GPT-5
20.5
11.2
12.0
32.4
17.9
27.4
10.5
25.9
27.4
3.1
Gemini 2.5 Pro
14.6
9.1
11.3
34.2
11.4
28.2
4.9
12.6
14.8
5.3
Doubao-Sced-1.6
10.9
5.3
3.9
34.0
6.1
15.6
6.1
17.9
8.0
4.7
Doubao-Sece 1-6-thnkiig
10.3
3.2
4.7
26.4
7.3
14.2
9.2
14.8
9.3
3.1
Grok 4
3.9
0.5
1.8
15.9
1.7
4.1
1.5
1.1
4.4
0.0
OCR LLM (Text Input)
Gemini 2.5
30.1
20.0
30.6
47.8
27.5
47.8
11.7
30.2
36.6
GPT-5
23.5
15.3
21.8
26.5
23.1
37.7
7.1
31.9
31.6
2.7
Grok 4
20.2
9.2
7.8
36.1
11.5
32.4
8.0
20.7
30.6
5.8
Doubao-Seed-Ii 1.6-
Seed-1.6 1.6-thinkinng
15.9
8.4
11.1
30.9
10.1
23.7
6.3
19.9
20.2
3.5
14.0
4.8
8.2
29.1
11.4
23.9
6.4
21.2
16.0
0.8
Claude Sonnet
3.8
1.4
11.1
3.9
9.4
8.7
6.5
3.1
52

---

Published as a conference paper at ICLR 2026
F HYPERPARAMETER SENSITIVITY ANALYSIS
We conducted a sensitivity analysis of all four hyperparameters involved scoring. We varied each independently and re-computed the overall score S across proprietary model configurations (Ta- bles and 10 The results demonstrate that our evaluation metric exhibits strong robustness.
Table 7: Sensitivity under image input: variations and μ (scaled by 100).
Model
A=0.6
x=1.0
==0.85
==
m=90955
GPT-5
19.3
19.2
19.0
18.5
19.2
19.9
Gemini 2.5 Pro
15.8
15.6
15.3
15.0
15.6
16.1
Doubao-Seed-1 66-thinking
10.4
10.2
10.0
9.9
10.2
10.5
Doubao-Seed-1.6
10.1
9.9
9.8
9.7
9.9
10.2
Grok
4.0
4.0
3.9
3.9
4.0
4.1
Table 8: Sensitivity under image input: variations of y and q (scaled by 100).
Model
7=0.4
0-6.6
0-0.8
1=1.0
q=1.5
q q=2.0 2.0
GPT-5 Gemini 2.5 Pro
19.4
19.2
19.0
19.3
19.2
19.1
15.7
15.6
15.5
15.6
15.6
15.5
Doubao-Sece Seetto--eee-1.66 1.dtt...-
10.4
10.2
10.0
10.3
10.2
10.1
Doubao-Seed-1.6
10.1
9.9
9.8
10.0
9.9
9.9
Grok 4
4.0
4.0
4.0
4.0
4.0
4.0
Table 9: Sensitivity under text input: variations of À and μ (scaled by 100).
Model
==066
A=0.8
X=1.0
p=0.85
p=0.9
1=0.95
Gemini 2.5 Pro
30.6
30.2
29.9
29.1
30.2
31.5
GPT-5
22.7
22.5
22.3
21.4
22.5
23.7
Grok
21.1
20.8
20.6
20.2
20.8
21.4
Doubao-Seed 111111.6thhhee 16-thinking
15.6
15.3
15.0
14.8
15.3
15.8
Doubao-Seed-1..
14.1
13.9
13.7
13.6
13.9
14.3
Claude Sonnet
6.0
5.9
5.8
5.6
5.9
6.1
Table 10: Sensitivity under text input: variations of y and g (scaled by 100).
Model
-=4.4
0=6.6
0-8.8
q=1.0
q=1.5
2-2.
Gemini 2.5 Pro
30.7
30.2
29.9
30.5
30.2
30.1
GPT-5
23.4
22.5
21.9
23.0
22.5
22.2
Grok
21.0
20.8
20.7
20.9
20.8
20.8
Baabbo--See--- 1.tth-..-ti
15.6
15.3
15.1
15.5
15.3
15,2
1oubao-Seed-1.6
14.2
13.9
13.7
14.1
13.9
13.8
Claude Sonnet 4
6.3
5.9
5.6
6.1
5.9
5.7
53

---

Published as a conference paper at ICLR 2026
G USE OF LLMS
L.MMs were used for language editing and stylistic refinement during manuseript preparation. In ad- dition, Gemini 2.5 Pro was used in controlled manner synthesize data for dataset construction. Details are provided in Section 3.2 Appendix [A] and Appendix [ All research ideas, experimen- tal design, evaluation protocols, and result analysis were conceived, implemented, and validated entirely by the authors. The use of LLMs did not influence the scientific conclusions of this paper.
54


Reviews:
[{"review_id": "WaRUQUn2XF", "reviewer": "Reviewer_6ouw", "summary": "The paper proposes ScholScan, a new benchmark meant to evaluate multimodal large language models (MLLMs) on “scan-oriented” scholarly paper reasoning rather than traditional “search-oriented” QA. This paper introduces a task setting where the model is given one or more full academic papers and a target-absent instruction (e.g., “Assess the Methods section for Measurement & Operationalization issues”) and curated 1,800 questions with 9 families of scientific errors. The authors found that existing models perform badly in this new task, and this can not be simply addressed by a RAG pipeline.", "questions": "1. Can you break down, as exact percentages (or counts): (1) generated-edited accepted papers (Gemini edits), (2) sampled-from-rejected-papers (ICLR reviews), (3) cross-paper citation consistency cases?\n\n2. You mention that human evaluation “confirms high agreement” between the GPT-4.1 extraction/metric pipeline and expert annotations. Can you report some quantitative results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. **Task**: The paper draws a sharp conceptual line between search-oriented QA (“find X in this paper”) and scan-oriented critique (“is anything off about this study?”). This framing is intuitive, well motivated, and reflects a real gap between “LLM as literature assistant” and “LLM as junior reviewer.” This is an important task with pressing needs. \n\n2. **Dataset scale, diversity, and annotation rigor**: the benchmark curation process covers good scale (715 papers across 13 scientific domains (including physics, chemistry, computer science, biology, etc.) and 1,800 instances spanning nine error families is impressively broad and goes beyond narrowly technical CS-only corpora. There is rigorous human review of the generated data instances. \n\n3. **Strong analysis and observations**: OCR text often outperforms raw page images overall (likely because long-context language modeling is currently better than long-context vision-language modeling). Longer reasoning chains actually correlate with worse scores. They report that classical retrieval does not solve the “scan-oriented” problem because the hardest part is deciding what to look for in the first place. These are very valuable insights immediately transferrable to the AI-for-science community.", "weaknesses": "1. **Synthetic edit to accepted papers**: A large fraction of benchmark items are created by prompting Gemini 2.5 Pro to insert coordinated edits into accepted/high-quality papers, and then asking models to catch those edits. The quality of this generated split of dataset is bounded by the capacity of Gemini 2.5 Pro. As the paper later finds, even best models so far struggle with the \"scan-oriented\" problems. This would hurt the quality of the benchmark. \n\n2. **Evaluation metric design**: The final score S(m) is an involved product of existence, evidence localization Dice (with squared penalties for over-reporting), reasoning prefix match, and a nonlinear hallucination penalty. There are many tunable constants (e.g. the 0.8 subtraction, exponent 1.5 in the hallucination penalty). It’s not obvious these specific choices are robust or interpretable across domains. \n\n3. **Leakage concern**: The benchmark is partially consisted of high-quality ICLR-accepted papers that future models will train on. Since the benchmark curation process involves extensive human annotations, data leakage problem becomes a major problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "chqZrMNEJW", "reviewer": "Reviewer_HMvn", "summary": "This paper proposes ScholScan, a new benchmark designed to evaluate multimodal large language models (MLLMs) on “scan-oriented” scholarly reasoning—i.e., the ability to read and cross-check entire academic papers like human researchers to identify internal consistency issues. Unlike conventional “search-oriented” paradigms that rely on pre-specified questions and localized retrieval, ScholScan presents target-absent queries that require holistic document understanding. The benchmark includes 1,800 expert-annotated questions across 9 scientific error types, drawn from 715 papers in 13 natural science domains, with detailed evidence localization and reasoning traces.", "questions": "1. Scan-oriented processing requires ingesting the full paper, which can be very long. Given current context-length and attention limitations, could this approach become impractical for some real-world use? How does performance scale with document length?\n\n2. Section 4.4 notes that stronger models have fewer zero-score cases but more hallucinations. Yet Figure 5 shows Gemini suffering from high omission and hallucination rates—seemingly at odds with its relatively higher scores in Table 1. Can the authors reconcile this apparent contradiction?\n\n3. Figure 6 suggests performance degrades with more reasoning steps or evidence locations. However, this may simply reflect inherently harder tasks rather than a causal effect of reasoning length. Could the authors provide an analysis of task difficulty (e.g., via human expert ratings) to disentangle complexity from model limitations?\n\n4. The Oracle RAG condition does yield significant gains, implying that high-quality retrieval is helpful. Doesn’t this suggest the core issue is not the RAG paradigm itself, but the inadequacy of current retrievers? Might future advances in retrieval close part of the gap?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly articulates a meaningful distinction between search-oriented and scan-oriented reasoning paradigms, highlighting a critical gap in current MLLM capabilities for full-document scientific verification.\n\n2. The dataset construction process is rigorous: 10 domain experts performed dual reviews with third-party adjudication. Of 3,500 initial candidates, 1,700 were discarded and 1,541 revised—demonstrating strong quality control.\n\n3. The evaluation is comprehensive: 15 models under 24 input configurations and 8 RAG frameworks are tested. Results consistently show poor performance across the board, and notably, RAG offers no significant gains—underscoring the challenge of scan-oriented tasks.", "weaknesses": "1. The definition of “scan-oriented” remains somewhat vague. Figure 1 contrasts target-prespecified vs. target-absent questions, which is clearer than the term “scan-oriented” itself. It would help to explicitly state that “scan-oriented” means processing the entire document (text or image) without relying on retrieved chunks—as opposed to RAG’s fragment-based approach.\n\n2. While the 9 error types span the scientific workflow (Figure 3), it is not convincingly demonstrated that all require full-document understanding. Many instances are derived from OpenReview comments, which may reflect localized, detail-level critiques rather than issues demanding cross-section or cross-paper reasoning.\n\n3. Model performance is universally low (e.g., even under Oracle RAG, average score is only 24.5/100). This raises the question: how would human experts perform on these tasks? Without a human baseline, it’s unclear whether the low scores reflect model limitations or simply extreme task difficulty.\n\n4. Several figures and tables lack sufficient clarity:\n   - Figures 4 and 5 omit axis labels.\n   - Figure 5’s “evidence locations” is not well-defined—how are these locations counted or validated?\n   - Table 2 reports RAG results but does not describe the detailed setting or related references of these RAG methods, making it hard to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KM8SU9A4iX", "reviewer": "Reviewer_78Mn", "summary": "This paper introduces ScholScan, a benchmark for scholarly reasoning that contrasts with existing search-oriented evaluations. ScholScan uses a scan-oriented question design: models are required to read entire papers and derive all necessary concepts and inferences solely from the provided documents, rather than answering pre-specified queries grounded in retrieval. The benchmark covers 13 scientific domains and 9 error types, enabling a comprehensive assessment of error detection. This paper evaluates multiple models and further highlights the paradigm difference by testing the RAG method, finding that simple retrieval offers no improvement on scan-oriented tasks. These results suggest that current MLLMs still struggle with addressing such scan-oriented tasks", "questions": "Have you analyzed why model performance remains low across all error categories, even when oracle evidence is provided? Is this primarily due to limitations in the models’ reasoning capability, or are there other contributing factors?\n\nAdditionally, have you evaluated human performance under the same oracle condition? From the current results, it seems that even with oracle evidence, the benchmark is still extremely challenging.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Introduction of a new paradigm:** This paper proposes a novel scan-oriented paradigm, which differs fundamentally from traditional search-oriented settings with pre-specified queries. In this setup, models must autonomously and comprehensively understand the entire document and identify the error by themselves. \n\n**Comprehensive evaluation and analysis:** This paper conducts large-scale evaluations covering both instruction-following and thinking-style models, providing detailed statistical analyses of errors. It further reveals that model performance deteriorates as the required number of documents and reasoning steps increases, offering valuable diagnostic insights. \n\n**Well-structured evaluation metric:** The proposed metric jointly measures error detection, evidence accuracy, and reasoning quality. It rewards correct identification of target errors, penalizes noisy or irrelevant evidence, and assesses reasoning completeness through prefix matching.", "weaknesses": "**Synthetic task generation and authenticity concerns:**\nAll benchmark tasks are generated by LLMs rather than collected from real-world review scenarios, which raises questions about the authenticity of the task distribution. Although the authors include human verification, the underlying errors are created by modifying accepted papers via LLMs. As a result, the error types and frequency distributions may still deviate from those naturally occurring in real research writing. Moreover, since the tasks are generated using Gemini-2.5-Pro, this may partially explain its superior performance on the benchmark.\n\n**Potential data contamination and long-term validity:**\nThere is also a risk of benchmark contamination as newer models may already include papers from sources such as ICLR 2024/2025 or Nature Communications in their pretraining corpora. Over time, this contamination could become increasingly severe, diminishing the benchmark’s long-term validity. In particular, for tasks generated from the \"sampling\" method based on the public reviews, future models may memorize or overfit to these examples, reducing the benchmark’s long-term validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4zEGtauGFR", "reviewer": "Reviewer_a2Tc", "summary": "The paper introduces ScholScan, a benchmark designed to evaluate multimodal large language models (MLLMs) on scan-oriented scholarly reasoning: instead of answering prespecified, “search-oriented” questions, models receive one or more full academic papers and must proactively scan the entire document to uncover consistency issues. Key aspects:\n- Task formulation: “target-absent” audit-style queries (e.g., “Assess the Methods section for Measurement & Operationalization issues”) requiring evidence construction across the whole paper.\n- Dataset: 1,800 questions across 715 papers from 13 natural-science domains, covering 9 error families (Research Question & Definitions; Design & Identifiability; Sampling & Generalizability; Measurement & Operationalization; Data Handling & Preprocessing; Computation & Formulae; Inference & Conclusions; Referential & Citation Alignment; Language & Expression). Instances come from two sources: (i) LLM-edited accepted papers (synthetic error injection) and (ii) sampling verifiable issues from ICLR public reviews of rejected papers. Cross-paper citation inconsistencies are generated.\n- Process-aware evaluation: A unified evaluation pipeline parses open-ended model outputs into existence, evidence localization (Dice-like score with over-reporting penalty), reasoning-chain prefix-match, and a penalty for unrelated/hallucinated errors.\n- Experiments: 15 models with 24 input settings (image vs OCR text) and 8 RAG frameworks. Main findings: (a) overall performance is weak; (b) reasoning-optimized models perform better; (c) text (OCR) inputs often outperform image inputs, but images can matter for formula/table-heavy tasks; (d) standard RAG brings little benefit; an RL-based visual-centric RAG (VRAG-RL) shows the most promise on image inputs.\n- Analyses: Correlations across error types suggest latent “skill” dimensions; evidence-and-reasoning workload correlates with failures; failure mode breakdown (omission vs hallucination); human–machine agreement for the judge-extractor shows strong Spearman correlations.", "questions": "- Beyond RAG - Other Potential Architectures: The RAG analysis is excellent. However, have the authors considered or could they comment on other architectural approaches that might be better suited for \"scan\" tasks? For example, models with explicit memory B, graph-based representations of the document's claims and evidence, or iterative self-refinement loops where the model repeatedly scans the paper to build and verify a \"consistency map.\" A brief discussion in the conclusion could enrich the paper.\n- LLM-as-Judge Robustness: Regarding the GPT-4.1-based evaluation, did the authors observe any failure modes? For instance, did the evaluator struggle with particularly verbose or poorly structured responses from certain open-source models? Providing some qualitative examples of the evaluator's performance could further strengthen confidence in this method.\n- Feasibility of Human Evaluation: The paper mentions high agreement between the GPT-4.1 pipeline and expert annotations on a subset. Given the complexity of the task, what would be the estimated cost or time for a full human evaluation of a single model's output on the benchmark? A brief comment on this could help contextualize the necessity of the automated pipeline.\n- Suggestion on Score Calibration: The final scores are quite low across the board (the best model scores ~30). This is a strong testament to the benchmark's difficulty. However, it can make it hard to differentiate between poorly performing models. Have the authors considered if a simpler, \"binary\" success metric (e.g., did the model identify the correct error type and at least one piece of correct evidence?) could complement the main score and provide another view on performance, especially for the lower-performing models? This is not a required change, but a suggestion for a potential additional analysis.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A New \"Scan-Oriented\" Task Paradigm: The paper formalizes a novel and challenging task setting that moves beyond simple question-answering to require exhaustive, document-level understanding and consistency checking without pre-specified targets.\n- The ScholScan Benchmark: A comprehensive and large-scale benchmark consisting of 1,800 questions derived from 715 scientific papers across 13 natural science domains. The benchmark covers 9 distinct families of scientific errors, from issues in experimental design to referential inconsistencies.\n- High-Quality Data and Annotation: The dataset was constructed through a rigorous process involving both LLM-based error generation/sampling and meticulous verification by 10 domain experts, ensuring the errors are subtle, realistic, and verifiable. The annotations include not just the final answer but also evidence locations and reasoning traces.\n- A Process-Aware Evaluation Framework: The authors propose a detailed evaluation protocol and a composite scoring metric that assesses a model's performance on multiple dimensions: correctly identifying an error's existence, localizing all relevant evidence, completing the correct reasoning chain, and avoiding the hallucination of unrelated errors.\n- Extensive Empirical Analysis: The paper presents a thorough evaluation of 15 state-of-the-art MLLMs (across 24 configurations) and 8 Retrieval-Augmented Generation (RAG) methods. The results show that all current models, including proprietary ones like GPT-5 and Gemini 2.5 Pro, perform poorly on this task, and that standard RAG techniques fail to provide any significant improvement, highlighting the unique challenges posed by the scan-oriented paradigm.", "weaknesses": "- Dependence on LLM-based Evaluation: The evaluation protocol relies on GPT-4.1 to parse model responses and extract structured information (evidence, reasoning steps). While the authors validate this with a human-machine consistency check (Appendix F), which shows strong correlation, this dependency introduces a potential point of failure or bias. The evaluation of one model is contingent on the capabilities of another, which is a common but not ideal practice. The evaluation could be influenced by the specific style or formatting of the model being evaluated.\n- Limited Scope of \"Generated\" Errors: The data generation process relies on an LLM (Gemini 2.5 Pro) to inject errors into high-quality papers. While curated by experts, these synthetic errors might have subtle, systematic patterns that differ from genuine human errors. The benchmark could be further strengthened by including more examples of errors \"sampled\" from real-world peer reviews or errata, although the authors are transparent about their methodology.\n- Ambiguity in the \"Reasoning Process Score\": The reasoning score (`S_reasoning`) is based on a prefix match of the reasoning chain. This assumes a single, linear \"gold\" reasoning path. However, a valid logical argument could potentially be structured in different ways or use slightly different intermediate steps. This metric might unduly penalize models that arrive at the correct conclusion through a logically sound but differently structured argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fuGV03x9Nb", "reviewer": "Reviewer_co8M", "summary": "This paper introduces ScholScan, a novel scan-oriented reading benchmark designed to address the limitation of Multimodal Large Language Models (MLLMs) in overlooking holistic and cross-document understanding. The benchmark's questions are initially generated through a sampling and generation process, then manually screened to form the final set. Corresponding metrics are employed to evaluate model performance, and multi-dimensional experiments provide valuable insights into future directions for model development.", "questions": "- On line 88, there appears to be a typo. \"Process-Aware Rvaluation Framework\" should be corrected to \"Process-Aware Evaluation Framework\".\n- Some entries in the bibliography seem overly long, which takes up considerable page space. Please consider condensing them.\n- In the section on evaluation metrics, several hyper-parameters are introduced (e.g., 0.8, 0.9, 0.6). The rationale for selecting these specific values is not entirely clear. To strengthen the paper, I would recommend including a justification for these choices. For instance, a sensitivity analysis or an ablation study could demonstrate the robustness and appropriateness of the selected parameters.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "ScholScan effectively captures a key aspect of human reading: the ability to process full documents or read across multiple articles. The data construction and filtering process is methodologically sound. The evaluation metrics are multi-faceted, reasonable, and well-aligned with human judgments. Consequently, the experimental results offer valuable insights that can guide the future development of Multi-modal Large Language Models (MLLMs).", "weaknesses": "The development of ScholScan heavily relies on the use of a Large Language Model (LLM), specifically Gemini-2.5 Pro. Although manual screening and validation were subsequently incorporated to ensure quality, the questions generated by the LLM may inherently differ from those posed by humans.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)