OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning
Authors: Mingxin Huang
Yongxin Shi
Dezhi Peng
Songxuan Lai
Zecheng Xie
Lianwen Jin
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal slow-thinking systems;text-rich image understadning;reasoning model
Primary area: datasets and benchmarks

Abstract: Recent advancements in multimodal slow-thinking systems have demonstrated remarkable performance across various visual reasoning tasks. However, their capabilities in text-rich image reasoning tasks remain understudied due to the absence of a dedicated and systematic benchmark. To address this gap, we propose OCR-Reasoning, a novel benchmark designed to systematically assess Multimodal Large Language Models on text-rich image reasoning tasks. Specifically, OCR-Reasoning comprises 1,069 human-annotated examples spanning 6 core reasoning abilities and 18 practical reasoning tasks in text-rich visual scenarios. Unlike existing text-rich image understanding benchmarks that only provide a final answer, this benchmark additionally provides a detailed step-by-step reasoning process. This dual annotation enables the evaluation of both the models' final answers and their reasoning processes, thereby offering a holistic assessment of text-rich reasoning capabilities. By leveraging this benchmark, we conducted a comprehensive evaluation of the latest MLLMs. Our results demonstrate that even the most advanced MLLMs exhibit substantial difficulties in text-rich image reasoning tasks, with none achieving an accuracy above 50\% on our benchmark, indicating that the challenges of text-rich image reasoning are an urgent issue to be addressed. The benchmark and evaluation scripts are available at https://github.com/SCUT-DLVCLab/OCR-Reasoning.

Published as a conference paper at ICLR 2026
OCR-REASONING BENCHMARK: UNVEILING THE TRUE CAPABILITIES OF MLLMS IN COMPLEX TEXT- RICH IMAGE REASONING
Mingxin Huang"', Yongxin Shilt, Dezhi P Songxuan Lai', Zecheng Xie Lianwen Jin South China University of Technology HHawwii Technologies Co., Ltd.
ABSTRACT
Recent advancements in multimodal slow-thinking systems have demonstrated re- markable performance across various visual reasoning tasks. However, their capa- bilities in text-rich image reasoning tasks remain understudied due to the absence of dedicated and systematic benchmark. To address this gap, we propose OCR- Reasoning, a novel benchmark designed to systematically assess Multimodal Large Language Models text-rich image reasoning tasks. Specifically, OCR- Reasoning comprises 1,069 human-annotated examples spanning 6 core reasoning abilities and 18 practical reasoning tasks in text-rich visual scenarios. Unlike ex- isting text-rich image understanding benchmarks that only provide a final answer, this benchmark additionally provides detailed step-by-step reasoning process. This dual annotation enables the evaluation of both the models' final answers and their reasoning processes, thereby offering a holistic assessment of text-rich rea- soning capabilities. By leveraging this benchmark, we conducted a comprehensive evaluation of the latest MI..Ms. Our results demonstrate that even the most ad- vanced MLLMs exhibit substantial difficulties in text-rich image reasoning tasks, with none achieving an accuracy above 50% on our benchmark, indicating that the challenges of text-rich image reasoning are urgent issue to be addressed. The benchmark and evaluation scripts are available at https://gitub..ommSSCC DLVCLab/OCR -LLaab//CR-Resooning.....
INTRODUCTION
Recently, slow-thinking systems in Large Language Models (LLMs), such as OpenAl-ol (Jaech ict al.] 2024). DeepSeck-RI (Guo et al.] (025a),, Gemini-Thinking (Team et al. 2023). and QwQ TTam 2025) have demonstrated significant progress in addressing complex math, coding, logical, and scientific problems. Building upon techniques like Chain-of-Thought (CoT) prompt- ing (Wei et al. 2022) and test-time compute scaling (Jaech et al. 2024 Guo al. 2025a), slow- thinking systems typically engage in critical thinking and reflection before providing the final an- swer. Moreover, emerging evidence suggests these systems may even experience 'Aha moments' when solving complex problems (Guo et al. 2025a). In order to broaden their ability across diverse contexts, multimodal dlwwthinnking systems have emerged as a rapidly evolving research direction, driven by the need for more versatile AI applications (Yang et al. 2025 Peng et al. 2225 Meng al. 2025; Chen al. 20256 Liu al., [202cc Wang et al. 2025b Liu al. 2025b Shen al., Wang et al. 2025a Liu et al. 2025a).
2025
To assess the reasoning capabilities of multimodal slow-thinking systems, researchers have devel- oped specialized reasoning benchmarks targeting distinct scenarios. For instance, Math Vista (Lu jet al.) 2023), Math Verse (Zhang et al., 2024), Olympiadhench (He et al. 2024). and Math Vi- sion [Wang et al.) (022a) are designed to evaluate the math-related reasoning ability of the model. In college-level subject knowledge domains, MMMU (Yue et al.) (023) focuses on advanced rea- soning in domains such as chemistry, physics, and scientific problem-solving. While these domains are thriving with the corresponding benchmarks, a critical gap persists in text-rich image scenarios.
Equul contribution.
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Social media users experience mix of positive, negative emotions using these platforms Sefiioiia tolk any layy
100
99.8
97.9
89.5
makon them fivl
81.5
73.4
Sometimee
Paruattgge
Amused
64
44
80
Anggy
71
tropdeed
Depresseed
18
36
40
Lonely
2.3
Question: What's the "NET" % of lonely users?
Answa
IsRVDA
DKYIA
ClarYQ4
TeeviqA
Answer:
(a) The ratio of Q&& pairs with answer included in OCR results.
(b) Example with answer included OCR results.
answer can directly through thinking.
Figure 1: (a) The percentage of answers in the benchmark's Q& pairs that can be retrieved from the OCR results. (b) An example where the answers can be retrieved from the OCR results.
Current benchmarks for text-rich images, such Doc VQA (Mathew et al. 2021 ChartQA (Masry et al. 2022), and OCRBench (Liu et al.) (024d), are designed primarily to assess the ability to merely extract textual content without requiring in-depth analysis (Mathew
al.) 202T: Xu al.
2025). However, text-rich images involve many reasoning-intens intennive tasks such as financial report analysis, invoice analysis, and cost-effective purchase decisions (Gan al., 2024, [Sun al. (021). There is still lack of benchmarks for systematically evaluating the reasoning ability within text-rich visual scenarios.
To bridge this critical gap in multimodal evaluation, we introduce mark designed to evaluate the text-rich image reasoning skills Multimodal Large Language Models (MLLMs). Specifically, our benchmark contains 1,069 meticulously collected and human- annotated examples, which span 6 core reasoning abilities and 18 practical reasoning tasks com- monly found in text-rich visual contexts. Furthermore, unlike other text-rich image understanding benchmarks that only annotate the final answers, OCR- OCa-Reasoning provides annotations for both the final answers and the step-by-step reasoning process. This comprehensive annotation scheme facil- itates a more in-depth evaluation of MILMs' reasoning capabilities. Additionally, through a simple comparison with existing benchmarks, shown Fig. [ we observe that most cases the an- swers in existing datasets are directly present in the images. whereas our benehmark contains very few samples of this type. This implies that in our benchmark. obtain the answer, the model needs to engage in reasoning rather than extracting it from the OCR results of the image.
a novel bench-
Using the OCR- Reasonnnn benchmark, we conduct extensive experiments to assess the text-rich im- age reasoning capabilities of popular LLMs and MLLMs. For pure LLMs, we replaced images with their OCR results and used these as input. The results show relatively low accuracy, which indicates that text alone is insufficient for solving text-rich image reasoning tasks. For MI.LMs, the strongest performer achieves only 46.8% accuracy, with none surpassing 50% on our benchmark. As for document-oriented MLLMs, their highest accuracy does not exceed 15%. These findings demon- strate that existing models still have significant room for improvement in handling text-rich image reasoning tasks. Additionally, perform poorly on text-rich image reasoning tasks. Designing reinforcement learning for text-rich image reasoning a potential direction for enhancing text-rich image reasoning capabilities.
find that most of the existing reinforcement learning methods
The main contributions of this work are summarized as follows.
We introduce OCR-Reasoning, a challenging rich-text image reasoning benchmark that provides a systematic evaluation framework for assessing the reasoning capabilities of MLLMs in text-rich scenarios. To the best of our knowledge, define various core sub-abilities for text-rich image reasoning and conduct systematic eval- uations.
the first to concretely
2

---

Published as a conference paper at ICLR 2026
We conduct systematic evaluation of leading MLLMs. Our results indicate that: 1) For text-rich image reasoning tasks, pure OCR input cannot effectively replace image input; 2) Even the most leading MLLMs struggle with proposed benchmark. Based on the experiment results, we find several potential directions for future improvements.
RELATED WORK
2.1 MULTI-MODAL BENCIIARK
Driven by innovations slow-thinking systems in LLMs, the evaluation of reasoning capabilities in Multimodal Large Language Models (MLLMs) has become highly focused and widely dis- cussed topic (Lu et al. 2023 Zhang et al. 2024 Wang et al.) 2024a Yue et al., 2023). Early benchmarks such CLEVR (Jonnson al. 2017) and GQA (Hudson & Manning (019 pioneered
the integration of compositional language- vision abstraction to assess visual reasoning in structured environments. Subsequent works expanded the evaluation of reasoning into diverse domains. For instance, ScienceQA (Lu et al., (2022) introduces scientific multimodal reasoning requiring domain knowledge. Meanwhile, the emergence of benchmarks like MMMU (Yue al., 2023) further pushes the boundaries by requiring a college-level reasoning across disciplines like physics and art. With the development of test-time compute scaling (Jaech et al. 2024; Cui al., 2025), mathematical benchmarks requiring complex reasoning processes to obtain the answer are emerging as critical benchmarks for evaluating the reasoning capabilities of MLLMs. For instance, Math Vista (Lu al.) 2023) systematically categorizes seven mathematical reasoning types through multimodal problem decomposition. Math Vision (Wang et al.) 2024a) curates competition-leves mathematical problems with authentic visual contexts. Mathverse (Zhang et al.) 2024) introduces comprehensive multi- modal benchmark specifically designed to assess the visual mathematical reasoning capabilities of MLLMs. Although these benchmarks have expanded the scope of evaluation to various domains, there is still a lack of systematic evaluation in the widely applied field of text-rich image understand- ing. The text-rich image encompasses numerous scenarios requiring reasoning, such financial report analysis, invoice analysis, cost-effective purchase decisions, and more.
2.2 TEXT-RICH IMAGE UNDERSTANDING BENCHMARK
The evolution of Multimodal Large Language Models (MLLMs) has driven corresponding advance- ments in text-rich image understanding benchmarks. Early benchmarks for text-rich image under- standing predominantly focused on assessing the perception capabilities of MLLMs within individ- scenarios, such as documents (Mathew et al. (02T), charts (Masry et al.
2022). infographic
images (Mathew et al. 2022), and scene text (Singh et al. 2019: Biten et al, 2019 In parallel,
recent advancements in high-resolution image processing Yee et al.) 2023 Li et al.) 2024c Huang
iet al. Hu et al., 2024b] Guan et al. 2025| Liu et al. 2024a) and optimized computational effi- ciency (Liu et al. 2202e)] Hu et al. 2025 Zhang et al. 2025a Yu et al. 2024) have significantly improved the performance of these benchmarks. To address the growing need for holistic evalua- tion of MLLMs, a series of benchmarks with broader, more diverse, and complex scenarios have emerged /Wadhawan et al. 2024 Li et al. 2024a Liu al. 2024c d Ouyang al. 2024)) For instance, OCRBench (Liu et al. (024d). CC-OCR Yang et al. 2024b), and OCRBenchy2 (Fu et al., 2024) concentrate on assessing the perceptual capabilities
MLLMs across multiple domains, while OmniDocBench (Ouyang et al. 2024) provides comprehensive evaluation of PDF docu- ment parsing. However, despite these advancements, with the emergence of slow-thinking systems requiring deliberate reasoning, current benchmarks reveal two critical limitations: 1. Overempha- sis on textual extraction tasks (Mathew et al., 2021 Xu et al., 2025), which can be solved through fast-thinking processes; 2. Lack of systematic assessment of reasoning capabilities in text-rich im- age understanding. This progression highlights the pressing need for next- next-generation benchmarks to evaluate MLLMs' complex reasoning capacities in text-rich visual understanding. To address this limitation, we propose a comprehensive benchmark specifically designed to assess multimodal slow-thinking systems in complex text-rich image reasoning tasks.
3

---

Published as a conference paper at ICLR 2026
Grmm
Lnternet
Open-Srunce
Iase
Human Expent
Quessons
Asswe: Proces
Step I
Step II Annotation Curation
Tmage Collection
Engen
Step Data Taxonomy
Step Dataset Correction
Reasoning
Sputisi Recccringg
Lugical Resoming
High Quality
Namenccll Analyss
Mathomatical Resuming
Resoning
Enumenntive Rocuning
Multidscisliaryy Kruncidyy Reasening
Amauate
Exgert
Quality Contol
Comection
Control
Figure 2: Data curation framework of OCR- Rea-Rnesnnnng. The framework includes: (1) dataset collec-
tion, (2) annotation curation, (3) data correction, and (4) detailed taxonomy.
3 OCR-REASONING
In Sec. 3.1 we first present the data curation framework of OCR- OCC-RResonnnn, comprising: (1) dataset collection, (2) annotation curation, (3) data correction, and (4) detailed taxonomy. The data cura- tion framework is shown in Fig. [ Then, in Sec. 3.2 we describe the statistics of OCR- Reasoninn,, including its total scale, categorical distribution, and detailed question-answer characteristics. No- tably, while existing benchmarks (Mathew al. 2022 Masry et al. 2022 Liu al. 2024d) focus solely on final answers, OCR- Reasonin provides annotations for both the final answers and the step-by-step reasoning process, facilitating more in-depth evaluation of MLLMs' reasoning capa- bilities. The statistics of the annotations are presented in Sec. 3.2
Additionally, OCR-Reasoning focuses challenges in single-image. This design choice is based on two well-founded design principles: 1. Capability isolation and focused evaluation: Multi-image or multi-document tasks primarily assess long-context processing capabilities, which require spe- cialized benchmarks for proper evaluation. Mixing single-image reasoning with multi-document challenges would confound the evaluation and make difficult to isolate specific reasoning defi- ciencies. 2. Model compatibility and fair evaluation: Several document-oriented MLLMs (Wang al. 2025c Xiao et al. 2025; Guan et al., 2025) only focus on single images and have not been trained on multiple images. Including multiple images in the benchmark would exclude these im- portant models from evaluation, potentially reducing the focus on the reasoning problem itself.
3.1 DATA CURATION FRAMEWORK
Dataset Collection. We constructed the OCR-Reasoning dataset by aggregating images from three
primary sources: (1) Internet-sourced images from publicly available online repositories, (2) real-
world photographs capturing street views and handwritten notes, and (3) images curated from es- tablished benchmarks including InfoVQA (Mathew et al. (022). DocVQA (Mathew et al.) 1021) ChartQA (Masry al. 2022), CharXiv (Wang et al. 2024c), WildReceipt (Sun al. [02 D. and MME-Finance (Gan et al. (2024). During data collection, we prioritized comprehensive coverage of text-rich scenarios commonly encountered in daily life. We also noted a severe lack of reason- ing data related to handwritten content. To address this, our annotators selected and transcribed college-level problems in chemistry, physics, geometry, functions, and statisties, which were then photographed to create a set of handwritten reasoning data. In addition to college-level problems, we also include a portion of handwritten data about logical reasoning tasks. We filtered out those with low resolution excessive noise. The final dataset comprises 1022 images, scale comparable to previous reasoning benchmarks (Lu et al.) 2023 Bi et al. 2025) Xu et al. (025). It consists of 476 Intemee-souured images, 253 real-world photographs, and 293 images from established benchmarks.
Annotation Curation. After collecting the images, we proceed to annotate them. First, our anno- tators will design questions based on the images evaluate the reasoning ability of MILMs. To
4

---

Published as a conference paper at ICLR 2026
Numurica Anabyb Reavelng
Makenakon Rowekk
brrtect Ei buy place por sricce
4
50
8
4
bevig son 4dl dena
and
Migh beli Be Firt aii..
3
3
86
Roch and apprect mode
eiy the colicing
7ED
Logical
Report
Quardio Thar Aoysl smttth thes
EXITE
S.C.% IN 12 14
Figure 3: Examples of different categories in OCR-Reasoning OCR-Reasoning includes six cate- gories: spatial Reasoning, numerical analysis reasoning, mathematical reasoning, enumerative rea- soning, logical reasoning, and multidisciplinary knowledge reasoning.
Figure 4: Subject Distribution of OCR- OCR-Reasoning.
Table I: Key Statisties of OCR- Reasoning.
Statistic
Number
questions
1069
Mulliple-choice questions
250 (23. (%8)
Loptol
Free-form questions
819 (76.6%%)
30.0%
Nowly collected question
987 (92.39%)
Newly collected reasoning path
(100.0%)
Number unique images
1022
5..
Number of unique quastions
1069
Number unique snswers
1069
Maximum question length
393
Clmnltt Partens
Maximum answer longth
3106
Average question length
Average answer length
421
Couddey
guarantee the quality of the data, engage PhD candidates in STEM fields as expert annotators. For each image, three annotators independently propose a question. Then, other annotators score and select the highest-quality question. Subsequently, we generate reasoning processes and answers through two parallel pathways: 1. Human Annotation: Annotators manually produce one reason- ing process along with the corresponding answer. Model-Based Generation: We input both the questions and answers into closed-soured MLLMs (c.g., Gemini 2.0 Flash) to generate an alternative reasoning process and answer.
Data Correction. After obtaining the questions, reasoming processes, and answers, three annotators evaluate the annotations from both pathways. The pathway with the highest average score chosen as the final reasoning process and the corresponding answers. Finally, manual review step is conducted to examine and correct all question-answe pairs and reasoning processes.
Data Taxonomy. After completing the data annotation process, we will categorize the data into six categories based on the reasoning skills required to answer the questions. To mitigate human bias, we implement two-stage classification approach. In the initial phase, three annotators inde- pendently classified each example into one of six predefined categories. Then, we implemented a majority voting system where the final category assignment was determined by plurality consensus among the three annotators. The definitions of each category are as follows: Spatial Reasoning fo- cuses the model's ability reason about spatial relationships between text and visual elements, as
5

---

Published as a conference paper at ICLR 2026
well as layout elements within text-rich images. Numerical Analysis Reasoning involves calcula- tions related to numerical variations in text-rich images, including cost-effective purchase decisions, growth rate estimation, financial report analysis, schedule planning, and data interpretation. Numer- ical Analysis Reasoning also encompasses samples related to scenarios involving web screenshots, financial documents, product manuals. Mathematical Reasoning entails solving mathematical problems (e.g., functions, geometry, statistics) in text-rich images using mathematical knowledge. Compared to other mathematical benchmarks, the mathematical reasoning-related data in our bench- mark is handwritten by our annotators, which requires models to possess stronger OCR capabilitiess accomplish these tasks. Enumerative Reasoning focuses on counting text and visual elements in text-rich images that satisfy specific query conditions. Logical Reasoning requires critical thinking and drawing inferences from provided text-rich images to arrive at conclusions, Multidisciplinary Knowledge Reasoning involves applying cross-domain knowledge (e.g., physics, chemistry) to in- terpret text and visual elements in text-rich images. We provide some examples in Fig.
3.2 DATASET STATISTICS
The key statistics of OCR- ReR-Reennnnng are summarized in Tab. 1 questions categorized into two distinet formats: multiple-choice (with provided answer options) and free-form responses. The free-form answers are further classified into three data types: integers, floating-point numbers, and strings. Notably, our benchmark contain extended analytical reasoning processes, evidenced by average combined length of 421 characters for reasoning chains and final answers. The maximum length reaches 3,106 characters, highlighting the complexity of the OCR-Reasoning, As shown in Fig. 4 the question distribution spans six reasoning categories: Spa- tial Reasoning (10.0%), Numerical Analysis (37.0%). Logical Reasoning (12.8%), Mathematical Reasoning (10.2%), Multidisciplinary Knowledge (13.5%), and Enumerative Reasoning (16.7%). Numerical Analysis Reasoning covers 5 real-world task types, more than the 2-3 task types in other categories, hence accounts for larger proportion. More examples are presented in the Sec. A.8
This benchmark contains 1,069
3.3 EVALUATION PROTOCOLS
Following previous methods (Lu et al. 2023 Zhang et al.] 2024), OCR-Reasoning adopts a three- stage evaluation framework: (1) Response Generation, (2) Answer Extraction, and (3) Score Com- putation. First, the multimodal large language model (MLLM) processes an input query to generate detailed responses. Subsequently, an LLM-based answer extractor (c.g. GPT-4o) extracts concise answer text from these responses through semantic parsing. Our preliminary study on 200 exam- ples shows that this extraction process achieves over 99.5% accuracy. Finally, the extracted answers undergo normalization into standardized formats (e.g., option letters, integers, or strings) before accuracy- based metric calculation for deterministic evaluation.
For the evaluation of reasoning processes, inspired by evaluation in large language models (Zheng et al. 2023 Chang et al. 2024), we employed the L.L.Mas-judge (Zheng al. 2023) approach to as- sess the reasoning process. Given question, detailed response from an MLLM, I.M, and a ground truth of the reasoning trajectory, an LLM judge is asked to directly assign a score to detailed responses.
Our adoption of this methodology is based on solid empirical justification: 1) Human evaluation costly, while LLM can quickly process large amounts of data. 2) LLM can reduce the variance among human evaluators. 3) LLM Judge is commonly used in NLP to evaluate reasoning pro- cesses. We tried to use human-grounded validation across different models to compare their scores with those of the LLM-as-Judge: the uroanddrounded validation score for DouBao- 1.5- is 53.1 (vs. LLM-as-Judge score 55.4), for (wwen..5-VL-725 is 50.2 (vs. LLM-as-Judge score 51.8), for Sluma4-Scout 109B-A1 is 43.8 (ys. LLM-as-Jude -Jddge score 44.9). and for OpenAI-ol is 47.6 (vs. LLM-as-Judge score 48.5), The scores of human-groundee validation are close to those of the LL---asJudge.
4 EXPERIMENT
In this section, conduct comprehensive evaluation of existing MLLMs OCR-Reasoning We first describe the experimental setup in Sec. 4.T Then, the overall results and the corresponding analysis presented in Sec. 4.2

---

Published as a conference paper at ICLR 2026
Table Accuracy scores on the OCR-Reesoningg The results include OCR LLM, closed-source MLLMs, and open-source MLLMs. Bold denotes the best performance.
Method
Overall
Spatial
Numerical Analysts
Mathematical
Eaumerative
Logical
Multidisciplinary Knowledge
OCR + LLM
OpenA--o--iin Open
DerpSeck-RR B---skk-R--i-iiii (wen2.5-328 Yang of 20246
2025p
26.5
23.5
34.6
18.8
30.7
26.5
13.7
29.9
16.6
29.1
26.4
31.4
Closed Source MLLM
Gemini Flash Team 2023
39.3
19.3
24.5
36.8
32.1
GPT-4e [flupt
OpenAl-ol Jach C A02
44.4
27.5
43.1
50.8
40.3
49.6
Clande-1 7-Somet (Anliruns 2055
35.8
20.2
35.4
23.5
60.3
30.6
32.
DonRoo- Pis
M125h
46.8
27.5
$4.0
33.3
50.8
34.7
58.4
Open Source MLLM
0wen2.5-VL-33
12.2
Qwen2.5-VLL-88
2072
15.7
20.1
35.8
DV-32.5-V1-3225
Fir 70775
36.2
21.1
38.7
25.5
46.9
34.7
36.5
Quen2.5.V1-728
Ban el 2023
37.5
34.8
44.7
22.5
47.5
28.5
34.5
IntemVL3-2B 2ht
10.8
2096
ImcmVL3-32BB
Z203 et
20P5
17.1
14.7
10.3
14.7
24.0
11.8
37.2
19.9
13.8
22.4
98
14.0
27.1
23.5
Llams4 Seour-corrrin 1198-1188 Iiter
E025
27.7
15.6
34.7
16.7
41.3
22.9
12.4
Kimi Be-i---L--BBBTaaae Thinking [eam
2015g
20.5
22.4
21.5
19.7
VL-Rothinkee-7B
MM--Encka (Mtiig
(Xtvin ce R0AS
13.2
15.3
27.0
IIAA-Thhinan-Q0e222 1-17 (CNirr 2025a
14.4
11.9
10.3
78
21.2
11.8
27.0
QvQ [Wang al 20244
32.7
24.8
34.7
15.7
44.1
31,9
32.
Keye-VL.:3F
2025h
22.6
17.4
30.7
Thyme RL-7B5-2hann
DeepEyesV2 [llong di R0463
20.9
11.9
18.8
13.7
27.9
18.85
32.8
MiMo VL. VL-RL-7B Assiin R0A4
38.8
20.2
41.2
22.5
$1.4
38,9
42.3
GLM-4.1V- 1V.M-..V.Thinkin 9B
2025h
44.1
22.9
49.2
35,3
53.1
35.4
30.4
Document- Oriented MLLMs
mPLUG-DexOw112-B
Docopilot Dean 2025
11.5
11.9
6.9
22.6
DocMark-22 2B A0d0 2
4006
7.4
8.3
0.3
6.9
11.5
3.5
22.6
TokenVL-8B 2025
14.3
10.1
8.9
25.7
13.9
23.4
4.1 EXPERIMENT SETUP
Evaluation Models. We evaluate three distinct types of foundation models on OCR-Reasoning
(a) Large Language Models (LLMs) with OCR results (Extracting by PP-OCRv3 (Li et al., 2022 )
including Deepseek-R1 (Zhou al. 2025) and OpenAl-o3-mini. (b) closed-source MLLMs, com-
prising Gemini-2.0-Fii (Dee 2025), GPT-4o (Hurst et al. 2024). OpenAl-ol (Jaech et al.
2024). Claude-3.1 3.ndde--.--Sonett (Anthropic, 2025), and DouBao-1.5.- Vision- (Guo al.] 2025b). (c)
Open-source MLLMs, represented by models like Qwen2.5-VL (Bai et al.
2025). Intern (Zhu
2025 Llamm4-Scout (Meta 2025). Kimi-VI. (Team et al. 2025a). VL-Rethinker (Wang al.
et al. 2025a), , MM-Fureka (Meng et al. (022), VLAA-Thinker (Chen et al. 2025a). (d) Document-
Oriented MLLMs, including mPLUG-Docome
(Hu al.) 2025). Docopilot (Duan
2025
DocMark (Xiao et al. 20255), and VL (Guan et al. (0255).
Implementation Details. To evaluate the generalization capacity of Multimodal Large Language Models (MLLMs), we adopt a zero-shot evaluation protocol without model fine-tuning or few-shot prompting. Following the standardized Srain-of-thoughn paradigm. we present MLLMs with both visual inputs (images) and textual questions, accompanied by explicit instructions: "Solve the com- plex problem through step-by-step reasoning." For text-only Large Language Models (LLLss,, we substitute visual inputs with the OCR results (using PP-OCRv3 (Li et al. 2022) to obtain the OCR results) while retaining identical textual queries. Given the inherent variability in output formats across text-rich image scenarios (e.g., monetary values like $15, temporal expressions like 20 days, or timestamps like 19:00:00), we implement format- format-specific prompting. This involves appending wemplemeen the directive: "The composition of the final answer should be: xxxxx" to each query. For instance, when expecting currency outputs "$15", the format-specifie prompting is: "The composition of the final answer should be: $ Integer""

---

Published as conference paper at ICLR 2026
Table Impact of Chain-of-Thoughh prompting on different MLLMs.
Method
Spatial
Numerical Amalysis
Mathematical
Enumerative
Logical
Multidiscpliiary Knowledge
Qwen2.5-VL-3228
Bai 2 A02S
33.0
12.8
33.7
24.5
28:4
38.7
S-V-2.5-V1-1288 15r ATAS
36.2
21.1
38.7
25.5
46.9
34.7
36.5
(PPT-4o Hurs a7 2024
16.5
11.9
33.4
15.7
29.
25.0
24.
GPT-4e
30.7
35,9
26.4
23.4
Kimi-Vi
20.5
19.7
V1--eethnker-7B Wane Sall 237330
19.1
13.7
16.6
98
25.7
14.6
33.6
VL-
ML-Reehinker-1B Wann of 20263
14.6
8.3
16.1
9.8
19.6
8.3
19.0
MM--Erekaa Owed-7B (slle- S
2025
12.2
14.6
25.5
MM-Eureka- Qwen- (Meng
Table 4: Reasoning scores on the OCR-R OCR-Reasoning benchmark. Bold denotes the best performance.
Method
Overall
Spatial
Numerical Analysii
Mathematical
Enumerative
Logical
Muttidisciplinary Knowlodge
Cloord-Source MILLM
Gemini
GPT-te [Hupt et p0Pel
45.4
18.9
33.0
18.7
48.0
45.5
OpenAhol Dacch 6 2024
48.5
36.9
53.9
50.0
39.4
49.4
51.8
Clande-3.7 7-ScomeT (Anficung 2025
50.3
37.7
$5.0
38.8
58.1
48.6
46.5
DonRoo-
55.4
38.2
Open- Speu-Source
Owen2.5-VL-38
Bai al. 2025
22.3
18.5
25.6
15.7
22.9
20.8
21.3
Owen25--VL-78 1010 A0243
34.0
24.9
39.2
27.5
11.5
28.9
27.
SVL-2.5-VL-322
$4.6
38.5
$8.9
45.9
55.8
56.3
$7.9
Qwen2.5-V1.2225
ImemVL3-8B
al 1026
16.3
16.1
14.9
13.8
21.5
15.0
16.6
IntmmVL3-3215
2200
01. 2013
42.5
32.1
43.9
38.0
45.3
43.6
46.
InimmVL3-78B
43.3
29.2
50.3
35.5
38.6
46.9
12.2
Llams4 1ooul-1199-AA7/11100000
2005
Kimi-VI..A Senn-VI..AAR-TThiaa
61125a
VL-Rothinker-7B Wang Vill PAYaa
29.8
23.1
23.7
31.1
26.5
29.8
MM-Euncka Ma-EEnna-QQa770 (atiig Le 2015
21.9
20.7
20.5
16.5
24,8
22.2
26.9
VIAAA-Thika--Quaa Maroik---enn22.111.
20755
24.8
22.2
25.8
18.0
25.9
24.2
28.5
Documeen-OOriented MI.LMs
mPLUG.DecOwI2-.
Docopilot-80 SB Dean 2025
20.5
17.1
22.4
25.2
19.6
19.1
DocMark-22 2B
X100 2003
15.3
13.9
13.5
12.3
22.7
15.3
13.9
4.2 OVERALL RESULTS
The use visual images as input crucial. To assess its importance, we replace the images with their OCR results and feed them into the LLMs for comparison. As shown in Tab. 2 substitut- ing image input with OCR text leads to a significant decline model performance. For instance, when using the same LLM, the performance of even strong reasoning model like DeepSeck-RI- Distill-Qwen-322 remains 9.7% % lower than that of Qwen2.5- 5- VL-32B. This demonstrates the critical importance of image input for text-rich image reasoning tasks. We present qualitative results in Appendix A.3
The performance of current MILMs still has significant room for improvement. Specifically, our analysis reveals several key ohservations: As shown in Tab. [ the top-performing model Doubao-I.5-Vision-PPe While Doubao- 1.s.aao--..--iioo-----
performs strongly on text-rich image un- derstanding tasks-Such as DocVQA (9..7%), InfoVQA (89.3%) and ChartQA (87.45 (87.4% (87.4%)its per- formance OCR-Reasoning does not exceed 50%. This highlights the particular challenge of integrating visual, textual, and logical information in reasoning scenarios. 2. Among different rea- soning types, MLLMs perform most strongly on enumerative reasoning, which consistently ranks as the first or second best capability in both closed-source and open-source models. 3. Furthermore, scaling up model parameters is positively correlated with performance gains, as illustrated by the Qwen2.5-VL series: the 7B model surpasses the 3B version by 3.5%, and the 32B model outper- forms the 7B version by 20.5%. Additionally, we observe that document-orientee MLLMs still face difficulties in complex reasoning. Although document-oriented MILMs effective basic comprehension, their limitations deeper reasoning underscore the need for innovations in model architecture training strategies.
CoT prompting performs differently across different models. The results of the influence of CoT prompts presented in Tab. 3 We observe that CoT prompting performs differently across
8

---

Published as a conference paper at ICLR 2026
Table 5: Impact of few-shot prompting on Qwen2.5-VL-7B.
Numerical Analysis
Multidisciplinary Knowledge
Method
Overall
Spatial
Mathematical
Enumerative
Logical
Qwen2.5-VL. Bai et 2025
15.7
13.8
11.6
8.8
20.1
9.0
35.8
One-shor prompting Three-shot prompting
16.1
12.8
14.6
10.8
22.3
13.9
21.1
22.9
13.2
different models. On most models, CoT prompting consistently enhances their capabilities. This improvement particularly pronounced in spatial reasoning, where CoT prompting exerts the most significant impact. Specifically, CoT prompting improves performance by 3.2% on Qwen2.5-VD 32B, and 4.2% on GPT-4o, respectively. However, for VL-Rethinker-7B (Wang al.) 2025a). the application of CoT prompting typically results in performance degradation. This phenomenon may stem from the forced rethinking machine on Vl.-Rethinker-7B. Adding an additional CoT prompt during inference creates discrepancy between training and testing conditions, ultimately leading to reduced performance.
Impact of Reasoning path. The scores of the reasoning path are presented in Tab. 4 Overall, observe that the ranking based on Reasoning Path scores aligns with that based on final answer accu- Specifically, the high scores of Gemini
racy, with the exception of Gemini and (laude-377-Sonee and Claude-3.7-Somee are primarily due to the high quality their reasoning path. We find that for many erroneous samples, the reasoning steps largely sound, with only minor errors leading to the incorrect outcome. Additional qualitative analyses are provided the appendix (Sec. A.2 to further illustrate these observations.
Some reinforcement learning methods perform poorly on text-rich image reasoning tasks. The performance of some reinforcement learning methods on text-rich image reasoning tasks is relatively poor compared to their baseline. There are several possible reasons for this. First, the reward function: The reward functions in these reinforcement learning methods are not specifically designed for text-rich image reasoning tasks. Most existing reward functions are tailored for mathematical reasoning tasks. How design reward function applicable to text-rich image reasoning tasks is a highly worthwhile research direction. Second, a notable discrepancy exists between the training data and the benchmark. The majority of training data primarily designed for printed mathematical problems, while our benchmark contains data from wide variety of scenarios. How select training data to improve OCR inference performance a highly valuable research direction.
Impact of Few-shot Prompting. We also conducted experiments to explore the impact of few-shot prompting. We annotated three additional samples as few-shot demonstrations and validated the performance of one-shot and three-shot prompting Qwen2.5-VL-7B. As shown in the Tab. 5 few-shot prompting improves overall coltrannce-partcclllle specific logical steps (e.g., Numerical Analysis Reasoning and Logical Reasoning). This demon- strates that moderate task-specific guidance helps the model understand and comply with task re- quirements. We observed decline in the performance of Multidisciplinary Knowledge Reasoning. The potential reasons may be: The increased length of input tokens caused by few-shot examples, combined with the extended reasoning path inherently required by multidisciplinary knowledge rea- soning, poses significant challenge to the model's long-text processing and reasoning capabilities.
on subtasks requiring adherence to
The "thinking with images" approach represents promising direction for enhancing rea- soning capabilities over text-rich images. "Thinking with images" has demonstrated tremendous potential in general-scent -cene reasoning; therefore, we also evaluated the two latest "thinking with im- ages" methods the OCR-Reasoning task. The experimental results are presented in Table 2 which illustrate the potential of this approach to enhance the model's reasoning ability on text-rich images.
4.3 COMPARISON WITH EXISTING BENCHMARKS
We conducted experiments to quantitatively compare our OCR-Reesoning dataset with existing benchmarks (including VQA, CharQA,, TextVQA, OCRBench) using three multimodal large language models (MLLMs): Twen2.5-VL-78 InterVVL--SB. and Token VL-8B. The experimental results are presented in Tab. 6 As observed, while existing methods achieve strong performance

---

Published as a conference paper at ICLR 2026
conventional datasets, they exhibit significant performance degradation on text-rich image rea- soning tasks. This discrepancy stems from the fundamental difference in evaluation focus: existing datasets primarily assess models' perceptual capabilities (e.g., text detection, recognition, and basic visual-linguistic alignment), whereas OCR- Reasonng requires the model to achieve accurate per- ception and further conduct thinking and reasoning. These findings indicate that the text-rich image reasoning ability of current MLLMs still has substantial room for improvement.
Table 6: Comparison with Existing Benchmarks
Models
DocVQA
CharQQ
OCRBench
TextVQA
OCR Reasoning
Qwen2.5-VLL V-VLBB
95.7
87.3
864
84.9
15.7
InternVL.3 8B
92.7
86.6
880
80.2
11.5
TokenVL--S3
94.2
86.6
860
79.9
14.3
4.4 ERROR ANALYSIS
We analyzed the enror types of DouBao-Ia -isionn
Knowledge application errors
Summary errors
Pro and classified them into six major categories: (1)
calculation errors (37.5%) (2) spatial comprehen-
sion errors (27.2%), (3) logical errors (19.9%). (4)
perception
crrors
Calculation errors
text perception errors (7.7%), (5) knowledge appli- cation errors (5.6%), and (6) summary errors (3.1%).
Calculation errors occur when layout understand- ing and text perception are correct, but mistakes are made during the calculation process. Spatial com- prehension errors arise when the model fails to prop- erly understand the spatial layout information of text in images. Logical errors result from unreasonable assumptions or inverted cause-and-effect relation- ships. Knowledge application errors occur in cases such as the incorrect application of theorems or mis- understanding of common sense. Text perception er- rors happen when text recognition incorrect. Sum- mary errors occur when the reasoning process is nor- mal but the final answer is incorrect. This distribu- tion reveals that the DouBao-La 1.ttao--1..... Vision-Pro pri- mary challenges lie in higher-level cognitive tasks rather than basic perception tasks.
Logical
errors
19.9
Spatial comprehension errors
27.2
Figure 5: Error analysis for DouBao- 1.5-- Vision-Pro reveals six main issues: Cal- culation errors, Spatial comprehension er- rors, Logical errors, Text perception errors, Knowledge application errors, and Summary errors.
5 CONCLUSION
In this paper, we introduce OCR-RRcaonning a comprehensive benchmark designed to systematically evaluate the reasoning capabilities of state-of-the-art Multimodal Large Language Models (MLLMs) in text-rich image scenarios. The benchmark provides structured framework comprising 1,069 human-annotated examples, which cover 6 core reasoning abilities across 18 distinct tasks. con- trast to existing benchmarks that focus solely on final-answer accuracy, OCR- ACC-RResonnig incorpo- rates annotations for both the final answers and the corresponding step-by-step reasoning processes. This dual annotation enables a more comprehensive evaluation of model reasoning. Through exten- sive experiments, we reveal critical limitations of current MLLMs and identify potential avenues for future improvement.
Limitations. This work has two main limitations. First, since most of our data collection and annotation processes are performed manually, the high costs associated with these processes have resulted in our dataset size being comparable to previous methods (Liu et al. 2024d; Xu et al.) 2025). In the future, we plan to combine automated annotation with human efforts to expand the dataset scale. Second, following the evaluation process of LLMs, we employed the LLMs-as JJsdgess approach assess model reasoning processes. However, issues such biases in LLMs-as-Judger adversarial attacks, and inherent weaknesses in the methodology may affect evaluation accuracy (Li
et al. 2024b). We intend to develop more robust evaluation approaches in future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research supported in part by the National Natural Science Foundation of China (Grant No.:62476093).
REFERENCES
Anthropic. Claude 3.7 sonnet extended thinking.
Anthropic System Card, 2025.
URL
https://asset.:
c claud--3-7-somme---yyteeemm
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502.1392 2025.
Jing Bi, Junjia Guo, Susan Liang, Guangyu Sun, Luchuan Song, Yunlong Tang, Jinxi He, Jiarui Wu, Ali Vosoughi, Chen Chen, et al. Verify: A benchmark of visual explanation and reasoning for investigating multimodal reasoning fidelity. arXiv preprint arXiv:2503.1 11557, 2025.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291-4301, 2019.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1 2024.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training rl-like reasoning large vision-language models, 2025a. URL https: //arxiv wwtgabs/////eooo//e//..ooe.essee 2 2504 11468
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. http://s///gittt..ccoo.eec Agent/
R1-V 2025b. Accessed: 2025-02-02.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. 2025.
DeepMind. Gemini 2.0 flash thinking. Google DeepMind website, 2025. URL https://
deepmind nd. www.gggggeeeeecoollii.... htt.:///eeeeni/llla---hh...aaaaa
Yuchen Duan, Zhe Chen, Yusong Hu, Weiyun Wang, Shenglong Ye, Botian Shi, Lewei Lu, Qibin Hou, Tong Lu, Hongsheng Li, et al. Docopilot: Improving multimodal models for document-level understanding. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 0026-4037, 2025.
Ling Fu, Biao Yang. Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang. Hao Lu, Mingxin Huang, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.0032.. 2024.
Ziliang Gan, Yu Lu, Dong Zhang, Haohan Li, Che Liu, Jian Liu, Ji Liu, Haipang Wu, Chaoyou Fu, Zenglin Xu, et al. Mme-finance: multimodal finance benchmark for expert-level understanding and reasoning. arXiv preprint arXiv:241 1,03314, 2024.
Tongkun Guan, Zining Wang, Pei Fu, Zhengtao Guo, Wei Shen, Kai Zhou, Tiezhu Yue, Chen Duan, Hao Sun, Qianyi Jiang, et al. A token-level text image foundation model for document under- standing. arXiv preprint arXiv:2503.022 020000003.0 2025.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 12948, 2025a.

---

Published as a conference paper at ICLR 2026
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seedl. 5-vl technical report. arXiv preprint arXiv:2505.0706, 2025b.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang. al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024.
Jack Hong, Chenxiao Zhao, ChengLin Zhu, Weiheng Lu, Guohai Xu, and Xing Yu. Deepeyesv2:
Toward agentic multimodal model. arXiv preprint arXiv:2511.05271, 2025a.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv e-prints, pp. arXiv-2507, 2025b.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understand- ing. arXiv preprint arXiv:2403 12895, 2024a.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. haplug-dcoow 1.5: Unified structure learning for ocr-free document understand- ing. arXiv preprint arXiv:2403. 20t.:22903./8995 12895, 2024b.
Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jin- gren Zhou. IPLUG-DocOw12:: High-resolution compressing for OCR-free multi-pan page document understanding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5817-5834, 2025.
Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. Mini-monkey; Allevi- ating the semantic sawtooth effect for lightweight milms via complementary image pyramid. In International Conference on Learning Representations.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1700-6709, 2019.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2127,, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint 2404 240..1679, 2024a.
Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, et al. Pp-ocrv3: More attempts for the improvement of ultra lightweight oer system. arXiv preprint arXiv:2206.0300: 2022.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yigun Liu. Limseas-judges: A comprehensive survey on Ilm-based evaluation methods, 2024b. URL https://arxtt httt::///aarav.oorra//////2222.
12

---

Published as a conference paper at ICLR 2026
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang. Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi- modal models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26763-26775 2024c.
Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, and Linli Xu. Hrvda: High-resolution visual document assistant. arXiv preprint arXiv:2404.0691 06918, 2024a.
Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jian- jian Sun, Chunrui Han, and Xiangyu Zhang. Focus anywhere for fine-graines multi-page docu- ment understanding. arXiv preprint arXiv:2405. 14295, 2024b.
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mmc: Advancing multimodal chart understanding with large-scale in- struction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1287-1310, 2024c.
Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haoman Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint 2r0: 13055, 2025a.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr large multimodal models. Science China Information Sciences, 67(12):220102 2024d.
Yuliang Liu, Biao Yang. Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint SXXv:22033.0473.. 2024e.
Zhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, and Jun Wang. Othink-mrl: Stimulating multimodal generalized reasoning capabilities through dynamic reinforcement learn- ing, arXiv preprint arXiv:2503. 16081, 2025b.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint (rXiv:2503.01788 2025c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. 2023.
Ahmed Masry, Xuan Long Do. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-222222222 2279, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvga. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu. Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning, arXiv preprint 0rXiv:2503.07365, 2025.
13

---

Published as a conference paper at ICLR 2026
AI Meta. The Ilama 4 herd: The beginning of new of natively multimodal ai innovation. checked on, 4(7):0025, 2025.
https://ai. meta. hatpi///lllaa-----mimimmmmm---ee
Open AI Team. Openai o3-mini.
2025.
Accessed: 2025-01-31.
Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, al. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. arXiv preprint arXiv:2412.07666, 2024.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprini 3rXiv:2503.07536, 2025.
Haozhan Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao.
Vlm-rl: A stable and generalizable rl r1-style large vision-language model. https:: htttps://github.
com/ fomm//omaat aomm-ai-- aia.aaaalllaa----aaa R1, 2025. Accessed: 2025-02-15.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Hongbin Sun, Zhanghui Kuang, Xiaoyu Yue, Chenhao Lin, and Wayne Zhang. Spatial dual-modality graph reasoning for key information extraction. arXiv preprini 103.::22033.44::: 2021.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang. Enming Yuan, Enzhe Lu, Fang Li, Flood Sung. Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang. Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng. Jiaming Li. Jianlin Su. Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Meng- fan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng. Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang. Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, and Ziwei Chen. Kimi-VI. technical report, 2025a.
URL httts::::: http:://xxvvvvvroo org abs/ 2504. 07491
Kwai Keye Team, Biao Yang, Bin Wen, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, et al. Kwai keye-vl technical report. arXiv preprint arXiv:2507.01949, 2025b.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https:// 1m. m...gitubl ho/blog//wg-3220
Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang. and Nanyun Peng. Contextual: Evaluat- ing context-sensitive text-rich visual reasoning in large multimodal models. arXiv preprint arXiv:2401.13311. 2024.
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837 2025a.
Ke Wang. Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math-visie vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024a.
14

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shengiong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.1029/ 2025b.
Zining Wang. Tongkun Guan, Pei Fu, Chen Duan, Qianyi Jiang, Zhentao Guo, Shan Guo, Junfeng Luo, Wei Shen, and Xiaokang Yang. Marten: Visual question answering with mask generation for multi-modal document understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14460-14471, 2025c.
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 17:113369-113666 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837. 2022.
Han Xiao, Yina Xie, Guanxin Tan, Yinghao Chen, Rui Hu, Ke Wang, Aojun Zhou, Hao Li, Hao Shao, Xudong Lu, al. Adaptive markup language generation for contexually-groondees vi- sual document understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19558-295666 2025.
LLM-Core-Teal Xiaomi. Mimo-vl technical report, 2025. URL htt.:tt://axxv..rr//ee 15506.3566)
Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504. 15279, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprini arXiv:2412.15115 2024a.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao. Minfeng Zhu, Bo Zhang, and Wei Chen. RI-onevision: Advancing general- ized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, Humen Zhong, Xuejing Liu, Mingkun Yang, Peng Wang, Yuliang Liu, al. Cc-ocr: A comprehensive and challenging ocr benchmark for evaluating large multimodal models in literacy. arXiv preprint arXiv:24/2.02210, 2024b.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
Ya- Qi Yu, Minghui Liao, Jiwen Zhang, and Jihao Wu. Texthawk2: A large lasion-language model excels in bilingual ocr and grounding with 16x fewer tokens. arXiv preprint arXiv:2410.05261, 2024.
15

---

Published as a conference paper at ICLR 2026
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi multi-disciplime multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311 2023.
Jiaxin Zhang, Wentao Yang, Songxuan Lai, Zecheng Xie, and Lianwen Jin. Dockylin: A large mul- timodal model for visual document understanding with efficient visual slimming. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 9923-9932, 2025a.
Renrui Zhang. Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, al. Mathverse: Does your multi-modal IIm truly see the diagrams in visual math problems? In European Conference Computer Vision, pp. 169-186. Springer, 2024.
Yi-Fan Zhang. Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint arXiv:2508. / /630, 2025b.
Liammin Zheng. Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
with mt- bench and
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsich. RI- zero's" aha moment" in visual reasoning on 2b non-sft model. arXiv preprint arXiv:2503.05132, 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv1:: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 10X49 2025.
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE. MODELS(LLMS)
We used LLMs only to polish and correct the grammar of this paper. This involved rephrasing sentences, checking grammar, and improving the overall flow for better readability and clarity. All research ideation, experiments, and analyses were developed by the authors, with no involvement from the LLM in these stages.
A.2 QUALITATIVE ANALYSIS OF REASONING PATH
Here, we illustrate this with example. As shown in Fig. aa), (b), and(c), observe that in mod- els such as Gemini and Slaude-3.7-Sonnee many errors occur where minor mistakes in the reasoning process lead incorrect final answers, while the overall reasoning process remains fundamentally sound. Consequently, the LLM assigns a relatively high score to their problem-solving procedures.
A.3 QUALITATIVE ANALYSIS OF OCR INPUT
We have provided some examples where OCR errors led incorrect final results, as shown in Fig. d). We found that LLMs generally struggle to provide correct answers when OCR results contain errors.
A.4 QUALITATIVE ANALYSIS OF ERROR CASE
Fig. o presents a systematic comparison of MILM reasoning processes. For OpenAl-ol, the main error lies its failure to correctly understand the problem. the scenario, the Green Salad had already been purchased and paid for, but OpenAl-ol still included it when calculating the new price. As for DouBao-La 1.5- Vision-Pro Pro. it exhibits similar issues to O1 and additionally made a mistake in understanding the product name. According to the menu, "6 jumbo shrimp" refers to single dish instead of six individual jumbo shrimp. However, both Qwen2.5- Qwen2.5-VL-722 and DouBao-L 1.5 Vision- Pro misinterpret it as six separate food items, made the same error as DouBao-1 1.5- ---Vision-Pro.
16

---

Published as a conference paper at ICLR 2026
Problem
OpenAh-01
you have already bought a Green Seled, and then want to 4-pecce fish and
Step by step:
Grewn Seled: $4.09
Two orders diprece frsh 12 $30.99);
$21.98
order of jumbo shrimp: $11.49 Sum:
$4.99 $21 $11.45 $38.40 $38.46
Salads
Qwen225-VL-728
To calculote the tocal cost, let's break down the
Seatood
slep stap:
**Green Salad The $4. 95.
"wo dipece fish
depaece $an
éilecce
10. 21.98
"Onne 6 jumbo shrimp"
Chilken
The price 50.090
New let's ads up all the costs: 4.99 21.90
9.89 + 36.96. the total cost $36.96.
DouBao-135 1.5 1.5-Vision-Pro
1. Find pr of each item:
The price of Green Salad is $9.99
The price a Piece Fish is $10.59, Since buy two, the for fish (2 10.99 21.9).
Sinee
Shrimp
step1: You buy Green Salad, but this is already bought, so we ighore it. The price of two piece fish is $10.99* 2-$2292111.
Add up total cost: 9.99).
kemm: (4.00-21.08
step2:
$9.99.
step3: The total cost is $21.98-59.99-533 The final answer in: $31.97
Fine, 99-99-2 98-26, 07).
Then, (26.97-998) 9.99999---- 36.96).
Sa the total cost (538.56).
Figure 6: Solution examples generated by different models. (a) presents the input image and cor- responding question; (b) shows the reference solution; (c)-(e) demonstrate outputs generated by different Al systems: (c) OpenAI-ol model, (d) Qwen2.5-VLL-2B model, and (e) DouBao-1.5. Vision-Pro model.
Table 7: Performance comparison of Qwen2.5-VI.-7B before and after fine-tuning on the synthetic data generated by our data curation framework.
Numerical Analysis
Multidisciplinary Knowledge
Method
Overall
Spatial
Mathematical
Knumerative
Logical
Qwen2.5.Vi 7B (base model)
15.7
13.8
11.6
20.1
9.0
35.8
Qwen2.5-VL-7B (Finc-tune)
17.0
14.7
10.6
21.6
22.9
17.4
26.3
A.5 USE THE DATA CURATION FRAMEWORK TO ANNOTATE TRAINING DATA
We leveraged our annotation logic to replace human annotators with DouBao- 1.-.o--.--Viiioo-PPP select- ing 5000 images from VOc, Info TextVQA, and ChartQA for synthetic data generation. As presented in Tab. 7 the results of fine-tuning Qwen2.5-VL-72 using this synthetic data. These results demonstrate that the proposed data curation framework cannot only be used for building benchmark data but also be migrated to the construction of training datasets.
A.6 A ROADMAP FOR OCR- RER-RRESNNNNG
We outline a concrete roadmap for how -CR-Reasoning might evolve into a standardized bench- mark suite for multimodal reasoning. We outline three core steps:
For reward design, we will leverage our annotation logic, replace the manual parts with leading models, and construct batch of training data through standardized processes. Using this data, we will train process-aware reward model that evaluates not only answer correctness but also the rationality of reasoning steps delivering more granular, objective signals than rule-based reward functions. The model will be fully open-sourced to help the community systematically enhance models' reasoning capabilities on text-rich images.
We will make the benchmark open-source on Hugging Face to facilitate downloading and use by researchers in the community. Subsequently, will open-source the code commonly used
17

---

Published as a conference paper at ICLR 2026
evaluation frameworks for MLLMs, enabling researchers to quickly evaluate their models' perfor- mance.
3. Existing benchmarks often focus evaluating capabilities in single dimension. This benchmark focuses on assessing models' reasoning abilities on single images. Therefore, for dataset expansion, will include complex tasks such reasoning multilingual text, domain-specific documents, or multi-page handwritten materials.
A.7 THE UNDERLYING MECHANISMS OF VL-RETHINKER-77
When testing RL-trained models, we used their official prompts to ensure consistency between training and testing. On this basis, we further added the CoT prompt: "Solve the complex prob- lem through step-by-step reasoning." Eureka, but degraded performance on VL-Rethinker. We hypothesize that this is due to a conflict between the generic CoT prompt and the model's 's built-in reflection/reasoning mechanism.
found that this approach improved performance on MM-
To further validate this. conducted an ablation study by removing the model's official built-in prompt and then testing the effect of the generic CoT prompt. The results are presented below. The performance improved when the official prompt removed (comparing the third and fourth rows). supports our hypothesis that the degradation occurs specifically because the two prompts conflict within the model's input processing.
Table Ablation study of the underlying mechanisms of VL-Rethinker--7.
Mithod
Official Prongt
CST
Overal
Spatial
Numerical Amalysi
Muthenutical
Enumcrative
Logicsl
Mulidiscipliaay Knowlldge
Official Setting
19.1
13.7
16.5
9.8
25.7
14.6
33.6
Original + CaT
14.6
8.3
16.1
9.8
19.6
8.3
19.0
Ablation (No Prompt)
17.
138
15.8
7.8
21.2
16.7
26.5
Ablation (CoT)
18.
14.7
13.1
11.7
27.9
18.8
31,3
A.8 PRACTICAL REASONING TASKS
In this Section, we provide some examples of the 18 practical reasoning tasks in text-rich visual scenarios, presented in Fig.
18

---

Published as a conference paper at ICLR 2026
belonging spottal menoniing
denctes belonging mettemattr
denotes belong
denotes belonging logical roasoning.
devoctes be poinning muttriuupplnn knowletge ressrring
denotes belonging nurner analysis reasorinng
Rouss Rassoring
Layose Reasasing
Relation Eutraction
Desetion: spitnal parh
whae What
The andaaedione (hat) adr
colams
Our Hiw 1ikme
Aniees
bugil uder unceel plycice bu
Sog The secodd uslccon balov (114
0009000
Brougf duggle Rocndd
Bead rakko Han debec 10 on
Bryg leai actuaaie ductte meual
Commoneatta Basccing
Gane-daoneiic Legie
Tw!
BIG
the game?
TALE
1+4=5
2+5
by
3+8=21
8+11=?
shec Caloiaate irovert
We (1 2
Theestrr
Orcmmbr 200d
The fiaal arower
Gooratyy Problome
Fundtion Problero
Aomelt candan ihea sag collticion
Arave:
mtsiirste aevvee ane
firk sinec. Firs the dirm piane proeto pour
Last Getl sEL San 2504 2e rakee or the virl
stgh
Grarmar that TC and cudior cocclle (o-g
lep dhs 2014 22112 3 ceeeen te- sereeiill center
Papi Gecals of dilaaia DnoG
exsterin meteiing option
out
tina oveer
Plylis Proliness
Cheriery Probdens
Quieiitttten Andlys
na coloe-hccce choce castoe dente pou de on
2000 past caloo Fot
til preses moii
Queedors foree ooders frade poyo??
Podoes--oo saaraae (0 Ci aiiiraa
Fagt: The cund HMdl Jttors
denely manted Danaadi Dammatilll 1918 msddial
A tee remenc
madi Suml the price deetitic The tral aroeer $33.
rapt
have etuchul aaraaaaz of he
caw Ef?
Analysie
Amswen
appir
Ecpee of socip?
bocen Eu LE PUNET aeso poiree
Anayer:
1962 Juth eol's
ruance
Mon't Thes mutoe sprer
be diluc are
Mad Thik numoer
apples lif...
Lay dy tive
411M5 and dis
aggeral
Eaph Cacuira dieence end.24
Cargeres bos
Cinebbatik
Schodalo
Queedor Him proticts there
Gandeer the un de tirk the
joice cacurt
conpe
contrate haw
Pirk it Ther ons Pric (977)) (rohh The sum of Broducic Product
OCC (igwerpon 707 Loner
Sappy depacuss draa 1i.4 0b00
ine by ban. 4s2 2.0 115.78
siec Caloaiile pamaet of ofeer produce 1% sleck nun other 7E3 fnelleioeal Bountt
1206 10
Figure 7: Examples of the 18 practical reasoning tasks in text-rich visual scenarios.
19

---

Published as a conference paper at ICLR 2026
Problem
Reference Solution
Gemini 2.0-Flash
anknaak have badkbee??
Image The inerrs
Ste3) enamat wifhor seckoann inertcteees, from
The zeincal that not love thane Enatritoons.
Classification
Networt
Coderrtaatn
Trotoupe v
trrenderrs
Canlentnonns e Worra.y /
Orstanens
Iriedt, Myrisss...
sted Count menyor the tneeidettt ciasss are hileet Frshoos Networms, Conteooaais, Aemolid shrrns bcheadere Mollics. lated.
mrads
groant
OCR Results
OpenAl
ui poweing treehee caincls ciniar oudiitis fictine
To Toraiie the problum and thenicaatt aun rehicue dupexet
insetty
antub with becdomes (nwetstrand
antrad
mub
divint sse saathoee, Pronooo, fotwrre, droelll eineeeeee,
antub
buplccee
Sup & Look Rocsied cverg in penrthere hhmppe ig the jot we soveral puts pronthaaest
Spiry
Alriccumpi, rusi rinry rids, (Tipesaer, eeeeeeennaaa skann (kuthanne, sw (routaiiee
saaly skin cakk onttal
Thaposorrs
fatesrru metttsrratall (tachworm Leadi These
Slarton See uschel Those en pesccrodeera tnseesaaass A Br Cocadial These reptiea Lesticastess Shher,
blooced, Ars cold soded, Ane cald
froe Ninel', Arth mopot, Hindineivia diniine, Crocddile, (Shrkchaal
Thoo codoraantas iewetttrttaal Sraih Umper Thes molisoo Inarideeiand
bedi (vetaiiutall 30 Dow, itamel Thon marm arrdrnool Beoler Scorpion Those are anachnde
Limpeti Marmul Hevelaateon
nourbar ryns find an bedy. Houdy
Ainwa batdee ware Handdel.
wws bectbor hrom 1 (lapnorr, 2 atmase Pan (Bethaoem laadt anmubo 3 Starfih, Soi undend arieate
lege Mare ind booky pamt gar rigaeees ee. (opoerooypen lpper)
ichowd modes par vedie to eemmb So tood number mnstabruts en rudh a tithes ane trai arower
Figure 8: Qualitative analysis of the reasoning path. (a) presents the input image and corresponding question; (b) shows the reference solution serving as ground truth; (c) demonstrates outputs gener- ated by Gemiini-2.0-Flashh (d) demonstrates outputs generated by O3-mini with OCR results input.
20


Reviews:
[{"review_id": "h4mO3sFF9d", "reviewer": "Reviewer_cM84", "summary": "This work addresses the lack of systematic benchmarks for evaluating Multimodal Large Language Models (MLLMs) in text-rich image reasoning. It proposes OCR-Reasoning, a benchmark with 1,069 human-annotated examples covering 6 core reasoning abilities and 18 practical tasks, featuring both final answers and step-by-step reasoning processes. Extensive evaluations of LLMs, MLLMs, and document-oriented MLLMs show that no model achieves over 50% accuracy, with image input outperforming OCR text alone and CoT prompting benefiting most models. The core contributions include the first reasoning process-annotated benchmark for text-rich images, systematic model evaluation, and identification of key improvement directions.", "questions": "The zero-shot evaluation effectively demonstrates out-of-the-box model capabilities. Have the authors explored or considered exploring few-shot prompting or fine-tuning scenarios on OCR-Reasoning? Could such experiments provide insights into whether models achieve substantial improvements with modest amounts of task-specific guidance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. OCR-Reasoning is the first benchmark to systematically assess reasoning processes in text-rich image scenarios, addressing a long-overlooked need.\n2. The comprehensive evaluation includes multiple model categories and zero-shot settings, ensuring generalizable results.\n3. Detailed error analysis and qualitative case studies deepen understanding of model limitations beyond accuracy metrics.", "weaknesses": "1. While the handwritten data in OCR-Reasoning provides valuable transcribed college-level STEM problems, it would be beneficial to consider incorporating more everyday real-world handwritten scenarios to further enhance the benchmark's coverage of diverse text-rich reasoning tasks commonly encountered in practice.\n\n2. The paper presents an interesting observation that CoT prompting may have backfired on VL-Rethinker-7B, potentially due to conflicting built-in reflection mechanisms. It would strengthen this finding if the authors could provide additional ablation studies or experiments to further validate this hypothesis and better understand the underlying mechanisms.\n\n3. The human validation for the LLM-as-Judge method demonstrates careful evaluation on DouBao-1.5-Vision-Pro. To further establish the robustness of this evaluation approach, it would be valuable to extend the validation across additional models and reasoning categories, which could help address potential concerns about judge bias and generalizability of the assessment methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nBZvZX72aK", "reviewer": "Reviewer_YFNZ", "summary": "This paper proposes OCR-Reasoning, a benchmark for evaluating MLLMs’ text-rich image reasoning, with 1,069 human-annotated examples (6 core reasoning abilities, 18 tasks) and annotations of both reasoning processes and answers (unlike existing benchmarks only annotating answers). Evaluations show top closed-source MLLM  doesn’t exceed 50% on it, while open-source ones perform worse.", "questions": "1. OCR-Reasoning annotates both reasoning processes and final answers, while existing text-rich image benchmarks (e.g., DocVQA, OCRBench) mostly only annotate final answers and their samples’ answers are often directly extractable from OCR results. What are the specific core differences between OCR-Reasoning and these benchmarks in terms of sample design and annotation logic? What key role does this difference play in evaluating the true reasoning capabilities of MLLMs?\n2. The paper states that existing reinforcement learning (RL) methods perform poorly on OCR-Reasoning, due to mismatched reward functions and a disconnect between training data and the benchmark’s scenarios. Does the study propose preliminary improvement directions (e.g., specific reward function design ideas, training data selection criteria)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Filling text-rich image reasoning evaluation gaps: Existing text-rich image benchmarks focus on text extraction but lack systematic reasoning assessment. OCR-Reasoning addresses this, measuring MLLMs’ reasoning in practical scenarios.\n2. Sample design forcing reasoning: Few answers in its samples are directly extractable from OCR results; models must actively reason, avoiding reliance on text extraction to truly reflect their reasoning levels.\n3. Comprehensive annotations for in-depth evaluation: Unlike benchmarks that only annotate final answers, OCR-Reasoning  labels both reasoning processes and answers, enabling holistic analysis of models’ problem-solving abilities", "weaknesses": "1. Limited dataset scale: Most of the data collection and annotation processes rely on manual work, and the high associated costs result in the dataset scale being only comparable to previous methods, failing to achieve larger-scale expansion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q0E89LXagb", "reviewer": "Reviewer_eZdh", "summary": "The paper introduces the OCR-Reasoning benchmark, designed to evaluate the reasoning capabilities of Multimodal Large Language Models (MLLMs) in complex text-rich image scenarios. Unlike existing benchmarks that focus on text extraction, OCR-Reasoning includes both final answers and step-by-step reasoning processes for 1,069 annotated examples across six core reasoning categories. The benchmark reveals significant limitations in current MLLMs, with top models achieving only around 50% accuracy in reasoning tasks, highlighting struggles in integrating visual, textual, and logical information. The study emphasizes the importance of Chain-of-Thought prompting and provides valuable insights into common failure modes such as calculation, spatial comprehension, and logical errors. This benchmark aims to inspire future advancements in multimodal reasoning systems.", "questions": "1. Dataset Scale and Coverage:\nCould the authors clarify whether there are plans to expand OCR-Reasoning beyond 1,069 samples? A larger and more diverse dataset (e.g., multilingual, domain-specific, or handwritten documents) would improve the benchmark’s representativeness.\n2. Evaluation Bias in LLM-as-Judge:\nHow do the authors mitigate potential bias when using LLMs to evaluate reasoning quality, especially if the judging model shares architecture or training data with the tested models? Would cross-model or partial human evaluation be feasible for validation?\n3. Reasoning-Type Analysis:\nThe paper shows category-wise results but lacks deeper diagnostics. Could the authors provide finer-grained error analyses or ablations—for example, separating failures due to OCR errors, visual reasoning, or logical inference?\n4. Comparison with Existing Benchmarks:\nIt would be helpful to include a more direct experimental comparison or transfer evaluation with existing datasets (e.g., DocVQA, ChartQA, OCRBench). How does OCR-Reasoning specifically challenge models beyond these benchmarks?\n5. Reinforcement Learning Methods:\nThe paper notes that RL-based methods perform poorly. Could the authors elaborate on how a better reward design for text-rich reasoning might look, or what specific factors caused these RL models to fail?\n6. Dataset Accessibility and Reproducibility:\nPlease clarify the intended release details—license, format, annotation schema, and evaluation scripts. Ensuring full reproducibility will significantly strengthen the paper’s long-term impact.\n7. Future Directions:\nThe authors mention possible improvements in reward design and dataset expansion. Could they outline a concrete roadmap for how OCR-Reasoning might evolve into a standardized benchmark suite for multimodal reasoning?\nThese clarifications and extensions could meaningfully strengthen the paper’s rigor, reproducibility, and long-term value to the community.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper demonstrates strong originality by introducing a new benchmark, OCR-Reasoning, that evaluates multimodal large language models on text-rich image reasoning—a domain previously underserved by existing datasets focused mainly on text extraction. Its dual annotation of final answers and reasoning steps represents a creative and meaningful extension of prior benchmarks.\n\nIn terms of quality, the study employs a rigorous and transparent methodology, including systematic dataset curation, expert annotation, and comprehensive evaluation across a broad spectrum of state-of-the-art models. The experimental design is sound, with clear baselines, detailed performance breakdowns, and thoughtful error analysis.\n\nThe paper exhibits strong clarity, with a well-structured presentation, clear motivation, and informative figures that effectively illustrate dataset design and results. The writing is precise and technically competent, enabling easy understanding of both the problem and its importance.\n\nRegarding significance, the work makes a timely and impactful contribution to the multimodal reasoning community. By exposing the current limitations of MLLMs in integrating textual and visual reasoning, it establishes an essential benchmark that will likely guide future research on improving model reasoning and evaluation frameworks.", "weaknesses": "The dataset is relatively small (1,069 samples), limiting generalization and coverage of diverse real-world scenarios. The reliance on LLM-as-Judge introduces potential bias; incorporating human or cross-model validation would improve reliability. The paper lacks deeper diagnostic analysis explaining why models fail, and provides limited quantitative comparison with prior benchmarks. Finally, details on dataset release and reproducibility are insufficient, which may hinder adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mbf00RcoUt", "reviewer": "Reviewer_KL6Q", "summary": "This paper reveals severe capability gaps in current advanced multimodal slow-thinking systems when performing text-rich image reasoning tasks, attributing this limitation fundamentally to the absence of a systematic and specialized evaluation benchmark. To address this challenge, this paper introduces the OCR-Reasoning benchmark, characterized by its large-scale manual annotations, coverage of multi-dimensional reasoning tasks, and core feature of \"dual annotation\" providing both answers and detailed step-by-step reasoning processes.", "questions": "see the weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper accurately identifies a crucial gap in multimodal reasoning research by proposing the first benchmark specifically designed for systematically evaluating \"text-rich image reasoning\" capabilities\n﻿\nThe constructed OCR-Reasoning benchmark surpasses traditional models that focus solely on answer correctness. By incorporating annotations of step-by-step reasoning processes, it achieves dual evaluation of both model reasoning paths and final answers, providing a more comprehensive and profound perspective for diagnosing model capability shortcomings.", "weaknesses": "1. The study primarily focuses on diagnostic evaluation but does not validate the effectiveness of its proposed \"thinking with images\" reasoning approach. A critical question remains unanswered: can explicitly training models to generate such visual reasoning chains on the proposed benchmark lead to significant gains in reasoning performance? The paper would be significantly strengthened by including fine-tuning experiments that demonstrate whether and how leveraging this benchmark for training, as opposed to merely for evaluation, improves model capabilities on text-rich reasoning tasks.\n\n2. The paper would benefit from a more thorough comparative analysis between the proposed OCR-Reasoning dataset and existing text-rich benchmarks (e.g., TextVQA, DocVQA, ChartQA). While the novel \"dual annotation\" is highlighted, a quantitative and qualitative comparison is needed to clearly delineate its advantages. This should explicitly detail the dataset's superiorities in terms of data quality (e.g., the depth and consistency of reasoning chain annotations), data characteristics (e.g., the diversity and complexity of reasoning types beyond simple QA), and coverage (e.g., the inclusion of scenarios that require multi-hop reasoning).\n\n3. Although the benchmark's quality is commendable, its scale (approximately 1,069 examples) may limit its comprehensiveness and statistical power. The covered visual domains might not fully represent the vast spectrum of real-world text-rich images. To enhance the robustness and generalizability of the findings, the authors could consider expanding the dataset to include more diverse sources, such as web screenshots (for UI/UX reasoning), financial documents (for complex table and report understanding), academic papers, or product manuals. This would ensure that the benchmark tests reasoning capabilities across a broader range of practical contexts.\n\n4. The benchmark is constructed manually, which ensures high quality but is not scalable for generating large-scale training data. A discussion on potential methodologies for scalable training data construction is a crucial missing piece. The authors could propose and discuss semi-automatic or synthetic data generation techniques that could produce large volumes of text-rich image reasoning data. For instance, exploring how advanced models (like GPT-4V) could assist in drafting reasoning chains for human review, or how to create synthetic tasks that inherently require visual-textual reasoning, would greatly enhance the practical utility and impact of this work beyond evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)