Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models
Authors: Zhentao he
Can Zhang
Ziheng Wu
Zhenghao Chen
Yufei Zhan
Yifan Li
Zhao Zhang
Xian Wang
Minghui Qiu
Venue: NeurIPS 2025 poster ()
Keywords: OCR
MLLM
hallucination
Primary area: deep_learning

Abstract: Recent advancements in multimodal large language models (MLLMs) have enhanced document understanding by integrating textual and visual information. However, existing models exhibit incompleteness within their paradigm in real-world scenarios, particularly under visual degradation (e.g., blur, occlusion, low contrast). In such conditions, the current response paradigm often fails to adequately perceive visual degradation and ambiguity, leading to overreliance on linguistic priors or misaligned visual-textual reasoning. This difficulty in recognizing uncertainty frequently results in the generation of hallucinatory content, especially when a precise answer is not feasible. To better demonstrate and analyze this phenomenon and problem, we propose KIE-HVQA, the first benchmark dedicated to evaluating OCR hallucination in degraded document understanding. This dataset includes test samples spanning identity cards, invoices, and prescriptions, with simulated real-world degradations and pixel-level annotations for OCR reliability. This setup allows for evaluating models' capacity, under degraded input, to distinguish reliable visual information and answer accordingly, thereby highlighting the challenge of avoiding hallucination on uncertain data. To achieve vision-faithful reasoning and thereby avoid the aforementioned issues, we further introduce a Group Relative Policy Optimization (GRPO)-based framework featuring a novel reward mechanism. By incorporating a self-awareness of visual uncertainty and an analysis method that initiates refusal to answer to increase task difficulty within our supervised fine-tuning and reinforcement learning framework, we successfully mitigated hallucinations in ambiguous regions. Experiments on Qwen2.5-VL demonstrate that our 7B-parameter model achieves a ~28% absolute improvement in hallucination-free accuracy over GPT-4o on KIE-HVQA and there is no significant performance drop in standard tasks, highlighting both effectiveness and robustness. This work advances the development of reliable MLLMs for real-world document analysis by addressing critical challenges in visual-linguistic alignment under degradation.

Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models
Zhentao Hel Can Zhangl.
Ziheng Wul.
Zhenghao Chen
Yufei Zhan 1.2 Yifan Li
Zhao Zhang' Xian Wang' Minghui Qiu'
ByteDance ?CASTA RUC
Abstract
Recent advancements in multimodal large language models (MLLMs) have en- hanced document understanding by integrating textual and visual information. How- ever, existing models exhibit incompleteness within their paradigm in real-world scenarios, particularly under visual degradation (e.g., blur, occlusion, low contrast). In such conditions, the current response paradigm often fails to adequately perceive visual degradation and ambiguity, leading to overreliance on linguistic priors misaligned visual-textual reasoning. This difficulty recognizing uncertainty fre- quently results in the generation of hallucinatory content, especially when precise answer is not feasible. To better demonstrate and analyze this phenomenon and problem, we propose KIE-HVQA, the first benchmark dedicated to evaluating OCR hallucination degraded document understanding. This dataset includes test sam- ples spanning identity cards, invoices, and preseriptions, with simulated real-world degradations and pixel-level annotations for OCR reliability. This setup allows for evaluating models' capacity, under degraded input, to distinguish reliable visual information and answer accordingly, thereby highlighting the challenge of avoiding hallucination uncertain data. To achieve vision-faithful reasoning and thereby avoid the aforementioned issues, we further introduce a Group Relative Policy Opti- mization (GRPO)-basee based framework featuring novel reward mechanism. By incor- porating a self-a welf-awwreness of visual uncertainty and an analysis method that initiates refusal to answer increase task difficulty within our supervised fine-tuning and reinforcement learning framework, we successfully mitigated hallucinations in am- biguous regions. Experiments on Qwen2.5-VV VL demonstrate that our -B-arrmmeter model achieves a ~~28% absolute improvement in hallucination free accuracy over GPT-4o on KIE-HVQA and there is significant performance drop standard tasks, highlighting both effectiveness and robustness. This work advances the devel- opment of reliable MLLMs for real-world document analysis by addressing critical challenges in risual-linguistic alignment under degradation. Data is available at httpps://uuggnggff otaae///eeeeeeltatttteteeeeeeeee Kertreceaaeeeeessssaarac--E--
Introduction
In recent years, there have been significant advancements in MLLMS [I 9 33
for document understanding [1]. These models integrate textual semantics with visual features, offering new paradigms for automated processing of identity cards, invoices, contracts, and similar applications.
MLLMs demonstrate near-human performance in documents understanding across several domains. Enhancements in language models have improved multilingual support and incorporated prior knowledge, leading to more accurate text parsing. Advancements in visual encoders [I [6 [5], such as increased resolution, have enhanced the ability capture image details. Additionally,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prompt
NUTRITION MP EDRMATTIONN TOW
Prompt of per 100g/ml?
What the value for total carbohydrate of per 100g/m/7
Sodium
Qwen2-VL-78
5%
(Halaihatio))
54.2g
Contrrhitz
Agec
24.35
10.04
(Urnongmennl Degnolstive)
224mg
511g
Our Model
22g
Our Model
5-eechaded--1 1g
Expart_occhoded--[///
Figure 1: The performance of Qwen2.5-VL-77B (left) and GPT-4o (right) in interpreting degraded text images. The Qwen2.5-VL- VL-7B model may experience hallucinations. identifying values not present the image, while the GPT-4o model struggles with unrecognized degradation due to partial text occlusion. Previous models have not thoroughly addressed these issues, whereas the model proposed in this paper effectively resolves them, providing more accurate recognition results.
refined layout-based instructions 335 20 have enabled systems to better understand document exhibit strong cross-modal reasoning capabilities when working with high-quality images and standardized layouts. However, current research has not yet fully addressed the incompleteness within their paradigm in real-world scenarios. The core challenge stems from models' inability to enforce strict adherence to visual signals. When confronted with practical complexities -iexxtttiesiicuuiinng image blurring or unconventional formatting theiiggg--hhhhet models frequently generate cross- modal hallucinatory content that deviates substantially from input data.
structures. In particular, current MLLMs 4 40
The OCR hallucination issue in MLLMs stems from three critical challenges across the model development process. First, during the pre-training phase, there a significant lack of key information extraction (KIE) data and clear annotations related to degraded visual scenarios which limits the model's ' ability process challenging visual inputs. Second, in the instruction fine-tuning phase, the paradigm for handling degraded visual scenarios is often overlooked, as researchers generally assume
OCR tasks involve non-degraded inputs 17 8 21 [4] Even MLLMs with strong visual capabilities fail to demonstrate the necessary reasoning abilities for real-world degraded documents. Third, in the evaluation phase, the absence of dedicated benchmarks for quantifying OCR hallucination in document understanding tasks impedes progress, as the field lacks both comprehensive metrics and sufficient annotated data due to the inherent challenges in collecting and labeling degraded samples. As result, when confronted with visually compromised inputs like glare- glate-obstructtd identity cards or low-contrast reports, models exhibit cognitive bias by defaulting to linguistic priors rather than anchoring decisions to observable visual evidence, leading to potentially catastrophic misinterpretations in critical applications 6 The examples are illustrated in Figure T
To address these pressing challenges, this paper introduces a comprehensive benchmark and a novel framework designed to tackle the critical issues of vision-faithful reasoning in degraded document understanding. We present KIE-HVQA, the first benchmark specifically designed to evaluate OCR hallucination under real-world noise conditions. This dataset includes 2, 2,000 annotated training samples and 400 rigorously curated test instances spanning diverse document types, including identity cards, receipts, and invoices. Each sample is carefully designed to simulate real degradation scenarios, such as motion blur and low contrast, necessitating fine-grained visual-textual alignment for accurate key information retrieval. For instance, the task may involve extracting ID numbers from partially occluded cards or resolving ambiguous dosage entries in faded prescriptions.
Inspired by the successful practices of reinforcement learning in computer vision tasks 40 we employ reinforcement learning as tool to provide feasible approach to addressing this issue. Unlike typical MLLM tasks such as VQA (2). the KIE [39] task benefits from having quantifiable standard answers, which allows for the construction of precise foundational rewards and the design of appropriate rewards for various degradation scenarios. By employing Group Relative Policy Optimization (GRPO) algorithm
[10], we can supervise the model to enhance its existing OCR capabilities and develop a self-reflective KIE instruction paradigm that addresses visual degradation. This approach encourages models to prioritize visual evidence over linguistic priors, ensuring that decisions are more robustly anchored to observable data, marking a significant advancement in overcoming the challenges of vision-faithful reasoning and cross-moue mooal OCR ballucination.
2

---

To validate the efficacy of our training methodology and dataset, we implemented our proposed approach to enhance Qwen2.5-VL 4 and conducted comprehensive experiments to benchmark our method against state-of-ther multimodal models. Our method achieves notable 288 absolute improvement in hallucination-free accuracy on the KIE-HVQA benchmark. The contributions of this paper are summarized as follows:
We propose KIE-HVQA, the first benchmark for evaluating hallucinations in degraded documents. This benchmark simulates real- world degradations with pixel-level annotations and OCR reliability scores, enabling comprchensive assessment of OCR hallucinations under degraded conditions.
Based on the characteristics of the KIE task, designed precise reward modeling for the GRPO algorithm. By integrating this with an appropriate coldstart, we successfully enhanced the model's ability reason effectively with degraded visual input, significantly reducing hallucination without sacrificing its original OCR capabilities.
Through extensive experiments, our model demonstrates superior reasoning capabilities. Our model achieves a 28% improvement in hallucination suppression compared GPT-4o.
Related Work
2.1 Reasoning in Multimodal Large Language Models
Recent developments in large language models (LLMs) I
30 5 demonstrate that simulating human-
like thought processes and implementing sequential reasoning strategies can significantly improve performance complex mrbbemmssllvngg tasks. A significant innovation [00 29] involves DeepSeek- RI's implementation extensive reinforcement learning 22 24
techniques to foster self-evolving cognitive pathways in LLMs, substantially enhancing their performance sophisticated reasoning challenges. Inspired by advancements LLM reasoning, researchers [36] 7 [4] (28) have applied CoT prompting and developed SFT datasets with step-level reasoning for MLLMs.
Despite recent advancements, there remains paucity of research focused on applying reasoning to OCR tasks, particularly in addressing hallucination issues. To our best knowledge, our approach is the first to utilize RL training to effectively tackle hallucination problems in OCR tasks.
2.2 OCR benchmarks
In the early era of deep learning, variety of specialized benchmarks emerged address different challenges, such as natural-scene text [13], web-scene text (25), and multi-directional and curved text recognition. The current OCRBench [17 g 37 for evaluating MLLMs primarily targets line- granularity recognition, Other benchmarks, such as DocLocal4K [ and FOX [1] curate data mainly from document images.
Currently, these OCR benchmarks predominantly focus on document understanding and key in- formation retrieval, often neglecting issues such as hallucinations and misrecognitionss caused by image degradation. Our newly proposed benchmark addressing these overlooked challenges for the first time.
takes a significant step forward by
KIE-HVQA
3.1 Task Description
To provide a comprehensive evaluation framework for degraded document hallucination tasks, the KIE-HVQA benchmark introduces risually-grounder question answering task. This task demands precise alignment between textual semantics and degraded visual evidence in real-world documents. When presented with a degraded document image, such as blurred ID card or occluded images, and a question, models are required to perform several key tasks. Initially, they must identify text elements relevant to the question through multi-modal grounding. Subsequently, they need assess recognition confidence at the character level by analyzing edge sharpness, measuring contrast ratios, and verifying contextual coherence. Finally, models should generate answers based on visually verifiable content, while clearly indicating regions of uncertainty.

---

Receipt
Document
Nutrition Facts
21:0
per
sedium
Calories
150
How
dp posped?
$4
Sene
Quaret
[G] Eopart scctadeatt(5)
[G] -accttded--Sgg
Mickerss Mark
Tioe leartedd
15
Santer
ttng Sad
$42.00
[P] 150mg
(P] 15mg
[Q] What the price of the Matter's Mark ardered by Guest 47 [G] S-oectatedd 0D
[P] $5.00
Card
TIMBERS
DENER
BRANY
TIME
5-22
DTV
ABOUTT
Local Niners Mote:
2.51
3.30
PATNTER
3.80
3.00
SEUMLICI 4013/08LD
2.00
1:00
EMOOT VOSING
2-30
2:53
[Q] What value on card?
[Q] What date of this
[Q] What time was the tranooction.co tranooction.cmmpleeee completed? Ei-qurt scttatd-(333122 21:33:22
[G] gert_ecteee---e (E -rcchaldd-rrecccdee-
[G] 2025 (eros--per_sttttee jodooeee-jjjj
[P] Emiyy
IP] 2025 02 01
Figure 2: Visualization of the three types of data in our KIE-HVQA benchmark. [Q] represents the question, [G] denotes the ground truth, and [P] indicates the prediction generated by Qwen2.5-VL with zero-shot prompt. The data exhibit varying degrees of degradation, such as blurriness or damage, which affect the model's predictive accuracy.
The benchmark focuses on evaluating models' ability to minimize reliance parametric knowledge biases situations of partial legibility, such as medical prescriptions where dosage units are clear, but frequencies not. This requires that models adhere strictly to the available visual evidence, ensuring accurate interpretation and response based on the information present in the document.
3.2 Annotation Curation
This section presents the curation of annotations in three stages: dataset collection, instruction formulation, and manual verification of results.
To tackle the challenge of fision-faithful reasoning in degraded document understanding. we as- sembled a diverse dataset from three main sources: OCRBench B WildReceipt (27], and GPT-40- generated images. Each source was carefully chosen to simulate realistic degradation scenarios and test the robustness of MLLMs.
OCRBench. Our methodology is benchmarked on 100 key information queries from OCRBench. We first employ a text detection model combined with a character localization model 3 trained under a weak supervision framework, to extract the precise coordinates of characters in the answers. Given the potential randomness in the reading order of image characters, we utilize GPT-4o to reconstruct the correct character sequence. Subsequently, these coordinates undergo random degradation process.
To ensure the accuracy of the degradation results and prevent model hallucinations, established rigorous dual-model evaluation mechanism. This process involves cropping each character from its degraded region in the original image and inputting these sub-images into two leading Multimodal Large Language Models (MLLMs) GPT-44 and Swwe2..5-VL-72B--1 model pairing was strategically chosen to leverage their complementary strengths: GPT-4o serves as the performance benchmark for closed-source models, excelling in complex visual pattern recognition, while Qwen2.5-VL-72BB is state-of-the-art open- spen-source model for OCR tasks. Each MLLM independently provides a binary judgment (visible/invisible), and any discrepancies between their outputs are resolved through expert human review. This design mitigates bias from any single model and ensures the accurate annotation of borderline cases, providing a solid foundation for subsequent character-level reliability annotations. The detailed format of our answer data presented in Section|4
parallel verification. This

---

eeedtt From the WildReceipt dataset, we extracted entity-type answers from the original dataset and used MLLMs to generate corresponding questions. The images were modified using the same techniques applied to OCRBench, and the answers were reconstructed in a similar manner.
GPT-4o-generated Images. We used GPT-4o image generator - create 200 synthetic templates of IDs and documents with fictional information, ensuring privacy compliance. The information on these IDs was generated by GPT-4o and then added using Photoshop. Based on this, we designed corresponding iestiionandd-nsww pairs. To evaluate the model's 's performance in handling complex visual information, we applied degradation techniques to the answers, including adding obfuscation and blur effects.
We provide some samples from the KIE-HVQA dataset in Figure 2
This comprehensive dataset allows for rigorous testing of the ability to maintain visual-textual alignment and avoid hallucinations, even under conditions of visual degradation.
3.3 Evaluation Criteria
To assess the performance of our model in understanding degraded documents, we have developed three comprehensive evaluation metrics. These metrics are designed to capture various aspects of OCR performance under different visual conditions.
Legible Character Accuracy. This metric measures the character-level accuracy in regions of the document with high visibility. It serves indicator for the model's basic OCR capabilities, reflecting its ability to accurately recognize text in ideal conditions. A high score indicates that the model can perform near-perfect recognition when the text clearly visible, thus setting a standard for its performance under optimal conditions.
This metric evaluates the recognition accuracy in predefined regions that have been annotated as degraded due to factors such as motion blur, occlusion, or low contrast. It is specifically designed test the model's robustness against visual ambiguities and its ability to maintain accuracy in challenging conditions. For words that are degraded but do not pose a risk of hallucination, the model should output the corresponding characters. However, for areas with high risk of hallucination, the model should demonstrate awareness to appropriately reject providing an answer. This metric ensures that the model can effectively handle and interpret text in degraded circumstances while minimizing errors due to hallucinations.
Degraded Character Accuracy.
Global OCR Performance. Focuses task-specific text extraction quality through two critical metrics: Accuracy of OCR results for question- critiicll text regions referenced in VQA answers and Normalized Levenshtein distance between the OCR-extracted text and ground truth specifically for information required to answer the question.
4 Method
In this section, we systematically model the OCR hallucination issue a fundamental problem with precise rewards, reflecting various degradation issues through different reward functions. We then extend this framework by introducing new reward paradigm and aligning model behavior using reinforcement learning. In Section 1.1 we provide an overview of the rule- based GRPO algorithm, which serves as the basis for our approach. In Section the cold-start initialization method and the data generation process are elaborated in detail. In Section (4.] we present the GRPO algorithm and the degradation-bart OCR reward function, explaining their roles in the training process.
4.1 Preliminaries
The training of DeepSeck-Ri R1 [10 employs Group Relative Policy Optimization (GRPO). a novel reinforcement learning algorithm that differs from conventional methods like PPO 23 GRPO assesses strategies by comparing groups of generated responses, climinating the need for a critic model and simplifying the training process.
For given input q, GRPO generates G responses {01, o2.... oG} using the current policy Te then evaluates each response via a predefined reward function to obtain rewards {ri, .2...., JG). To
generation
5

---

1. Data Collection
Data Generation
0419A
21411
poer
Filter
Info. Extration (CIE)
Mail
A
ALLM
Bend
QuurteR
201
Meld
Degncd
prwp
CV
Reports
1419
V-L19A 84
Input Data
Char Detect
Degrade
RL Generate
CoT Generate
Supervised Fine-tuning
Qwen2.
Model
imae
Image
Image
Image
Extracted
Gleerly displeyed
The recage time chons
El Una lis shoped
wari caflan
stroke undar the flotruction shoee cleor one structure of the number with
Rewara
#RPO
Dupat
reprosunta the hour, 35 represants altunac
Question:
cloens).
Fornot rewerd
Gloo-llvel reward
Findl reward
What hime Trer soct on congre stod?
Cartura
Teep
watthrsh--der dur les
wurch le
Ground 23 (B3]3
The tannsction completion 53:22.
Ausnert
Answer:
transsation completion
roart cl (3]
Reinforcement Learning
Overview of our framework: The cross-modal reasoning pipeline begins with compre- hensive data collection, incorporating visual formal descriptions to enhance reasoning capabilities. We utilize multimodal approach to generate training data. After training, the process includes supervised fine-tuning to address OCR hallucination issues and improve reasoning accuracy. Finally, rule-based reinforcement learning is applied to enhance generalization across multimodal tasks.
Figure 3:
determine the relative quality of each response, GRPO normalizes the rewards:
mean (rran(( A std .rd(rr...,,ra))
(I)
In the training procedure, GRPO initializes trainable policy model me and frozen reference model Tref- The policy model iTy is optimized by maximizing the following objective function of G.
N
no (a:lg))) Tee (o: (0:14) A; B (CL((o (o):::/))
(GRPO(#) = N Σ
Here, N represents the number of completions in a group, and 3 a hyperparameter. This objective function encourages the model to prioritize completions with higher advantages within the group while maintaining proximity to the initial model.
4.2 Cold-start Initialization
Recent studies have focused on developing multimodal reasoning datasets that build upon existing fine-tuned data, with the objective of enhancing the reasoning capabilities of MILMs and improving their overall performance. This paper built multimodal CoT-OCR dataset that encompasses complex OCR degradation scenarios, enabling models to reason in a human-like manner. Several reasoning models, such as DeepSeek-RI 1 natural cognitive processes using CoT reasoning. These models can generate high-quality CoT data that includes human-lik like self-reflection processes. However, these models are purely language-based and cannot directly process multimodal data to produce CoT data.
[10] and Kimi K1.5 22), already possess the capability to perform
To address the challenge of processing multimodal data with language- -ased models, we integrate existing MLLMs with DeepSeek-Re R1. First, we convert multimodal information, such as images and text, into purely textual information using GPT-4o. This involves inputting image-question answer pairs and prompts into GPT-4o to generate a pseudo-CoT that includes both image descriptions and reasoning processes. Next, we merge these image-question pairs with the generated pseudo-Co'T and prompts, and feed them back into the MILLM produce detailed image descriptions. These descriptions then combined with the textual information and input into DeepSeck RR1, allowing

---

to execute high- -uality CoT process. This approach ensures that the resulting CoT data captures complex reasoning in way that mimics human cognitive processes.
Finally, we pair the pure textual CoT data generated by DeepSeek-R] with the corresponding images to create an integrated multimodal CoT dataset for cold-start initialization, as illustrated in the data generation process shown in Figure The CoT data obtained through this method closely aligns with human cognitive behavior, allowing the reasoning process to exhibit natural and logical thinking.
4.3 RL with OCR reward
We implement GRPO with hard formatting result rewards to enhance the self-learning capabilities of the model. For each question g, GRPO samples group of generated output set {0r 022-... oG) from policy model Te.ia Then GRPO maximizes the objective function in Equ. 2 and optimizes the model и.. Specifically, we introduce a rule-based reward for degraded OCR scenarios. This reward function is designed to ensure that OCR models maintain fidelity to visual input when generating textual output. is specifically tailored to handle varying levels character clarity within visual data, categorizing them into three distinct cases for accurate recognition and transcription. The criteria and objectives our reward function are as follows:
Legible Character: For characters that are entirely clear and unambiguous, the model is required to accurately recognize and retain these characters the final OCR output. This ensures that any fully legible text is preserved without alteration.
Characters: In cases where the characters are partially obscured or blurred but still recognizable by a human observer, the model should identify these as "anomalous" characters. Although these characters lack perfect clarity, they must be included in the final OCR output, reflecting the human ability to infer their identity.
Partially Obscured but Staman-Recogniie
Unrecognizable Characters: Characters that are entirely obscured and cannot be identified should not be included in the OCR output. Instead, these should be represented by a space to prevent any hallucination erroneous inference by the model.
For the degraded input text Figure www./uueebb/eeaauoocclleeeeeeddd The reward function enforces visual-textual fidelity through multi-stage analytical process. During the character-level classification phase, clear characters ("B, u, I"") are preserved verbatim, moderately blurred characters such as "e" (marked with tags) retained partially occluded anomalies, while severely obscured character clusters like "t, 1" are classified unrecognizable units. Building upon these assessments, quantitative evaluations yield legible characters, 1 partial occlusion instance, and completely obscured characters.
Beau.ffit
[B, u, u, l]:
clear partial occlusion completely obscured
[e]:
[t,
Figure 4: The figure illustrates the degrade criteria of each letter in the word "Beautiful", The letters "B, a, u. f. u, I" are clearly visible; the letter "c" is partially occluded; the letter "r, E" is completely obscured.
This reward function integrated into the GRPO training objective to systematically guide the learning process of the model. The reward calculation is formalized as:
F = G.), (1 Σ
95 (A((
(3)
Here, i represents different categories or types of OCR results, including clear characters, unclear characters, and the final answer. involves counting specific elements within the OCR results. A and G represent the model's 's output and the ground truth, respectively. f and g denote the OCR edit distance evaluation metric and the numerical calculation evaluation metric, respectively. The structured reward signals enable precise alignment between visual faithfulness and textual accuracy, with error type differentiation driving targeted model improvement. The training loop continuously evaluates the model's performance against diverse set of degraded OCR samples, allowing for iterative improvement. As the model encounters varied visual challenges, the reward function dynamically adjusts, promoting adaptability and robustness in handling
OCR scenarios.
7

---

Algorithm Reward Function for OCR Task
1: function CALCULATE METRRS truth) if len(pred) = 0 and len(truth) = 0 then return 1.0
2:
3:
4: end
5: edil_dist levenshtein distance(pred, truth)
6:
max max_len +- max max((en(pred)) len (truth))
7:
similarity < 1 (edit_ dist max__en)
8: return similarity
9:
end function
10: function REWARD(answer,, gt)
_cdear_metrrc,,,, f'nal_metric, ddear_metrer + calculate caleulate_metrics metrics(answerr
cleor_metric 12: bs - -1 not cll__lear__a clear_metric + C2 x clear _ettiicereee + C2 x final_metria
return bs
14: end function
OCRbench-Kie subset
Wildrecippt subset
Card subbett
Models
Final
Avg
Avg
GPT-46 (1120)
Chude335 Sonnet
24.30
29.49
27.13
26.97
23.13
18.75
24.54
22.14
30:24
22.92
29.98
27.71
26.92
21
26.22
26.25
Chmle3.7.50 S-Smmet
25.76
40.3
33:95
31.2
23.57
26.32
34.87
26.77
29:32
19.77
33.73
26.17
Gemani2.5 pro
35.34
53.85
56.18
55.12
27:33
24.22
29.13
26.89
47.71
46.94
26.77
40.47
36.94
34.64
33.53
25.03
1stemVI,3-88 [1
4.05
4.42
4.09
4.19
7.26
7.61
7,46
7.44
12.49
16.65
11:22
13.45
7,83
9:03
7.68
8.18
LatemVL3 38
1
9.34
14.29
9:00
10.58
10.97
18.93
11.20
13.70
21.96
26.45
21.03
23.15
13.11
19.75
12.98
15.28
latenNE.37888 1
43
1.50
2.00
2.00
1:97
5.00
7.60
6.00
6.20
12.49
16.65
11.22
13.45
6.00
8.59
6.43
7.04
Qwen2 5-VL
29.06
37.68
26.84
31,20
14.69
19.15
15,66
16.50
26.95
26.70
27.74
27.13
20.00
24.19
20.37
21.53
Qwen2 3-h
28.54
42.31
29:43
33.43
10.34
30.1
10.66
17.04
85
32.25
23.65
26.60 60
16.64
32.81
16.95
22.14
Model
22.25
45.00
50.01
49.72
Table 1: Evaluation results of closed-sourced, open-sourced and our models on KIE-HVQA bench- mark. Clr, Nc, Final, Avg represent clear characters, not clear characters, final OCR and average results, respectively.
In summary, this enhanced GRPO framework, featuring novel reward function, enables a more balanced training approach. By guiding the model to prioritize both high textual accuracy and faithfulness visual input, this methodology leads to more reliable and trustworthy outputs.
Experiment
5.1 Experiment Settings
Training Dataset. To obtain the cold-start dataset, we created custom data by generating word images using random fonts and varying degrees of degradation. We utilized the bounding boxes from TextOCR (26] to acquire relatively accurate character-by- chhharcte coordinates, thereby generating set of cold start data with "think" phase. In the GRPO phase, we mixed part of TextOCR (26). WildReceipt 27 and other OCR datasets 12 as our reinforcement learning training dataset.
Implementation Details. For the cold-start dataset preparation, we utilized GPT-4o and the reasoning LLM DeepSeek-R1. We then processed the VQA datasets using GPT-4o and DeepSeek-RI over approximately 12 hours. For the cold-start initialization, we used base model and performed supervised fine-tuning for 5 epochs with learning rate of 1e-6 and data rollout batch size of 512. This process required approximately hours, using the LLaMA-Fach framework [42]. Following the cold-start phase, we trained the model using the collected dataset with the GPRO method over several hours, employing the Easy-RI R1 framework [41].
1-a---..--V-7--Issttte as the
5.2 Main Results
Table D provides detailed evaluation of document understanding performance the KIE-HVQA benchmark. Our model sets a new standard with an average distance score of 58.05%, outperforming close-sourced models GPT-4o, Claude and Gemini. This substantial improvement underscores our

---

model's superior ability to maintain visual- texttal alignment even under challenging degradation conditions.
In scenarios simulating partial occlusion, our model achieves a remarkable 61 61.34% accuracy in not clear character-level OCR distance evaluations, surpassing GPT-4o'ss ' 36. 13%. This success is attributed to our uncertainty- grounding mechanism, which effectively reduces hallucination.
The results validate the robustness of our approach across various metrics, including average distance, clear character recognition, and handling of occluded text. Our model demonstrates balanced performance across all dimensions, proving its capability adapt to different levels of text degradation and complexity. Crucially, these findings highlight path towards more robust and trustworthy OCR by addressing the inherent limitations of previous methods. Rather than simply relying feature- based estimation prone to critical errors under degradation, this work enables a more nuanced understanding and processing of real-world documents, particularly in challenging scenarios
5.3 Abalation Study
Analysis of Training Strategy. To evaluate the effectiveness of training data, we compared the model's performance under two training strategies: (1) applying Supervised Fine-Tuning dataset, and (2) optimizing the SFT-trained model with Reinforcement Learning, As shown in Table [ SFT significantly improved the model's performance the KIE-HVQA benchmark while applying RL afterward led to additional performance gains, enabling the model to tackle more complex problems. This study demonstrates that our training data is crucial for enhancing model performance, and the combination of SFT and RL is a powerful and effective strategy for maximizing reasoning and thinking capabilities in KIE-HVQA
Analysis of General OCR Capability Preserva- tion. To investigate whether enhanced degrada- tion handling will affect general OCR capabilities, we conducted comparative evaluations in three standard OCR domains of OCRbench [17] Scene Text-centric VQA, Doc-oriented VQA, and Key Information Ex- traction. As shown in Table 2 our model achieves comparable performance to specialized baseline mod- els. This demonstrates that our uncertainty-awart grounding mechanism specifically targets degraded regions without affecting general text recognition ca- pabilities.
Model
Scene
Doc
Info
GPT-4o (1120) II
180
167
163
Claude3.7-Somne
159
130
125
Qwen2.5-VL.-27 4
181
181
182
MiniCPM-o-2.6 38
187
182
187
Our Model
180
179
183
Table 2: Ablation studies demonstrating the preservation of general OCR capabilities.
Analysis of Reward Setting. Ablation studies on our dataset demonstrate the necessity of integrating composite rewards, as shown in Table 3 The format reward primarily ensures that the model adheres the expected format in its responses. Therefore, the ablation experiments focus mainly on the character matching aspect of the final reward. When consid- ering only the clear character reward, the model's
Reward Setting
Clr
Ne
Final
only clear only final
50.64
44.15
53.34
51.06
54.06
54.24
all rewards
55.45
61.34
57.35
Table 3: Ablation reward setting.
performance on not clear characters significantly declines. Similarly, when focusing solely on the final character reward, the results are inferior compared the combination of all rewards. Our framework significantly outperforms single-reward variants. showing marked improvements across all evaluation dimensions. This validates that multi-objective reward synthesis crucial for handling real-world document degradation patterns.
6 Conclusion
This paper addresses the challenge of cross moosl OCR hallucination in degraded document un- derstanding by introducing KIE-HVQA, the first benchmark designed to evaluate vision-faithful reasoning under real-world noise conditions. Our benchmark simulates practical degradation scenar- ios, facilitating comprehensive assessment MILMMs performance in challenging environments. We propose novel GRPO-based framework with multi-objective reward mechanism to enforce

---

vision-faithful reasoning. This framework incorporates uncertainty rejection behaviors, ef- fectively suppressing hallucinations ambiguous regions and enhancing adaptability to complex tasks. Extensive experiments demonstrate the efficacy of our approach, with our 7B-parameter model achieving ~28% absolute improvement in hallucination-free accuracy over GPT-4o on the KIE-HVQA benchmark. This highlights our model's robustness and computational efficiency in maintaining visual-textual alignment under visual degradation.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint WXXiv:2303.08774, 2023.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[3] Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. Character region awareness for text detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9365-9374, 2019.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song. Kai Dang. Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 1923,, 2025.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901 2020.
[6] Shuo Chen, Zhen Han, Bailan He, Jianzhe Liu, Mark Buckley, Yao Qin, Philip Torr, Volker Tresp, and Jindong Gu. Can multimodal large language models truly perform multimodal in-context learning? In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6000-6010 IEEE, 2025.
[7] Zhangquan Chen, Xufang Luo, and Dongsheng Li. VisrI: Intention-driven visual perception via reinforced reasoning. arXiv preprint arXiv:2503.07533 2025.
[8] Ling Fu, Biao Yang, Zhebin Kuang. Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.00321, 2024.
[9] Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha, Exploring the frontier of vision-language models: A survey of current methodologies and future directions, 2024.
[10] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[11] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprini arXiv:2403. 12895, 2024.
[12] Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Saan-Phllippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1-6. IEEE, 2019.
[13] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. ledar 2015 competition on robust reading. In 2015 13th international conference on document analysis and recognition (ICDAR), pages 1 1156 160. IEEE, 2015.
10

---

[14] Chenglong Liu. Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge. Zining Zhu, Liang Zhao, Jianjian Sun, Chumrui Han, and Xiangyu Zhang. Focus anywhere for fine-gratt roe-grained multi-page document understanding. arXiv preprint a4Xiv 14295, 2024.
[15] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[16] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[17] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024.
[18) Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint 2024.
[19] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning arXiv preprint arXiv:2503.0/785, 2025.
[20] Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. Layoutllm: Layout instruction tuning with large language models for document understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15630-15640, 2024.
[21] Tengchao Lv, Yupan Huang, Jingye Chen, Yuzhong Zhao, Yilin Jia, Lei Cui, Shuming Ma. Yaoyao Chang, Shaohan Huang, Wenhui Wang, et al. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309. 1419, 2023.
[22] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 05:53728-537411 2023.
[23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
[24] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.00 03000, 2024.
[25] Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai. lcdar2017 competition on reading chinese text in the wild retww117). In 2017 14th iapr international conference on document analysis and recognition (ICDAR). volume 1, pages 1429-1434 IEEE, 2017.
[26] Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. Tex- toer: Towards large-scale end-to-end reasoning for eebitrarysshaper scene text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8802-8812, 2021.
(27] Hongbin Sun, Zhanghui Kuang, Xiaoyu Yue, Chenhao Lin, and Wayne Zhang. Spatial dual- modality graph reasoning for key information extraction. arXiv preprint arXiv:2 14470, 2021.
(28) Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.2075: 2025.
[29] Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025.
11

---

[30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini wwwii:2307.092260 2023.
[31] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual rision-languee encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.147585 2025.
[32] Wenzhuang Wang, Yifan Zhao, Mingcan Ma, Ming Liu, Zhonglin Jiang, Yong Chen, and Jía Li. Ficgen: Frequency-inspired contextual disentanglement for layout-driven degraded image generation. arXiv preprini ar arXiv:2509.00 107, 2025.
[33] Ziheng Wu, Zhenghao Chen, Ruipu Luo, Can Zhang, Yuan Gao, Zhentao He, Xian Wang, Haoran Lin, and Minghui Qiu. Valley2: Exploring multimodal models with scalable vision- language design. arXiv preprint arXin:2501.05901, 2025.
[34] Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio. Cha Zhang, and Furu Wei. Xfund: a benchmark dataset for multilingual visually rich form understanding. In Findings of the association for compulational linguistics: ACL 2022, pages 3214-3224, 2022.
[35] Tao Yang, Yingmin Luo, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. Posterllava: Constructing a unified multi-modal layout generator with 1lm. arXiv preprint Stiv:2406.02885 2024.
[36] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. Rl-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint 161615 2025.
[37] Zhibo Yang. Jun Tang. Zhaohai Li, Pengfei Wang, Jianqiang Wan, Humen Zhong, Xuejing Liu, Mingkun Yang, Peng Wang, Yuliang Liu, et al. Cc-oer: A comprehensive and challenging ocr benchmark for evaluating large multimodal models in literacy. arXiv preprint arXiv:2412.02210,, 2024.
[38] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0 08.4008.0800, 08000 2024.
[39] Wenwen Yu, Ning Lu, Xianbiao Qi, Ping Gong, and Rong Xiao. Pick: processing key informa- tion extraction from documents using improved graph Gaaninng-covoluttinnn networks. In 2020 25th International conference on pattern recognition (ICPR), pages 4363-4370. IEEE, 2021.
[40] Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang. Ming Tang, and Jinqiao Wang. Vision-rl: Evolving human-free alignment in large vision-language models via vision- guided reinforcement learning. arXiv preprint arXiv:2503. / 18013, 2025.
[41] Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi multi-modalliyy rl training framework. ltttps://githu/ com/hiyouga//aasyR] 2025.
[42] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403. 13372, 2024.
[43] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint vrXiv:2504.10477 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope. The paper introduces KIE-HVQA, a benchmark for evaluating OCR hallucination under degraded conditions, and proposes GRPO-based framework to enhance u'sion-faithii reasoning. The claims are supported by the experimen- tal results showing significant improvement in hallucination-Fie free accuracy, demonstrating the effectiveness and robustness of the proposed approach.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have included separate "LLimitations" section the supplementary materials to analyze the limitations of our method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In this paper, we elaborate on the data collection and annotation process (Section 3), the overall details of the method (Section 4), the open-source datasets used (publicly available), and the implementation details (Section 5.1).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
14

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We regret that we are unable to provide the source code at this time as is currently undergoing institution's 's internal review and clearance process for open access. The code will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co public/gui
wwtpsuwwwwssssoooooooooooooo soiooooicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results, See the NeurIPS code and data submission guidelines (https //nips. wttpccccccpppppecccccccccccccccp httpllcc/giiee//oooooo
ietooisssiin ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please refer to Section 5.1 for more implementation details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Although we did not explicitly report error bars due to resource constraints, we conducted a detailed ablation analysis the experimental section, which demonstrates that our method has significant advantages under different experimental settings.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 5.1, under the implementation details, we provide a detailed explanation of the GPUs, quantity, and experimental time required.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::
Answer: [Yes]
Justification:
We confirm that our paper complies with the NeurIPS Code of Ethics in all
aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
16

---

Justification:
Please refer to the "Broader Impacts" section in our supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper mainly focuses addressing the issue of OCR hallucination in degraded document analysis, and there is no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the data used in the paper has been properly cross-referenced and complies with their licensing requirements.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, ppaper swi thcode. com/ com/datasets. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The new dataset constructed in the paper based on existing open- data or generated through simulation. Once our internal review process for open-sourcing the dataset is completed, the dataset and corresponding code will be made open-source, along with detailed documentation.
Justification;
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
18

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We have described the use of LLM tools clearly in the Method Section.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy http:s///eerrrrssscc/ooooorrrres for what should or should not be described.
19


Reviews:
[{"review_id": "GyOhEswjyp", "reviewer": "Reviewer_cy4C", "summary": "This paper introduces KIE-HVQA, the first benchmark explicitly designed to evaluate OCR hallucinations in multimodal large language models (MLLMs) under degraded visual conditions. The benchmark includes degraded document images with pixel-level OCR reliability annotations, enabling precise assessment of hallucination behavior. To address this challenge, the authors propose a novel reinforcement learning framework using Group Relative Policy Optimization (GRPO), combined with a new reward function that accounts for visual uncertainty. The approach incorporates both supervised fine-tuning and reinforcement learning to encourage models to recognize and appropriately respond to degraded visual information. Experiments on Qwen2.5-VL show substantial improvements over state-of-the-art models like GPT-4o, especially in hallucination-prone settings, while preserving general OCR performance.", "questions": "Prior works[3] have tried to address a similar problem of models not focusing on visual details. Under settings like TextVQA[1], STVQA[2], have the authors done any experiments or have any thoughts as to how the authors' proposed method would work?\n\n[1] Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D. and Rohrbach, M., 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8317-8326). \\\n[2] Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C.V. and Karatzas, D., 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 4291-4301). \\\n[3] Hegde, S., Jahagirdar, S. and Gangisetty, S., 2023. Making the v in text-VQA matter. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5580-5588).", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper presents a rigorous and thorough experimental setup, including well-designed benchmarks, precise annotation strategies, and a custom reward function tailored to OCR degradation. The results clearly demonstrate strong gains in hallucination suppression and robustness across diverse degradation types.\n- This work tackles a critical and underexplored problem in OCR image understanding, vision-language hallucination in degraded inputs. It offers not only a novel dataset (KIE-HVQA) but also a method to mitigate OCR errors.\n- The proposed GRPO-based framework, with a multi-objective reward function, is a novel approach to mitigating OCR hallucinations. The emphasis on \"vision-faithful reasoning\" and incorporating uncertainty-driven refusal behaviors directly into the training process via reinforcement learning is innovative and well-aligned.\n\n**Weakness**\n- While the paper effectively demonstrates the problem and its proposed solution, it lacks a comprehensive comparison with other potential or existing mitigation techniques for hallucination in MLLMs (beyond just showing that other models hallucinate). A discussion or comparison with methods that might approach hallucination from different angles (ex., uncertainty quantification, confidence estimation, or alternative training paradigms) would provide a broader context and further highlight the unique advantages of the proposed GRPO-based approach.\n- The reinforcement learning approach is adapted from prior work (ex., DeepSeek-R1), and while the integration with OCR-specific reward design is well-executed, it builds incrementally on known RL optimization principles.\n- Lack of comparison and evaluation on other widely used datasets like TextVQA[1], ST-VQA[2], OCR-VQA[3]. These datasets contain OCR images and bounding box data along with them.\n\n[1] Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D. and Rohrbach, M., 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8317-8326). \\\n[2] Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C.V. and Karatzas, D., 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 4291-4301). \\\n[3] Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A., \"OCR-VQA: Visual Question Answering by Reading Text in Images,\" 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 947-952", "was_revised": true, "final_justification": "The authors have addressed my initial concerns satisfactorily. The paper is technically sound and the contributions are clear, I believe the overall impact, novelty, and scope remain somewhat limited to OCR recognition. I believe the work would be much more effective when aligned with reasoning rather than just recognition which is although important, would be better with certain revisions. Therefore, I am maintaining my rating.", "rebuttal": "**Q1:** \n\nThank you for your valuable suggestion. Current mainstream research on multimodal hallucinations[1] mostly focuses on objects, attributes, or relationships, whereas our paper concentrates on OCR hallucinations in degraded documents, which can be viewed as an exploration of existence and attribute hallucinations for text—a special category of objects. Our method essentially constitutes an uncertainty quantification modeling approach, introducing a rejection mechanism in degraded scenarios that aligns with the underlying ideas of most approaches addressing existence hallucinations. What differentiates our work is the use of natural language descriptions such as \"occluded\" and \"invisible\" to measure uncertainty, combined with reinforcement learning modeling, thereby providing a clearer solution tailored to hallucination-prone scenarios specific to OCR tasks. We will incorporate comparisons with traditional hallucination tasks in the related work section.\n\n**Q2:**\n\nOur core innovation lies in task-driven refinement and integration. Traditional reinforcement learning methods typically require models to perform complex chains of reasoning to arrive at correct answers. In contrast, this paper innovatively unifies the GRPO method with refusal-to-answer tasks and, for the first time, proposes a quantifiable dataset addressing the hallucination problem in degraded text OCR, providing a foundation for future research.\n\nInspired by counting tasks[2], we introduce an indirect reasoning step: during the GRPO phase, the model first infers character quantities and then decides whether to refuse answering based on this inference. Building upon this mechanism, we propose the first hierarchical reward rules for character degradation, quantifying visual uncertainty into three trainable signals. Based on experimental results, our proposed method, which constructs vision-text reasoning chains through the collaborative integration of GPT-4o and DeepSeek-R1, successfully alleviates the hallucination problem in degraded text OCR.\n\n**Q3:** \n\nThis paper primarily evaluates the model’s general OCR capability (as shown in Table 2, demonstrating comparable results to leading models). It should be noted that although these datasets contain OCR images and bounding box information and are widely used to train MLLMs, they consist almost entirely of clear and easily readable text, with little to no degraded text samples. As a result, they are insufficient for assessing model performance on degraded OCR inputs, which is the focus of our evaluation. Therefore, our evaluation framework cannot be directly applied to these datasets. As future work, we plan to construct more datasets by combining degradation techniques with large model annotations and manual verification, and will release these publicly to better support the evaluation methodology proposed in this paper.\n\n**Q4:** \n\nWe sincerely thank the reviewer for the careful attention to the work \"Making the V in Text-VQA Matter.\" This prior work indeed makes important contributions toward addressing the issue of models overly relying on textual cues and neglecting visual details in TextVQA tasks. By employing a multimodal embedding fusion framework, it maps image objects, OCR texts, and question texts into a unified representation space, effectively balancing the visual and linguistic modalities.\nIn contrast, our work focuses on a fundamentally different and challenging problem:\n- Firstly, we mainly focus on the hallucination problem caused by the model’s over-reliance on large language model (LLM) prior knowledge in degraded scenarios (such as low-quality images), which is different from previous works that address the issue of weak attention to text within images.\n- Secondly, we propose a character-level fine-grained alignment mechanism, which differs from the coarser multimodal fusion approaches in prior work, aiming to capture textual details more precisely;\n- Lastly, we design an end-to-end architecture that directly learns text representations from pixels **without relying on external OCR systems**, thereby avoiding additional error propagation and improving overall robustness.\nTherefore, while both works share a common interest in vision-language fusion, our research emphasizes distinct challenges and adopts different technical approaches. We hope this explanation clarifies the differences in focus and contributions between the two works. We greatly appreciate the reviewer’s insightful comments and valuable feedback.\n\n[1] Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., & Shou, M. Z. (2024). Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930.\n\n[2] Tan, H., Ji, Y., Hao, X., Lin, M., Wang, P., Wang, Z., & Zhang, S. (2025). Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752."}, {"review_id": "UhOTuvdk6B", "reviewer": "Reviewer_AmPt", "summary": "The paper presents KIE-HVQA, the first benchmark for measuring OCR hallucinations in degraded document images with character-level annotations. A 7B model is trained via supervised CoT followed by GRPO with a reward that penalizes hallucinations. The system achieves ~28% higher hallucination-free accuracy than GPT-4o on KIE-HVQA while matching OCR baselines on standard tasks, combining a novel benchmark and RL method for robust document understanding", "questions": "1. Have you tested the model on entirely unseen corporate or governmental documents to verify generalisation beyond the synthetic portion?\n2. How does hallucination suppression change if degradation masks are obtained from an automated detector rather than ground truth?\n3. Could the authors report results for a system that first runs a state-of-the-art OCR engine, then feeds text and images to the LLM? This would clarify the specific advantage of GRPO.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. First, the benchmark itself fills a clear gap: existing OCR/VQA suites rarely test degraded inputs or explicitly label which characters are uncertain. By supplying pixel-level reliability masks and diverse degradation types, KIE-HVQA offers the community a concrete target for measuring and analysing hallucinations in safety-critical settings such as ID verification and medical forms.\n2. Second, the technical solution is well motivated and experimentally validated. The GRPO framework, with its uncertainty-aware reward, yields large gains over strong proprietary and open-source baselines without eroding general OCR accuracy.\n\n**Weaknesses**\n1. Despite its realism, the dataset is modest in scale and partially synthetic, relying on GPT-4o-generated documents; it is unclear whether 2 400 samples cover the full variability of real-world paperwork or support robust generalisation to unseen layouts and languages.\n2. The reward formulation assumes oracle-level degradation labels during training, practical deployment would lack such pixel-wise ground truth, so the paper should discuss how performance degrades when degradation detection itself is noisy or learned.\n3. Finally, the evaluation omits specialised vision-only OCR engines or hybrid pipelines that could be combined with LLM reasoning; without these baselines it is hard to disentangle the benefit of the new learning paradigm from that of simply adding more OCR-centric supervision.", "was_revised": true, "final_justification": "While your clarifications addressed many of my concerns, considering the overall quality, scope, I will keep my original score, which was already positive.", "rebuttal": "**Q1:** \n\nFirstly, the training data in our work primarily originates from the TextOCR dataset, whereas the validation set includes other categories, such as cards, corporate documents, which were not encountered during training. Therefore, there exists a certain domain difference between the training and validation data. The experimental result of 58.05% demonstrates the generalizability of our method.\n\nFurthermore, we have conducted evaluations on the OCRBench dataset (as shown in Table 2), which further demonstrate the strong generalization capability of our method in conventional OCR scenarios.\n\nTo improve the diversity of data types in the validation set, we plan to expand it in future work to enhance the comprehensiveness and representativeness of our evaluation.\n\n**Q2:**\n\nOur training process does not rely on bounding box annotations. The reason we use bounding boxes in the pipeline is primarily to generate accurate degradation samples, which in turn provide precise labels for training and enable controllable degradation through image processing techniques. Therefore, the entire process does not require assistance from automatic detectors. Our method adopts an end-to-end SFT-RL training framework that directly optimizes from pixel-level input to text output. In fact, if we can obtain a large amount of real training data with OCR annotations and accurate character-level degradation information, we do not depend on any coordinate-based detection tools, and the model performance is expected to improve further. We appreciate your valuable suggestion and will continue to explore the potential application of automatic detectors in this stage in our future work.\n\n**Q3:**\n\nThank you for the valuable suggestion. We conducted experiments on a system that first runs a state-of-the-art OCR engine, then feeds the recognized text along with the images into the large language model. The results are shown in the table below:\n\n| Dataset   | Clr | Nc | Final |\n|-----------|----------------------|-------------------------------|--------------------------|\n| Card      | 14.11                | 0.0                           | 15.13                    |\n| OCRBench  | 38.85                | 0.0                             | 42.50                    |\n| Wild      | 5.25                 | 0.0                           | 4.85                     |\n\nAs can be seen, the model blindly trusts the OCR outputs in the OCR+LLM system, leading to a significant drop in recognition performance on unclear or blurred text."}, {"review_id": "LxXd1oKnBB", "reviewer": "Reviewer_fNhj", "summary": "This paper proposes the first benchmark specifically designed to evaluate OCR hallucination in degraded images, covering diverse document types such as identity cards, invoices, and prescriptions. To address this real-world challenge, the authors further introduce a Group Relative Policy Optimization (GRPO)-based training framework, incorporating a novel reward mechanism to fine-tune Multimodal Large Language Models (MLLMs). Experimental results demonstrate the effectiveness of the proposed framework on the newly introduced benchmark.", "questions": "Please refer to the listed weakness.", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n1.\tThe paper proposes the OCR benchmark with degraded documents—a challenging yet realistic task setting that is crucial for enhancing the reliability of MLLMs.\n2.\tThe proposed GRPO-based training framework effectively improves model performance on the newly introduced benchmark.\n\nWeakness\n1.\tThe benchmark lacks essential details. It would be better to provide comprehensive dataset documentation by including key statistics (e.g., size, category distribution) and a clear explanation of the dataset construction process. Additionally, incorporating human expert validation can improve the annotation quality and enhance the benchmark's overall credibility. \n2.\tAs illustrated in Figure 2, the benchmark introduces numerous novel tokens (e.g., <part_occluded>) that are unseen during MLLM pretraining. Consequently, existing MLLMs are inherently incapable of generating these tokens. So the authors should provide more details about how to control comparison fairness between the trained model and the original model on generating unseen tokens.\n3.\tThe proposed GRPO-based framework only modifies the reward design without introducing significant architectural or algorithmic innovations. So the contribution of this paper from a technical perspective should be strengthened. \n4.\tThe method does not show improvement on traditional OCR benchmarks, which raises concerns that the observed gains on the proposed benchmark may result from overfitting to specific data patterns. So it would be better to validate the model’s effectiveness and generalization ability with broader experiments.", "was_revised": true, "final_justification": "The rebuttal addressed some of my concerns, but I'm still not convinced regarding the technical contributions and model generalization toward other benchmarks.", "rebuttal": "**Q1:**\n \nWe appreciate your attention to the details of the dataset. In Section 3.2, we provide a detailed description of the construction process of the KIE-HVQA benchmark: this dataset integrates 100 key information queries from OCRBench, entity data from WildReceipt, and 200 synthetic templates generated by GPT-4o (all using compliant fictional information), totaling 400 carefully designed test samples (see lines 129–156).\n\nRegarding degradation simulation, we applied pixel-level degradation to key regions using Python scripts and Photoshop, including various real-world degradation types such as motion blur, low contrast, and partial occlusion. Visual examples of these degradation effects can be found in Figure 2.\n\nFurthermore, to ensure annotation quality, we implemented a rigorous double-blind cross-validation mechanism — each data sample was independently reviewed by at least two domain experts. Additionally, we employed multi-model consistency checks to verify the reliability of the degradation region annotations. This layered quality assurance system significantly enhances the credibility of the benchmark.\n\n**Q2:**\n\nOur paper does not contain [object Object] tokens. You may be referring to the special tokens such as <part_occluded> and <occluded> shown in Figures 1 and 2. We clarify that these tokens are used solely as visual indicator and do not participate in the actual input or output processes of the models. To ensure fair evaluation, we designed a unified zero-shot evaluation prompt template (the full prompt is provided in the appendix), which effectively prevents evaluation bias caused by differences in model output formats.\n\n**Q3:**\n\nOur core innovation lies in task-driven refinement and integration. Traditional reinforcement learning methods typically require models to execute complex reasoning chains to obtain correct answers; by contrast, this paper innovatively unifies the GRPO method with refusal-to-answer tasks. Inspired by counting tasks, we introduce an indirect reasoning step: during the GRPO phase, the model first infers character quantities and then decides whether to refuse answering based on this inference. Building upon this mechanism, we propose the first hierarchical reward rules for character degradation, quantifying visual uncertainty into three trainable signals. Furthermore, through collaborative integration of GPT-4o and DeepSeek-R1, we construct vision-text reasoning chains that effectively overcome the limitations of pure text models in processing multimodal data.\n\n**Q4:**\n\nAs shown in Table 2, our model maintains a comparable accuracy level to the original model on  OCR benchmarks (Scene: 180 vs. 181, Doc: 179 vs. 181), which clearly demonstrates that the optimization scheme does not compromise the fundamental OCR capability. It is important to emphasize that the core objective of KIE-HVQA is to enhance the model’s robustness against hallucinations in degraded scenarios, rather than directly optimizing traditional OCR metrics. On the degraded subsets of WildReceipt and OCRBench, the model achieves a breakthrough accuracy of 61.34% in recognizing blurred characters, strongly validating the approach’s generalization ability to real-world degradation patterns. Compared to GPT-4o and Qwen2.5-VL, this represents a relative improvement of +29.6% and +37.2%, respectively."}, {"review_id": "UWCOTF0cFV", "reviewer": "Reviewer_A3fL", "summary": "This paper aims to address MLLM’s OCR hallucination issue in degraded document understanding. The authors propose a dataset named KIE-HVQA, consisting of various documents with real-world degradation and pixel-level annotation. For annotation, they tag the degraded area (blur, occlusion, low contrast) using special tokens like `<part_occluded>` and `<occluded>`, following the correct content. They also propose a new RL-based KIE MLLM, which is trained using GRPO and self-designed rewards. The authors conduct experiments to validate their methods’ effectiveness.", "questions": "For rebuttal questions, please see weaknesses. The major concern of this paper lies in the lack of many critical model implementation and experimental details. If the authors can address these issues, I will consider raising my score.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "> Strengths:\n\n1. The authors discover a novel perspective: the hallucination of OCR on degraded documents. This is interesting and can encourage the community to pour more attention into this underexplored issue.\n2. The KIE-HVQA dataset is a valuable contribution to the community. There is a lack of document analysis datasets with such real-world, character-level degradation. This pioneering effort fills this gap and potentially contributes to broader applications in the document field beyond this paper’s task.\n3. The KIE-HVQA includes diverse data sources, including receipts, documents, and ID cards. The authors generate ID card images using GPT-4o to avoid privacy issues. This is an effective and important approach to producing privacy-sensitive data.\n4. In Sec. 4.2, the authors generate CoT data using an iterative approach, which is reasonable for producing high-quality CoT data.\n\n> Weaknesses:\n\n1. In Sec. 3.2, L141-142, the authors only say they use GPT and Qwen2.5-VL-72B to evaluate the degraded results, without giving a concrete evaluation process. Why are two models used here? How do they operate? The authors should provide more details.\n2. In Sec. 3.3, how do the authors implement the measurement of legible and degraded character accuracy? The authors provide only definitions without implementation details.\n3. In Fig. 3, the bottom-right panel “3. Supervised Fine-Tuning” is unclear and confusing. What is the role of Qwen2.5-VL here? SFT is a training procedure. If it takes the QA pair as input, it looks more like a data generation procedure, which should not occur during training. In addition, what does the “CoT” output by Qwen2.5-VL mean? I think the SFT model is trained using the CoT data generated during “2. Data Generation”, and the CoT here makes me confused. Can the authors provide some clarifications and make Fig. 3 clearer?\n4. The title of Sec. 4.2 is “Cold-start Initialization”. As far as I know, cold-start initialization refers to supervised fine-tuning (SFT) on certain data. Yet, this sub-section mainly describes the CoT data generation process rather than the training process. If the authors aim to describe data generation, then they should change the title. Also, it seems that the authors have omitted the illustration of SFT (despite its presence in Fig. 3) since Sec. 4.3 is about RL training. An SFT-related sub-section should precede RL training.\n5. In Sec. 4.3, the authors define three rewards (legible character, partially obscured but human-recognizable characters, unrecognizable characters), but do not provide their detailed implementations. Algorithm 1 appears to compute rewards; however, the definitions inside do align with the three rewards (the outputs’ names are not_clear_metric, final_metric, and clear_metric, which are different from the above three). The authors should give necessary details for reward computing.\n6. For Algorithm 1, the CALCULATE_METRICS function simply outputs one similarity. However, in line 11 of this algorithm, this function outputs three values. This inconsistency is confusing.\n7. In experiments, what prompts are used in testing the closed-source models (e.g. GPT, Claude)? Are the prompts the same as those used for training the proposed model to ensure fairness?\n8. In Table 1, the proposed model exhibits better performance than other models not trained on the proposed dataset. However, for recognition of those occluded texts, what is the rationale behind the outperformance of the proposed models? Recognizing visual text mainly depends on the ability of the visual encoder. Since the authors did not specifically improve visual modeling and text is occluded, performance on these texts should be close. Can the authors provide a deeper analysis of such outperformance?\n9. In L322-323, what does the format reward refer to? I did not found any definition of the format reward in either the methodology section or the experiment section.\n10. typo: L106: MLMMs -> MLLMs; L145: We -> we; L307: KIE-HVQA -> KIE-HVQA. L320: setting -> Setting", "was_revised": true, "final_justification": "Most of the concerns are addressed. Therefore, I raise the rating to borderline accept.", "rebuttal": "**Q1，Q2:**\n\n  Thank you for your comment. We will clearly specify these details in the final version. In the degradation results verification phase (L141-L144), we established a rigorous dual-model evaluation mechanism. This process begins with a character localization model extracting the coordinates of degraded regions, followed by cropping each degraded character into a sub-image and inputting it into GPT-4o and Qwen2.5-VL-72B for parallel verification (Lines 141-142). These two models were selected based on their complementary strengths: GPT-4o serves as the performance benchmark for closed-source multimodal models and excels in complex visual pattern recognition, while Qwen2.5-VL-72B is currently the leading open-source model in OCR capability. Each model independently outputs a binary judgment (visible/invisible), and discrepancies between their judgments are resolved through expert human review. This design avoids bias from any single model and ensures accurate annotation of borderline cases through human intervention, providing a solid foundation for subsequent character-level reliability annotations.\n\n Regarding the implementation of legibility measurement (Section 3.3), we adopt a multi-model consensus mechanism as the core criterion. Specifically, when both GPT-4o and Qwen2.5-VL-72B correctly recognize a character independently (i.e., without relying on contextual inference), the character is labeled as \"Partially Obscured but Human-Recognizable Characters\"; when both models fail, it is marked as an \"Unrecognizable Characters.\" All model outputs are ultimately verified and refined through expert human review to ensure annotation accuracy. \n\n**Q3，Q4:**\n\nYes, there is some ambiguity in the illustration of Figure 3. Specifically, Qwen2.5-VL and SFT MODEL refer to the baseline reference model and the cold-started model, respectively. The contents shown correspond to the original response generated by these models and the chain-of-thought (CoT) outputs from the cold-started model. It should be clarified that CoT here does not refer to the model outputs themselves, but rather to the training data generated in the data generation phase used for supervised fine-tuning (SFT). This design intuitively demonstrates the training effect through comparison, avoiding confusion with the data generation process in Stage 2.\n\nRegarding Q4, we appreciate your precise correction of the term \"cold-start initialization.\" We will clearly distinguish between the data generation and training phases in the main text and revise the figure accordingly: first, a dedicated dataset is constructed via the multimodal CoT data generation pipeline (currently described in Section 4.2), followed by an independent subsection detailing the SFT process. Specifically, we performed supervised fine-tuning (SFT) on the pretrained Qwen2.5-vl-7B model as the foundational multimodal large language model (MLLM) for cold-start initialization. The model after cold-start initialization is named the SFT MODEL. At this stage, the base MLLM has already learned complex reasoning patterns from DeepSeek-R1; however, this also leads to an \"overthinking optimization problem,\" where the SFT MODEL tends to generate unnecessarily lengthy reasoning processes on certain problems, whereas correct reasoning typically occurs within shorter cognitive chains. Based on this, the model requires additional GRPO training to further refine its performance.\n\n**Q5:**\n\nThank you for your comment. We have designed a corresponding reward mechanism for the three different categories of characters. For clearly recognizable characters, such as obvious text, we use the clear_metric for evaluation, which requires the model’s output to exactly match the visual input. For partially occluded characters that both MLLM and humans can still infer, we apply the not_clear_metric, which allows the model to retain labeled anomalous characters, reflecting a certain degree of tolerance.\n\nAs for completely unrecognizable characters, the model’s performance is indirectly constrained by the **final_metric**. For example, in the word “Beautiful” shown in Figure 4, the letters **t** and **i** are severely blurred and cannot be recognized. If the model attempts to guess and output specific content for these characters, the overall OCR similarity score will significantly decrease, thereby penalizing the model and encouraging it to avoid generating fabricated or uncertain information in such regions.\n\n**Q6:**\n\nSorry for the confusion caused. The contradiction you observed in the function output actually arises because both pred and truth inputs are triplets, and consequently, the function returns a triplet as well. This is a Pythonic style of handling multiple related values together. To avoid any ambiguity, we will provide a clear and detailed definition of this triplet input data structure in the revised version.\n\n**Q7:**\n\nThe prompts used are detailed in the appendix. Unlike the prompts used during model training, we specifically added zero-shot output format examples to prevent the model from producing incorrect responses due to a lack of clear format guidance. In fact, without providing these examples, the model’s responses often exhibit format inconsistencies or fail to meet the expected requirements. For instance, in the following case, the ground truth is: {\"clear Char-level OCR\": \"2 . 0\", \"not clear enough Char-level OCR\": \"\", \"clear number\": 3, \"not clear enough number\": 0, \"Final OCR\": \"2. 0\"}, while the model’s response is: \"The price of the \"NIVEA FOR MEN\" on the receipt is £0.94.\" This demonstrates that without format examples, the model tends to deviate from the expected output format.\n\n**Q8:**\n\nThank you for your insightful discussion on this important issue. Your observation is highly accurate — the performance advantage of our model in recognizing occluded text does not stem from improvements in the visual encoder, but rather from our innovative approach to addressing OCR hallucination problems. The key reason why traditional models perform poorly in degraded scenarios lies in their excessive reliance on LLMs, which leads to hallucinations. Our method fundamentally overcomes this challenge through three key innovations:\n\nFirst, the character-level reliability assessment mechanism enables the model to finely distinguish readable information from noise interference. As shown in Figure 4, when processing the word “Beautiful,” the model analyzes each character independently: clearly visible letters “B, a, u, f, u, l” are directly retained; the partially occluded “e” is marked with low confidence; and the completely blurred “t, i” are deemed unrecognizable. This pixel-level fine-grained analysis frees the model from the chain hallucinations caused by whole-word recognition errors in traditional approaches.\n\nSecond, the dynamic refusal strategy completely overturns the conventional output paradigm. When the system detects a blurry receipt, it no longer blindly outputs a guess as existing multimodal large language models (MLLMs) do, but instead opts to refuse to answer, thereby avoiding the generation of incorrect information.\n\nFinally, the precise guidance of the reward function, implemented via the GRPO algorithm, ingrains these behaviors as inherent model capabilities. As shown in Table 3, removing the refusal weight from the format reward immediately causes a 12% drop in the model’s performance on occluded character recognition.\n\nThese three innovations together empower the model to achieve outstanding performance in complex degraded environments, significantly enhancing the reliability and accuracy of recognition.\n\n**Q9:**\n\n Thank you for your comment. As shown in Figure 3, the format reward consists of two parts: the template format and the content (json) format. The purpose of the format reward is to encourage the model to follow a predefined template, facilitating the extraction of answers. For each generated result, the validation function checks whether the output adheres to the specified template format (i.e., \"not clear enough Char-level OCR,\" \"clear Char-level OCR,\" \"Final OCR\") as described in the Table1's caption. Additionally, it verifies whether the content complies with the standard, specifically whether it is in valid JSON format. We will provide more detailed descriptions in the main text in the future. Once again, thank you for your insightful comment.\n\n**Q10:**\n\nThank you for your careful review and valuable feedback. We have thoroughly checked the manuscript and corrected all the typos and capitalization errors you pointed out (e.g., \"MLMMs\" to \"MLLMs\" at L106, \"We\" to \"we\" at L145, as well as the corrections for \"KIE-HVQA\" at L307 and \"setting\" at L320)."}]

Author rebuttal:

Meta-review: The paper focuses on hallucinations for MLLM in OCR. 
They also propose a RL-based approach with an heuristic reward to improve or mitigate hallucinations. 

From the reviews, the paper is technically sound with clear contributions. A major limitation is the scope, being focused on OCR recognition. While agreeing that OCR might be limited, the novelty of the approach and tackling this relevant problem might inspire other approaches in other domains. Therefore leaning towards acceptance.

Importantly, reviewers acknowledge the efforts made to address all the comments
Decision: Accept (poster)