When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding
Authors: Yan Shu
Hangui Lin
Yexin Liu
Yan Zhang
Gangyan Zeng
Yan Li
Yu ZHOU
Ser-Nam Lim
Harry Yang
Nicu Sebe
Venue: NeurIPS 2025 poster ()
Keywords: Large Multimodal Models
Hallucination
Scene text spotting and understanding
Primary area: deep_learning
TL;DR: Exploring and Mitigating hallucination in LMMs towards scene text spotting and understanding

Abstract: Large Multimodal Models (LMMs) have achieved impressive progress in visual perception and reasoning. However, when confronted with visually ambiguous or non-semantic scene text, they often struggle to accurately spot and understand the content, frequently generating semantically plausible yet visually incorrect answers, which we refer to as semantic hallucination.
In this work, we investigate the underlying causes of semantic hallucination and identify a key finding:  Transformer layers in LLM with stronger attention focus on scene text regions are less prone to producing semantic hallucinations.
 Thus, we propose a training-free semantic hallucination mitigation framework comprising two key components: (1) ZoomText, a coarse-to-fine strategy that identifies potential text regions without external detectors; and (2) Grounded Layer Correction, which adaptively leverages the internal representations from layers less prone to hallucination to guide decoding, correcting hallucinated outputs for non-semantic samples while preserving the semantics of meaningful ones. To enable rigorous evaluation, we introduce TextHalu-Bench, a benchmark of  1,740 samples spanning both semantic and non-semantic cases, with manually curated question–answer pairs designed to probe model hallucinations.
Extensive experiments demonstrate that our method not only effectively mitigates semantic hallucination but also achieves strong performance on public benchmarks for scene text spotting and understanding.

When Semanties Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding
Yan Shu
Hangui Lin
Yexin 2
Yan Zhang
Gangyan Zeng
Yan Li
Yu Zhou? Ser-Nam Lim
Harry Yang² Nicu Sebe'
University of Trento Honn Kong University of Science and Technology
University of International Relations
Institute Information Engineering, Chinese Academy of Sciences
School of Cyber Security, University of Chinese Academy of Sciences Nanjing University of Science and Technology
VCIP & TMCC DISSec, College of Computer Science, Nankai University University of Central Florida
[yan. shu, ni niculae.c sebee@uniiieeeeeeeeeeeeeeeuuueee inniiin.it it
https://githut: com/ sww.ssssyyyyyyaaaaeaa -ea----eemntniialaaaiiiiiee
Abstract
Large Multimodal Models (LMMs) have achieved impressive progress in visual perception and reasoning. However, when confronted with visually ambiguous or non-semantic scene text, they often struggle to accurately spot and understand the content, frequently generating semantically plausible yet visually incorrect answers, which we refer to as semantic hallucination. In this work, we investi- gate the underlying causes of semantic hallucination and identify key finding: Transformer layers in LLM with stronger attention focus on scene text regions are less prone to producing semantic hallucinations. Thus, we propose training-free free semantic hallucination mitigation framework comprising two key components: (1) ZoomText, a coarse-to-fi strategy that identifies potential text regions without external detectors; and (2) Grounded Layer Correction, which adaptively leverages the internal representations from layers less prone to hallucination guide decod- ing, correcting hallucinated outputs for non-semantic samples while preserving the semantics of meaningful ones. To enable rigorous evaluation, we introduce TextHalu-Bende benchmark of 1,740 samples spanning both semantic and non- semantic cases, with manually curated question-answer pairs designed to probe model hallucinations. Extensive experiments demonstrate that method not only effectively mitigates semantic hallucination but also achieves strong performance public benchmarks for scene text spotting and understanding.
Introduction
Scene text, as self-descriptive visual element, conveys rich semantic information that is crucial for downstream applications such autonomous driving, product analysis, and assistive technologies. Effectively spotting and understanding scene text [1, 2, 3, 4, 5, 6, 7, 8] thus attracts growing attention from the deep learning community.
'Equal contribution.
Conresponding author <liyan @ uir edu.cns.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Non-sementic
Semontic
Semontic
Semontic
Non-semontic
85.30
74
64.82
58.40
95.13
44
Mogy
24
4.77
User: What is the text under Pocific?
User: Recognize all the text the picture
User: Is the text the first line?
LLoVA15
Qwen2-VL Miri Miri-Morkey (b)
Qwendvt-78 under Pacific
Qwendvi-
Qwen2vi-7B: The text in first "PUBLIC.
Minimonkey: The text in the mage "PULL" on the glass dioor.
Minimenkey: The Text under "Pacffic S "MOTEL"
Minimenkey: The text the first line "PUBLIC."
Semanttc
Non-semontic
30
User: Is the text visible on the right of the picture PONDOK") ?
Scene Text
10.00
PONDO
Owen22i-78: Yes, text "PONDDII" visible the right
6:47
side of the
Minimonkey: Yes, the text "PONbOK" visible on the right of the picture.
0.00
LLeVA15 Qwen2-VL Mini-Morkey
(6)
Figure 1: (a) LMMs hallucinate scene-text answers by relying on semantic priors rather than grounding in the actual visual content. For instance, when edit "MOTEL" and "PULL" to "MMOTEL" and "PULLa", the models still answer the original ones. (b) and (c) illustrate the performance of LMMs OCRBench and ICDAR 2015, with separate evaluations semantic and non-semantic text samples.
To spot and understand scene texts, traditional approaches [9, 10, 11, 12] rely on multi-stage meth- ods, separately addressing text detection, recognition, and language modeling, which limits their generalization ability in diverse real-world settings. As a general solution for vision-language tasks, Large Multimodal Models (LMMs) [13, 14, 15, 16, 17] have shown remarkable capabilities in image captioning and visual question answering by combining visual encoders with Large Language Models (LLMs). Motivated by this progress, researchers have begun adapting LMMs for OCR-related tasks, including document question answering [18, 19, 20, 21J, GUI analysis agents [22, 23], and unified OCR frameworks [24].
However, whether LMMs can reliably address scene text spotting and understanding remains under- explored. In this work, we investigate this question through "TextTrap" challenge. As illustrated Fig. I, LMMs such as Qwen2-VL [13] perform well when scene texts are semantically coherent. However, introducing subtle character-level perturbations that disrupt semantic meaning often leads these models to produce semantically plausible yet visually incorrect answers, a phenomenon we refer to as semantic hallucination. Further experiments ICDAR 2015 [25] and OCRBench [26] provide solid evidence that LMMs frequently hallucinate scene text answers based on semantic priors rather than actual visual grounding.
Motivated by the intuition that semantic priors mainly originate from the LLM, we analyze the causes of hallucination from two perspectives. Inspired by prior observations [27, 28, 29] that different layers in LLMs capture different types of information, we further reveal that these layers exhibit varying tendencies to hallucinate, with certain intermediate layers showing higher likelihood of correctly predicting ground-truth tokens. Building upon this insight, we further quantify and inspect the spatial distribution of attention maps within the LLM, and observe that layers allocating greater attention to ground- -round-truth text regions are less prone to hallucination. thereby suggesting a causal relationship between accurate attention allocation and the mitigation of semantic hallucination.
Based on these findings, we propose a semantic hallucination mitigation framework composed of two key components: ZoomText, which takes gglimpse-refocus steps to first localize contextual regions related the scene text, and then refines its focus estimate scene text regions. This coarse-to- toarse-to-fine grounding strategy eliminates the need for external model intervention. Grounded Layer Correction (GLC): Given the anchor regions produced by ZoomText, GLC adaptively selects the transformer layer with the strongest scene text grounding and fuses its hidden state representations into the decoding process. This design helps mitigate hallucinations for non-semantic samples while preserving the semantics of meaningful ones. Notably, method is training-free free and can be

---

seamlessly integrated into existing LMMs to effectively mitigate semantic hallucination in scene text
spotting and understanding.
Our main contributions are summarized as follows: 1) We identify the problem of semantic halluci- nation in LMMs when spotting and understanding scene text. We further investigate its underlying causes, revealing that attention drift across different layers within the LLM contributes significandy to hallucination. We propose a training-free hallucination mitigation framework that can be seamlessly integrated into existing LMMs without requiring any architectural modifications. 3) We conduct extensive experiments on multiple benchmarks, demonstrating the effectiveness of our method. For example, when applied to the Mini-Monkey [30] and Qwen2.5-VL [13], our framework yields substantial accuracy gains on ST-VQA [5] and TextVQA [31]. Additionally, we introduce SeetHall--encche new benchmark designed to evaluate semantic hallucination, where our framework consistently improves existing methods by approximately 4%.
2
Related Works
Large Multimodal Models for OCR. LMMs have demonstrated strong performance in general visual understanding tasks such image captioning [13, 14, 15, 32, 33], visual question answering [16, 34, 17, 35, 36, 37, 38, 39, 40], and video understanding [41, 42, 43, 44, 45, 46]. However, the increasing demand for text-grounded visual reasoning has revealed its limitations in accurate OCR. Recent works have proposed OCR-specific enhancements for LMMs, which can be broadly categorized into three strategies. (1) Resolution-aware processing: UReader introduces shape- adaptive cropping [18], while Monkey [19] and TextMonkey [20] adopt patch-wise division to better handle high-resolution text regions. Ocean-OCR [47] further utilizes native-resolution VIT to support variable input sizes. (2) Token compression and layout encoding: MPLUU-DDoCOu [21]
and TextHawk2 [48] reduce visual token redundancy while preserving spatial structure. Vary [49] introduces SAM-style [50] visual vocabulary tailored for document and chart understanding. (3) Redesigned OCR paradigms: GOT-OCR [24] proposes a new task formulation and architecture specifically optimized for OCR scenarios. Despite these advances, current models still rely heavily on semantic priors and often fail when the input contains visually plausible but meaningless words. This indicates lack of text grounding. Our work investigates this failure mode and proposes an attention-based inter-layer fusion mechanism to enhance robustness in text-level reasoning.
Hallucination in Large Multimodal Models. Hallucination in LMMs refers to the generation of outputs that are not grounded in the visual input, often leading to content that is irrelevant or factually incorrect. Prior work has systematically explored hallucination along several dimensions, including object hallucination [51, 52, 53, 54, 55], knowledge hallucination [56, 54, 57], relational misinterpretation [58, 59, 57, 60], attribute hallucination [58, 57, 60, 55], and hallucination induced by spurious visual patterns [61, 62]. Recent studies have also revealed inconsistencies in model responses across question types [63, 64, 65]. To mitigate hallucination, various strategies have been proposed, including self-correction decoding [66, 67, 68]. contrastive decoding [69, 70], and adversarial training [64, 56). However, most of these studies focus on object- or fact-centric hallucinations, while ope-sicc hallucinations remain underexplored. In this work, we identify a novel form of semantic hallucination in scene text spotting: LMMs accurately recognize semantically meaningful words, yet fail when those words are replaced with syntactically valid but semantically meaningless tokens. This behavior indicates that models rely heavily on semantic priors rather than truly grounding their predictions in visual evidence.
3 Methods
In this section, we first provide background on the generation paradigm of LMMs and analyze the underlying causes of semantic hallucination in scene text spotting and understanding. These analyses reveal that semantic hallucination is closely tied to attention drift within LLMs, where attention deviates from ground-truth text regions. Building upon these insights, we introduce our training-free hallucination mitigation framework.
3

---

Amsterdam (response)
P
Phat
21,90
Large Multimodal Models
Am
Stor
cm
Probability
0.5
Shat
Phon
Doyakt
E05
Pios
P
Amsterd (ground truth)
-gt
Phut
What is the text between 'NEU' "b21,99''
8m
EOS
A,(1= 0)
4,1 25)
A = 35)
Loyer 25
S
Layer 35
Layer
Loyer
Loyer 25
Layer
Figure 2: Visualization of the iotuuciiatiin-nnalllle pipeline. For each input image, we (1) identify hallucinated text tokens and compute their layer-wiee wise hallucination--enddee tendency scores, (2) calculate the ratio of the ground-truth text score to the hallucinated text score for each layer, and (3) overlay these normalized ratios onto the corresponding attention maps. We observe that layers with lower propensity to hallucinate concentrate their attention more strongly on the text regions.
3.1 Preliminaries of LMMs Generation
Most current LMMs adopt the minimalist architecture of LLaVA [35], which comprises a visual encoder, vision-language projector, and an LLM. Given input image, the visual encoder extracts a sequence of visual tokens V {U1, 82 ........ Un where n denotes the number of output patches. Similarly, the text input is tokenized into a sequence of text tokens T {t, f,, .tm............... These two token sequences are concatenated as X concat( (V, 7) and fed into the L.I,, parameterized by for auto-regressive generation. At each decoding step i, the model predicts the probability distribution over the next token yi in an auto-regressive manner:
ply: V,T,y<s) softmax (logit V, ycc))
To generate the final output, decoding strategies such as greedy decoding or beam search are employed to select the next token. The predicted token y4 is then appended to the previous input sequence, and the process is repeated until a stop condition met.
3.2 Investigating the Mystery of Semantic Hallucination
LMMs are pretrained on large-scale corpora primarily composed semantically coherent texts, which may impose strong semantic priors the model. In scene text spotting and understanding, such priors may cause the model to incorrectly interpret visually meaningless or random character patterns as meaningful words. To gain deeper insights into how these hallucinations arise within the model, we focus on the internal processing of the LMM. Prior work shows that different layers capture different types of information [68]. Building this, hypothesize that different layers of the LLM may exhibit varying tendencies to produce semantic hallucinations. To validate this hypothesis, we design an analysis pipeline consisting of steps:
Hallucinated Token Extraction. For each generated output, tokenize both the generated answer and the ground-truth answer using the LMM's 's predefined text tokenizer. We then compare the two token sequences and identify the first token in the generated sequence that diverges from the ground-truth as a hallucinated token.
Hallucination Tendency Scoring. We compute the hallucination tendency score at each layer l by comparing the output probabilities of the hallucinated token and its ground-truth counterpart. Specifically, each decoding step t, the model computes probability distribution over the entire

---

Gámpam
LLM
Transfoma Book
Tranffrmm Block
Nosmal commmaliiniion
Taps Sewettn
is code number heloopter?
Figure 3: Visualization of the ZoomiTxx process and examples.
vocabulary based on the prefix T From this distribution, we extract the probabilities assigned to both That and Y/tt as candidate tokens.
Phu softmax (W out out b) yhur
P softmax W Woulg + b)
where W out and b are the parameters of the output head. The hallucination score Shal is then calculated by / + Pa). A higher Sha, indicates that the model is more likely to favor the hallucinated output over the correct one at layer f.
As shown in Fig. 2, different transformer layers within the LMM exhibit varying tendencies toward semantic hallucination, with more examples provided in the Supplementary Material.
Based on this observation, we aim to further investigate the underlying mechanisms driving these differences, particularly focusing on the visual grounding behavior of different layers (i.e., how they attend to relevant scene text regions). This leads us to pose key question: Is there a relationship between a transformer layer's 's visual grounding ability (specifically, its attention to scene text regions) and its lendency produce semantic hallucinations?
To answer this question. we propose quantitative measure of visual grounding for each layer, termed the Text-region Attention Score (At). This score evaluates how much attention a transformer layer allocates to ground-truth text regions, which is calculated as:
Eier ESEr A,
(3)
EiEr Ejer oig
where I denotes the of all image tokens, and T I represents those image tokens located within the provided ground-truth text bounding boxes. af, is the self-attention weight from the i-th image token to the j-th image token layer f. Higher values of A₁ reflect increased allocation of attention score to the correct text regions, indicative of more robust visual grounding at layer t.
Based on Qwen2.5-VL [13] and Mini-Monkey [30], we evaluate our method on OCRBench [26], ST-VQA [5]. and TextVQA [31], and analyze the Spearman correlation [71] between layer-wise hallucination tendeney scores and their corresponding text-region attention scores. We observe strong negative correlation across all datasets, indicating that layers with lower attention to ground-
truth text regions more susceptible to semantic hallucination. Additional experimental details are provided in Appendix E.
3.3 Toward Semantic Hallucination Mitigation
Building on this key observation about semantic hallucination, we aim to leverage the connection between visual grounding ability and hallucination tendency design an effective mitigation strategy. This naturally raises two questions: 1) How estimate scene text regions without relying
5

---

additional modules? 2) Once we identify the layer with the strongest scene text grounding, how can we guide the decoding process using this information to reduce hallucinations?
ZoomTTxx. Unlike naturally salient objects, scene text often difficult to localize, especially in the absence of external text detectors. To address this challenge, we propose glimpse-refocus strategy for estimating scene text regions. We begin by observing that scene text frequently appears on semantically meaningful backgrounds, such as signs, posters, or product packaging, which naturally attract model attention during question answering [68]. Leveraging this intuition, we perform glimpse step that identifies text-related regions by computing the muery-to-image cross- attenttnnnn,on, which measures how much each image token contributes to the query understanding. The highlighted attention regions serve coarse estimation of potential text positions. Specifically, we extract the softmax nortuxxinomaaieed cross-attention from the query tokens to all image tokens at the final layer of the I.LM, resulting in Agav E RR x where H is the number of attention heads, Q the number of query tokens, and N is the number of image tokens. We average across heads and query tokens to obtain a global image attention map:
H
Q
Auss
(b.g) 2
E
(4)
HQ
h=1g=1
We then apply thresholding to select the top-K image tokens as coarse text region candidates.
However, not all high-responss tokens are truly relevant to the query, as LLMs often utilize certain tokens as "registers" to aggregate global context across the image. To mitigate this bias toward irreevant regions, we introduce Refocus step that filters out spurious activations. This step is based on the hypothesis that background or non-semantic tokens exhibit relatively stable attention patterns across layers, as they do not actively participate in the visual reasoning process. Accordingly, we compute a normalized attention shift score among the top-K candidate tokens identified in the Glimpse stage, which quantifies how much each token's importance evolves throughout the forward pass. Let = ( .K} denote the set of top-K image token indices selected from Atext- We extract the self-attention submatrices A and A/2. KxK from the first and last transformer layers, and then compute a normalized attention shift score as:
1/4) (1) V2V
normalized
(5)
text
(1)) +
where is a small constant for numerical stability. As shown in Fig. 3, most noisy tokens are effectively filtered out, leaving accurate text regions that can be used to guide the decoding process.
Grounded Layer Correction. After identifying grounded text regions, select the most visually grounded transformer layer t = arg max, Ag, and propose three strategies to correct the decoding process. All strategies operate on the final-layer hidden states (L) before decoding, producing revised representations H that integrate information from the grounded layer (+)), Specifically, we either: (1) replace all hidden states with the grounded ones (Replacement), (2) apply a weighted fusion with factor w (Fusion), (3) selectively replace tokens with high grounding scores based on the refined attention map (Selective Replacement).
(Replacement)
H.
(1 w) H I.)
w H!!
(Fusion)
(6)
H (L) if & S; H!")) ifies (Selective Replacement)
Empirical results (Sec. 5.3) show that among the three strategies, Fusion achieves the best balance between hallucination mitigation and semantic preservation. Accordingly, we adopt Fusion as our default decoding strategy.

---

Industry (200)
(401)
Non-semantic Rate
SQ: What are oll the texts the image?
SQ:: What are all the TexTs the image? ANS browhcus
7120
ANS: 4FO 118A DE 253 211E
options line
UQ: Which text from the following four
"What the establishment shown??
60
AFFURECIA
Abbrowhuuse B.browhous
Pabilic facilities (277)
Traffic (350)
Carrowrkusse D D.brokehuus
Dally life (201)
30
21
SQ2: What are all the texts imoge?
ANS: RPM 71 WATTS
sQ:
SQt What arc The
158
textss in the image?
16 + 15B
DMK br Faisol..
ANS W WTL2266 PALERMO
UG: Where is Data office?
What the licerse number? BPALERMO
4 8000
UGQ heart rate now?
What's
your
STYD4
TenvaA
Terribba
A.br-Faisd 8.2005
F3 170
A.167
0.71
(Gar)
C.M 0.2.17
<PLAYERNO WIT266
C158
D.17
Figure 4: (a) Examples of TextHalu- Bench.. (b) Comparison of non-semantic answers ratios between existing scene text benchmarks and TexcHalu-Bench. SQ, UQ, and ANS represent spotting, under- standing questions, and answers, respectively.
4 TextHalu-Bench
Previous scene-text benchmarks such ST-VQA [5] and TextVQA [31] have notable limitations: their test sets are dominated by semantically meaningful and visually clear samples, as shown in Fig. 4. This may overestimate the visual grounding ability of LMMs, models often rely language priors rather than true visual perception to answer correctly.
To address these limitations, we introduce TextHalu- a new benchmark comprising 1,740 carefully curated samples by select collected from diverse public datasets, including ICDAR 2013 [72]. ICDAR 2015 [25], ICDAR 2019 [73]. CCPD [S4).MSRR-TDSOO [75] RoaddText [76] and MPSC [77]. The curation process specifically targets instances containing non-semantic text elements, such as isolated numbers, incomplete words, and rare or out-of-vocabulary tokens.
Industry. Transportation, Public Facilities, and Daily Life-with balanced distribution and emphasis on visually challenging cases (e.g., occlusions, low-contrast text, unconventional fonts). It features two subtasks: Spotting, which requires models to extract text directly from images, and Understanding. which evaluates whether models can semantically ground the recognized text. The detailed data construction pipeline is provided in the Appendix C.
Our benchmark covers five representative scenario sategories-Busiene
5 Experiments
5.1 Experimental Setup
Baselines. To validate the effectiveness of our proposed semantic hallucination mitigation framework, we integrate into three contemporary open-source LMMs with diverse LLM backbones: Mini- Monkey [30]. Qwen2.5-VL [78], and LLaVA-NeXTs [37]. For all three models, we follow the default configurations provided in their official implementations to ensure fair comparison. In addition, we evaluate our method alongside 10 representative LMMs, including both open-source and proprictary models, across multiple public benchmarks.
Benchmarks. In addition to our proposed TextHalu-Bench, we evaluate our method on six public benchmarks encompassing scene text spotting and understanding. ST-VQA [5] and TexiVQA [31] focus on real-world images containing scene text, requiring models to understand and reason over both visual and textual information. A12D [79] centers on scientific diagrams, emphasizing structured reasoning and domain-specine knowledge. OCR-VQA [80] involves book covers and challenges models to incorporate OCR-derived content into question answering. SEED-Bench [81] offers broad suite of vision-language tasks; we evaluate its Text Understanding subset, which tests general VQA and grounding capabilities. Finally, GOT [24] contains 400 natural images with multilingual scene text; we its scene text subset and report character-level Fl scores.
7

---

Table Experimental results on TextHalu-Bende and mainstream scene text spotting and understand- ing benchmarks. We report the performance on STVQA and GOT by using their official weight.
Method
LLM
Textiaalu Bench
STVQA
ToaV
G0
OCRVQAcvez
SEEDBenchaw
AEDD
Proprictary Models
Geminil.S P.Proo [82]
43.2
61.6
18.5
79.1
GPT-4o (85)
45.3
71.0
18.7
70.2
85.9
Open
MILMSs
LLaVA1.5 [85]
Vicuna-7B
21.4
51.9
46.0
38.8
60.6
36.9
55.5
JPLUGG-Oi2 [84]
LLaMA-7B
24.3
49.8
56.4
29.8
65.2
32.1
55.7
Molmo-D (85]
Quen2:7B
24.7
62.3
67.5
42.3
15.9
71.4
81.0
PixtralB [86]
Neme- 12B
32.8
$2.9
64.3
35.4
64.7
47.6
79.0
Moakey (87)
Qwen-7B
LLaVA-OV [88]
Qwen2-7B
21.4
51.9
78.5
43.9
61.7
82.8
Ovis1.6 [89]
Llama-3.2
38.4
72.6
78.2
25.2
71.2
52.4
84.4
[ntem/LL2.5 [90]
12..7L.M2..-7B
LLaVA-NeXT (38)
Llama-3-SB
27.9
65.1
65.3
41,9
60.7
50.0
72.8
LLAVA-NeXT Ours
Liuma: 3-5B
28.5 (+0.6)
65.2 (+0.1)
65.5 (+0.2)
42,0 (+0.1)
61.5 (+0.8)
(+12()122)
(+0.2)
Monkey [91]
InkerLLM2-1
46.5
Monkayy Ours
M2 1SS
50.6 (+4.1)
70.6 (+3.9)
75.0 (+0.9)
89.2 (+0.4)
39,9 (+0.2)
$4.5 (+1.2)
74.7 (-0.1)
Qwen2.5 (L (78)
Qwen2.5:3B
48.3
67.3
79.1
85.2
70.2
66.7
78.1
Qwen2.5.VL. Ours
Qwen2.5:B
53.8 (+5.5)
67.6 (+0.3)
80.3 (x1.2)
86.0
70.5
70.2 (+355
78.3 (+0.)
Implementation Details. Our method is training-free and test-time adaptive plug-in module. In Text we set the number of top image tokens K to 128. In Grounded Layer Correction, we adopt the Fusion strategy and set the fusion factor w to 0.1. All experiments are conducted on single NVIDIA A800-80G GPU during inference. Importantly, our algorithm introduces no additional modules trainable parameters. Test-time efficiency analysis is provided in Appendix
5.2 Experiment results
We conduct extensive experiments on the seven benchmarks, as demonstrated in Tab. 1, in which we derive three primary conclusions.
Semantic hallucination remains significant challenge for existing LMMs. On our proposed TextHalu-Benn even the best-performing proprietary model, GPT-4o, achieves only 45.3 F1 score, while most open-source models perform considerably worse, far below human performance (96.8). This difficulty arises from two key aspects. First, compared to document-basee OCR tasks, scene text spotting and understanding are inherently more challenging due to the presence of complex visual distractors and highly diverse text styles. Second, non-semantic texts require accurate visual grounding rather than reliance on semantic priors, an area where many LMMs still suffer from severe hallucinations. These findings highlight the urgency of addressing semantic hallucination and underscore the importance of TextHalu-Bente which incorporates diverse non-semantic texts to robustly evaluate and analyze the hallucination behavior of LMMs.
Effectiveness of the proposed hallucination mitigation method. We integrate our method into three LMMs with different underlying LLM architectures. Mini-Monkey and Qwen2.5-VI. achieve 4.1% and 5.5 improvements in Fl score respectively, indicating that our method effectively helps models remain faithfully grounded on visual cues for scene text spotting and understanding. In contrast, LLaVA-Next shows only a marginal improvement of 0.6%, which we attribute to its limited OCR-related capabilities. These results suggest that method can bring greater benefits when applied to models with stronger scene text perception abilitics.
Generalization to other benchmarks. Beyond TextHalu-Benn our method demonstrates promis- ing results on range of public vasion-language benchmarks centered on scene text understanding and spotting. All baseline models show consistent improvements when integrated with our frame- work. Notably, Mini-Monkey achieves an accuracy gain approximately 4% on ST-VQA, while Qwen2.5-Ve VI. improves by around 3% on SEED-Bench. These results suggest that our hallucination mitigation approach serves as generalizable solution, effectively enhancing visual grounding without compromising the original recognition capabilities semantically valid samples.

---

Table 2: Comparison of different hallucination mitigation methods. "Adv." means adversarial training method, and "CoT" means Chain-of- Thought testing strategy.
Methods
TexHHal-Bench
STVQAu
TextVQAva
OCRVQAcuoa
SEEDBendhige
A12D
Baseline
46.5
66.7
74.1
39.7
83.3
74.8
Adv.
(+1.0)
66.8 (+0.1)
(-0.4)
89.1 (+0.3)
39.9 (+0.2)
(+0.0)
(-0.3)
CoT
46.8 (+0.3)
68.2 (+1.5)
75.2 (+1.1)
89.2 (+0.4)
39.7 (+0.0)
83.3 (+0.0)
74.9 (+0.1)
Ours
50.6 (+4.1)
70.6 (+3.9)
75.0 (+0.9)
89.2 (+0.4)
39,9 (+0.2)
84.5 (+1.2)
74.7 (-0.1)
Table 3: Ablations about the effectiveness of Zexm
Methods
TextHaln-Bemm
STVQAtea
TexVQQvi
GOT
OCRVQAcORE
SFHDRenchyau
A121)
Baseline
46,5
66.7
74.1
88.8
39.7
83,3
74,8
with text detector
50.4 (+3.9)
70.8 (+4.1)
(+1.1)
89.0 (+0.2)
39.9 (+0.2)
83.3 (+0.0)
74.7 (-0.1)
w/o Glimpse
50.2 (+3.7)
70.2 (43.5)
75.0 (+0.9)
88.7 (-0.1)
39.8 (40.1)
84.5 (+1.2)
74.8 (+0.0)
Refocus
(+3.3)
69.5 (+2.8)
74.9 (+0.8)
88.7 (-0.1)
39.7 (+0.0)
83.3 (+0.0)
(+0.0)
50.6 (+4.1)
70.6 (+3.9)
75.0 (+0.9)
89.2 (+0.4)
(+0.2)
84.5 (+1.2)
(-0.1)
5.3 Ablation Experiment
We conduct extensive ablation studies to evaluate the robustness and generalization capability of our proposed semantic hallucination mitigation method. Mini-Monkey is used as the primary baseline, and results on additional models are provided in Appendix F.
Comparison with other hallucination mitigation methods. As most existing hallucination mitiga- tion techniques, such as contrastive decoding and self-correcting decoding, are not directly applicable to our setting, we design two tailored baselines for comparison. (1) Truining-based adversarial training: Following [64], construct leading question-answer pairs augment the training set with adversarial examples, and retrain the LMM using this data. (2) Training-fre Chain-of-Thought (CoT) prompting: We apply CoT prompts guide the model first attend text regions before generating answers. Further implementation details are provided in the Appendix F. As shown in Tab. 2, adversarial training yields only marginal improvements on TextHalu-Benee While the CoT strategy enhances attention to text regions and improves performance on general scene text tasks, it fails to fundamentally address semantic hallucination.
Effectiveness of Zoom' Text. Our proposed Zoom module estimates potential scene text regions without relying external text detectors. We validate its effectiveness in ways. First, compare ZoomText with a baseline that incorporates accurate region proposals obtained from an off- the-shelf pretrained detector [92]. Second, we ablate ZoomText's two key components, Glimpse and ReFocus, to evaluate their individual contributions. As shown in Tab. 3, ZoomTTex achieves performance comparable to models equipped with external detectors, demonstrating its standalone effectiveness. Moreover, we observe that both Glimpse and ReFocus contribute significantly to performance, highlighting the importance of coarse-to-fin region localization.
Ablation on Grounded Layer Correction (GLC). To demonstrate the effectiveness of GLC, we first ablate the impact of our layer selection strategy. Specifically, randomly select a layer from the early, middle, and late stages of the LLM and apply the same Fusion strategy. As shown in Fig. 5, intermediate layers can indeed help mitigate hallucination. However, they may also overwrite valid semantic knowledge, as reflected by performance drops on general VQA benchmarks such as ST-VQA. In contrast, our method adaptively selects the layer with the strongest scene text grounding. leading to reduced hallucination on non-semantic samples while preserving the semantic integrity of meaningful ones. Furthermore, we evaluate the three correction strategies introduced in Sec. 3.3: Replacement, Selective Replacement, and Fusion. For fair comparison, all methods operate on the same grounded layer identified by our selection strategy. Naive Replacement performs poorly across all benchmarks. likely due to significant domain gap between training-time representations and directly injected hidden states. contrast, both Selective Replacement and Fusion effectively reduce hallucinations. However, similar to the trend observed in layer selection, Selective Replacement substantially degrades performance on general scene text understanding tasks. We attribute this to its aggressive overwriting of final-laycu hidden states, which may disrupt the learned alignment between

---

51
Towtine
Boodin
Bla Ropla
sec Radk
Selpp.
Eaily
Extty MiAI
TexHHlu
STVQA
Tocllslo-ltecch
STVQA
Figure 5: Ablation on the Grounded Layer Correction, (Left) Different layer selection method. (Right) Different correction strategy. "Base": Baseline; "Repla.": Replacement; "S-Repla.": Selective Replacement; "Fuse": Fusion.
g
User
Qwen2.5-VL.. 5-VL-38
Qwen2.5 Srren2.5-VL--. Our Method
Duchess
DRIVET HINU
Recognize all the the image:
Recagetze all image:
DEU.
TSCHLAND?
The hort the image "Duchess FRESHFOCD FDCO SERVES BREVETHRU
The teot the rmagais:
"The text ofter "OEUTSO-LAND in the imoge
sord under Guitmsss image 'Drought
"Ouchess FRESH Foob SERVED FAST DiLL
image
DEUTSCHLAND "BR"
The Innaan
1307020"
Figure 6: Visualization our proposed methods.
visual text and multimodal context. In light of these results, we adopt Fusion, weight-controlled integration, as our default strategy. Details on fusion weight selection are provided in Appendix
Visualization
We provide visualization results in Fig. 6, including synthetic images generated with [93] and challenging samples from our proposed TextHalu-Bench As shown, base models such as Qwen2.5- demonstraaee limited ability accurately spot texts without reasonable semantics, such sign names and numbers. Moreover, these models are prone to hallucinating semantic words (e.g.. "Berlin") that do not appear in the images. In contrast, our proposed method enables models to respond to questions grounded in the target regions of images, thereby improving the reliability of scene text spotting and understanding.
7 Conclusion
In this work, we identify the problem of semantic hallucination in Large Multimodal Models, where models often produce semantically plausible but visually incorrect answers when spotting and understanding scene text. We analyze its underlying causes and establish strong correlation between accurate intra-layer attention allocation and the reduction semantic hallucination. Building on this insight, we propose training- free hallucination mitigation framework comprising two key components. First, ZoomText adopts a coarse-to- seurse-to-fine fine strategy to estimate scene text regions without relying external detectors. Second, Grounded Layer Correction leverages the hidden states from the most visually grounded layer to guide the decoding process. Furthermore, we introduce TextHalu- Bench,, benchmark designed to robustly evaluate scene text spotting and understanding in the presence of non-semantic text. Extensive experiments demonstrate the effectiveness and generalizability of our approach across multiple LMMs and benchmarks.
10

---

Acknowledgements
This work was supported by the Fundamental Research Funds for the Central Universities (Grant No. 3262025T82) and partly supported by the EU Horizon projects ELIAS (No. 101 120237) and ELLIOT (No. 101214398).
References
[1] Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang. Abenet: Real-time scene text spotting with adaptive bezier-curve network. In CVPR, pages 9809-9818, 2020.
[2] Mingxin Huang, Yuliang Liu, Zhenghao Peng, Chongyu Liu, Dahua Lin, Shenggao Zhu, Nicholas Yuan, Kai Ding, and Lianwen Jin. Swintextspotter: Scene text spotting via better synergy between text detection and text recognition. In CVPR, pages 4593-4603, 2022.
[3] Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai. Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes. In ECCV, pages 67-83, 2018.
[4] Yan Shu, Wei Wang, Yu Zhou, Shaohui Liu, Aoting Zhang, Donghao Yang, and Weipinng Wang. Perceiving ambiguity and semantics without recognition: an efficient and effective ambiguous scene text detector. In ACM MM, pages 1851-1862, 2023.
[5] Ali Furkan Biten, Ruben Tito, Andres Mañla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, pages 4291 13001, 2019.
[6] Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, and Manmatha. Latr: Layout-aware transformer for scene-text vqa. In CVPR, pages 16548- 16558, 2022.
Gangyan Zeng, Yuan Zhang, Yu Zhou, and Xiaomeng Yang. Beyond ocr+ vqa: Involving into the flow for robust and accurate textvqa. In ACM MM, pages 376-385, 2021.
[8] Zhoufaran Yang, Yan Shu, Zhifei Yang, Yan Zhang, Yu Li, Keyang Lu, Gangyan Zeng. Shaohui Liu, Yu Zhou, and Nicu Sebe. Vidtext: Towards comprehensive evaluation for video text understanding. arXiv preprint arXiv:2505.228/00 2025.
[9] Siyang Qin, Alessandro Bissacco, Michalis Raptis, Yasuhisa Fujii, and Ying Xiao. Towards unconstrained end-to-end text spotting. In ICCV, pages 4704 4714, 2019.
[10] Wenhai Wang, Xuebo Liu, Xiaozhong Ji, Enze Xie, Ding Liang, ZhiBo Yang, Tong Lu, Chunhua Shen, and Ping Luo. Ae textspotter: Learning visual and linguistic representation for ambiguous text spotting. In ECCV, pages 457 473. Springer, 2020.
[1I] Chuhui Xue, Jiaxing Huang, Wenqing Zhang, Shijian Lu, Changhu Wang, and Song Bai. Contextual text block detection towards scene text understanding. In ECCV, pages 374-391. Springer, 2022.
[12] Min Liang, Jia-Wei Ma, Xiaobin Zhu, Jingyan Qin, and Xu-Cheng Yin. Layoutformer: Hierarchical text detection towards scene text understanding. In CVPR, pages 15665-15674, 2024.
[13] Shuai Bai. Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song. Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2XXi::250211392 2025.
[14] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185-24198, 2024.
[15] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:33392--3991111 2023.
[16] Jean- ManntBapisste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. NewrIPS, 35:23716 16-2373, 2022.
11

---

[17] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22 Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, pages 19730-19742. PMLR, 2023.
[18] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye. Ming Yan, Guohai Xu, Chenliang Li, Jun- feng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situtedd language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023,
[19] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, pages 26763-26773, 2024.
[20] Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403. 04473,, 2024.
[21] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, al. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499 2023.
[22] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong. Ming Ding, et al. Cogagent: A visual language model for gui agents. CVPR, pages 14281-14290, 2024.
[23] Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, and Xiangyang
Ji. -i: Understanding gui before following user instructions. arXiv preprint urXiv:2412009302 2024.
[24] Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, et al. General ocr theory: Towards ocr-2.0 via a unified end-to-end model. 2024.
[25] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In ICDAR, pages 1156-1160. IEEE, 2015.
[26] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(1))220000, 2024.
[27] Amit Ben-Artzy and Roy Schwartz. Attend first, consolidate later: On the importance of attention in different IIm layers. arXiv preprint arXiv:2409.03300 2024.
[28] Jingcheng Niu, Wenjie Lu, and Gerald Penn. Does bert rediscover a classical nlp pipeline? In Proceedings of the 29th International Conference on Computational Linguistics, pages 3143-3153, 2022.
[29] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.149) 2020.
[30] Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. Mini-monkey: Alleviating the semantic sawtooth effect for lightweight milms via complementary image pyramid. arXiv preprint arXiv:2408. 02034,, 2024.
[31] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317-8326,
2019.
[32] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0m 087744, 2023.
[33] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jeean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.118800 2023.
[34] Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprint arXiv: 24055 10739, 2024.
12

---

[35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892-34916, 2023.
[36] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296-26306, 2024.
[37] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[38] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabílities in the wild, May 2024.
[39] Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402. 1530, 2024.
[40] Haochen Xue, Feilong Tang, Ming Hu, Yexin Liu, Qidong Huang, Yulong Li, Chengzhi Liu, Zhongxing Xu, Chong Zhang, Chun-Mei Feng, et al. Mmre: A large-scale benchmark for understanding multimodal large language model real-world conversation. arXiv preprint arXiv:2502. 1903, 2025.
[41] Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409. 14485, 2024.
[42] Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Re- constructive token compression for extremely long video understanding. arXiv preprint arXiv:2503. 18848 2025.
[43] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406. 04264, 2024.
[44] Huaying Yuan, Zheng Liu, Minhao Qin, Hongjin Qian, Y Shu, Zhicheng Dou, and Ji-Rong Wen. Memory-enhance retrieval augmentation for long video understanding. arXiv preprint urXiv:2503.09149 2025.
[45] Qi Li, Runpeng Yu, and Xinchao Wang. Vid-sme: Membership inference attacks against large video understanding models, 2025.
[46] Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of- offhuuttt dataset for fine- grained video reasoning via core frame selection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26181-26191, 2025.
[47] Song Chen, Xinyu Guo, Yadong Li, Tao Zhang, Mingan Lin, Dongdong Kuang, Youwei Zhang, Lingfeng Ming, Fengyu Zhang, Yuran Wang, et al. Ocean-ocr: Towards general ocr application via vision-language model. arXiv preprint wrXiv:2501.15588 2025.
[48] Ya-Qi Yu, Minghui Liao, Jiwen Zhang, and Jihao Wu. Texthawk2: A large vision- language model excels in bilingual ocr and grounding with 16x fewer tokens. arXiv preprint 0rXiv:2410.0526.1 2024.
[49] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang. Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language model. In ECCV, pages 408-424. Springer, 2024.
[50] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer W Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, pages 4015-4026, 2023.
[51] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large lanion-language models. arXiv preprint arXiv:2305. 2023,
[52] Suzanne Petryk, David M Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, and Trevor Darrell. Aloha: A new measure for hallucination in captioning models. arXiv preprint M.kiv:2404.02900 2024.
[53] Yusu Qian, Haotian Zhang, Yinfei Yang, and The Gan. How easy is to fool your multimodal llm? empirical analysis deceptive prompts. arXiv preprint arXiv:2402.13220, 2024.
13

---

[54] Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. Phd: A prompted visual hallucination evaluation dataset. arXiv preprint arXiv:2403. 11111,111111111111111 2024.
[55] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models. arXiv preprint arXiv:23 10. 14566, 2023.
[56] Fuxiao Liu, Kevin Lin, Linjie Li. Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. ICLR, 2023.
[57] Chaoya Jiang, Wei Ye, Mengfan Dong, Hongrui Jia, Haiyang Xu, Ming Yan, Zhang, and Shikun Zhang. Hal-eval: A universal and fine-grat fineggrinned hallucination evaluation framework for large vision language models. arXiv preprint arXiv:2402.1572. 2024.
[58] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Yc, Bosheng Qin, Siliang Tang, Qi Tian, and Yucting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. arXiv preprint wrXiv:2311.13614 2023.
[59] Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang, and Ming Tang. Mitigating hallucination visual language models with visual supervision. arXiv preprint arXiv:2311.
2023.
[60] Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, and Nanyun Peng. Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models. arXiv preprint arXiv:2404. 13874, 2024.
[61] Tianyang Han, Qing Lian, Rui Pan, Renjie Pi, Jipeng Zhang, Shizhe Diao, Yong Lin, and Tong Zhang. The instinctive bias: Spurious images lead to hallucination in milms. arXiv preprint XXiv:2402.03355: 2024.
[62] Haz. Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, and Rifat Shahriyar. Illusionvqa: A challenging optical illusion dataset for vision language models. arXiv preprint arXiv:2403. 15952, 2024.
[63] Yuan Zhang, Fei Xiao, Tao Huang, Chun-Kai Fan, Hongyuan Dong, Jiawen Li, Jiacong Wang. Kuan Cheng. Shanghang Zhang, and Haoyuan Guo. Unveiling the tapestry of consistency in large vision-language models. arXiv preprint urXiv:2405. 14156, 2024.
[64] Yexin Liu, Zhengyang Liang, Yueze Wang, Muyang He, Jian Li, and Bo Zhao. Sceing clearly, answering incorrectly: A multimodal robustness benchmark for evaluating milms on leading questions. arXiv preprint arXiv: 10638, 2024.
[65] Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma. Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vasion-language models on natural adversarial samples. arXiv preprint arXiv:2410. 14669, 2024.
[66] Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhutdinov. Louis Phiil-P Morency, Katia Sycara, and Yaqi Xie. Self-correcting decoding with generative feedback for mitigating hallucinations in large vision-language models. arXiv preprint arXiv:2502.06.1 2025.
[67] Seil Kang, Jinycong Kim, Junhyeok Kim, and Seong Jac Hwang. See what you are told: Visual attention sink large multimodal models. arXiv preprint arXiv:2503.03321, 2025.
[68] Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Mllm can see? dynamic correction decoding for hallucination mitigation. arXiv preprint arXiv:2410.11779 2024.
[69] Shunqi Mao, Chaoyi Zhang, and Weidong Cai. Through the magnifying glass: Adaptive per- ception magnification for hallucination-free vlm decoding. arXiv preprini 10xiv:2503.10183. 2025.
[70] Sicong Leng, Hang Zhang. Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Li- dong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In CVPR, pages 13872-13882, 2024.
14

---

[71] Joost CF De Winter, Samuel D Gosling, and Jeff Potter. Comparing the pearson and spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychological methods, 21)322733, 2016.
[72] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In ICDAR, pages 1484-1493.
IEEE, 2013.
[73] Chee Kheng Chng. Yuliang Liu, Yipeng Sun, Chun Chet Ng. Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. lcdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In ICDAR, pages 1571-1576. IEFF, 2019.
[74] Zhenbo Xu, Wei Yang, Ajin Meng, Nanxue Lu, and Huan Huang. Towards end-to-end license plate detection and recognition: A large dataset and baseline. In ECCV, pages 255-271, 2018.
[75] Jiaming Liu, Chengquan Zhang. Yipeng Sun, Junyu Han, and Errui Ding. Detecting text the wild with deep character embedding network. ArXiv, abs/190/ 0ess/1901.00663, 2018.
[76] Sangeeth Reddy, Minesh Mathew, Lluís Gómez, Marçal Rusiñol, Dimosthenis Karatzas, and
C. V. Jawahar. Roadtext- Ik: Text detection & recognition dataset for driving videos. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 11074-11080,
2020.
[77] Tongkun Guan, Chaochen Gu, Changsheng Lu, Jingzheng Tu, Qi Feng, Kaijie Wu, and Xinping Guan. Industrial scene text detection with refined feature-attentive network. TCSVT, 2022.
[78] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint urXiv:2502. 13923, 2025.
[79] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, pages 235-251, Springer, 2016.
[80] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pages 947-952. IEEE, 2019.
[81] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge. Yixiao Ge. and Ying Shan. Seed- bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307. 1615,, 2023.
[82] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.00 240.00553, 2024. May 2024.
[83] OpenAL. Gpt-4o, https: /pwpaii.pnaaaa//////////
[84]
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In CVPR, pages 13040-13051, 2024.
[85] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, al. Molmo and pixmo: Open weights and open data for trate-of-the-art multimodal models. arXiv preprint arXiv:2409. 1tn.22499./1446 1716,, 2024.
[86] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073. 2024.
[87] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, pages 26763-26733, 2024.
[88] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li. Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0 03326, 2024.
[89] Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.2077 2024.
15

---

[90] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-souree source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv:24/2.05271, 2024.
[91] Mingxin Huang, Yuliang Liu, Dingkang Liang, lianwen Jin, and Xiang Bai. Mini-monkey: Alleviating the semantic sawtooth effect for lightweight milms via complementary image pyramid. arXiv preprint SSII:22408.020:44 2024.
[92] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2961-2969, 2017.
[93] Weichao Zeng, Yan Shu, Zhenhang Li, Dongbao Yang, and Yu Zhou. Textetrl: Diffusion- scene text editing with prior guidance control. Advances in Neural Information Processing Systems, 37:138569-133 1138896999-33944..-- 2024.
[94] Yan Shu, Weichao Zeng, Fangmin Zhao, Zeyu Chen, Zhenhang Li, Xiaomeng Yang, Yu Zhou, Paolo Rota, Xiang Bai, lianwen Jin, et al. Visual text processing: A comprehensive review and unified evaluation. arXiv preprint arXiv:2504.21662, 2025.
[95] Zhenhang Li, Yan Shu, Weichao Zeng, Dongbao Yang. and Yu Zhou. First creating back- grounds then rendering texts: A new paradigm for visual text blending. arXiv preprint arXiv:2410. 10168, 2024.
[96] Yan Shu, Bin Ren, Zhitong Xiong, Danda Pani Paudel, Luc Van Gool, Begum Demir, Nicu Sebe, and Paolo Rota. Earthmind: Towards multi gratulg and multi-sensor earth observation with large multimodal models. arXiv preprint arXiv:2506.01667, 2025.
[97] Sagar Soni, Akshay Dudhane, Hiyam Debary, Mustansar Fiaz, Muhammad Akhtar Munir, Muhammad Sohail Danish, Paolo Fraccaro, Campbell D Watson, Levente J Klein, Fahad Shah- baz Khan, et al. Earthdial: Turning multi-sensory earth observations to interactive dialogues. In CVPR, pages 14303-14313, 2025.
[98] Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning, arXiv preprint arXiv:2506.07044, 2025.
[99] Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large mudti-modality models. In ACM MM, pages 11198-1 1 1201, 2024.
[100] Peng Li, Wei Li, Zhengyan He, Xuguang Wang. Ying Cao, Jie Zhou, and Wei Xu. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering, 2016.
[101] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024.
[102] Chaoyou Fu, Peixian Chen, Yunhang Shen. Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[103] Yang Shi, Huanqian Wang, Wulin Xie, Huanyao Zhang, Lijie Zhao, Yi-Fan Zhang, Xinfeng Li, Chaoyou Fu, Zhuoer Wen, Wenting Liu, et al. Mme-videoocr: Evaluating bar-based capabilities of multimodal 1lms in video scenarios. arXiv preprint arXiv:2505.21333 2025.
[104] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
[105] Liu Yuliang, Jin Lianwen, Zhang Shuaitao, and Zhang Sheng. Detecting curve text in the wild: New dataset and new solution, 2017.
[106] Chee Kheng Ch' ng. Chee Seng Chan, and Chenglin Liu. Total-text: Towards orientation robustness in scene text detection. IJDAR, 23:31-52, 2020.
16

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Sec. B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[NA]
Justification:
All the data and code will be made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
onlolicy))))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.coo. wwtppblic/gguiiieecccceoooocccce gttee/gggee///eee CCodeSubmiieeeeece
hotooi/igguddeesodeeeubbiiiiiiii wwribbmmssiioooolllioioo (eionnllccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
We do not report error bars on the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / http///eurrpp..e/ee ctt.ccccpubbll hc/ppbiicc/tthiccillleennses
Answer:
[Yes]
Justification:
We follow the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Sec. B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the models we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
21

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
22

---

The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy lattps:/nneuris. /tearrpp..ccCCoofeeeeeeee. for what should or should not be described.
23

---

A Overview of Appendix
B: Limitations and Broader Impact.
C: Details of TexcHhalu-Bench.
D: Experimental Settings.
Detailed Experimental Results.
More Ablation Studies.
Checklist
B Limitations and Broader Impact
Limitations. While our method shows promising performance on scene text spotting and understand- ing, it still has two key limitations. First, it requires token selection and attention map computation during the prefilling stage, which introduces additional inference time and computational overhead. Second, the effectiveness of our method heavily relies on the underlying OCR perception ability of the base model. As result, it performs suboptimally when applied to LMMs with weak scene text understanding capabilities.
We propose a training-free semantic hallucination mitigation framework with broad impacts across multiple domains. For the OCR community, our method facilitates the adaptation of LMMs to text- intensive tasks, potentially benefiting downstream applications including document understanding, autonomous driving, assistive technologies, and low-level text processing techniques [94, 95] such as editing and generation. Beyond OCR, our findings and mitigation strategy provide valuable insights for developing more reliable and hallucination-resiliee framework is generalizable and can be extended to other domains where visual-semana alignment is critical, such as remote sensing image interpretation [96, 97] and medical image analysis [98], where hallucination mitigation is equally crucial for accurate and trustworthy predictions.
multimodal large models. Importantly this
C Details of TextHalu-Bench
Dataset Collection Process. To promote coverage and diversity, we carefully curated samples across five representative scenario types: Business, Industry, Transportation, Public Facilities, and Daily Life. These categories were selected based on their prevalence in real-world OCR applications and their variance in textual layout, typeface complexity, and visual background noise. In addition, during sample construction, we emphasized the inclusion of challenging edge cases, such as low-contrast text, occlusions, unconventional fonts, or partial visibility, stress-test the visual grounding ability of MLLMS and better surface hallucination tendencies.
Scene Text Spotting Task Definition. Given an image, the model is required to extract all visible textual content from the scene. The task is treated as word-level prediction problem and its output compared to the ground-truth words using case-insensitive exact match.Spotting examples include questions such as "What the texts in the Answerr the question in only words you recognize."
Scene Text Understanding Task Definition. To measure the higher-level comprehension ability, we adopt a multiple-choic format, with at least one correct answer and at most three well-crafted distractors. These distractors are designed with the following strategies:
Glyph-based distractors: visually similar characters (e.g.. '0 vs. '0', '1' vs. '1')
Semantic distractors: misleading but contextually related words (e.g., "apple" vs. "apole")
Context-based distractors: co-occurring spatially nearby words within the same image
Understanding task examples include questions such as "What is the texts on the boat? A.aa B.bb C.ce D.dd
Metric. To quantitatively measure hallucination behavior, we report the average score across both subtasks as our evaluation metric which captures both the model's accuracy in extracting visible text and its ability to semantically interpret visual information, distinguishing genuine visual understanding from language-pric hallucination.
24

---

Spotting
1084600510
3110403
ABV-760
Recognize the Reply in words:
Recognize the image.Reply in words:
Recognize all the text image.Reply in words:
Honda 6A03 TD411
ABV760 EL SANTUARIO
A1664600510 3110403
Understanding
g
What is the text visible the commercial kitchen equipment?
What's 's the first text the penultimate line?
What was the book purple? A.1958 8.1956 C.1955 D.1957
A.HOBAR CHOBERT
HOBER D.HOBERI
CAPP23-CCM C.07-11-27
8.00
D.PSA
8,1956
A.HOBAR
A.CAPP23-CEM
Figure 7: Visualization of TextHalu-Bende
Table 4: Spearman Correlation between Hallucination tendency score and scene text region attention score, with performance on STVQA and VeA.VQA.
Benchmarks STVQA
Model
OCRBench
TextVQa
Mini-Monkey
-0.72
-0.78
-0.74
Qwen2.5-VL.
-0.68
-0.80
-0.76
Visualizations. We provide some qualitative cases of our benchmark in Fig. 7.
D Experimental Settings
Our method is training-free and thus does not require any additional fine-tuning or parameter updates. All models are evaluated under their official default configurations without modification. For evaluation, we test on TextHalu-Bench, ST-VQA and GOT using our own implementation to ensure consistent handling of OCR and visual inputs. For other benchmarks, we utilize the VLMEvallit [99] toolkit, and follow the original leaderboard results published by each model for fair comparison. All experiments are conducted server equipped with 4 x NVIDIA A800 GPUs.
25

---

Spearman p -0.89 Pearson -0.89
Data Points
Scene Text Atterttin Score
0.6
0.4
0.2
0.2
0.4
0.6
0.8
Hallucination Tendency Score
Figure 8: An example of the correlation (Spearman and Pearson coefficient) between hallucination tendency score and scene text region attention score.
Table 5: Generalization performance of our method on other domains.
Method
SREDBench RealWorkIQA
MathVista
POPE
MME-P
MMF-VideeOCK
A-OKVQA
Qwen2.5VL
74
65.5
61.2
85.9
1567.7
59.2
85.2
Qwen2.5VL.
65.8
86.7
1572.2
85.6
E
More Experimental Results
Correlation between Hallucination and Attention Distribution in LLMs. Leveraging our auto- matic hallucinated token identification mechanism, we compute hallucination tendency scores and corresponding scene text region attention scores across all transformer layers for each hallucinated sample. As shown in Tab. 4, Spearman correlation analysis reveals a strong negative correlation, indicating that stronger attention to scene text regions associated with reduced semantic halluci- nation. A layer-wisc visualization for representative sample is shown in Fig. 8, where each point corresponds to a transformer layer with its hallucination score (x-axis) and scene text attention score (y-axis).
Generalization to other domains. To assess the generalization ability of our method beyond the scene text domain, we apply it to four diverse vision-language benchmarks: SEED-Bench consists of 19K multiple-choice questions with accurate human annotations, covering 12 evaluation dimensions across both image and video modalities; Real [100] evaluates real-world spatial understanding in physical environments, contributed by XAI; Math Vista [101] is challenging benchmark requiring visual mathematical reasoning over charts, diagrams, and textual math problems; POPE [51] focuses on object hallucination, comprising three evaluation tracks: random, popular, and adversarial hallucination. MME [102] is large-scale comprehensive multimodal benchmarks toward the perception and reasoning ability LMMs. MME- VideoOR [103] focuses on the multi level ability the video text understanding. A-OKVQA [104] is challenging benchmark that requires commonsense and world knowledge to answer.
As shown in Tab. 5, our method consistently improves performance across all benchmarks for instance, achieving +0.3 accuracy gain on RorlWAA and +0.8 on POPE. These results suggest that our approach not only enhances scene text understanding but also generalizes well to broader multimodal reasoning tasks, without degrading the pretrained models' core alignment or reasoning abilities.
26

---

Table Efficiency analysis of our methods, which use the same prompt to calculate the first token generation time.
Method
prefilling
decoding
total
Qwen2.5VI.
0.53
1.14
1.67
Qwen2.5VL Ours
1.08
1.15
2.23
Qwen2.5VL CoT
0.56
3.44
4.00
Table 7: Ablation about the effectiveness of ZoomText on Qwen2.5-VL-3B..
Methods
TextHalu Bencc
STVQATe
TexiVQAu
AI2D
OCRVQACORE
SEEDBenchya
GOTscne
Baseline
48.3
67.3
79.1
78.1
70.2
66.7
85.2
with text detector
53.4
67.9
80.3
78.2
70.4
67.9
85.8
w/o Glimmse
52.9
67.3
78.8
78.2
70.2
85.2
w/o Refocus
53.5
67.3
78.9
78.2
69.8
70.2
85.1
Ours
$3.8
67.6
80.3
78.2
70.5
70.2
86.0
Efficiency Analysis. As training-free free method, we report the inference time overhead introduced by our approach. As shown in Tab. 6, our method inevitably incurs additional computation in the prefilling stage, where attention maps from all layers extracted and stored before decoding. However, we argue that this overhead acceptable, as our approach remains more efficient than other test-time scaling methods such as Chain-of-Though prompting (introduced in Sec. F). Furthermore, our method does not require any additional modules or external models to assist decoding, maintaining streamlined and lightweight inference process.
F More Ablation Studies
Additional Implementation Details: Comparison with Other Hallucination Mitigation Methods.
(1) Adversarial Training. To construct adversarial training data, we employ the image-text editing tool TextCtrl [93] to synthetically perturb the textual content of images from existing scene text datasets, including C7W1500 [105], ICDAR 2015 [25], and TotalText [106]. Following a targeted editing strategy, we generate up to three adversarial variants per image, depending on the number of text instances it contains, as shown in Fig. 9. Editing operations include character-level insertions, deletions, substitutions, and replacements with visually similar but semantically misleading characters. This process yields approximately 10,000 adversarial image-text pairs designed to introduce non- semantic visual perturbations that challenge both grounding and recognition. We fine-tune both Mini- Monkey and 2wwe22.5-V/ 5-VI. on the augmented dataset, using the original fine-tuning hyperparameters and training for one epoch. The fine-tuned models are then directly evaluated on downstream benchmarks to assess their robustness against semantic hallucination.
(2) Chain-of- Thhhughht (CoT) Prompting. Our CoT-based hallucination mitigation strategy follows two-stage inference process. In the first stage, the model generates an initial answer using standard inference procedures. In the second stage, feed the model with both the original image and its previously generated answer, along with CoT-style prompt that explicitly instructs the model to reflect on and verify its initial prediction by more carefully grounding it in the visual text regions. We design the following Chain-of Thought (CoT) prompt to guide the second-star stage reasoning:
" "Your previous answer was: '[[answer]]' Please carefully examine the text in the image again and verify whether the answer is fully supported by the visual evidence. If necessary, correct the answer based the actual content in the image."
Therefore, further demonstrated in Tab. 8, Qwen2.5-VV 5LV also exhibits consistent improvements, providing additional evidence of the effectiveness our approach.
Additional Effectiveness Results of Zoom'Tes As shown in Tab. 7, the experimental results further validate the effectiveness of Zoom In particular, incorporating both the Glimpse and ReFocus modules leads to notable performance gains, demonstrating the benefit of our progressive refinement
27

---

Table 8: Comparison of different hallucination mitigation methods on DL-
Methods
TexHHalu-Bench
STVQAn
TextQQAva
G0T
OCRVQAcoRE
SEEDBeachue
A12D
Baseline
48.3
67.3
79.1
78.1
70.2
66.7
85.2
Adv.
49.1
67.2
78.6
78.1
70.4
67.9
85.2
CoT
48.5
67.7
79.4
78.2
70.4
70.2
85.5
Ours
53.8
67.6
80.3
78.2
70.5
70.2
86.0
Smasked BURGERS
Smashedu BURGERS
MARKET
MARKET
CRUST TRUOT
CRUST
Original image
Adversarial image
Original image
Adversarial image
No edition
Smashed -> Smashedu
No edition
CITY mCITY
Figure 9: Visualization of adversarial training data.
strategy for region localization. By incrementally narrowing the model's attention to relevant visual areas, ZoomText effectively reduces ambiguity in visual-language alignment. These findings highlight the importance of hierarchical attention in improving scene text spotting and understanding.
Our proposed ZoomTTxx module is based attention can effectively highlight relevant visual regions such as signs or posters, and (2) that background non-semantic tokens exhibit relatively stable attention dynamics across layers. To empirically validate these assumptions, we conducted the following analyses:
assumptions: (1) that query-to-image
First, to assess the effectiveness of ZoomText in capturing relevant text regions, we manually annotated 100 samples across TextHalu-Bench, TexiVQA, and ST-VQA, with quadrilateral bounding We then performed IoU-based ablation study Qwen2.5- VL-3B, comparing three variants: (a) a baseline selecting top-k tokens using final-layer attention, (b) baseline Glimpse (which uses query- query-to-image attention, Eq. 4), and (c) baseline Glimpse + Refocus (which incorporates attention variation dynamics, Eq. 5). As shown in Tab. 9, results show consistent improvement in IoU scores, confirming the benefit of each step in refining the focus on relevant visual regions.
boxes marking the target text
Second, to validate the Refocus assumption, computed the coefficient of variation (CV) of attention scores across layers for each visual token. Tokens were categorized into: (i) foreground (within bounding boxes), and (ii) background (outside boxes but with high attention). We compute the CV for each token i, defined as:
CV, Std .g...a)).....a Mean 0 ,2,,..,07 af,)
where af is the attention of token i layer L. The analysis reveals that foreground tokens exhibit significantly higher attention variation across layers (higher CV), while background tokens remain relatively stable (lower CV), supporting our hypothesis that "sink tokens" can be identified through their consistent attention profiles.
Analysis of Weighting Strategies for Cross-Layer Hidden State Fusion. To mitigate the limitations of relying solely on the final output layer for text recognition and understanding, we adopt a weighted fusion strategy that combines hidden states from different transformer layers, modulated by fusion coefficient X. We perform a grid search over À E {0.1, 0.2, 0.4, 0.6, 0.8) to investigate its effect on performance. As shown in Tab. 10, the optimal performance is achieved at À = 0.1, resulting in average accuracy improvement 1.67% over the baseline.
Moreover, the results reveal nuanced trade-off: when the selected hidden layer carries richer visual information, higher values of À (e.g., À 0.6 or 0.8) tend improve performance on text spotting tasks. However, these higher weights lead to diminished performance in text understanding tasks,
28

---

Table 9: Ablation studies Zoom'Txt (Left) and attention variation analysis (Right).
Method
Mini-Monkey
Qwen2.5-VL
Token Type
Mini-Monkey
Qwen2.5-VL.
Baseline
42.3
46.1
Foreground Background
2.76 0.00054
2.86
Glimpse
45.9
48.9
0.00051
Refocus
47.8
52.7
Table 10: Analysis of weights for cross-layer hidden state fusion Qwen2.5VL-3B.
Weights
TextHalu-Bende
STVQAtes
TextVQA
A12D
OCRVQAcuRE
SEEDBenchyet
GOT
48.3
67.3
79.1
78.1
70.2
66.7
85,2
53.8
67.6
80.3
78.2
70.5
70.2
86.0
0.2
53.4
66.7
77.7
78.3
67.9
69
86.2
50.7
73.2
72.6
0.6
45
58.2
65.5
80.3
56.9
76.2
85.4
0.8
27.2
42.5
45.7
80.1
43
69
35.8
Table 11: Analysis of layers for hidden state fusion Qwen2.5VL-333
Layer Index
TextVQAvi STVQAnes
A12D
OCRVQAcORE
SEEDBenchres
GOT
0-10
52.5
67.3
78.6
78.2
69.7
86.1
10-20
52.9
67.4
78.9
78.2
70
70.2
85.4
20-35
53.1
67
79
78.1
70.4
69
85
random
53.1
67.1
78.9
78.2
69.9
70.2
85.4
53.8
67.6
80.3
78.2
70.5
70.2
86.0
likely due to an overemphasis on visual features at the expense of semantic comprchension. This finding underscores the importance of carefully balancing visual and linguistic cues in cross-layer fusion for different types of downstream tasks.
Analysis of Hidden Layer Contributions to Fusion. To better understand which layers are most beneficial for visual grounding. we conduct an ablation study examining the effects of fusing hidden states from different model depths. Specifically,
compare five fusion strategies: (a) early layers (layers 0-10), (b) middle layers (layers 10-20), (c) late layers (layers 20-35). (d) randomly selected layers, and (e) our proposed layer selection method. As shown in Tab. 11, fusing hidden states from our selected layers yields the most substantial performance gain. These results indicate that not all layers contribute equally to grounding, and that carefully chosen subset can more effectively capture the visual information necessary for ballucination mitigation.
29


Reviews:
[{"review_id": "IB0cQw5oKf", "reviewer": "Reviewer_LbB3", "summary": "This paper presents “semantic hallucination”, a hallucination problem in MLLMs where the target task answer that appears in the input image is not semantically relevant (e.g., OCR on a text that has a typo), causing MLLMs to hallucinate a semantically relevant answer based on its strong language prior in their pre-trained LLMs. The author(s) presented an investigation of the underlying causes of this issue through cross-attention visualization and layer-wise token outputs likelihoods, and presents a mitigation framework that automatically augments hidden states within the LLM to reduce semantic hallucination. In addition, the author(s) also introduce TextHalu-Bench, a new benchmark targeting this particular issue, and show that the proposed framework is both effective in reducing hallucination and improving scene-text understanding tasks.", "questions": "Based on the above weaknesses, I have the following questions:\n\n- Can the author(s) report the proposed method’s performance on knowledge-based, non-scene-text benchmarks, and compare them against baselines, such as OK-VQA [1]?\n- Can the author(s) investigate the effectiveness of ZoomText in extracting areas of interest in the image?\n- How frequently does Grounded Layer Correction change the proposed method?\n\nI am currently on the fence about this paper. However, if the author(s) can conduct a more thorough exploration of the capabilities and limitations of the proposed method, I will be willing to raise my score.\n\nReference:\n[1] https://okvqa.allenai.org/", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n**Presentation of an important hallucination issue and contribution of the associated benchmark**: As MLLMs are increasingly used to conduct text-rich tasks, hallucinating texts that are semantic but misaligned with the image is an important problem that can cause performance degradation and user frustration. Hence, the presented issue is important, and the associated benchmark contributed by the author(s) will be very useful for future research.\n\n**Proposed method is effective in mitigating hallucination and improving scene-text-based tasks**: The proposed method is effective in targeting the presented semantic hallucination problem, and also improves some benchmarks focusing on scene-text tasks.\n\n**Novel technical ideas in proposed method are potentially generalizable**: There are some novel ideas in the Glimpse-Refocus strategy that could be highly useful in not only mitigating semantic hallucination, but grounding useful regions/tokens in the problem in general.\n\n\n**Weaknesses**:\n\n**Unclear Technical Soundness of the Exploration**: It is unclear to me whether computing probability from intermediate states and the output weights is technically sound, given the output projection layer was trained only to handle the output of the final LLM layer. It appears that doing so will only isolate layers where clear concepts of output tokens have not yet been formed, instead of actually investigating the “hallucination source”.\n\n**Impact on other capabilities**: Similar to the above point, as the output projection layer is only trained to handle the output from the final layer, it is unclear to me how the proposed method of “Grounded Layer Correction” would affect the models’ capabilities, especially on knowledge-based capabilities that rely more on text-based reasoning. In my opinion, the proposed method provides shortcuts for more localized, lower-level detail knowledge to be considered directly by the final layers, which would potentially degrade higher-level, knowledge-based capabilities. It would be great if the author(s) can include more general benchmarks outside of scene-text tasks, which would be important in cases where the LLM needs to handle not only scene-text-based tasks but also general tasks (hence cannot determine ahead of time whether the proposed method should be applied).\n\n**Limited evaluation of the effectiveness of individual components of the framework** While the proposed method is evaluated end-to-end and has shown some improvements in downstream benchmarks, it would be helpful to understand the performance of individual components in the framework. In particular, the effectiveness of ZoomText can be measured with datasets where the ground-truth areas of interest are known. Similarly, the author(s) can also report the frequency where Grounded Layer Correction changes the output prediction. Reporting these metrics can significantly improve our understanding of the proposed method.", "was_revised": true, "final_justification": "All my concerns were addressed by the author(s) in the rebuttal, and thus my score was raised.", "rebuttal": "# To reviewer LbB3\n\nWe thank the reviewer for the detailed and encouraging feedback. We're pleased that you found our identification of the semantic hallucination issue in multi-modal large language models (MLLMs) to be both timely and significant, especially in light of the increasing use of MLLMs for text-rich tasks. We appreciate your recognition of the benchmark we contributed as a valuable resource for future research. We're also encouraged by your positive assessment of our method’s effectiveness in addressing semantic hallucination and enhancing performance on scene-text-focused benchmarks. Furthermore, we are grateful for your appreciation of the novel ideas behind our Glimpse-Refocus strategy and your insight that these ideas may generalize to broader applications involving grounded reasoning and alignment in vision-language tasks.\n\nBelow, we respond to each Weakness (W) or Question (Q) with a corresponding Answer (A):\n\n## [W1/Q1]: Unclear Technical Soundness of the Exploration\n\n[A1]: Thank you for your insightful question. First, we would like to clarify that the hidden states from intermediate layers are **not merely underdeveloped representations.** In fact, prior work has shown that intermediate layers in large language models can offer better representations for downstream tasks. For example, [1] concludes that intermediate layers provide stronger embeddings for natural language understanding tasks (see Sec. 4.1). This insight motivates our exploration in MLLMs.\n\nBuilding on this hypothesis, our experiments (Table 1) demonstrate that incorporating intermediate representations significantly improves grounding and reduces hallucination.\n\nThat said, we acknowledge the potential mismatch between the intermediate hidden states and the output projection layer, since the model was only trained with final-layer outputs. To address this, we conduct an ablation study (Table 9), which shows that **assigning large weights** to intermediate layers can hurt performance, likely due to semantic disruption. However, **a moderate fusion weight preserves the semantic capacity of the final layer while benefiting from improved visual grounding** in intermediate layers.\n\n\n## [W2/Q2]: Impact on other capabilities.\n\n[A2]: Thank you for your thoughtful comment. We would like to clarify that our proposed method, Grounded Layer Correction (GLC), is specifically designed to mitigate hallucination in scene text spotting and understanding by injecting localized visual grounding signals. However, this design does not compromise the model’s **general reasoning or knowledge-based capabilities.**\n\nAs shown in Table 5, we evaluated our method on several general-purpose benchmarks beyond scene text. Still, we observed **minor improvements or comparable performance,** demonstrating that our method preserves the model’s broader abilities.\n\nTo further validate this, we conducted **additional ablation studies on A-OKVQA [2],** a challenging benchmark that requires commonsense and world knowledge. Results show that applying our method **does not degrade the model’s performance,** confirming that GLC does not interfere with higher-level semantic reasoning.\n\nWe do agree that our method does not significantly boost performance on knowledge-intensive tasks, primarily because it is **training-free** and does not introduce new external knowledge. In future work, we plan to explore combining hallucination mitigation with external knowledge grounding to further enhance general reasoning while preserving robustness in scene text understanding.\n\n\n| Model                 | A-OKVQA |\n|-----------------------|---------|\n| Qwen2.5-VL            | 85.2    |\n| Qwen2.5-VL + Ours     | 85.6    |\n| Mini-Monkey           | 77.7    |\n| Mini-Monkey + Ours    | 77.9    |\n\n\n## [W3/Q3]: Limited evaluation of the effectiveness of individual components of the framework\n\n[A3]: Thank you for the valuable suggestion. We agree that evaluating the effectiveness of each individual component would provide deeper insight into our framework.\n\n**ZoomText Evaluation:**\n\nTo assess the localization accuracy of ZoomText, we manually annotated 100 samples across TextHalu-Bench, TextVQA, and ST-VQA. Each sample was labeled with quadrilateral bounding boxes indicating the ground-truth text regions of interest. Since, to our knowledge, there is no existing benchmark that simultaneously offers both VQA annotations and dense spatial annotations, this manual annotation was necessary.\n\nWe compare the following configurations on Qwen2.5-VL to measure the impact of each component in ZoomText:\n\n(1) Baseline: Selects top-k tokens based on visual attention from the final LLM layer.\n\n(2) + Glimpse: Incorporates query-to-image attention (Eq. 4) for token selection.\n\n(3) + Refocus: Further filters irrelevant tokens based on relative attention dynamics (Eq. 5).\n\n\nWe compute the average IoU between the predicted tokens and ground-truth bounding boxes. The results show that ZoomText significantly improves text region localization, demonstrating its effectiveness in focusing on relevant visual regions.\n\n\n**Grounded Layer Correction (GLC)**:\n\nAs for GLC, it is applied uniformly across all samples during decoding—there is **no conditional switching**. Therefore, we cannot meaningfully report a frequency of invocation. However, its effectiveness has been validated through extensive experiments showing consistent hallucination mitigation and semantic preservation (e.g., Table 1 and Table 5), confirming its utility as a general decoding enhancement.\n\nWe will include these additional analyses and experimental results in the revised version of the paper.\n\n\n\n| Method      | Mini-Monkey | Qwen2.5-VL |\n|-------------|-------------|------------|\n| Baseline    | 42.3        | 46.1       |\n| + Glimpse   | 45.9        | 48.9       |\n| + Refocus   | 47.8        | 52.7       |\n\n\n\n[1] Does Representation Matter? Exploring Intermediate Layers in Large Language Models [NeurIPS 2024 workshop]\n\n[2] A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge [ECCV 2022]"}, {"review_id": "70CDVfMGQ4", "reviewer": "Reviewer_1eg9", "summary": "This paper tackles a problem in Large Multimodal Models (LMMs) called \"semantic hallucination.\" This is when a model reads text based on what makes sense semantically, rather than what is actually in the image. For instance, it might see \"MMOTEL\" but incorrectly output \"MOTEL\" because that's a real word. The authors discovered a key insight: within the LMM, some internal layers are much better at focusing on the actual text regions and are therefore less likely to hallucinate. Based on this, they propose a training-free framework to fix the problem: First, it finds the text regions in the image using a coarse-to-fine approach, without needing an external tool. Then, Grounded Layer Correction identifies the \"best-grounded\" internal layer (the one focusing most on the text) and uses its information to correct the model's final output. To test their method, they also created a new benchmark called TextHalu-Bench. Experiments show their approach effectively reduces these hallucinations and improves the model's accuracy on scene text understanding tasks.", "questions": "* How is the arg max operation performed in line 190: l* = arg max_l A_l. Is it that the attention map for a layer, A_l, is a matrix? It would be great to provide more details about this.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n* This work investigates the significant issue of 'semantic hallucination,' a phenomenon where Vision-Language Models exhibit a propensity to rely on their internal semantic priors to generate textual output, rather than faithfully grounding their responses in the visual evidence. The formalization and subsequent mitigation of this problem represent a valuable contribution toward enhancing the reliability of LMMs for real-world, text-centric applications.\n* The creation of the TextHalu-Bench benchmark will be a valuable resource for future research in this area.\n\nWeakness\n* The methodology described for Equation (2) seems to contradict the fundamental principles of auto-regressive models. In line 132, The paper states \"feed both the hallucinated token y_hal and the ground-truth token y_gt into the LMM's output head.\" This means that it feeds both y_hal and y_gt into the output head to produce their respective probabilities. How is this reconciled with the fact that an auto-regressive model's hidden state at a given step is determined only by its previous inputs, not by the token it is about to predict?\n* While OCRBench is used for analysis in Figure 1 and mentioned in Section 3.2, its results are conspicuously absent from the main experimental results in Table 1 (Section 5.2). Could the authors clarify why this benchmark was excluded from the final evaluation and it would be helpful to include it in the evaluation.", "was_revised": true, "final_justification": "Thanks to the authors for the response. It mostly addressed my concern and I will raise my score.", "rebuttal": "# To reviewer 1eg9\n\nWe thank the reviewer for the detailed and encouraging feedback. We're glad that you recognized the significance of the semantic hallucination issue, particularly the tendency of Vision-Language Models to rely on internal semantic priors rather than grounding their outputs in visual evidence. We appreciate your acknowledgment of our formalization and mitigation of this problem as an important step toward improving the reliability of LMMs in real-world, text-centric applications. We're also pleased that you found the introduction of the TextHalu-Bench benchmark to be a valuable resource for future research in this area.\n\nBelow, we respond to each Weakness (W) or Question (Q) with a corresponding Answer (A):\n\n\n\n## [W1]: The concern of the hallucination score calculation.\n\n[A1]:  Thank you for your thoughtful comment. We agree that the wording in Line 132 may have caused some confusion. Specifically, the phrase “feed both the hallucinated token $y\\_{\\\\text{hal}}$ and the ground-truth token $y\\_{\\\\text{gt}}$ into the LMM's output head” was not precise.\n\nWhat we intended to convey is that, at each decoding step $t$, the model computes a probability distribution over the entire vocabulary based on the prefix $x\\_{\\\\leq t}$. From this distribution, we **extract (not feed) the probabilities assigned to both $y\\_{\\\\text{hal}}$ and $y\\_{\\\\text{gt}}$,** as they are among the candidate tokens.\n\n\nThis process is fully consistent with the autoregressive property of LLMs: the model’s hidden state is determined solely by the preceding tokens, and the comparison between $y\\_{\\\\text{hal}}$ and $y\\_{\\\\text{gt}}$ is made at the output level, not by modifying the input sequence or hidden states.\n\n\nWe will revise the original sentence to clarify this distinction and appreciate the opportunity to improve the presentation.\n\n\n\n\n## [W2]:  The absence of OCRBench in the main experiment.\n\n[A2]: Thank you for the suggestion. OCRBench was not included in the main results table because it primarily **evaluates text recognition from cropped text regions**, where scene text understanding plays a limited role. In contrast, our method is designed for open-world scene text spotting and understanding.\n\n\n\nNonetheless, to provide a more comprehensive evaluation, we have conducted additional experiments on OCRBench. The results show **consistent improvements across multiple LMMs**, especially in scene-text-related VQA subtasks, confirming the generalization ability of our method. We will include these results in the revised version of the paper.\n\n\n\n\n| Model                 | Text Recognition | Scene Text-centric VQA | Doc-oriented VQA | Key Info Extraction | Handwritten Math Exp. | Final Score |\n|-----------------------|------------------|--------------------------|-------------------|----------------------|------------------------|--------------|\n| LLaVA-NeXT            | 191              | 156                      | 84                | 100                  | 0                      | 531          |\n| LLaVA-NeXT + Ours     | 188              | 159                      | 85                | 105                  | 0                      | 537          |\n| Qwen2.5-VL            | 249              | 172                      | 162               | 180                  | 60                     | 823          |\n| Qwen2.5-VL + Ours     | 250              | 177                      | 164               | 181                  | 60                     | 832          |\n| Mini-Monkey           | 250              | 168                      | 141               | 168                  | 71                     | 798          |\n| Mini-Monkey + Ours    | 250              | 175                      | 139               | 169                  | 71                     | 804          |\n\n\n## [Q1]:  The confusion about how to select the best layer.\n\n[A1]: Thanks for your question. As defined in Equation (3), $A\\_\\\\ell$ is a scalar attention score that quantifies how much the $\\\\ell$-th layer attends to the scene text regions.\n\nWe compute $A\\_\\\\ell$ for each layer and collect them into a 1D array $\\\\{A\\_1, A\\_2, \\\\dots, A\\_L\\\\}$. We then apply the $\\\\arg\\\\max$ operation:\n\n\n$$\n\\ell^\\star = \\arg\\max_\\ell A_\\ell\n$$\n\n\nto select the layer that focuses most strongly on the text regions."}, {"review_id": "zSswsmPBIC", "reviewer": "Reviewer_6wHJ", "summary": "This paper identifies and addresses a specific failure mode in Large Multimodal Models (LMMs) which the authors term “semantic hallucination.” This phenomenon occurs when LMMs, confronted with visually ambiguous or non-semantic scene text, generate semantically plausible but visually incorrect answers by relying on language priors instead of visual evidence. The authors first conduct an analysis and find a strong negative correlation between a Transformer layer’s tendency to hallucinate and the amount of attention it allocates to the ground-truth text regions.\n\nBased on this insight, they propose a novel, training-free framework to mitigate this issue. The framework consists of two main components: (1) ZoomText, a detector-free, coarse-to-fine strategy to localize potential text regions within an image, and (2) Grounded Layer Correction (GLC), a mechanism that adaptively selects the most visually-grounded layer and fuses its representations into the final decoding step to correct potential hallucinations.", "questions": "- Regarding the claim of a potential causal link between attention and hallucination, could the authors elaborate on this? Is it possible that a layer’s strong attention to a text region is a consequence of it correctly identifying the tokens, rather than the cause of the correct identification? How might one design an experiment to further probe this causal relationship?\n- The fusion weight w in GLC is set to 0.1, and the top-K in ZoomText is 128. How sensitive is the performance to the choice of K? Does the optimal K vary significantly across different models or image resolutions? A brief discussion on the robustness or a principled way to set these parameters would be beneficial.\n- The method shows impressive gains, but what are the primary remaining failure modes? Does the framework sometimes “over-correct” and harm the recognition of highly semantic or idiomatic text? Or does it primarily fail on extremely distorted or occluded non-semantic text where even the best internal layer struggles? Providing a few qualitative examples of failure cases in the appendix would offer a more complete picture of the method’s boundaries.\n- The efficiency analysis in Table 6 transparently shows the inference overhead during the prefilling stage. Could you contextualize this for a real-world application? For instance, how does this overhead compare to the overall latency of generating a long-form answer? Is the prefilling stage a one-time cost per query, regardless of the generated answer length? Clarifying this could help readers assess the practical trade-offs.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n- The paper tackles a timely and significant problem. The concept of “semantic hallucination” in the context of scene text is a novel and precise framing of a critical weakness in current LMMs.\n- The proposed solution is both clever and well-motivated by the initial analysis. The core finding that links hallucination tendency to layer-wise attention provides a solid, empirical foundation for the entire framework.\n- The creation of TextHalu-Bench is a significant contribution in itself. It provides the community with a much-needed tool to measure a specific, challenging aspect of LMM performance that is overlooked by existing benchmarks. Figure 4 clearly justifies its necessity.\n\n### **Weaknesses**\n- The paper states that its analysis suggests “a causal relationship between accurate attention allocation and the mitigation of semantic hallucination.” While the negative correlation is strong and provides a compelling motivation, **claiming causality might be an overstatement**. It is also plausible that both poor attention allocation and hallucination are symptoms of a common underlying cause (e.g., the model’s failure to recognize the region as containing text from the outset). A more tempered discussion of this correlation would strengthen the scientific claim.\n- The results on LLaVA-NeXT show a marginal improvement, which the authors rightly attribute to its limited base OCR capabilities. This highlights a key limitation: the method is a correction mechanism, not a fundamental capability enhancer. **It can help a model that can see the text to report it faithfully, but it cannot help a model that is effectively blind to the text**. This dependency could be discussed more prominently as a scope-defining weakness of the approach.\n- **Heuristic Nature of Refocus**: The “Refocus” step is based on the heuristic that background or “register” tokens exhibit stable attention patterns across layers. While the ablation shows this works well, the paper could benefit from a deeper analysis of this phenomenon. Visualizing which tokens are actually filtered out by this step would provide stronger evidence for the hypothesis.", "was_revised": true, "final_justification": "The authors have solved my concerns. I believe this is a weak-accept-level paper. So I suggest that AC and PC decide on this paper based on the batch situation.", "rebuttal": "# To reviewer 6wHJ\n\nThank you for the thoughtful feedback. We're glad you found our focus timely and novel, and appreciated both the framing and attention analysis. We also appreciate your recognition of TextHalu-Bench.\n\nBelow, we address each Weakness (W) or Question (Q) with a corresponding Answer (A):\n\n## [W1/Q1]: Discussion about the causal relatioinship between visual attention grounding and halluction generation.\n\n**[A1]**: Thank you for the thoughtful question. We agree that claiming a direct **causal relationship** may be too strong, and we appreciate your suggestion to phrase it more cautiously. It is indeed plausible that both poor attention and hallucination stem from a shared issue—such as the model’s failure to recognize meaningful text.\n\nThat said, we believe it's valuable to explore whether **inaccurate attention contributes to hallucination**, given the **auto-regressive decoding** process in LLMs. Misalignment during the **encoding stage** could bottleneck downstream reasoning.\n\nTo examine this, we ran a new ablation on 100 samples from TextHalu-Bench, TextVQA, and ST-VQA. We manually annotated ground-truth text boxes and **artificially reallocated attention** from these to background regions using:\n\n$$\n\\\\alpha' = \\\\frac{(1 - \\\\lambda) \\\\cdot \\\\alpha_{\\\\text{gt}} + \\\\lambda \\\\cdot \\\\alpha_{\\\\text{bg}}}{\\\\sum \\\\left[(1 - \\\\lambda) \\\\cdot \\\\alpha_{\\\\text{gt}} + \\\\lambda \\\\cdot \\\\alpha_{\\\\text{bg}}\\\\right]}\n$$\n\nHere, $\\alpha_{\\text{gt}}$ and $\\alpha_{\\text{bg}}$ denote the original attention scores over ground-truth and background tokens, respectively.  The denominator computes the total adjusted attention score to ensure that $\\alpha'$ remains a valid probability distribution.\nAs **λ** increases, hallucination scores rise significantly (same metric as Fig. 2), confirming that **misallocated attention worsens hallucinations**.\n\nWe will incorporate this analysis in the revised paper and revise our claim from a **“causal relationship”** to a **“strong correlation”**, ensuring a more accurate and robust interpretation.\n\n| λ (Attention Shift Ratio) | Hallucination Score |\n|---------------------------|---------------------|\n| 0.1                       | 0.36                |\n| 0.3                       | 0.38                |\n| 0.5                       | 0.54                |\n| 0.7                       | 0.62                |\n\n## [W2]: Discussion about the model`s ability limitation.\n\n[A2]: Thank you for raising this important point. We agree that our method functions as a correction mechanism, not a capability enhancer. Hallucinations in multimodal models broadly fall into two types:\n(1) Faithfulness Errors – the model perceives the information but responds inaccurately;\n(2) Knowledge Gaps – the model fails to perceive or understand the required input.\n\nOur training-free framework targets Type 1 errors and cannot improve models lacking fundamental OCR ability (e.g., LLaVA-NeXT). Still, we believe addressing faithfulness is a necessary step before tackling deeper knowledge gaps.\n\nWe have noted this scope in Appendix \"Limitations\" (Lines 597–602) and will make it more explicit in the revised version. We also plan to explore external knowledge integration in future work to expand the framework’s coverage.\n\n## [W3]: Heuristic Nature of Refocus.\n\n[A3]: We agree that a deeper analysis of the Refocus step would enhance the clarity and credibility of our method. Although we are unable to include visualizations during the rebuttal phase, we conducted two ablation studies to empirically support the underlying heuristics: 1. **IoU Evaluation**: On the same validation set in [A1]. We compared three configurations using Qwen2.5-VL: (1) Baseline: Selects top-$k$ tokens using attention from the last LLM layer. (2) +Glimpse: Utilizes query-to-image attention (Eq. 4) to select tokens. (3) +Refocus: Filters irrelevant tokens using relative attention dynamics (Eq. 5).\n\nThe average IoU between predicted and ground-truth regions increases notably after adding Refocus, indicating improved localization precision.\n\n| Method      | Mini-Monkey | Qwen2.5-VL |\n|-------------|-------|-------|\n| Baseline    | 42.3  | 46.1  |\n| + Glimpse   | 45.9  | 48.9  |\n| + Refocus     | 47.8      | 52.7  |\n\n2. **Coefficient of Variation (CV) Analysis**. To test the assumption that background tokens exhibit stable attention, we computed the coefficient of variation (CV) of attention scores across layers for each visual token:\n\n$\n\\mathrm{CV}_i = \\frac{\\mathrm{Std}(\\alpha^i_1, \\alpha^i_2, \\dots, \\alpha^i_L)}{\\mathrm{Mean}(\\alpha^i_1, \\alpha^i_2, \\dots, \\alpha^i_L)}\n$\n\nwhere $\\\\alpha^i\\_L$ is the attention score of token $i$ at layer $L$. We categorized tokens into: (1) Foreground: Tokens inside bounding boxes; (2) Background: Tokens outside bounding boxes but with high attention\n\nOur analysis shows that foreground tokens have significantly higher CV, while background tokens remain stable (lower CV). This validates our heuristic: **stable tokens likely represent less semantically relevant areas and can be effectively filtered via Refocus.**\n\n| Method      | Mini-Monkey | Qwen2.5-VL |\n|-------------|-------|-------|\n| foreground tokens    | 2.76  | 2.86  |\n| background tokens   | 0.00054  | 0.00051  |\n\n## [Q2]: The analysis of hyperparamter K.\n**[A2]**: We agree that understanding the role of **hyperparameter K** in ZoomText is essential for clarity and reproducibility. We conducted ablations on **Qwen2.5-VL** and **Mini-Monkey** across four datasets with varying image resolutions, leading to three key observations: 1. **Model-Agnostic Behavior**:   The optimal **K** generalizes well across architectures. Performance is not highly sensitive to the backbone. 2. **Robust Value Range**:  \nWhile there's no fixed best value, performance remains stable when **K ∈ [96, 160]**. Small **K** may miss text; large **K** may introduce noise. We use **K = 128** as a balanced default. 3. **Resolution Robustness**:   Most datasets (≤1000px) show no performance drop with **K = 128**. On high-res images from **ReCTS** (>2000px), the optimal **K** shifts to **[160, 224]**, suggesting larger values help in ultra high-resolution scenarios. We acknowledge that **K** is empirically chosen. Still, its consistent performance across diverse datasets demonstrates robustness. We will highlight the lack of a resolution-aware selection method in the **Limitations** section.\n\n| K   |  TextHalu-Bench | TextVQA |  SEEDBench$_{Text}$  | OCRVQA \n|-----|---------|---------|-------|--------|\n| 64  | 50.1    | 78.8    | 67.9  | 70.0   | \n| 96  | 53.4    | 79.9    | 69.0  | 70.2   | \n| 128 |**53.8** |**80.3** |**70.2**|**70.5**| \n| 160 | 53.6    | 80.1    | 70.2  | 70.0   |\n| 192 | 52.1    | 80.1    | 70.2  | 70.0   | \n| 224 | 52.2    | 79.2    | 69.0  | 69.9   | \n（Experiments on Qwen2.5-VL）\n\n| K   |  TextHalu-Bench | TextVQA |  SEEDBench$_{Text}$  | OCRVQA \n|-----|---------|---------|-------|--------|\n| 64  | 48.9    | 74.8    | 83.3  | 38.9   |\n| 96  | 50.1    | 75.0    | 83.3  | 39.8   | \n| 128 |**50.6** |**75.0** |**84.5**|**39.9**|\n| 160 | 50.5    | 74.8    | 84.5  | 39.9   |\n| 192 | 49.9    | 74.1    | 83.3  | 38.2   | \n| 224 | 49.9    | 73.8    | 83.3  | 38.2   |\n（Experiments on MiniMonkey）\n\n| K   | Qwen2.5-VL \n|-----|------------|\n| 64  | 51.2       | \n| 96  | 51.6       | \n| 128 | 52.7       | \n| 160 | 53.1       | \n| 192 | **53.8**   | \n| 224 | 53.7       | \n(Experiments on High-resolution ReCTS-VQA)\n\n[1] Icdar 2019 robust reading challenge on reading chinese text on signboard. ICDAR 2019.\n\n## [Q3]: Discussions about the failure cases.\t\t\n\n[A3]: Thank you for raising this important point. Due to the limitations of the rebuttal stage, we are unable to include visual failure case examples at this time, but we will provide more qualitative samples in the appendix of the revised version. Regarding failure modes, we clarify the following: (1) **Minimal Over-correction**: Our method does not impair recognition of semantic or idiomatic text. As shown in Table 1, OCR performance is slightly improved, indicating no adverse effects. (2) **Preserved Generalization**: Table 5 shows stable performance on general benchmarks, suggesting hallucination mitigation does not come at the cost of reasoning ability. (3) **Typical Failures Reflect Model Limits**: Errors mainly occur in inherently difficult cases—e.g., distorted/occluded text, rare scripts, or questions requiring external knowledge—where even base models struggle. As a training-free method, ours ensures faithful outputs but cannot recover missing knowledge.\n\nWe will include detailed failure cases in the revised version to clarify the scope and limitations.\n\n## [Q4]: Further explanation about efficiency.\n\n[A4]: Thank you for the insightful question. The added latency from our method occurs only during the prefilling stage, where intermediate attention scores are computed. This is a **one-time cost per query**, independent of the output length. In long-form generation tasks (e.g., captioning), our experiments with Qwen2.5-VL show that prefilling adds less than 35% to the total inference time when outputs exceed 50 tokens.\n\nThis overhead is modest compared to the **dominant cost of autoregressive decoding**, especially for longer outputs. We thus consider the trade-off acceptable for real-world use.\n\n| Output token Length | Method    | Prefilling (s) | Decoding (s) | Total Time (s) |\n|---------------------|-----------|------------|----------|------------|\n| (0,20]              | Baseline  | 0.62       | 1.03     | 1.65       |\n|                     | Ours      | 1.22       | 1.04     | 2.26       |\n| (20,50]             | Baseline  | 0.65       | 1.92     | 2.57      |\n|                     | Ours      | 1.32       | 1.94     | 3.26      |\n| (50,]               | Baseline  | 0.66       | 2.64     | 3.30      |\n|                     | Ours      | 1.34       | 2.65     | 3.99      |\n（ablation studies on efficiency in 80G A-100 ）"}, {"review_id": "6M2gnfN58f", "reviewer": "Reviewer_wv1z", "summary": "This paper investigates the phenomenon of semantic hallucination in Large Multimodal Models (LMMs), specifically within scene text spotting and understanding tasks. The authors identify that LMMs often rely on semantic priors and fail to recognize visually ambiguous or non-semantic text accurately. To address this, the paper proposes a training-free framework with two key components: (1) ZoomText, a coarse-to-fine text region estimator, and (2) Grounded Layer Correction (GLC), which guides decoding by leveraging transformer layers with better visual grounding. A new benchmark, TextHalu-Bench, is also introduced to evaluate hallucination behavior in LMMs. Experiments across several benchmarks demonstrate the effectiveness and generalizability of the proposed approach.", "questions": "**1.Incomplete Evaluation on OCR Benchmarks:**\nThe proposed method is not evaluated on MME or MME-VideoOCR, both of which include OCR-relevant tasks and are widely used for assessing multimodal LLMs. Given that the method aims to mitigate semantic hallucinations involving visual text, such evaluation is necessary to verify its robustness and generalizability across diverse scenarios.\n\n**2.Unverified Methodological Assumptions:**\nThe method assumes that visually prominent text (e.g., signs or posters) draws model attention, and that background tokens have stable attention patterns across layers. However, no analysis or visualization is provided to support these assumptions. Without empirical validation, the method’s motivation remains speculative and its design less convincing.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\n**1.Novelty:**\nThe work tackles the underexplored issue of semantic hallucination in OCR scenarios, distinct from prior object or attribute hallucination studies. The concept of layer-wise hallucination tendency analysis and visual grounding correlation is original and empirically well-motivated.\n\n**2.Motivation and Clarity:**\nThe paper provides a clear and compelling motivation by identifying a failure mode in existing LMMs. The problem is illustrated with intuitive examples and well-analyzed causes rooted in LLM architecture.\n\n**3.Effectiveness:**\nAcross multiple benchmarks, the method consistently improves performance, especially on hallucination-prone samples. Gains are shown to be more pronounced on models with stronger OCR capabilities, which aligns with the method’s design.\n\n**Weakness:**\n\n**1. Incomplete Evaluation on OCR Benchmarks:**\nThe MME benchmark,a widely used general-purpose benchmark for multimodal large models,includes OCR-related tasks. Therefore, the proposed method should also be evaluated on MME and MME-VideoOCR to more rigorously assess its effectiveness in mitigating semantic hallucinations. Without such evaluation, the generalizability and robustness of the method across broader multimodal scenarios remain uncertain.\n\n[1] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. CoRR 2023\n\n[2] MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios. CoRR 2025\n\n**2. Unverified Methodological Assumptions:**\nThe design of ZoomText relies on two key assumptions: (1) that signs, posters, or product packaging naturally attract model attention during question answering; and (2) that background or non-semantic tokens exhibit relatively stable attention patterns across layers. However, the paper does not present any experiments or visualizations to directly validate these assumptions. Without empirical support, the motivation behind the proposed method remains speculative. This lack of validation undermines the credibility of the approach and weakens the explanatory power of its design.", "was_revised": true, "final_justification": "This is a well-structured and thoughtful rebuttal. The authors have effectively addressed the weaknesses I raised.\n\nReviewer wV1z\nI appreciate the authors' thorough response to my feedback. The rebuttal effectively addresses the two main weaknesses I raised.\n\nRegarding Weakness 1, the authors have provided new experimental results on both the MME and MME-VideoOCR benchmarks. These results, which demonstrate consistent performance improvements, significantly strengthen the claim that the proposed method is robust and generalizable. I am satisfied with this additional evaluation.\n\nRegarding Weakness 2, the authors have provided empirical validation for their key methodological assumptions. The new ablation study, which uses IoU scores to evaluate the effectiveness of the ZoomText module, and the analysis of the attention coefficient of variation (CV) for foreground versus background tokens directly address my concerns. These results provide clear evidence supporting the design choices of the proposed method and successfully move the motivation from speculative to empirically grounded.\n\nOverall, the authors' response has successfully resolved the weaknesses I identified. My initial assessment remains unchanged, and I maintain my recommendation of borderline accept.", "rebuttal": "# To reviewer wv1z\n\nWe thank the reviewer for the detailed and encouraging feedback. We appreciate your recognition of the novelty of our work in addressing the underexplored issue of semantic hallucination in OCR scenarios, which is distinct from previous studies on object or attribute hallucination. We're glad you found the analysis of layer-wise hallucination tendency and its correlation with visual grounding both original and empirically sound. We also value your positive comments on the clarity and motivation of the paper, particularly in highlighting a failure mode in existing LMMs through intuitive examples and architectural insights. In addition, we’re encouraged by your acknowledgment of the method’s consistent performance improvements across benchmarks, especially on hallucination-prone samples, and how these gains align well with the method’s design.\n\nBelow, we respond to each Weakness (W) or Question (Q) with a corresponding Answer (A):\n\n## [W1/Q1]: Incomplete Evaluation on OCR Benchmarks\n\n[A1]: Thank you for your valuable feedback. We agree that incorporating more OCR-related benchmarks can better demonstrate the robustness and generalizability of our method. To address this, we have complemented our evaluation with experiments on both **MME and MME-VideoOCR** (a concurrent work). The results show:\n\nOn MME, our method brings improvements in **perception-based subtasks such as OCR, counting, and positioning**. However, due to the binary (yes/no) nature of many MME questions, where models can often guess the answer, the absolute improvement is relatively modest.\n\nOn MME-VideoOCR, we observe significant gains, especially in **complex scene-text-centric video question answering tasks**, where visual text grounding is more challenging.\n\nThese results, along with the existing experiments (e.g., Table 5), provide strong evidence of the robustness and general applicability of our proposed method across diverse OCR and multimodal reasoning benchmarks. We will include these new results in the revised version of the paper.\n\n| Task                   | minimonkey | minimonkey+ours | qwen2.5vl | qwen2.5vl+ours |\n|------------------------|------------|------------------|-----------|----------------|\n| perception             | 1451.78    | 1461.91          | 1567.70   | 1572.2        |\n| reasoning              | 427.5      | 427.5            | 601.07   | 601.07          |\n| OCR                    | 95         | 98               | 132.5     | 136             |\n| artwork                | 146.25     | 146.25           | 143.75    | 143.75         |\n| celebrity              | 118.529    | 118.529          | 127.35   | 127.35        |\n| color                  | 168.333    | 168.333          | 170.0   | 170.0        |\n| count                  | 143.333    | 148.0          | 155.0   | 156.0        |\n| existence              | 190        | 190              | 185.0    | 185.0            |\n| landmark               | 156.5      | 156.5            | 177.0     | 177.0          |\n| position               | 153.333    | 155.8          | 151.67   | 151.67        |\n| posters                | 129.252    | 129.252          | 171.43   | 171.43        |\n| scene                  | 151.25     | 151.25           | 154.0    | 154.0         |\n| code_reasoning         | 92.5       | 92.5             | 155.0    | 155.0           |\n| commonsense_reasoning  | 115        | 115              | 128.57 | 128.57            |\n| numerical_calculation  | 42.5       | 42.5             | 125.0    | 125.0           |\n| text_translation       | 177.5      | 177.5            | 192.5     | 192.5          |\n(Experiment on MME)\n\n\n\n| Task      | Qwen2.5-VL-7B | Qwen2.5-VL-7B+ours |\n|-----------|-----------|----------------|\n| TR        | 70.1      | 70.1           |\n| VTQA      | 70.2      | 73.4           |\n| TG        | 57.6      | 58.9           |\n| AR        | 69.2      | 69.2           |\n| CDT       | 46.8      | 46.8           |\n| STP       | 66.4      | 66.4           |\n| CFTU      | 17.1      | 17.3           |\n| TBR       | 49.8      | 49.6           |\n| TBVU      | 52.5      | 52.5           |\n| RVT       | 71.3      | 71.3           |\n| **Total** | **59.2**  | **60.8**       |\n(Experiment on MME-VideoOCR)\n\n\n## [W2/Q2]: Unverified Methodological Assumptions.\n[A2]: Thank you for pointing out this important aspect. Our proposed ZoomText module is based on two core assumptions: **(1) that query-to-image attention can effectively highlight relevant visual regions such as signs or posters**, and **(2) that background or non-semantic tokens exhibit relatively stable attention dynamics across layers.** To empirically validate these assumptions, we conducted the following analyses:\n\n\nFirst, to assess the effectiveness of ZoomText in capturing relevant text regions, we manually annotated 100 samples across TextHalu-Bench, TextVQA, and ST-VQA, with quadrilateral bounding boxes marking the target text areas. We then performed an **IoU-based ablation study** on Qwen2.5-VL-3B, comparing three variants: **(a) a baseline selecting top-k tokens using final-layer attention**, **(b) baseline + Glimpse (which uses query-to-image attention, Eq. 4)**, and **(c) baseline + Glimpse + Refocus (which incorporates attention variation dynamics, Eq. 5).** Results show a **consistent improvement in IoU scores**, confirming the benefit of each step in refining the focus on relevant visual regions.\n\n\n\nSecond, to validate the Refocus assumption, we computed the **coefficient of variation (CV)** of attention scores across layers for each visual token. Tokens were categorized into: **(i) foreground (within bounding boxes)**, and **(ii) background (outside boxes but with high attention).** We compute the CV for each token $i$, defined as:\n\n$\n\\mathrm{CV}_i = \\frac{\\mathrm{Std}(\\alpha^i_1, \\alpha^i_2, \\dots, \\alpha^i_L)}{\\mathrm{Mean}(\\alpha^i_1, \\alpha^i_2, \\dots, \\alpha^i_L)}\n$\n\nwhere $\\\\alpha^i\\_L$  is the attention score of token $i$ at layer $L$. The analysis reveals that \\*\\*foreground tokens exhibit significantly higher attention variation across layers (higher CV)\\*\\*, while \\*\\*background tokens remain relatively stable (lower CV)\\*\\*, supporting our hypothesis that \"sink tokens\" can be identified through their consistent attention profiles.\n\n\nDue to rebuttal constraints, we are unable to include visualizations at this stage, but we will incorporate these empirical results and more qualitative examples into the revised version of the paper to strengthen the methodological soundness of our design.\n\n\n\n| Method      | Mini-Monkey | Qwen2.5-VL |\n|-------------|-------|-------|\n| Baseline    | 42.3  | 46.1  |\n| + Glimpse   | 45.9  | 48.9  |\n| + Refocus     | 47.8      | 52.7  |\n(ablation studies of ZoomText )\n\n\n| Method      | Mini-Monkey | Qwen2.5-VL |\n|-------------|-------|-------|\n| foreground tokens    | 2.76  | 2.86  |\n| background tokens   | 0.00054  | 0.00051  |\n(abalation studies of relative attention)"}]

Author rebuttal:

Meta-review: Final rating: 4: Borderline Accept/  4: Borderline Accept/ 4: Borderline Accept/ 4: Borderline Accept. The paper tackles semantic hallucination in LMMs on ambiguous scene text, finding that layers with stronger attention to text hallucinate less. It proposes a training-free framework—ZoomText for coarse-to-fine text region focus and Grounded Layer Correction to decode with hallucination-resistant layers—and introduces TextHalu-Bench (1,730 samples) for evaluation. Experiments show effective hallucination mitigation and strong scene-text spotting/understanding performance.

Reviewers praise the timely problem, sound motivation, and the benchmark’s community value, but questioned (pre-rebuttal) the technical soundness of using intermediate-layer logits with the final head (and y_hal/y_gt in an autoregressive setup), possible side effects on broader capabilities, limited component attribution, overstated causality, reliance on baseline OCR (a corrective rather than capability-adding method), heuristic assumptions, and missing OCR benchmarks (OCRBench, MME/MME-VideoOCR).

After the rebuttal, reviewers report their concerns were satisfactorily addressed. The ACs concur that the problem about semantic hallucination is important, the method of the Glimpse-Refocus strategy is effective and the TextHalu-Bench benchmark is a substantial contribution. Therefore Acs recommend acceptance
Decision: Accept (poster)