Imitating the Truth: Attention-aware Truth-Guided Enhancement for Hallucination Mitigation in Large Vision-Language Models
Authors: Hairui Ren
Zixuan Wang
Yibo Yang
He Zhao
Fan Tang
Dandan Guo
Yi Chang
Venue: ICLR 2026 Poster (2026)
Keywords: Hallucination Mitigation
Large Vision-Language Models
Attention Intervention
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose AGE, a training-free framework that mitigates hallucinations in LVLMs by imitating truth-grounded attention patterns of real tokens, yielding more accurate and trustworthy multimodal reasoning.

Abstract: Large Vision-Language Models (LVLMs) achieve impressive multimodal reasoning but remain prone to hallucinations, generating content inconsistent with visual evidence.
Existing mitigation methods often rely on auxiliary modules or coarse decoding-time adjustments, overlooking the fine-grained dynamics that distinguish truthful (real) tokens from hallucinatory ones.
In this paper, we introduce \textbf{AGE (Attention-aware Truth-Guided Enhancement)}, a training-free framework that performs fine-grained, layer-wise interventions guided by attention patterns of real tokens. 
Our analysis reveals that real and hallucinated tokens follow distinct stage-specific attention behaviors, and hallucinations emerge when models fail to reproduce these behaviors.
AGE addresses this by introducing two lightweight interventions: (i) Imitating the image attention, derived from discrepancies between real and hallucinated tokens, and (ii) Imitating the text attention when semantic grounding is required.
Extensive experiments on widely used benchmarks, including COCO Image Captioning, POPE, and MME, demonstrate that AGE consistently mitigates hallucinations across diverse LVLMs such as LLaVA, MiniGPT-4, and mPLUG-Owl2, without additional training or loss of fluency.
Our results highlight that imitating truth-grounded attention dynamics is a simple yet powerful principle to improve the reliability of LVLMs.

Published as a conference paper at ICLR 2026
IMITATING THE TRUTH: ATTENTION-AWARE AWARE TRUTH- GUIDED ENHANCEMENT FOR HALLUCINATION MITI- GATION IN LARGE VISION-LANGUAGE MODELS
Hairui Ren', Zixuan Wang', Yibo Yang", He Zhao Fan Tang', Dandan Guo! 2+
Yi Chang 67.
School of Artificial Intelligence, Jilin University'; KAUST²;
CSIRO's D Department of DSAL, Monash University*;
Institute of Computing Technology. Chinese Academy of Sciences International Center of Future Science, Jilin University Engineering Research Center of Knowledge- Shooledge-Driven Driven Human- Intelligence, MOE, China? (renhr22, z zixuan24 [emails..........lll jlu. pew..eduuueeeoo en, yibo. yang 930 gmail...... com, he zhao@data61 csiro.au tangfan www.aaniictaac..oo {guodandan, yi richangee ful. edu.
ABSTRACT
Large Vision-Language Models (LVLMs) achieve impressive multimodal reason- ing but remain prone to hallucinations, generating content inconsistent with vi- sual evidence. Existing mitigation methods often rely on auxiliary modules or coarse decoding-time adjustments, overlooking the fine-grained dynamics that distinguish truthful (real) tokens from hallucinatory ones. In this paper, we in- troduce AGE (Attention-aware Truth-Guided Enhancement), a training-free framework that performs fine-grained, layer-wise interventions guided by atten- tion patterns of real tokens. Our analysis reveals that real and hallucinated to- kens follow distinct stage-specific attention behaviors, and hallucinations emerge when models fail reproduce these behaviors. AGE addresses this by introduc- ing two lightweight interventions: (i) Imitating the image attention, derived from discrepancies between real and hallucinated tokens, and (ii) Imitating the text at- tention when semantic grounding is required. Extensive experiments on widely used benchmarks, including COCO Image Captioning, POPE, and MME, demon- strate that AGE consistently mitigates hallucinations across diverse LVLMs such LLaVA, MiniGPT-4, and mPLUG-Ow12, without additional training or loss of fluency. Our results highlight that imitating truth wuth-grounded attention dynamics is simple yet powerful principle to improve the reliability of LVLMs.
INTRODUCTION
Large Vision Languuge Models (LVLMs) (Chen et al. 2023 Li et al. 2023a Liu et al.
2023b Zhu
ct al. 2023 Ye et al. 2023) have demonstrated remarka aall capabilities across a wide range of mul- timodal tasks, including image caption (Hu al. 2023b visual question answer (Liu et al. 2024c), and instruction following grounded in visual content (Hong et al. 2024 Liu et al. (022a)) Despite their impressive capabilities, LVLMs often suffer from critical flaw: hallucination, the generation of content that is not supported or contradicted by the visual input. This issue poses a substantial threat to their reliability and limits deployment in high-stakes scenarios such as autonomous sys- tems (Chen et al.) |2024c] Mai et al.) (203)) and healthcare diagnostics (Hu et al.) [2023a] WWan et al.) 2024b). Understanding the causes of hallucinations LVLMs and devising effective strategies for their mitigation are essential steps toward enhancing the reliability of these models.
Given the complexity of LVLM architectures and their reliance on multimodal fusion reasoning, eliminating hallucinations remains highly challenging. Recent studies have explored two major directions: introducing external auxiliary modules (Zhou al. 2023 Yin et al. 2024) and inter- vening during decoding (Huang et al. 2024
Leng et al. 2024 Yang et al. 2025b). Among these,
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Hest tabee
-6thh31th Layer
How many animals in the image?
31 Layer
16 Layer
Weaken Image Alt.
26th Layer
What are the disparities the Attention behaviors of real and hallucinated tokens?
Image
16th Layer
Thos Weaken Text Att.
Dackse
-11
4th Layer
Imitating Real Responses
Figure 1: An illustration of our motivation. Left: The attention behaviors for real and halluci- nated tokens exhibit clear, stage-dependent disparities. Right: A conceptual example demonstrating the consequences of these disparities. Weakening the dominant modality a given stage (e-g-, text attention for counting, image attention for final grounding) leads to specific hallucinated outputs.
decoding-tinne time approaches such OPERA (Huang et al. 2024 and VCD (Leng et al. 2024 have drawn particular attention due to their model-agnostic nature and ease of deployment. These meth- ods typically mitigate hallucinations by reweighting visual attention or counteracting biased text priors during generation. However, most of them (Huang et al., 2024 Chen et al.) 2024a Park al., 20255 Leng al.) 2024; Zou et al.) 2025) operate in a coarse- grained manner, applying uni- form enhancements across layers or modalities. Such oversimplification fails to capture the nuanced dynamics of multimodal reasoning, resulting in reduced robustness and less effective hallucination mitigation.
coarse-grained
To move beyond coarse-grained interventions, we adopt finer-grained analysis, where we examine hallucinated responses by decomposing them into "truthful tokens" (real tokens image-grounded) and "hallucinatory tokens" (unsupported) to analyze their attention behaviors layer by layer. As visualized in Fig. [ (more details in Fig. 2), this investigation reveals systematic, stage-dependent discrepancies across both modalities: the attention patterns for real tokens, both visual and textual domains, differ significantly from those of hallucinatory tokens, and this pattern of divergence is both model- and stage-specific. For instance, when the model fails to sufficiently attend to text in real response would, it may default to a salient visual object (e.g., "Elephant")
the middle stage instead of the correct count. Conversely, insufficient visual attention in the late stage can lead to over- reliance on incomplete textual priors (e.g.. generating "Two" instead of "Three"). These findings suggest that hallucinations are caused by a failure to reproduce the token-level, stage-sensitive attention dynamics characteristic of real responses. This insight motivates our core hypothesis: hallucinations can be mitigated by guiding the model to imitate the internal attention behaviors of real tokens through adaptive, stage-specific interventions.
Motivated by this insight, we introduce AGE (Attention-aware Truth-Guided Enhancement), training-free, decoding-time framework that mitigates hallucinations by imitating the stage-specific attention dynamics of real tokens. Rather than applying uniform adjustments, AGE implements targeted interventions aligned with the distinct attention characteristics of each model, focusing on the stages where discrepancies interventions that require: (i) Imitating the Image Attention, derived from the attention disparity between real and hallucinatory tokens, restore visual grounding in late reasoning layers; and (ii) Imitating the Text Attention where analysis reveals a stronger reliance on text, as observed in the middle stage of VA. By imitating attention behaviors in real tokens, AGE enables more accurate, fluent, and trustworthy multimodal generation. We summarize our primary contributions as follows:
most pronounced. Concretely, it employs two lightweight
1. We conduct novel token-level, layer-wise analysis of attention within hallucinated re- sponses, identifying the cause of hallucinations as failure to reproduce stage-specific attention behaviors of real tokens.
2. We design and propose AGE, a framework whose lightweight interventions effectively translate our analytical insights into practical, decoding-time solution.
2

---

Published as a conference paper at ICLR 2026
3. We provide extensive experimental validation showing that AGE significantly and consis- tently reduces hallucinations across multiple LVLMs and benchmarks, without sacrificing fluency completeness.
2 RELATED WORK
Large Vision- Language Models. The remarkable success large language models (LLMs) (Tou- vron al., 2023a b) has spurred increasing interest in extending their capabilitiie to the multimodal domain. With the open release of influential LLM backbones such as LLaMA (Tourron et al. 2023a)
and Vicuna CChing al. 2023). large vision- language models (LVLMs) (Bai et al. 2023 Chen
eet al. 2024b Liu et al. 2024a 2023b; Zhu et al. 2023) have rapidly emerged as powerful systems capable of comprehensively processing and generating content across multiple modalities, includ- ing text, images, and even audio. Building on these foundations, models such as LLaVA (Liu et al.,
2023b), mPLUG-Owl2 (Ye et al., 2023). and MiniGPT-4 (Zhu al., 2023) further advance inter- activity by supporting joint image text prompts, enabling richer and more context-aware responses. Most LVLMs employ two-stage training paradigm: an initial vision- laiiuage feature alignment phase followed by instruction tuning, which equips them to interpret and follow multimodal queries effectively. However, despite these advancements, hallucination, where generated content diverges from or contradicts the visual evidence, remains persistent and widespread limitation across cur- rent LVLMs.
Mitigating Hallucinations in LVLMs. Hallucination, content that is irrelevant, inaccurate, in- consistent with the visual input Bai et al.
2024). has been linked to limitations visual encod-
ing (Tong al.) 2024; Liu et al.) 2022b] Shi et al. 2024] Liu et al., 2024d). overreliance para- metric knowledge (Zhou et al. [202] Leng er al. al.] (024). and noisy supervision (Liu et al. 2023a Yu al. 2024). Mitigation strategies include training-basete improvements via cleaner reweighted datasets (Yue et al. 2024 Jiang et al. 2024) and decoding-ting timiing--mmee adjustments ( (Yang et al. 2025a
Zhang | 2025 An et al. 2024) Wang et al. 2024a) such as VCD (Leng al. 2024), which con- trasts output distributions derived from original and distorted visual in puts, or attention calibration OPERA (Huang al. 2024), which penalizes over-trust and refines token selection based on previ- ous outputs. However, most of these approaches require costly retraining, depend on most-generation filtering, or apply coarse global attention changes. contrast, our method directly calibrates atten- tion behaviors during inference by imitating real response patterns, offering a fine-grained interven- tion across reasoning stages without modifying model training or architecture.
UNCOVERING HALLUCINATION--R REAL RESPONSE DISCREPANCY
3.1 PRELIMINARY
Large Vision-Language Models (LVLMs) process image V and textual instruction to generate a response Y = (y....., y The image is encoded into visual tokens {ri...., "n and the instruction into text tokens (41...., ......,.... These models generate text autoregressively using L.- layer Transformer decoder. At each decoding step & and layer I, the model computes an attention (I.A) weight vector (1.A) E t This vector is composed of a visual component avisoon E R² over the visual tokens and a textual component a E Rm+* over the textual (instruction and previously generated) tokens. These attention weights then used to update the hidden state h E IRd from the corresponding value matrices (V( E Raxd and (((.k) E (mm++)xx) via a vision
}.
residual connection:
k
((+1)
(1) h
AttentionSubLayer
(1.b) (3.b) (
(issio::
vision a
(L.K)
v((k))
(1)
The probability of the next token is predicted from the final layer's hidden state, h()) (L),
p(uale<) (f
where ʃ(-) is an affine layer. Hallucination may occur when the generated text Y is inconsistent with or contradicts the image V.
3

---

Published as a conference paper at ICLR 2026
3.2 METHODOLOGY FOR ANALYZING ATTENTION DISCREPANCY
While prior work often attributes hallucination general lack of visual grounding textual prior interference, we hypothesize that the underlying cause is a more nuanced, dynamic behavior in atten- tion. Specifically, conduct fine-gr fine-grained analysis of the internal attention dynamics of LVLMs to uncover stage-dependent patterns that can inform a more targeted intervention. Our analysis focuses three representative LVLMs: LLaVA-1.55
(Liu et al. 2023b), MiniGPT-- (Zhu et al. 2023),
and mPLUG-Ow12 (Ye et al. 2023). For these models, we curated set of (Vit== with N 100
challenging images from the COCO training set, specifically selected for their known propensity to elicit hallucinated responses { Our core idea is to investigate if model exhibits different attention behaviors when generating real versus hallucinated content within the same response.
For each generated response ( that contained inaccuracies, distinguish real and hallucinated tokens by comparing the predicted objects with the ground-truth annotations. token was labeled as "real" only if it corresponded an object explicitly present in the ground-truth labels, and otherwise
marked as hallucinated. The resulting sets are: (1) Real Tokens y(t) Tokens corresponding to
objects verifiably present in the image; and (2) Hallucinated Tokens I(a): Tokens corresponding to objects confabulated by the model and not present in the image. To quantify the difference in attention behavior between these two token sets, we first define the Average Attention Sum, s, per-sample metric for a specific modality and token type. For instance, the average visual attention sum for real and hallucinated tokens in sample layer I be expressed as:
(1,5) vision)
(1.A)
(1.8) (hall,vision)
(1.k) a
[T(
sum
visioo
sum
vision
kE
JE (9
where (-) is the sum of all elements in a vector. The corresponding terms for the textual modality defined analogously. After computing these per-sample scores, aggregate them across all N images to derive our final layer-wise difference metrics:
N Σ
Diff' imuge
1
(1.i)) visain
(l.i) (hall, vision)
Diff' Σ
((ii))) (real,texi)
(1.1) (hall,
Consequently, a positive value in Diff signifies that real tokens, on average, allocate greater attention to the visual modality than hallucinated tokens layer l. This finding forms the basis of our intervention: guiding the model to align with the attention patterns truthful tokens.
Attention Differnnce
0.03
0.01
0.00
Diff
Attention Difference
0.00
0.01
0.02
0.08
0.06
0.04
-0.02
-0.02
Attention
-0.03
-0.03
-0.02
0..04
-0.05
8 12
20 Layer Index
16
24
28
-0.05
8
12
16
20
24
28
8
12
16
20
24
28
Layer Index
Layer Index
(a) I.L.VA-1.5
MiniGPT-4
mPL.UG-OW2
Figure 2: A layer-wise characterization of attention disparities between real and hallucinated re- sponses, which vary across -L..5-1.55 MiniGPT-4, and mPLUG-Ow12.
3.3 ANALYSIS OF ATTENTION BEHAVIORS
As shown in Fig. 2 our analysis of real and ballucinated token sets reveals distinct, model-specific attention behaviors. To facilitate this analysis, we partition each model's architecture into three stages based on the observed dynamics: Early (Layers 0-16), Middle (Layers 16-26), and Late (Layers 26-31). The attention dynamics are notably model-specific. LLaVA-1.5, for instance, re- veals a complex non-monotonic relationship where real responses show stronger visual attention in the early and late stages but rely more heavily on textual context in the middle stage. In con- trast, MiniGPT-4 and mPLUG-Ow12 present more consistent dynamic, with visual attention for real responses decisively dominating across nearly all stages. Despite these differences, a critical commonality emerges: Across all tested models, the final reasoning stage (Layers 26-31) exhibits
4

---

Published as a conference paper at ICLR 2026
Hallucination Generation
Attention-Aware Truth-Gulded Enhancement
Calibrated Generation
Allenton Map
Deep layer
Irag
Deep layer
ONQ Oxec
User
User
Pleas cerente tisimuge in Ietall
Paus duounder Use imaeiindaa..
Opertiion
A
AttentionSubl. ayer
The trues fnsturt bouslah
pooildy chanceer from
Target Discciinaa Vestor
Sratnnaom) stnoe)
itt curdor bencuet tabl,
adorred with strast
pave "Sking pols" par formen
Shallow layer
rupbente and bls berien takes certer stags, sumourded plitss dhoownr Ltresk srongee masy for pantss The sxckirp indudes pree catiggmmooion
decorated
vee sppoors
phoncssngg lis mrposii skl.
Real
layer
There soarn hyng
Tokerss
Hallucination
bwssen the budgetore
Tokens
Image Tokens
Text Tokens
Figure 3: AGE first samples hallucinatory responses and distinguishes real from hallucinated tokens. The visual attention gap between them yields the target vector 5. During inference, AGE applies 5 and/or textual self melf-multipiicative enhancement, guiding LVLMs to align with real responses and mitigate hallucinations.
stable and pronounced positive gap where real responses attend more to visual tokens than halluci- nated ones. This provides a universal and impactful opportunity for intervention. See Appendix A.8 for more results about different size N.
This crucial finding demonstrates that hallucination is not caused by a simple, uniform lack of visual attention, but is instead tied to failure to replicate nuanced, model-specific, and stage-specific attention dynamics. Consequently, naive, one-size-fits-all interventions that globally enhance visual attention are insufficient and may even be counterproductive. Our analysis, therefore, points to more effective strategy: we hypothesize that hallucinations can be mitigated by guiding a model to imitate the attention behavior of real responses. This requires interventions that are not only targeted to specific reasoning stages but are also adaptive to the unique attention patterns of each individual model. Motivated by this core insight, introduce novel framework in the following section, designed to implement this principle through targeted attention interventions faithfully.
4 OUR PROPOSED METHOD
This work introduces AGE (Attention-awar) Suihh-Gidded Enhancenment), AGE a training- free, decoding-time framework that corrects these attention disparities by guiding the model to imitate attention patterns of real responses.
4.1 IMITATING THE ATTENTION BEHAVIOR
Imitating the Image Attention. Our analysis in Section 3.3 revealed critical, universal finding: across all tested LVLMs, the late reasoning stages consistently exhibit stable pattern where real responses allocate significantly more attention to visual tokens. While a simple self- mulipluvtileeeative enhancement could increase visual focus, it is agnession- and coarse. This work proposes a directional calibration of visual attention.
As shown in Fig. our strategy is to compute a single, robust directional vector, E R", that captures the essential shift from hallucinatory to a real attention pattern. The dimension corre- sponds to the number of visual tokens produced by the image encoder, as introduced in Section 3.1 To compute this vector, first randomly select AM samples from the COCO training set that known to elicit hallucinatory responses. The vector is designed to capture a general corrective tendency in the model's attention space, rather than overfitting to the specifics of the M samples. For each sample E [1, M). we follow the methodology from Section 3.2 and partition its generated
tokens into distinct sets: real tokens (7(0) and hallucinatory tokens (T()). To quantify the atten-
5

---

Published as a conference paper at ICLR 2026
tion patterns for these token sets, we define the Average Attention Vector R". This vector represents the averaged attention distribution over the n visual tokens. It is calculated by averaging the attention vectors from a specific layer for given set of tokens. We choose to compute this from the final decoder layer (T.,, as contains the most semantically integrated representations just prior to token generation. For given sample i, the average image attention vector for its hallucinated tokens and real tokens are formally defined as:
(I.Kk vision
vislinvssion)
vision 7 ((real.vision)
|
(5)
The target directional vector is then calculated as the weighted average of the difference be- tween these real and hallucinatory vectors across the M samples (see Appendix(A.3 the adaptive weighting scheme):
1 M
6
W (arreal.visioo)
[hall
(6)
During inference, we inject this universal adjustment vector into the visual attention of layers within the pre-defined late stage:
aision +
Au x
Fisioo
where A is scaling factor. This encourages the model to reproduce the universal image attention behavior characteristic of truthful responses.
Imitating the Text Attention. As analysis also highlighted, attention dynamics can be highly model-specific. For models like MiniGPT-4 and mPLUG-Ow12, where visual attention dominates throughout, no textual intervention is necessary. However, for LLaVA-1.5, the middle stage shows that real responses rely substantially more on textual context. To address this model-specific need, reinforce text attention in the designated middle stage of LLaVA. Since text attention vectors dynamically change in dimension, pre-computing a fixed directional vector is infeasible. Therefore, adopt self-multiplicn ceff-luuiillicative enhancement effective proxy amplify the model's focus on its generated context:
aLox
afext A aext
where A is a scaling factor. This targeted adjustment ensures our framework adapts to the unique reasoning patterns of each LVLM.
While this method does not specify a corrective direction, it effectively amplifies the model's ' focus on its generated context, thereby imitating the attention behavior uncovered in our analysis. Unlike prior work, our approach modulates multimodal attention a stage-specific manner to faithfully reproduce the attention behavior of real responses.
4.2 CALIBRATED AUTOREGRESSIVE GENER ATION
The stage-specific adjustments described previously are integrated into the standard autoregressive generation process. At each decoding step & and for each decoder layer I, the model applies the corresponding intervention conditionally based on the layer's designated stage. Specificall, for all LVLMs, choose 1 from Late Stage and shift the visual attention using the directional vector as defined in Eq. 7 for LLaVA, we additional enhance the textual attention via self-multiplication in Eq. g by choosing / from Middle Stage. Attention in layers outside these defined stages remains unchanged. The model then computes the subsequent hidden state h ([+1) using these potentially calibrated attention scores, a (t.k) sented by: (1+1) h + AttentionSubLaaerr( vision vision a
vision and a (L.))
form the context vector. This conceptually repre-
(1.k) (
v(. text
By coordinating these interventions in different LVLMs, AGE calibrates attention behavior through- out the reasoning process guided by the direction of real responses. This design not only suppresses hallucinations but also ensures faithful alignment with visual evidence, providing a principled and interpretable pathway toward more trustworthy multimodal generation. The algorithm of AGE is summarized in the Appendix A.2

---

Published as a conference paper at ICLR 2026
Table 1: Hallucination rates (%) are reported using CHAIRs (Cs), ,HAIR, (C1), and BLEU (%) on COCO image captioning tasks, where lower CHAIR and higher BLEU are better. The mar new token is set to 64. The best results are highlighted in bold while the second-best results are marked with underline. represents the results reported from Chen et al. (0224d). represents the results reported from the corresponding original paper.
Method
Ce
MiniGPT-4-7B C7 L
BLEUT
Cs 4
LLaVA-I.5-7B
C₁
BLEUT
Cs 4
mPLUG-OW12-78
C/ 4
BLEUT
Cs 4
C₁ 4
Ave
BLEU
Greedy
30.87
14.33
20.80
6.77
15.93
8.33
15.37
24.95
9.14
15.21
Beam Search
29.56
11.36
14.94
18.67
6.30
16.17
21.67
7.63
15.77
23,30
8.43
15.62
DeLA
Chuang cl
2023
30.87
11.70
14.93
21.00
6.70
15.93
24.60
8.73
15:40
25.49
9.04
15:42
OPER
Houne
2014
30.00
11.67
14:87
21.13
6.73
16.27
22.13
7.57
15.53
24.42
8.65
15.56
VCD
[Lnn al 2024
30.27
12,60
14.33
23.33
7,90
14.67
27.27
9.73
14,40
26.95
10.07
14.46
Woodepecker Vin
2024
28.87
10.20
15.30
23.85
7.50
17.05
26.33
8.43
16.43
26.35
8:71
16.26
LURB
(2ou ot gl 20725
27.88
10.20
15.03
19.48
6.50
13.97
21.27
7.67
15.65
22.87
8.12
15.55
VISTA
19.80
20.40
Ours
15.62
6.00
15.79
16.43
5.58
16.48
19.40
7.47
16.21
17.15
6.35
16.16
EXPERIMENTS
Baselines. We evaluate our approach on three representative LVLMs: LLaVA-1.5 (Liu et al. h023h),
mPLUG-OWI2 (Ye et al. 2023), and MiniGPT-4 (Zhu et al.] 2023). To examine the impact of model
scale, we additionally experiment with the 13B variants of LLaVA- We use the default greedy
decoding strategy and compare AGE against recently SOTA methods: DoLA (Chuang et al. 2023
OPERA (Huang et al. 2024)), VCD (Leng et al. (2024), Woodepecker et. 2024). LURE (Zhou
ct al. 2023). ICD (Wang et al. 2024c), and VISTA et. 2025).
Evaluation Metrics. We assess our method using suite of standard hallucination benchmarks, in line with prior studies (Huang et al. 2024 Leng et al.) 2024). (1) First, for image caption- ing, we employ the Caption Hallucination Assessment with Image Relevance (CHAIR) (Rohrbach) jet al. 2018). This involves generating descriptions for 500 COCO validation images and comparing them against ground-truth objects to measure hallucination at both the sentence (CHAIRs) and in- stance (CHAIR,) levels. To ensure responses remain faithful to the visual content, we also report an instance-lever level BLEU score (Papineni al. 2002). (2) Second, evaluate perceptual hallucinations using the POPE benchmark et. (002b)), which probes a model's ability to identify the presence or absence of visual concepts through 3,000 binary questions. We report Accuracy and F1 scores across its Random, Popular, and Adversarial settings. (3) Finally, we use the MME benchmark (Fu al. 2023). focusing on four hallucination--rlatee subtasks: Existence, Count, Position, and Color. Following the protocol of Yin et al. (2024), we report the overall accuracy as the evaluation metric.
Please see more details in Appendix A.4
Hyperparameter setting, In our experiments, interventions are applied at the 20th layer for the Middle stage and at the 30th and 31st layers for the Late stage. The scaling factors are set to A 100 for visual attention and A = 3 for textual attention. The number of M (samples to generate 8) is set to 10. Please refer to the Appendix A.7 for more details.
5.1 MAIN RESULTS
Results on COCO Image Captioning. To evaluate the effectiveness of our method on the caption- ing task, we adopt the CHAIR metric on COCO. Specifically, we conduct experiments on three rep- resentative models: LLaVA-1.5-77 (Liu et al. (023b), MiniGPT-4 (Zhu et al. 2023), and mPLUG- Owl2 (Ye et al. 2023). As shown in Table our method outperforms the latest state-of-the-art (SOTA) method by a margin of 2.85% and 0. on CHAIR& and CHAIR,, ,, respectively, demon- strating that our AGE substantially mitigates hallucinations during multimodal reasoning. Impor- tantly, our approach maintains comparable BLEU scores to the baseline models, even a 0.95% im- provement, indicating that the mitigation in hallucination is not achieved at the expense of response quality. It is worth noting that both LURE (Zhou et al.) 2023) and Woodpecker (Yin et al. 2024) rely on additional training data auxiliary models to mitigate hallucinations. In contrast, AGE achieves superior performance using a minimal set of only 10 images. This highlights that the improvements of our approach do not stem from external data augmentation, but rather from faithfully replicating the nuanced, stage-specific attention dynamics observed in real responses by imitating their attention behaviors. Please refer to the Appendix A for more results about different sizes of models.
Results on POPE. We further validate our method on the POPE benchmark. Following the proto- col of Leng et al. 20244, conduct experiments MiniGPT-4 (Zhu et al. 2023). As shown in
7

---

Published as a conference paper at ICLR 2026
AGE consistently achieves state-of-the-an art performance across all evaluation settings, sur- passing the baseline by an average margin of 17.09% in Accuracy while maintaining a comparable Fl score. is worth noting that, although OPERA (Huang al. (024) also intervenes attention, it primarily penalizes certain textual attention patterns. In contrast, AGE emphasizes the dominant shifts between visual and textual attention across reasoning stages. This distinction leads to substan- tial performance gains: AGE outperforms OPERA by average margins of 20.09% in Accuracy and 1.25% in F1, underscoring that interventions guided by real responses and targeted real attention behaviors yield more effective hallucination mitigation.
Table 2
Results on MME. Fig. reports results on the MME benchmark, where we follow the evaluation setup of Leng et al. (2024) and focus specifically on the ballucination subset. All experiments are conducted on I.I. VA-15-- (Liu et al., 2023b). Compared with VCD Leng et al. (004)) which mitigates hallucinations primarily by suppressing text bias, our AGE achieves superior performance at both the object and attribute levels. This suggests that merely reducing textual priors is insuff- cient; effective hallucination mitigation requires faithfully aligning text and visual attention with the dynamics observed in real responses across different reasoning stages.
Table 3: Ablation study of AGE on
Evaluation with POPE in random, popular, and adversarial settings. We report the accuracy($)) and FI score(%). The best results are highlighted in bold. i represents the results from Chen et al. (024dd)
Table 2:
CHAIRs (C'), CHAIR; (C2), and BLEU. SMA: meltiplltipiicaiive ampliffcaaaa of at- tention; AGEr: text attention intervention; AGE,: image attention intervention. M az new Lokens is 128. Best results are in bold.
amplification
Methoos
Random
Popular
Adversarial
Avp.
1
F17
Ace
Fl
F1?
Acc
SMA
AGET
AGE,
Cs 4
Cr
BLEU 1
Gredd
55.33
68.69
54.00
67.76
56.77
69.32
53.4
14.2
10.5
OPERA
Seacc
Hougg 2004
69.86
69.97
65.71
51.00
50.35
57.33
66:42
6521
66.21
52.67
52.00
53.67
07.58
53.44
53.77
67.68
64.35
68.12
/
43.1
50.4
14.9
13.1
10,4
10.1
71.88
74.85
68.73
76.23
67.81
75.66
35.4
10.9
10.4
31.8
10.5
5.2 FURTHER ANALYSIS
Contribution of Each Intervention. We conduct ablation experiments on the COCO dataset with LLaVA-1.5. By incrementally enabling individual intervention and reporting CHAIR and BLEU scores, we examine how each intervention affects model performance. To further test robustness under different output settings, we increase the maz new token to 128, double the 64-token set- ting in Table D. thereby validating the method's effectiveness across varying generation lengths. As shown in Table SMA refers to visual attention intervention via seff-multtplicattive augmenta- tion, a straightforward strategy that directly amplifies the original attention values. AGEEy denotes text attention intervention through direct amplification of attention values, and AGE, represents our directional vector-based visual attention intervention. Relative to the baseline, SMA improves CHAIRs by 10.3%, showing that even simple amplification of real attention behaviors can bring notable gains. More importantly, our proposed AGE, achieves a further boost, raising CHAIR from 10.3% to 18.0%, which highlights that precise, vector-guided interventions are substantially more effective than coarse, direction-agnostic scaling. In addition, applying AGE, alone yields 3.0% improvement, indicating that imitating text attention patterns guided by real responses is like- wise effective. Finally, combining AGE, and AGEr yields the best overall performance, indicating that the two approaches offer complementary benefits. These results validate not only the effective- ness of each intervention in isolation but also the synergistic improvements achieved when applied together. Please refer to the Appendix A.5 for more results about the different intervention layers.
Ablation Study on Sample Size M. In Sec. 4.1 we generate the target directional vector 5 from M sampled instances. Here, we analyze how different values of M affect CHAIR and BLEU on LLa 1.5 with COCO, as shown in Fig.5) Regardless of the value of M (from to 200), AGE con- sistently outperforms the baseline on both CHAIR and BLEU, indicating that the target directional vector 5 can be reliably estimated with only small number of samples, without requiring large- scale aggregation. This confirms that the performance gains of AGE do not stem from introducing additional samples. Moreover, as M increases, CHAIR scores show a slight increase, suggesting that excessive samples introduce noise and reduce the accuracy of ö. Balancing performance and computational efficiency, we set M = 10 in our experiments.
8

---

Published as a conference paper at ICLR 2026
200
30
Regular
CHARRS
VCD
CHAIR
MME Score
180
ICD
Ours
20
Score
160
140
120
Existence
Count
Position
Color
Avg.
5
10
20
50
100
200
Figure 4: Evaluation MME benchmark across the 'Existence', "Count', 'Position', 'Color', and 'Avg'' settings. Best results are bold.
Figure 5: Impact of varying the sample size M for generating 8 model performance, evaluated with CHAIR (%) and BLEU (%).
Table 4: Comparison of CHAIR (%), and BLEU (%) with different variants of AGE. AGE ô generated from the cor- responding intervention layer. AGEM: Intervention in all Middle stage layers. AGE,; Intervention in all Late stage layers. The best results are highlighted in bold while the second-best results are marked with underline.
Variants of AGE. To further investigate AGE, we de- sign several variants based on LLaVA-1.5, as shown in Tab. R Specifically, AGED denotes using 8 generated from the corresponding intervention layer, AGEM applies interventions to all Middle-stage layers, and AGE, inter- venes all stte-stage layers. AGEp outperforms the bascline, confirming that computing ò within the corre- sponding layer is effective. However, its performance is still weaker than that of AGE obtained from the final output layer, suggesting that the attention state closer to the output carries richer visual grounding information. In contrast, AGE underperforms the bascline, indicating that coarse, uniformly enhancing textual attention across all Middle layers degrades model quality. We attribute this the fact that attention disparities between real and hallucinated responses evolve dynamically across layers; thus, indiscriminate interventions can distort attention patterns instead of aligning them. A similar trend is observed with AGE,, although applying interventions to all Late-stage layers yields slight improvements over the baseline, remains inferior to the selective interventions AGE. Overall, these findings underscore that targeted, fine-grained interventions, not blanket modifications, are key to effective hallucination mitigation.
Method
Cs
C₁
BLEUt
Baseline
20.80
6.77
15.93
AGEp
19.40
5.86
16.69
AGEM
28.40
8.64
16.46
AGEz
19.20
6.22
16.73
AGE
16.43
5.58
16.48
Visualization of Responses. To more intuitively demonstrate the effectiveness of our AGE, duct visualization experiments on LL with the query "Please describe this image in detail." As shown in Fig. 6 the baseline model hallucinates objects (red), such as "a dining table" which is absent from the image. In contrast, responses generated with AGE are more faithful and precise (green), e.g., correctly noting that "broccoli occupies most of the bowl." These results indicate that AGE, by imitating truth-grond outh-grounded attention behaviors, substantially enhances both the factual ac- curacy and descriptive richness of generated captions. Furthermore, we visualize the cross-moda. attention difference (a (5 vision) - (1)
(realhall across layers for real (blue) and hallucinated (red) tokens, and compare them with responses produced after applying AGE (green). The visualiza- tion reveals clear disparities between real and hallucinated tokens in baseline LLaVA, whereas AGE narrows this gap, aligning generated responses more closely with real, hallucination-fiie attention patterns. This alignment not only validates our analysis in Sec. 3.3 but also demonstrates that imitat- ing real- real-response attention enables more faithful multimodal reasoning and significantly improves response quality. Additional visualizations for MiniGPT and mPLUG-OWl are provided the Ap- pendix A.9
Sec.3.33
6 CONCLUSION
In this paper, we introduce AGE, a novel attention intervention approach designed to mitigate hal- lucination by imitating the attention behavior of real responses. In this way, capture the fine-grained dynamics of multimodal reasoning and improving response quality with less hallucana- tions. Extensive experiments across three benchmarks and three LVL.Ms validate the effectiveness

---

Published as a conference paper at ICLR 2026
Baseline
Model
aouistant, gives hopri, detailed, and polki annawess the user's qu modione LSfiit: vimagas Preese deeraiio image
Deal Desparse
Hallucintioo Response
withooa gence rosietent
u22 start gios he heu, desailod, and pollio
0.2
answe's BiD uso's questors. USER,
<Iragee mass desrrioe vraggn filed
ASSISTANT ThThe mage featurcs filed
00
consaiingg Searns Insezaal and pocedtt other imgetaciees I'e bost BlSeN on mane and the boons sottered Phoouhhout
-07
including Boans, brocool and posstaty other vegutadi hos The Bransiil to the muse
-0.4
D0000 throughout
12
36
20
24
28
The corerdd and
Figure 6: Visualization of responses and cross-modality attention difference for the baseline model LL. versus our AGE. Red and blue text/line highlights hallucinated and real tokens generated by the baseline, respectively. while green text/line marks the calibrated tokens produced by AGE.
of our approach. We hope this work provides new insight into the internal dynamics of multimodal reasoning, offering principled path toward building more reliable LVLMs.
7 ETHICS STATEMENT
This research was conducted in accordance with established ethical standards for scientific inquiry. We carefully considered issues related human subject involvement, dataset usage and release, potential societal harms, research methodologies and applications, conflicts of interest, fairness and bias, privacy and security, legal compliance, and overall research integrity (e.g.. IRB approvals and ethical documentation). Our study does not involve human participants or personally identifiable information, and thus no Institutional Review Board (IRB) approval was required. All datasets em- ployed are publicly available and distributed under appropriate licenses. Potential risks-including bias, fairness, privacy, and unintended misuse-were critically assessed, and steps were taken to minimize such concerns. We affirm that this work adheres principles of research integrity, ensur- ing accuracy, transparency, and reproducibility.
REPRODUCIBILITY STATEMENT
We have taken extensive measures to facilitate the reproducibility of our findings. The main pa- per provides detailed descriptions of the model architecture and methodology. while the appendix further documents implementation specifics and hyperparameter settings. All datasets used in our experiments publicly available. To ensure faithful replication, release our source code, config- uration files, execution environment, and preprocessing scripts in an anonymous repository, enabling researchers to reliably reproduce our experiments.
ACKNOWLEDGMENTS
The authors would like thank the anonymous referees for their valuable comments. In this work, Hairui Ren, Zixuan Wang and Dandan Guo are supported by the National Natural Science Foun- dation of China (No. 62306125). Yi Chang is supported by the National Key R&D Program of China under Grant (No. U2341229). This work was carried out during Dandan Guo's visit to King Abdullah University of Science and Technology (KAUST).
the National Natural Science Foundation of China (No.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qian Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations large vision-language models with assembly of global and local attention. arXiv preprint 2rXiv:2406.1277 2024.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar3iv 12966, 1(2:3, 2023.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He. Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint
arXiv:2404. 18930, 2024.
Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. let: -ojaee- cross-level trusted intervention for mitigating object hallucination in large vision- lision-language models. arXiv preprint arXiv:2411. 2024a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306.15195 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Confer- ence on Computer Vision, pp. 370-387. Springer, 2024b.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with Ilms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 14093-14100. IEEE, 2024c.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint vrXiv:2403.00425, 2024d.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% a chatgpt quality. See https://vicuna, Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03888 03883, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 2023.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 14385,
June 2024.
Wenyi Hong, Weihan Wang, Ming Ding. Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408 16500, 2024.
Mingzhe Hu, Shaoyan Pan, Yuheng Li, and Xiaofeng Yang. Advancing medical imaging with language models: A journey from n-grams chatgpt. arXiv preprint arXiv:2304.049200 2023a.
Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311. 2023b.

---

Published as a conference paper at ICLR 2026
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection alllocatioo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
Chaoya Jiang, Haiyang Xu, Mengfan Dong. Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2024.
Recognition, pp.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 13872-13882, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Zhuowei Li. Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. arXiv preprint arXiv:2502.03628, 2025.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigat- ing hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34911 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 36296-26306 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge. 2024b.
Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful lideo-language model supporting long-context video input. arXiv preprint arXiv:2408. 15542, 2024c.
Sheng Liu, Haotian Ye, Lei Xing, and James Zou. Reducing hallucinations in vision-language models via latent space steering. arXiv preprint arXiv:2410.15777 2024d.
Jinjie Mai, Jun Chen, Guocheng Qian, Mohamed Elhoseiny, Bernard Ghanem, et al. Llm as a robotic brain: Unifying egocentric memory and control. 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Woohyeon Park, Woojin Kim, Jaeik Kim, and Jaeyoung Do. Second: Mitigating perceptual hallu- cination in lasioanlanguage models via selective and contrastive decoding. In Proceedings of the 42nd International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, PMLR, 2025.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv: arXiv:1809.000 1809.02/66 2018
12

---

Published as a conference paper at ICLR 2026
Min Shi, Fuxiao Liu, Shihao Wang. Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang. Hongxu Yin. Karan Sapra, Yaser Yacoob, et al. Eagle: Exploring the design space for multimodal Iims with mixture of encoders. arXiv preprint arXiv:2408. 15998, 2024.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Open and efficient foundation language models. Preprint at arXiv. https://doi. 0rg/10.48550/am
2302(3), 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Milm can see? dynamic correction decoding for hallucination mitigation, 2024a. URL
htttps:/aarxie org/ abs aws///110000 2410. 11779
Sheng Wang, Zihao Zhao, Xi Ouyang, Tianming Liu, Qian Wang, and Dinggang Shen. Interac- tive computer-aided diagnosis on medical image using large language models. Communications Engineering. 3(1):133, 2024b.
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint 1877:5 2024c.
Jinge Wu, Yunsoo Kim, and Honghan Wu. Hallucination benchmark in medical visual question answering. arXiv preprint arXiv:2401.05822 2024.
Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. Nullu: Mitigating object hallucinations in large lasion-language models via halluspace projection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025a.
Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Mitigating hallucination in large vision-language models via modular attribution and intervention. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2025b.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan. Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. Science China Information Sciences, 67(12):220105, 2024.
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12944-12953, 2024.
Zihao Yue, Liang Zhang, and Qin Jin. Less more: Mitigating multimodal hallucination from an cos decision perspective. arXiv preprint 2xXiv:2402.14444 2024.
Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q. Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis-Philippe Morency, Katia P. Sycara, and Yaqi Xie. Self-correctng de- coding with generative feedback for mitigating hallucinations in large vision-language mod- cls. The Thirteenth International Conference on Learning Representations, 2025. URL
https:: /tts:///ppeeeeeeeweee net. Forum? http://ommmmmmmmmeeeeeee d=t..deePR..K...
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large lisiuaelanguage models. arXiv preprint 4rX04:2310.00754, 2023.
13

---

Published as a conference paper at ICLR 2026
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing usiio-langusl understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Xin Zou, Yizhou Wang. Yibo Yan, Yuanhuiyi Lyu, Kening Zheng, Sirui Huang, Junkai Chen, Peijie Jiang, Jia Liu, Chang Tang, and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. Forty-second International Conference on Machine Learning (ICML), 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LLM
In this work, we utilize Large Language Models to perform grammatical verification.
A.2 ALGORITHM
Algorithm Workflow about AGE method for hallucination mitigation
Require: Intervention Image Attention Layer Set Limage Intervention Text Attention Layer Set Luxu, Hallucination samples M
1: Construct hallucinated tokens set z(())
2: Construct a real tokens set ((a))
3: Compute hallucinated average image attention vector (hall.vision)
ExErcO (G)
a
(L,k) vision
14
2all
4: Compute real average image attention vector af visiin)
Σ ke (10
a ayssion
5: Compute the sample weight w/ = ={ {{exp( (-9 (C² T)),
= [C//T))
if if <
if C TT,
6: Generate target directional vector = M Ei=1 Wi (a (a vision) a hall, vision) for each decoding step & do for each decoding layer do if layer / in Limage then Imitating Image Attention / a + Au 8 end if
7:
8:
9:
11:
if layer / in Lust then Imitating Text Attention aLext = afext + de x afext
13:
end if
15: Compute
next-layer
hidden (I.A)
state
h
(1+1)
AttentinSSuLaayer vision
(I.A) (
vision 4 v((k))
16: end for
17:
end for
A.3 ADAPTIVE WEIGHT STRATEGY
In sec. 4.1 calculate the target directional vector d with M samples. However, assigning equal weights overlooks that different samples, due to their varying proportions of hallucinated content, contribute unequally to the corrective signal. To address this, introduce Adaptive Weight Strategy using sample-specific weights w, based on the CHAIR instance-level score (CH). The key intuition is that samples with a balanced ratio of real-to-hallucinale richer comparative signals, making them more informative for constructing the difference vector. Conversely, samples heavily dominated by hallucinations (C) > T) can inject misleading bias into 6, and should therefore be penalized. A piecewise weighting function is adopted to reflect this asymmetric importance: one branch smoothly increases the weight as Oj approaches 7 from below, while the other rapidly decreases it when Cj exceeds the threshold, ensuring that excessively noisy samples do not dominate the correction: We define w;
tokens (Cj т) contain
ifC <T. W; = Teep (-q (-q (C; if 2T.
[CH/T))
exp
where 7 denotes the balance threshold. The exponent governs the amplification rate for balanced samples, and q controls the suppression strength for cntteciinniionddmnnattt ones. In our imple- mentation, p and 30, determined empirically small held-out validation
15

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of CHAIR and BLEU with different image attention intervention layers. For VL, we do not intervene with image attention in the Middle Stage; only intervene with text attention needed, while other models intervene all Stages. The best results are in bold.
MiniGPT-4-7B
-LaVA-1.5-7B
mPLUG-OW12-77
Method
Cs +
C₁
BLEUT
Cs 4
C₁ J
BLEU
Cs +
C₁ J
BLEUt
baseline
30.87
12.33
14.33
20.80
6.77
15.93
23,20
8.33
15.37
1,2
8.80
4.73
12.38
0.00
0.00
1.03
22.20
8.55
16.12
3,4
2.20
3.31
4.93
19.00
6.05
16.60
20.60
7.84
16.29
5,6
2.00
15.85
3.68
0.20
1.03
2.96
20.40
8.15
16.25
7,8
14.60
9.71
14.82
20.00
6.63
16.70
19.20
7.70
16.30
9.10
13.80
9.44
14.60
19.00
6.36
16.50
21.80
8.20
16.34
11,12
0.20
2.12
0.56
17.80
5.87
16.17
23.20
8.21
16.23
13,14
18.80
7.61
15.64
17.80
5.96
16,43
22.60
8.66
16.38
15,16
11.20
4.00
15.37
19.40
6.66
16.52
23.00
8.50
16.29
17,18
15.60
5.57
16.99
22.20
8.06
16,24
19,20
15.80
6.12
15.78
20.80
7.56
16.30
21,22
18.20
7.52
15.54
21.40
8.02
16.26
23,24
17.20
5.78
16.12
21.20
7.97
16.29
25,26
17.00
6.13
15.68
22.00
8.18
16.26
27,28
15.91
6.19
15.86
18.00
6.10
16.60
21.40
8.03
16.26
29,30
19.00
6.76
16.05
17.40
5.75
16.64
21.40
7.99
16.32
Ours (30,31)
15.62
6.00
15.79
16.43
5.58
16.48
19.40
7.47
16.21
A.4 EVALUATION METRICS
The CHAIRs and CHAIR, scores computed by comparing the model-generated answers with the ground truth object annotations as
|hallucinated objects}| [all mentalionde objects)[]
Meoptions with hallucinated objects]]
CHAIR =
CHAIRs
[Hlll caplions]l
(10)
In addition. following the standard BLEU definition (Papineni et al., 2002). we incorporate an instance-level BLEU score evaluate whether the generated descriptions faithfully capture the nec- essary visual content from the image as:
BLEU BP exp w,logpn)
(11)
where pn, is the precision n-grams between the generated and reference captions, w,, is the weight assigned to n-grams (typically w, 1,) and BP denotes the brevity penalty to penalize overly short predictions. We report the average BLEU score, computed as the mean of BLEU-1 through BLEU-4
A.5 INTERVENTION ON DIFFERENT LAYERS
In our experimental setup, interventions on Late-stage image attention are applied to layers 30 and 31, while Middle-stage interventions target layer 20. This naturally raises the question: what hap- pens if interventions are applied to other layers? To investigate, we conduct further exploration.
As shown in Tab. intervening in Early-stage layers (c.g., 1, 2, 5, and 6) produces extremely low C's and C7 values-neart zero zero- Surg-sugggsning strong hallucination suppression. However, this comes at the cost of sharp drop in BLEU, as the model fails to generate meaningful text and instead outputs repetitive or corrupted tokens (e.g.. long strings "nobody" or garbled symbols like "&&#"). We attribute this to the role of carly layers, which primarily capture low-level features of images and text. Although differences in attention exist between real and hallucinated responses at these shallow layers, they do not reflect the high-level semantic inconsistencies that drive hallucinations. As
16

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of CHAIR and BLEU with different text attention intervention layers in
LLa VA. The best results are in bold.
Layer
Cs
C₁
BLEU
Baseline
20.8
6.7
15.9
16
20.6
7.0
16.0
17
22.6
7.5
16.8
18
19.0
6.3
16.5
19
20.0
6.5
16.2
20
16.4
5.5
16.4
21
20.0
7.1
16.7
22
18.8
6.0
16.8
23
17.0
5.7
16.5
24
20.0
6.4
16.5
25
19.8
7.1
16.7
26
19.4
6.5
16.8
Table 7: Comparison of CHAIR and BLEU with LLaVA-1.5-13B as baseline. The best results in bold.
Method
Cs 4
C7 4
BLEU
LLaVA-I.5-13B
48.60
12.38
10.57
AGE-I3B
45.40
12.55
10.75
result, interventions here disrupt the extraction of essential features, leading the model to deviate during later inference. Therefore, we avoid applying interventions in shallow layers.
In the Late stage, we compared interventions applied to layers 27-28 with those applied layers 30-31. When targeting layers 27 and 28, Css and C₁ decreased by 2% and 0.67%, respectively, while BLEU improved by 0.67%, indicating that ballucinations are partially mitigated. However, the overall performance remained inferior to interventions at layers 30 and 31. We attribute this to the proximity of layers 30 and to the output: the closer the intervention to the decoding output layer, the more directly it influences the model's final predictions. Consequently, intervening at layers 30 and yields the most substantial benefit, and we adopt them as the Late-stage intervention layers.
For the Middle stage, we experiment with layers 19, 20, 21, and 22, as shown Tab, 6 All tested layers reduce hallucinations varying degrees, and at layer 22, BLEU reached high as 16.87. This consistency arises because Middle-stage layers emphasize textual attention, and rein- forcing it-regardless of the exact lmyer-improves performance. The relatively stable gains across different layers demonstrate the robustness of our approach: interventions need not be tied to single fixed layer but rather stage interval, within which similar benefits can be achieved. In practice, we select the best-performing candidate (e.g., layer 20) as our Middle-stage intervention layer.
Taken together, these findings highlight that interventions in shallow layers are detrimental, Middle- stage layers yield broadly consistent improvements by reinforcing textual attention, and Late-stage layers closest the output provide the strongest influence on hallucination mitigation. Accord- ingly, we adopt layer 20 for Middle-stage intervention and layers 30-31 for Late-stage intervention, striking a balance between stability and effectiveness.
A.6 INTERVENTION ON LARGE-SCALE LVLM
To evaluate the scalability of approach, conducted experiments LLaVA-1.5-13B (Liu ct al. 2023b). As shown Tab. 7 AGE effectively reduces hallucinations in larger models, yielding 3.2% decrease in Cg and 0. 18% in BLEU, while C₁ shows only marginal rise of 0. 0.17%. These
17

---

Published as a conference paper at ICLR 2026
results highlight that hallucinations caused by attention misalignment persist across model scales, and demonstrate the cross-scale generalization and robustness of AGE in mitigating such errors.
25
25
Jaselre
Barelinn
20
20
15
115
10
2
80
90
100
110
120
Figure 7: Comparison of CHAIR and BLEU with different values of X (left) and Xn (right).
A.7 HYPERPARAMETER
To assess the sensitivity of AGE hyperparameter choices, we use LLaVA-1.5-7B as the baseline and vary A from to and A from 80 to 120, reporting the corresponding CHAIR and BLEU scores. As shown in Fig. 7 performance consistently surpasses the baseline across all settings, suggesting that once the correct intervention mode is applied, hyperparameter variations have limited impact on overall gains. We therefore set X and X 100 as the default configuration in our experiments.
Attention Difference
Diff
Diff
Attention Difference
0.03
0.02
Diff
Diff
Difference
0.02
Diffi
Diffber
0.01
0.01
0.02
0.00
0.00
-0.01
-0.01
0.00
-0.02
-0.02
-0.02
-0.04
-..03
-0.04
-0.03
16 20 28
-0.05
16 20 Layer index
12
24 26
16 20 24
Layer Index
Layer Index
(a) Num: 10
(b) Num: 100
(c) Num: 1000
Figure 8: LLaVA-1.5 layer-wise characterization of attention disparities between real and halluci- nated responses with different numbers of samples N.
A.8 ANALYSIS MORE SAMPLES
In Sec. 3.3 examined attention disparities by sampling N 100 responses and concluded that hallucinations do not stem from a uniform lack of visual attention, but rather from the failure reproduce nuanced, model-specific, and stage-dependent attention dynamics. A natural question arises: does this conclusion hold under different sample sizes? To test this, we compare the attention behaviors of real and hallucinated tokens LLaVA-1.5 with N 10, 100, and 1000 samples. As shown in Fig. g while the detailed attention curves fluctuate with sample size, the dominant modality at each reasoning stage remains consistent, This robustness further confirms the reliability and generalizability of our statistical findings.
A.9 MORE VISUALIZATIONS
As shown in Fig. 12 we provide additional visual comparisons between the baseline models and AGE the image captioning task.
A. VISUALIZATIONS OF DISTINGUISHING BETWEEN HALLUCINATION AND REAL OBJECTS
We illustrate the full procedure for distinguishing hallucinated objects from real ones in Tab. g Specifically, we first extract objects from the LVLM's output and from the ground-truth labels. Then, leveraging COCO's synonym vocabulary, match the sets identify overlaps, thereby
18

---

Published as a conference paper at ICLR 2026
separating hallucinated from real objects. The detailed steps as follows: Following prior work (Rohrbach et al. 2018). we use the COCO ground-truth object annotations as supervision. The procedure is: (1) Extract noun tokens. We first identify all noun tokens in the model's gen- erated response using POS-tagging parser. Only nouns are considered candidate visual entities. (2) Map each noun to object categories. Each noun is normalized lowerrased. lemmatized, and matched to COCO object categories using synonym and alias dictionary (the same strategy as prior CHAIR-style evaluation). This allows us automatically determine whether generated noun corresponds to a visually grounded category. (3) Compare against ground-truth annotations. If noun (or its synonym) appears in the COCO ground-truti object list for that image, it is labeled as real token; otherwise, it is labeled as hallucinated token. (4) Fully automated pipeline. No human annotators participate token-level labeling. The entire process-nose noun extraction, synonym matching, and GT comparison- -is fully automated and reproducible.
A.11 DOMAIN SHIFT GENERALIZZATIN
We conduct extensive domain-shift experiments across multiple LVLMs. Specifically, directly apply the å computed on COCO to five out-of-domain settings, including Medical (Wu et al. 2024). Video, Math, Map. and Table (Guan al.) 2024). which differ substantially from natural images. For the medical benchmark, we follow Wu et al. (2024) and report Accuracy under FAKE and SWAP settings; for the remaining domains, we report Accuracy/F1 following Guan al. (024)). As shown in Table AGE consistently improves over the baseline across all domains and architectures (e.g., +3.2% in Medical for LLaVA, +3.8% in Math for MiniGPT-4, +1.6% Table for mPLUG-OwI2), signal encoded by and the
These results demonstrate that the calibrated hallucinationntto-rae chosen stages remain effective even under severe distribution shift, indicating strong robustness and transferability rather than datasct-specific tuning. Our domain-shil appriments-cooveee coverrin medical images, math diagrams, tables, and video without object-level supervision.
that AGE generalizes well even in tasks
A.12 LATENCY AND THROUGHPUT
To assess the computational efficiency of AGE, we report both latency and throughput across three different LVLM architectures (Table I0). The results show that AGE introduces only negligible latency overhead while maintaining high throughput on all tested models.
A.13 3
A DYNAMIC VERSION OF AGE
To completely eliminate dependencies on manual layer selection and hyperparameter tuning, we further develop Dynamic AGE (see Appendix A.13). This variant replaces fixed settings with adaptive mechanism that monitors the live attention gap at every decoding step. The procedure consists of three phases:
Constructing Reference Profiles (Offline). We first aggregate attention statistics from the M auxiliary samples build "Hallucination Reference Profile," representing the typical attention state of hallucinated tokens: V*sisual Reference Vector (0(halt)):* The weighted average of visual attention vectors from hallucinated tokens:
Phase I:
M
((hall = M
Σ wi (Aipall) 1=1
Textual Reference Sum (s)hat)) ): The weighted average of attention sums allocated to text tokens in hallucinated responses:
8 thall)text) = .M Σ ws ((hall.tezt))
o
Phase 2: Calculating Dynamic Gaps (Online). During inference, for each layer I and decoding step k. we calculate how much the current attention state deviates from the hallucination profile: Visual Instantaneous Deviation (62): We define da the vector difference between the current
19

---

Published as a conference paper at ICLR 2026
I.k visual attention Dision token:
and the hallucination reference. This captures the *live* drift of the current
8
(ll
1.k vision "hhalllession
a
Dynamic Visual Scaling (g.): We determine the intervention strength by normalizing the magnitude (RMS) of this instantaneous deviation (6a) against the global calibration vector (6 ,े MM
W,
viaeeaivisionn visiiiiis derived in Eq. 6). This ratio quantifies how "strong" the current deviation is relative to the expected correction direction:
RMS ())
g.
RMS(6)
Dynamic Textual Scaling (g:): Similarly, we measure the ratio of the current text attention sum to the hallucination reference sum:
g. (1.k) Steat
teall))e
Phase 3: Adaptive Calibration. Crucially, intervention is triggered only when the dynamic gap implies significant mismatch with truthful patterns (i.e., exceeding a statistical threshold r). The model automatically self-calibrates using the computed dynamic strengths:
Visual Calibration: If the dynamic gap gh exceeds Tus we inject the global correction vector á scaled by the live strength of:
Distonn dession + 8,
if > T
Textual Calibration: If the textual attention ratio of exceeds Ti (indicating a need for stronger context grounding), we perform self- saff-muutiplicative enhancement:
atext abezt + gi aleats if Te
Result: This mechanism enables the model to automatically determine both which layers to inter- vene in and how much modulation strength apply at each step (via g. g)), without any manual tuning. Following the logic used for X and A we initialize their base values to 100 and 1, re- spectively. The terms Ty and 7; capture the instantaneous discrepancy between the current token's attention and the hallucination pattern, and simultaneously act as dynamic weights that guide the cal- ibration strength. We evaluate this dynamic variant of AGE in Table TT. While AGEamiiiic offers strong flexibility in both layer choice and weighting, its slightly higher Css and C₁ scores-relate to manually tuned AGE AGE-suggest a modest performance trade-off. We attribute this to the more aggressive and unconstrained adjustments occasionally interfering with the preservation of low- level visual or textual features, reducing the net benefit of semantic-level calibration. Nevertheless, AGE Aynamiii remains substantially better than the baseline across all metrics, demonstrating that the core mechanism of AGE is inherently robust, adaptive, and transferable, even without manually specified parameters.
A. AN ONLINE VERSION OF AGE
Fully online estimation promising. However, without ground-truth supervision during inference, online estimation is challenging. Since the model cannot reliably distinguish real tokens from hal- lucinatory ones, making impossible to compute an accurate vector as we do offline. our original submission, we had devised a more direct online calibration mechanism that avoids using 5. Instead of relying on real-hallucination discrepancies, the online variant applies self-multiplicative enhancement to both image and text attention, allowing the model to internally amplify reliable signals. As shown in Table I2, this online version does not match the performance of the offline d-guided method, but it still achieves consistent and notable gains over the baseline, demonstrating that effective hallucination mitigation is possible even without offline supervision. These results further support the core motivation of our work: The model benefits from imitating truth-aligned at- tention behaviors, and this principle remains effective even when the calibration is performed online without explicit 5.
20

---

Published as a conference paper at ICLR 2026
Query: Please describe this image in detail.

A chat between a curious user and artificial intellgence assistant The assistant gives helpful, datailed, and polite ansvers to the users
LLaVA-1.5:
questions. USER: <lmage> Please describe this image indetal.. ASSISTANT:
The imge features large, orntt clock mounted on a wall, with reflaction of lenal, which side scene. reflection of the clock the mirror crestes en interesting visual effect, making the clock appeer
the clock visible mirror. The clock positioned
acated
A chat between curious user and artificial intelliencee assistant. The assistant gives helpful, detaled, and polis answers to the user's questions. USER: <lmage>> Please describe this image indelail. ASSISTANT: The image featuree large, old-fashioned
AGE:
positioned the middle of the wall, and appears to be a praminent feature in the room.
The clock surrounded by few other decorative elements, including &mall ngght side the
Figure 9: Generated image description examples.
A.15 COMPARISON WITH MORE METHODS
Some studies (VTI (Liu et al. 2024d Nullu (Yang al. 2025a). AGLA (An et al. 2024). Deco (Wang et al. 2024a), and DeGF (Zhang et al. 2025)) have similarities with our AGE, so further compare and analyze AGE with these studies. VTI (Liu et al. 2024d), which mitigates hallucinations by stabilizing visual representations through noise-perturbed image augmentation: Nullu (Yang al., 2025a), which contrasts truthful and hallucinated responses at the sample level and intervenes in weight space; DeGF (Zhang et al. (025),, which leverages text-to-image generative feedback to guide correction; AGLA (An al. (024). which assembles global and local attention to better capture object semanties; DeCo (Wang et al. 2024a), which dynamically corrects decoding by aligning confidence patterns across layers.
We directly compare AGE with AGLA, DeCo, and DeGF, and also integrate AGE into each system. As shown in Table 13 AGE alone achieves the best or comparable performance, and when combined with AGLA/DeCo/DeGIF, it consistently improves them in most of settings. This confirms that AGE introduces non-overlapping, complementary gains, supporting that our layer-wise, truth-guided at- tention imitation captures fundamentally different source of hallucination. (2) Additionally, originally did not compare AGE with HALC (Chen et al. 2024d) because HALC relies on an exter- nal detector (Grounding DINO), which introduces additional supervision beyond our method. For completeness, we now include HALC in Table [3 The results show that AGE-despite requiring auxiliary modellsstil achieves SOTA performance. This further demonstrates the effectiveness and independence of our hallueinatioon-o--ral----- include two visualization studies Fig. 14 and Fig. 15. As the results show, DeGF, DeCo, and AGLA all hallucinate the same nonexistent objects- such as the "chair" in Fig. [4 and the "wine glasses" and "cups" in Fig. [S] contrast, AGE produces descriptions that are both more accu- rate and more detailed, offering a clearer and more intuitive demonstration of its effectiveness in reducing hallucinations.
calibration. (3) For qualitative analysis, we
In summary, AGE differs from prior approaches by (1) identifying the token-level, stage- spec-fie attention patterns that characterize truthful responses, (2) revealing model-dependent text-image attention requirements, and (3) intervening directly along these dynamics. These insights form the core conceptual novelty of our work.
A.16 FAILURE CASE ANALYZE
We further examine the failure cases-images that still exhibite hallucinations after applying AGE, and observe a consistent pattern: as shown in Fig. 13 AGE's calibration ability diminishes in scenes containing a large number of distinct small objects. In such complex settings, the abundance of visual instances increases the difficulty of accurate grounding, leading the model to continue pro- ducing hallucinations. We hypothesize that these cases require more fine-grained extraction and reinforcement of visual cues, which offers clear direction for future improvements.
21

---

Published as a conference paper at ICLR 2026
Query: Please desoribe thes mage detall
<anA chal beiwwen curious and arifiii inteligence ansialant The asdiste's givia helpfl detailed and pollo enswers the user's questions USER <Images Please describe this imige Indetall ASSISTANT: The imace features snowy landsoape with snowboard lying the ground partially covered by green tarp. The snowboard postiinned the center the scene. and the tarp spread out amannd Banggin them
LLaVA-1.S: 5::
pecple
smool
betweon curious and ardiical intlligenc
gives helptul.
poite
questions, <imagen Please image indetall
ASSISTANT
postioned cooupying signfcantparioon surfece. snowboard background, sha the
Figure 10: Generated image description examples.
Query: Please describe this Image detail
MiniGPPT4: Image
boets
docked at the shore. The river is surrounded by buildings and trees, and the sky clear and blue. The boats are moored at the dock and there and still,
are lied
image shows view city bridge. city on the barkes of river and has a number of bosts the water. The buildings the city made of while and gray concréte and The",
sky light gray
Figure 11: Generated image description examples.
Query: describe image delail.
mPLUG-OwI2: image depicts
street
traveling
down the tracks, passing through a city with tall buildings. The train
positioned middle of the occupying significant partion of
image./illeee WTere are several people visibbl on the street, some closer
the Irain others furthel
AGE:
The image depicts City street with a train traveling down the passing
buildings. The positioned the middle
the scene, occupying significant portion of the image. The buildings
surrounding the train various heights, creating dense urbani almosphere inin There are
Figure 12: Generated image description examples.
Baseline Model
Ours AGE Model
A chat beiteee a tariteu cser and antiical rrtelligena
detailed, siellignnc
assistant, The assistant gives hepru, dotallos, and polito answers to
assiitant
quaedions simagai desecib inage
cucstions, USER: simage Please desorbe this image
Indetail, ASSISTANT.HH image features Bitnol large standing
ancwbedl ASS ST ANT: The lestumee fllmmi with larpe.
refrgerstor
metallic refigrrator treezor stancing in the middl the foor. The
refizaentor suumorraee by varkous famms indludin a fow fislinal and
verious indluting
There
There are also coople Boon and tpos p placed the
-SoV char beWcor curlous Use and ontiical mollliennc
betwoon
artilial Inteligence
waabeber. The woodelrt gleen helplii chtoebed and coliee anseents to
meriaaani The Manidaar gine helpfd, detained and pod arnonee
the use''s gunstions. USER: dimages Please desoribe thes mage
the user's quections, USER: <image> Please descrboe this image
inslstai,, ASSISTANT
kptop Bommulie (iall siinng ai
maga
indetalll ASS STANT: The imagu features a fiail siting at Biail in
front of Biater communee He weering
antilived repworss
working of using comcunal dosk clutored
Heme, including
There
Figure 13: An example of the failure case in AGE. Real token is marked green, hallucination is marked red.
22

---

Published as a conference paper at ICLR 2026
Table 8: An example to demonstrate how to distinguish between hallucinatory objects and real objects.
Instruction
Please describe this image in detail
LLaVA Output
The image features two boats floating on a body of water, possibly the ocean. One boat is larger and positioned towards the left side of the scene, while the other boat is smaller and located closer to the center. The smaller boat appears be speedboat, while the larger boat yacht. There are three people visible in the scene, with one person standing on the larger yacht and two others on the smaller speedboat. The people seem to be enjoying their time on the water, possibly engaging in recreational activities or simply enjoying the view.
Generated Objects
boats, speedboat, yacht, people, person
Ground Truth Caption
an old photo os small boat near bigger boat in a large body of water; Two boats being spied on from distance; One large and one small boat floating in the ocean; One boat sailing next to one canoe in a body of water; two white boats next to each other in the water;
Ground Truth
boat
Objects
Hallucination
people, person
Objects
Real Objects
boats, speedboat, yacht
23

---

Published as a conference paper at ICLR 2026
[Prompt]
[Image]
Please describe this image in detail.
[DeGF]
The image features a man and woman standing next to colorful cake, both smiling and preparing to it together. The cake decorated with flowers and is placed on a dining table. The couple appears to be enjoying the moment and sharing special occasion. CocasionnIninTTere There are a few bottles on the table, possibly containing beverages for the celebration. A handbag
chaif can seen the background, is placed nearby. The scene captures the joy and excitement of the couple they cut into the cake together.
[Deco]
The image depicts man and woman standing together, both holding onto knife as they into birthday cake together. They appear be celebrating a birthday party or similar occasion. The cake itself is colorful and decorated with flowers, making it look festive and delicious. Int.uus..nnnnneerrn There the party guests. A chair can also be seen the background, possibly for guests sit down and enjoy their refreshments.
several bottles placed around the scene, likely containing drinks for
[AGLA]
The image depicts together and cutting a cake. They are smiling and enjoying the moment. The cake placed on a dining table, which occupies significant portion of the scene. Inenhrrre are a few other items in the scene, such positioned at the far left side the image. The man and woman appear to be the main focus of the scene, as they share the experience of cutting the cake together.
and woman standing next to each other, both holding knife
bottle located near the left side of the table, and chail
[AGE]
The image depicts a man and a woman standing next to each other, both holding knife and cutting large, multi-colored cake. The cake is placed bottle beer on the table as well. The couple appears to be celebrating a special occasion, possibly their wedding anniversary, as they share the joyful moment together.
dining table, and there
Figure 14:
An example of generated captions by different decoding methods. Hallucinated content is marked in red.
24

---

Published as a conference paper at ICLR 2026
[Prompt]
[Image]
Please describe this image in detail.
[DeGF]
The image features luxurious bathroom with a large, black bathtub sitting tile floor. The bathtub positioned the center walls that are painted beige. A mirror is placed on the left side of the room, and a sink be left side as welh.nninThe bathroom is well-stocked with various toiletries, including multiple bottles and pups placed around the room. There are also couple wine glasses and a bowl, adding to the elegant atmosphere the space. The bathroom appears to
room, surrounded by
the
be a part hotel suite, providing a comfortable and relaxing environment for guests.
[Deco]
The image depicts a large bathtub placed against wall in bathroom. The bathtub occupies most the space the room, extending from one end the room the other. Next to the bathtub, there is sink with mirror above it. The sink area also contains a the
bottle placed on top of it. (t.nnnnnn addition these items, there two bottles placed bathtub, one on each side of it. Furthermore, there are two pups placed close to each other near the bathtub as well. The bathroom appears to be well-equipped with bathroom essentials.
[AGLA]
The image features large, white claw-foot bathtub situated in bathroom. The tub is placed on floor, appears to empty. The bathroom is well-equipped sink, a mirror, and towel rack. A bottle can be seen on the sink counter, and a wine glass placed hearby.ininin addition to the main elements, there is counter, adding to the overall decoration the bathroom. The bathroom also has loilef. which is located in the background.
and cup the
[AGE]
The image features a spacious bathroom with a large, claw-foot tub positioned the center of the room. The tub is surrounded by tiled floor, and appears to be the focal point of the bathroom. The walls are adomed with a beige wallpaper, and the room is illuminated by dim light. In'nln addition to the tub, there are several other items in the bathroom. A sink is located on the left side of the room, and mirror is placed above it. There bottles, one placed on the left side of the tub and the other on the right side. A towel is hanging on the wall, and vase can be seen on the right side of the room.
also two
Figure 15:
An example of generated captions by different decoding methods. Hallucinated content
is marked in red.
25

---

Published as a conference paper at ICLR 2026
Table 9: Source 8 is generated by COCO. Comparison with different domain benchmarks: Medi- cal (Wu et al. 2024). Video (uuan et al. 2024), Math (Guan et al. 2024p, Map (Guan et al. 2024). and Table (Guan et al, 2024). FAKE question: Fake or nonsensical questions are used examine model's s ability to detect incoherent questions. Image SWAP: In this scenario, we swap the images with unrelated ones to evaluate the model's ability to detect mismatches between the image content and the question. The best results are highlighted in bold.
Medical
Video
Methods
FAKE 1
NONE 1
Acc 1
F1 1
LLaVA
20.9
5.8
57.7
53.2
Ours
24.5
8.6
58.7
54.0
Math
Map
Acc
F11
Acc 1
F1 1
MiNiGT4
66.6
40.0
25.0
24.9
Ours
68.2
46.0
26.9
26.8
Table
Map
Acc
F1
Acc
F1 1
MPLUG-Ow12
72.7
52.2
36.5
41.4
Ours
74.5
53.5
41.3
44.8
Table 10: Comparison of latency (seconds/tokens) and throughput (tokens/seconds) with different LVLMs.
MiniGPT-47BB
LLaVA-1.5-7B
mPLUG-OWl2-7B
Method
lateneg(s/0
(eronghgnd(((a))
Intency(s/() 4 (hroughpat(/ss) lalency(s/1)) 4. throughpuut//4
Baseline
7.1e-5
14104.4
17c-
5700.8
2.1e-1
4635.4
Ours
14646.9
1864
1.944
5234.2
Table 11: Comparison between the AGE-Dynamic variants on VA-L5. The best results are shown in bold.
Method
Cs
C
BLEU1
Baseline
20.80
6.77
15.93
AGE D Dynamic
17.77
5.89
16.80
AGE
16.43
5.58
16.48
Table 12: Comparison between the AGE-Online variants on LLaVA-1.5. The best results are shown in bold.
Method Cs
C1
BLEUt
Baseline
20.80
6.77
15.93
AGEo 16.93
6.05
16.13
AGE
16.43
5.58
16.48
26

---

Published as a conference paper at ICLR 2026
Table 13: Hallucination rates (%) reported using CHAIRs (Cs), CHAIR, (Cr), and BLEU (%) on COCO image captioning tasks, where lower CHAIR and higher BLEU are better. Intervention Target and Intervention Modality are reported with each method. The mar new token is set 512. The best results are highlighted in bold. represents the results reported from the corresponding original paper. represents the results reported from the corresponding official code.
Method
Intervention Target
Intervention Modality
Cs 4
C₁ 1
Recall
Baseline
51.0
15.2
75.2
VTI (Liu et al. 2024d
Hidden State
Text & Image
35.8
11.1
Nullu Yang al.
2025a
Weights
Image
54.6
14.1
81.3
HALC Chen
2024d
Logits
Image
41.8
12.2
80.3
Ours
Attention
Text & Image
35.6
10.8
77.2
DeGF (Zhang et al.
2025
Logits
Image
46.0
14.2
78.5
Ours D&GF
Attention & Logits
Text & Image
45.8
14.0
79.3
AGLA (n al.
2024
Logits
Image
43.0
14.1
78.9
Ours
AGLLA
Attention Logits
Text & Image
40.1
13.0
81.2
Deco Wang et
2024a
Hidden State
Image
37.8
11.1
77.6
Ours Deco
Atlention & Hidden State
Text & Image
32.6
10.9
77.5
27


Reviews:
[{"review_id": "UZVp0xclTN", "reviewer": "Reviewer_sf9o", "summary": "The paper proposes AGE, a training-free, decoding-time framework that mitigates hallucinations in LVLMs by imitating the stage-specific attention patterns observed in truthful tokens. From a token- and layer-wise analysis, the authors find that real tokens systematically allocate more visual attention in late layers, while some models (e.g., LLaVA-1.5) also require stronger text attention mid-stage. AGE injects a directional visual correction vector δ (estimated from a small set of COCO samples) into late-stage layers and applies self-multiplicative text attention in the mid-stage when model-specific analysis indicates it helps. Across COCO/CHAIR, POPE, and MME, AGE outperforms recent SOTA methods while maintaining or slightly improving BLEU, and requires no additional training. Ablations show the vector-guided visual intervention (AGEI) is substantially more effective than coarse self-scaling, and that targeted, stage-aware interventions beat blanket modifications. The method is simple, interpretable at the attention level, and appears robust to the number of samples used to estimate δ.", "questions": "- How sensitive are results to the layer indices and λᵥ/λₜ? Could you automatically pick stages (e.g., by monitoring live attention gaps) to avoid manual choices?\n- δ is computed from COCO captions—how does AGE transfer to domain-shifted settings (e.g., medical images, charts) or text-heavy VQA where “real vs. hall” labels are less object-centric?\n- Have you considered online estimation of δ during decoding (or per-image calibration) to reduce dependence on an offline sample pool and improve robustness to image/content diversity?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- A precise, stage-aware recipe—directional visual correction in late layers + optional mid-stage text boost—plugs into existing LVLMs without retraining or extra modules.\n- The token-/layer-wise diagnosis directly motivates the interventions; ablations (SMA vs. AGEI, AGEM/AGEL variants) convincingly show why direction + stage targeting matter.\n- Strong gains on CHAIR/POPE/MME across LLaVA-1.5, MiniGPT-4, mPLUG-Owl2, with BLEU preserved; δ estimated from M=10 samples yet still competitive—good efficiency story.", "weaknesses": "- The approach assumes attention patterns of “real” tokens are the right supervisory signal; while effective, this remains heuristic and may not universally reflect grounding quality across tasks or architectures.\n- Fixed layer choices (20, 30–31) and large λᵥ=100 / λₜ=3 suggest potential sensitivity; it’s unclear how easily settings transfer to unseen models, domains (e.g., medical, diagram, chart), or alternative decoders.\n -“Real vs. hallucinated” token labeling depends on object annotations (COCO); generalization to open-ended VQA without dense GT, long-context reasoning, or video remains untested, and the latency/throughput overhead of per-step intervention isn’t reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RzMmS0GY2", "reviewer": "Reviewer_s179", "summary": "The goal of \bthe paper is to address the hallucination in LVLMs. The paper conducts the layer-wise analysis by introducing the concept of Average Attention Sum and computing the difference between non-hallucinated and hallucinated ones. The analysis shows the patterns of $Diff^{l}_{image}$, which is positive at the last stage. The paper proposes an attention direction to imitate the attention pattern, which is used during inference. The paper validates the proposed method on hallucination benchmarks and conducts an ablation study, including sample size and variants of AGE.", "questions": "1. In Sec. 3.2, do humans manually annotate tokens as 'real/hallucinated', or is it done automatically?\n2. In Eq. (3), the left term is divided by $I^{(i)}_{hall}$. Is it correct?\n3. What common characteristics between MiniGPT and mPLUG-Owl lead to the positive $Diff^{l}_{image}$ for all layers?\n4. After applying the proposed method to LVLMs, are the general task capabilities preserved?\n5. After steering the attention, does the total attention still sum to 1?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The inference cost is lower than that of other methods, such as VCD. Once the attention direction is computed to imitate attention, inference requires only a simple shift operation.\n2. Latent steering is widely adopted to mitigate hallucinations. But attention steering is a novel approach.", "weaknesses": "1. There is a logical gap in the hypothesis of the layer-wise analysis.\n- Both MiniGPT-4 and mPLUG-Owl2 exhibit a consistently positive difference across all layers. Given this observation, is it valid to imitate Early and Late attentions for MiniGPT-4 and mPLUG-Owl2?\n- In contrast, the difference in text is negative in general. This suggests that the corresponding $\\lambda_t$ should be negative to reflect this. However, the paper assigns the positive value to $\\lambda_t$.\n2. The way to select the final hyperparameters, including the choice of layers and the scaling factor, is not clear. Details on the hyperparameter tuning method would be helpful.\n3. The proposed approach steers the attention after computing the attention direction, which is similar to the latent steering methods [VTI, Nullu]. A detailed discussion and comparison are needed to validate the advantages of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EhQxsqRKka", "reviewer": "Reviewer_Y1U5", "summary": "This paper presents AGE (Attention-aware Truth-Guided Enhancement), a training-free, decoding-time framework aimed at reducing hallucinations in large vision-language models (LVLMs). It leverages an in-depth analysis of attention patterns distinguishing truthful from hallucinated tokens across layers and modalities, applying targeted, stage-specific interventions to realign attention toward grounded responses. The approach is evaluated on benchmarks including COCO (with CHAIR metric), POPE, and MME, across three LVLMs (LLaVA, MiniGPT-4, mPLUG-Owl2), showing reduced hallucination rates while preserving or enhancing overall performance metrics like BLEU.", "questions": "1. Given the similarities to prior works like AGLA, DeCo, and DeGF, could you clarify the novel contributions of AGE and provide comparisons (both qualitative and quantitative) to these methods? Additionally, please address discrepancies in baseline metrics compared to their original reports to ensure fair benchmarking.\n\n2. Could you provide examples of failure cases for AGE, including their common characteristics (e.g., specific image types, prompt complexities, or distributional shifts)? This would help assess the method's limitations more concretely.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. AGE offers a lightweight, training-free, and model-agnostic solution that can be integrated seamlessly during inference, making it practical for deployment.\n\n2. The experimental setup is comprehensive, covering three diverse LVLMs and multiple benchmarks (COCO/CHAIR, POPE, MME), with results indicating hallucination mitigation alongside maintained or improved generation quality.", "weaknesses": "1. The core idea of using attention calibration or intervention to mitigate hallucinations by aligning with truthful patterns has been explored in several prior works, such as AGLA, which assembles global and local attention to reduce object hallucinations, DeCo, which employs dynamic correction decoding, and DeGF, which uses generative feedback for self-correction. Token-level and layer-wise analyses of hallucinations are also featured in these and related studies. While AGE contributes through finer-grained, stage-specific interventions, the paper does not sufficiently delineate its novelty beyond these approaches and lacks citations or direct comparisons to them.\n\n2. The intervention stages (e.g., layer 20 for text, 30–31 for vision) are determined empirically (Appendix A.5), but the paper provides limited guidance for adapting these to novel architectures or LVLM variants. It notes that early-layer interventions severely degrade BLEU scores yet fails to propose a systematic methodology for identifying optimal stages or assessing transferability to deeper or wider models.\n\n3. The formulation of δ as a \"universal\" intervention vector is empirically motivated and tuned, but lacks theoretical grounding on its effectiveness, particularly under distribution shifts between calibration and test data.\n\n4. Notation in equations for attention weights exhibits inconsistencies, reported baseline metrics appear inconsistent with those in the original papers, potentially leading to unfair comparisons.\n\n5. The method relies on numerous empirically set hyperparameters, with insufficient analysis of their robustness or generalizability across different settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ATF6ofQbyF", "reviewer": "Reviewer_yxZC", "summary": "This paper observes layer-wise attention disparities between real and hallucinated tokens across three LVLMs, which motivates a method for calibrating stage-specific attention behaviors using a pre-calculated directional vector on a labeled subset of data.", "questions": "1. In Section 3.3 and Figure 3, the statement that “the final reasoning stage (Layers 26–31) exhibits a stable and pronounced positive gap” (lines 204–205) is less convincing. The earlier layers (Layers 0–4) appear to show a more consistent and significant gap.\n\n2. The related work section could be more up to date. most of the papers are from 2024.\n\nHappy to raise score if my concern is addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is clearly motivated by empirical observations, offering new insights into attention behavior in LVLMs. Comprehensive experiments demonstrate that it effectively mitigates object hallucinations and outperforms prior work.\n\n2. The paper is well written and supported by clear figures.", "weaknesses": "1. My main concern is that this method relies on access to a labeled subset of data to calculate the directional vector that captures the attention correction tendency. This limits its applicability in real-world scenarios where labeled data from the same distribution are not available, and there is no validation showing whether the learned attention behavior remains valid.\n\n2. Related to W1, it is also unclear how the stage-specific settings (line 300–302) are determined in practice, since they seem to be tuned like hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)