Hallucination Begins Where Saliency Drops
Authors: Xiaofeng Zhang
Yuanchao Zhu
Chaochen Gu
Xiaosong Yuan
Qiyan Zhao
Jiawei Cao
Feilong Tang
Sinan Fan
Yaomin Shen
Chen Shen
Hao Tang
Venue: ICLR 2026 Oral (2026)
Keywords: LVLMs-Saliency; Saliency-Guided Rejection Sampling;  Local Coherence Reinforcement; Hallucination
Primary area: foundation or frontier models, including LLMs

Abstract: Recent studies have investigated attention dynamics in large vision language models (LVLMs), yet existing methods remain limited in reliably distinguishing hallucinated from correct outputs — primarily because they rely solely on forward-pass attention, ignoring gradient-based signals that reveal how token influence propagates through the model. To bridge this gap, we introduce \textbf{LVLMs-Saliency}, an \textit{gradient-aware diagnostic tool} that quantifies the grounding strength of each output token by fusing attention weights with their gradients. Through analysis, we identify a decisive pattern: \textit{Hallucinations occur when prior output tokens shows low saliency to the next token prediction}, indicating a failure of contextual memory. Building on this insight, we propose a dual-mechanism inference-time framework: (1) Saliency-Guided Rejection Sampling (SGRS), which dynamically filters candidate tokens during decoding by rejecting those with saliency below a context-adaptive threshold, thereby preventing coherence-breaking tokens from entering the sequence; and (2) Local Coherence Reinforcement (LocoRE), a lightweight plug-and-play module that strengthens attention from the current token to its most recent outputs, actively counteracting the “forgetting” behavior identified by LVLMs-Saliency. Experimental results demonstrate that our method significantly reduces hallucinations across multiple LVLMs, offering a robust and interpretable solution to improve model reliability.

Published as a conference paper at ICLR 2026
HALLUCINATION SALIENCY DROPS
BEGINS
WHERE
Xiaofeng Zhang*1 Yuanchao Zhu Chaochen Gu Xiaosong Yuan Qiyan Zhao
Jiawei Cao', Feilong Tang². SSnan Fan? Yaomin Shen' Chen Shen? Hao Tang'
1 Shanghai Jiaotong University Alibaba Group
Monash University
*Peking University
"Equal contribution *Corresponding author @ejueeduu.et..u...cn
ABSTRACT
Recent studies have investigated attention dynamics in large vision language models (LVLMs), yet existing methods remain limited in reliably distinguishing halluci- nated from correct outputs- primarily because they rely solely forward- pass attention, ignoring gradient-based signals that reveal how token influence prop- agates through the model. To bridge this gap, introduce LVLMs-Saliency, an gradient-awars awadie diagnostic tool that quantifies the grounding strength of each output token by fusing attention weights with their gradients. Through analysis, we identify a decisive pattern: Hallucinations occur when prior output tokens shows low saliency to the next token prediction, indicating failure of contextual memory. Building on this insight, we propose dual-mechanism inference-time framework: (1) Saliency-Guided Rejection Sampling (SGRS), which dynami- cally filters candidate tokens during decoding by rejecting those with saliency below context-adaptive threshold, thereby preventing coherence- coherence-breaking tokens from entering the sequence; and (2) Local Coherence Reinforcement (LocoRE), a lightweight plug-and-plas module that strengthens attention from the current token to its most recent outputs, actively counteracting the "forgetting" behavior identified by LVLMs- Saliency. Experimental results demonstrate that our method significantly reduces hallucinations across multiple LVLMs, offering robust and interpretable solution to improve model reliability. The code can be accessed actps::/https:: //qittt com/ zhangbai Littn/LLMM--S00
INTRODUCTION
Large Vision Language Models (LVLMss) have made significant strides in cross-modal tasks. However, hallucinations remain a key challenge, particularly in visual question answering and image captioning.
Current mitigation strategies such as incorporating external knowledge, retraining with additional
data Li et al. (2023a): Liu et al. (2023); Park et al. 2024); Ma el al. (2025e a b 2024; 2025c) or training-free methods Neo Chen (2024):
3 Li al.
2025a Zhang et al. (2025a) Wu
(2225a);
Liu et al. (2024c); Gong et al. (2024); Zhou et al. 2024); Shang et al. 2024); Min et al. (2024) Liu
al. (2224)): et al. (2025): Wu et al. (205b)). Although the above methods have made great
progress, their interpretability is insufficient, especially without a clear explanation of the causes of hallucinations the autoregressive generative model.
Recent studies on attention sinks have provided new perspectives for understanding hallucinations. For example, OPERA Huang et al. (2024). DOPRA Wei & Zhang (2024), L et al. (2024d), VastV Chen et al. (2024b), EAH Zhang et al. 2024a). TAME Tang al. (2025a) and Farsight Tang et al. (0225b have revealed the relationship between attention sinks and hallucinations. They prove that when token continues to attract high attention weights in subsequent tokens, this over-reliance may cause ballucinations in the model output. However, the relationship between attention maps and hallucinated tokens remains inadequately explained. This is because attention maps only reflect the

---

Published as a conference paper at ICLR 2026
Please describe this Image in detall
The image depicts a cozy, vintage-style room with a mix of traditional and rustic elements. The room features a sloped ceiling with a striped wallpoper pattern in shodes of blue. white, and red. The walls are also odorned with similar striped wallpoper.
Attention map
Saliency map
Attention map
Saliency map
No pattern
No pattern
high sallency
pattern
sallency
System
Onlpet
Saliency Store
Sofieny Stwee
hish
Josr
Toleed...
Tikee
Symw
Pasep:
Ougat
Syron
Prcept
Ouppat
Correct pattern
Wrong pattern
Figure 1: Attention vs. Saliency Maps for Correct and Hallucinated Tokens (Qwen2-VL-77)) Left (correct token wallpaper): Attention maps show no distinctive pattern, while our LVLMs- Saliency maps reveal strong, structured grounding to prior outputs. Right (hallucinated token blue): Attention maps remain visually similar, but saliency maps collapse, signaling loss of contextual dependency.
model's decision-making in the forward pass, without capturing how changes in input tokens influence the final output. Moreover, existing methods often overlook gradient information, which is essential for understanding the interdependencies among different tokens during the generation process. As illustrated in Figure [] it is nearly impossible to discern meaningful patterns in attention maps that distinguish correct outputs from hallucinated ones. Therefore, a token-level, interpretable observation tool is essential to uncover the mechanistic origins of hallucinations in large vision- lision-language models, revealing not just when they occur, but why and where in the generation process they emerge.
To address this limitation mentioned above, we draw inspiration from the concept of information flow introduced "Label Words" Wang et al. (2023), which highlights how information within LLMs tends to converge specific user-specified tokens. Adapting this insight the autoregressive generation setting of LVLMs, we propose an unsupervised metric called LVLMs-Saliency. defined the element-wise product of attention weights and their corresponding gradients. This measure quantifies how strongly each previously generated output token influences the prediction of the next token, offering fine-grained, token-level view of contextual grounding or its absence during generation. As shown in Figure I and Figure 2 we observe saliency patterns in Qwen2-VL and VL- that distinct from conventional attention maps:
Pattern: Hallucinations occur when prior output tokens shows low saliency to the next token.
which reveals a breakdown in contextual grounding that attention-only methods fail to capture. When generating the correct token, the model maintains high saliency on previous related tokens, thereby ensuring the coherence context tokens. However, hallucinations occur when the model "forgets" the past context, resulting in weak dependencies between tokens and low saliency of previous output. By the way, although there a noticeable difference in the saliency of user prompts for correct versus hallucinated tokens, our analysis of 500 samples indicates that these saliency scores do not significantly affect the model's 's predictive accuracy. This finding suggests that although prompt saliency plays role in the model's behavior, it is not the primary cause of hallucinations.
2

---

Published as a conference paper at ICLR 2026
Saliency map
wallpaper
Query
System system
System prompt
Prompt prompt pt
System
Tromp
Output
outpet
ettpu
ontput
high saliency
low salieney
Systens User tnkens next prediction token
Syste
Osipat
sentence1
The image depicts a cozy, vintage-style
sentence2
with mix of traditional and ruseic elements,
sentence3
The image features sloped ceiling with striped wall wallpaper pattern shades of blue.
high saliency
low saliency
Figure 2: Output Token Saliency Patterns in Qwen2-VL--B. When generating correct token (e.g., wallpaper), the current token assigns high saliency to recent output tokens, typically decaying with distance. In contrast, when generating hallucinated token (e.g.. blue), saliency toward all prior outputs collapses - signaling contextual disconnection.
Unlike previous methods of intervening in image attention (Zhang al. 2024a Liu et al. 2024d
Jiang et al., 2024; Tang et al. 2025a by to alleviate hallucinations, we focus exclusively on the dynamics of output token saliency during autoregressive generation. To mitigate hallucinations caused by context loss when the model outputs tokens, we propose dual-intervention approach in the inference phase that incorporates saliency:
Saliency- Guided Rejection Sampling (SGRS): proactive filtering mechanism that evaluates the grounding quality of each candidate output token before it is committed to the sequence. By computing the token's saliency, SGRS rejects candidates that exhibit weak contextual dependencies (i.e., low saliency), forcing the model to resample until contextually grounded token is selected. This directly prevents the injection of "coherence- "coherencee-rrakking breaking" tokens that trigger cascading hallucinations.
Local Coherence Reinforcement (LocoRE): A reactive stabilization mechanism that activates after a token is accepted. LocoRE strengthens the attention weights from the current query token to the (P) + B DT(((P-)) S w..). This ensures that even as the sequence grows, the model maintains strong attentional links to its immediate past, counteracting the "forgetting" behavior observed Pattern
recent w. output tokens, using distance-aware gain factor ?
Together, SGRS and LocoRE form a closed-loop coherence preservation system: SGRS acts as gatekeeper, blocking low-saliency tokens at the point of entry: LocoRE acts as a stabilizer, reinforcing contextual dependencies after commitment. With extensive experiments, our method demonstrates sigmificant hallucination mitigating performance across different LVLMs on image hallucination and generation benchmarks, proving its effectiveness. Our contributions are as follows:
We propose MsS-Saliency, an unsupervised, gradient-based metric for quantifying token-level hallucination in autoregressive LVLMs. Through systematic analysis, we establish direct causal link between low output token saliency and hallucination: when the model fails to maintain attention recently generated tokens (Pattern D. contextual memory collapses, leading to semantically inconsistent outputs.
We introduce Saliency- Guidedd Rejection Sampling (SGRS), the first inference-tine mechanism that dynamically filters candidate tokens based on their saliency with respect to prior output context. By rejecting low-saliency tokens before commitment, SGRS proactively prevents the injection of

---

Published as a conference paper at ICLR 2026
voherence-breaking elements into the generation stream directly mitigating the root cause of context- drift hallucinations.
We introduce Local Coherence Reinforcement (LocoRE), lightweight, plug-and-plas module that strengthens attention from the current token to its most recent w, predecessors. Unlike prior methods that rebalance cross-modal attention, LocoRE operates purely within the output stream. SGRS ensures only coherent tokens enter, LocoRE ensures they are not forgotten.
ANALYSIS AND MOTIVATION
2.1 HALLUCINATION TOKEN SALIENCY ANALYSIS
We propose a gradient-based attention analysis framework for quantifying token-level hallucination saliency in autoregressive language models. Given an input sequence E V", where V denotes the vocabulary space and n represents the sequence length, we process through the model M to obtain:
(y. (A (1hh) I.H 11 (
M(x),
(1)
where A (1,b) E [0, 1|axn denotes the attention weight matrix at layer E (1,..., J.} and head E {1...., H) 8 E R/V represents the logits corresponding to the target hallucination token, y E RM is the model's output probability distribution. The cross-entropy loss function L IRIV x R M + R+ defined as:
((y,8) = Σ yt log (8t):
where (-) denotes the softmax function and indexes the token position in the sequence. The gradient of the loss with respect to attention matrices is computed as:
Oc (A
V (L.)
E <
(3)
The saliency matrix (l.,)) _ for each attention head is obtained through the Hadamard product followed by triangular masking:
S(LN)) tril (AaCA) VA
where tril(-): Ruxn + R x preserves the lower triangular portion to maintain causal structure, and denotes element-wise multiplication. The layer-wise normalized saliency S(1) R is computed by averaging across attention heads and applying fz-normalization:
Σ 1=
((LA)
5())
(5)
Σ S(LA)
As demonstrated in Figures differences in saliency patterns between veridical and hallucinated tokens across both Qwen2-VL- 7B Yang al. (0224) and LLaVAII557D Liu et al. (2024a) architectures.
and 5 our quantitative analysis reveals statistically significant
3
METHODOLOGY
3.1
SALIENCY-GUIDE REJECTION SAMPLING (SGRS)
SGRS dynamically evaluates the grounding quality of each candidate token before commitment; the complete algorithm is formalized in Algorithm At the decoding step corresponding absolute position P, given context x and image I, the model produces logits (P) E RIV, We sample K
candidates C(P) via top- K sampling, For each E C(P)) we compute its hallucination saliency S(c,) as:
S(c:) = Lurgel .IT
Σ
( P3
(6)
LE Cueee JET
4

---

Published as a conference paper at ICLR 2026
where ( is the layer-wise normalized saliency matrix defined in Section 2.1 Lurget denotes the set of target layers (e.g., middle to-deep layers), and J = {j Sys, + Img.. S P} is the set of positions corresponding to previously generated output tokens, with Sys, = 35 and Img 576 for VA-1.5.
A candidate is accepted only if S(a) 2 r(P), where the adaptive threshold is computed the most recent W output tokens:
(P)
-a
Σ S(r)), JEH
= - W),
HH
with E (0,1) controlling sensitivity and W the history window size. scales the historical average saliency to control: "How many times the saliency of the current candidate token needs to reach all candidates are rejected, we fall back selecting
the historical average before it is accepted". the token with the highest saliency score. This mechanism directly operationalizes our finding in Pattern I low output-token saliency precedes hallucination. By rejecting such tokens, SGRS enforces generation path grounded in textual context - specifically, the model's own prior outputs.
3.1.1 LOCAL COHERENCE REINFORCEMEN (LoCORE)
While SGRS ensures token-level grounding, LocoRe addresses sequence-level context drift by explicitly reinforcing attention dependencies among output tokens, the complete algorithm is formalized in Algorithm 2 Formally, at absolute position P (where P Sys, + Img,). let previously generated output tokens. For the prediction of token at position P + 1, we enhance the attention weights from query + keys Jp within local window size ws.
Jp = (j E N Sys, + Img S < P) denote the of positions corresponding
Define the disghnce-weighted gain for each j E Jr as:
(P) + (-PP((((-)))) sw,,
(8)
where 3 > 0 is the reinforcement strength, and II(-) is the indicator function. Let A (P+1) e [Bxxxx(P41)x((P1) denote the attention weight matrix computed during the forward pass for position P+1. We modify the submatrix corresponding to attention from query P to keys in Jp:
A h, < (P+1) h P 1,
Ab E [B), h E [nn), j E Jp.
(9)
Equivalently, in vectorized form, let (P) RIJrl be the gain vector with entries 7/ 7/ and let
A (P+1)
P41.Jp E BBBxx x|Jrl denote the slice of attention weights from query P + 1 to keys in Jr. The
update is:
A
(P+1) A (P+1) P41.Jp @ (P)))
(10)
P41.3P
where denotes element-wise multiplication broadcasted over batch and head dimensions. The modified attention weights are then used in the softmax and weighted sum operations of the self- attention mechanism, ensuring that the model's prediction for token P+1 is more strongly grounded in its recent output history. This operation amplifies the influence of recent context on the prediction of token P+ 1, directly countering the saliency decay observed in Pattern I Crucially, LocoRE operates purely the attention structure no gradient computation or model parameter modification is required.
Synergistic Workflow. SGRS and LocoRE operate sequentially at each decoding step: SGRS filters and selects the current token pp based on its saliency to prior outputs; LocoRE then modifies the attention weights used in the next forward pass (for position P + 1) to reinforee dependencies recent tokens. This closed-loop design ensures that each accepted token is both well- gell-groudded (SGRS) and unlikely be forgotten (LocokE)..
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 SGRS
Algorithm 2 LoCORE
Require: M, x, K, R. a, W, L, 3=35, /-576,
Require:
Ensure: ZP: accepted token at position P
1: A ( E
attention
I: logits <- M(xipuu. KV) -.:
weights for step P+1
2: C (opK(sofimx(logits), K). accepted
2: S = 35,
576: system and image token
False
lengths
3: for r- 1 to R do
3: w.: local window size, 3 2 0: gain strength
4: em Sample(C)
Ensure: A (P+1); modified attention weights for step
5: S(c) SALIENCY(M,,C. c, Lurga P. P.S.I)
P+1
Eq. (1)
4: P + current position
Last generated token
Tr + -
Output token
position
positions
f P-P(5 (S +5++
7: HP E Jp (P -
6: if 0 then return A (P-1)
Recent W outputs
end if
D No output yet
T T- a Rpp ΣjEN, H[i
Eq. (2)
8: Jp < f < Pj b Historical output
Sc) 2 then
positions
10:
xp +- e, H.append(S(c)) accepted +
9: if Jp 2 then return (P+1)
Truc, break
10: end if
11: else
11: for all j Jr do
12:
C+ (c)
12:
dj f -P
Distance current position
end if
13:
% f- 1 3-1, S w.)
> Eq. (3)
14:
end for
14:
for all [B].
[ma] do
15: not accepted then
15:
A AP+1) [b, h P
1
16:
+ arg maXenanignal S(c)
> Fallback:
( [b, h, P 1, %
Eq. (4)
best saliency
end if
16:
end for end for
17:
18: return xp
18: return A (P+1)
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Baselines. To demonstrate the broad applicability of our method in LVLM architecture, we applied and evaluated the latest models, including LLa VA-v 15-7/13B Liu al. (2024a). Ww-n2-VL-7BB Wang et al. 2024) and Intern-VL-7713BB Chen et al. (2024d). This study used the following data sets as evaluation sets, representing the expertise in reducing hallucination and general fields.
Evaluation Benchmarks. We conduct evaluations on image benchmarks. For image benchmarks, we assess three categories: (1) Comprehensive benchmarks (LLaVA Liu al. (2024a). MM- Vet Yu et al. (2023), MME Yin et al. 20233: (2) General VQA benchmarks (VizWiz Gurari et al. (2018)
Science Lu al. (2022); ; (3) Hallucination benchmarks (POPELi (2023b). CHAIR Rohrbach
et al. 2018)).
4.2
EVALUATION RESULTS ON HALLUCINATION BENCHIARRS
CHAIR and POPE Evaluations. As shown in Table I methods for mitigating hallucinations can be broadly categorized into third groups The first group, including OPERA Huang et al.) (024)) DOPRA Wei & Zhang 2024), DOLAChuang et al. (2023), VCD et al. 2024). HALC Chen
al. (204cc An al. 2024h. ICD Zhanghang et al. (2023). RITUAL Woo al. (204). AGLA An et al. (2024). SID Huo et al. (002). Only Wan et al. (2025), focuses on modifying the decoding process to address hallucinations. The second group, represented by SFT methods such as LESS is more Yue et al. (024), CCA-LLaVA Xing et al. (2024) and Reverse- VLM Wu et al. (2025b), adjusts the logits of the end-of-sequence (EOS) symbol to control its positioning, allowing the model to terminate earlier, thus reducing hallucinations. The third group includes Vissink Kang al. (2025). EAH Zhang et al. (2024a). TAME Tang et al. (2025a), MemVR Zou et al. (2024) and Farsight Tang et al. (2025b), which aim to enhance the truthfulness of the model's 's output during inference by adjusting attention heads. Among these methods,reaching SOTA on the POPE dataset, and achieved significant results second only to EAH descriptive datasets such CHAIR. Compared with EAH's approach

---

Published as a conference paper at ICLR 2026
Table 1: Compare results of LocoRE with other SOTA methods on POPE, CHAIR and MME datasets. The best performances within each setting are bolded, baseline: LLaVA-1.5-7B.
Method
Venue
F17
POPE
Acc?
C-4
C,
CHAIR
Recall)
length
Exist.
Countt
MME Pos."
Color?
Total?
Beam Search
85.4
84.0
51.0
15.2
75.2
102.2
175.67
124.67
114.00
151.00
565. 34
Colanh
ICLR 2004
80.2
78.2
180.10
127.40
119.30
154.60
$34.10
VCDLESS
77.2
128.65
SPERAIIIAE.. el di
2014
CVPR 2024
84.2
85.2
47.0
14.6
78.5
95.3
180.67
133.33
111.67
123.33
549.00
DOFRA Wey Ziini
2024
2024
84.6
84.3
46.3
13.8
78.2
96.1
185.67
138.33
120.67
133.00
S77.67
HALC Chien II. 20144
ICML 2024
83.9
84.0
50.2
12.4
79.4
97.2
190.00
143. 30
128. 30
150.00
621.60
CCA-LraVeXIIg
NeuriPS 2024
86.4
86.5
43.0
80.4
190.00
148.33
153,00
641.66
2024
78.3
164_17
616.25
2han2 cLoli Mla
EMNLP 2025
85.7
56.0
36.4
9.9
74.9
97.7
190.00
108.33
145.00
150.66
603.99
SID Fiin PiR PL/A3
ICLR 2025
85.6
85.8
44.2
12.2
73.0
99.4
183,90
132.20
127.80
155.90
599 80
TAME Jin ar 0135g
ICIR 2025
85.4
85.7
41.3
12.2
74.4
98.8
193.00
157.3 33
130
154.67
634.00
Vissin
2025
148.33
33
155.00
631.33
2025
170.00
AGLA Andriin 20144
CVPR 2025
84.6
85.5
78.9
98.8
195.00
153.89
179:44
161.67
640.000
Farsigh einreriin 201-5b
CVPR 2025
41.6
13.2
75.5
100.6
Mcm
2013
ICML 2025
87.1
97.4
46.6
13.0
80.8
99.6
190.00
155.00
133.33
170.60
648.30
ONLY at
ICCV 2025
85.5
85.1
75.9
99.7
191,67
145.55
136.66
161.66
635.55
Reverse at.
NeurlPS 2025
75.2
LocRE:
$6.9
87.3
38.4
11.2
75.4
98.2
190.00
158.33
133.33
175.00
656.66
SGRS + LLccoRE
$7.0
87.5
35.6
8.2
73.4
98.2
195.00
158,33
140.00
175.00
668.33
Table 2: Comparison of different LVLMs and LocoRE across all image benchmarks. Notably, in the Hallucination Benchmark, lower scores on CHAIR; and C indicate better performance, while higher scores are preferable for other metries.
Comprehensive Benchmark
General VQA
Hallucination Benchmark
Method
LLaVA
MM-Vet
VizWizt
SQA
CHAIR:
CHAIR,
POPE-R1 POPE-FI" POPE-A1
I.AVA-L55-78
72.5
30.5
48.5
65.5
48.0
13.9
87.0
85.4
84.0
+ICD
69.7
30.4
46,9
62.8
47.7
13.6
87.9
84.9
84.0
+VCD
70,9
43.4
63.3
87.0
85.3
85.0
+OPERA
72,0
31.4
50.0
64.9
45.2
12.7
88.8
$4.2
85.2
+SID
73.4
31.2
50.9
67.8
44,2
14.0
89.4
85.6
85.8
+TAME
73.9
88.9
Vissink
74.1
33.5
53.8
67.0
52.4
14.5
87.7
8499
85.8
FFarSight
74.7
32,5
50,8
67.4
41.6
13.2
90.5
85.5
85.8
LLocoEE
74.8 (+2.3)
33.8 (+3.3)
54.8 (+6.3)
67.5 (+2.0)
38.4 (49.6)
10.2 (+3.7)
89.5
(42.5)
86.9 (+1.5)
87.3 (+3.3)
-SGRS+LocoRE
76.7 (+4.2)
36.0 (+5.5)
54,9 (46.4)
67.8
35.6 (+12.4)
89.8
87.0 (+1.6
87.5 (+3.5)
LLaVA-15--1B
72.5
36.1
60.5
71.6
47,2
13.6
82.5
86.6
87.2
+ .LocoRE
(+.55)
(42.3)
62.1 (+).6) 72.5 (+0.9)
43.8 (+3.4)
12.8 (+0.8)
87.8 (45.3)
87.7 (+1.1) (+02)
SGRS LocoRE
76.8 (+4.3)
(+5.9)
(43.5) (+3.4)
(+7.4)
8.8 (+4.8)
(45.5)
88.1
(+1.5)
(+0.4)
Intern-VI--70 VB.
51.6
31,2
51.7
6622
46.6
12.4
80.0
85.3
86.2
+ LLcoRE
5288 (+1.2)
33.7 (+2.5)
54.5 (+2.8) 66.4 (40.2)
40.2 (+6.4)
(01.9)
85.8 (+5.8)
87.2
(+1.9) (+13
SGRS + LocoRE
55.5 (+3.9)
35.0 (+5.0)
56.2 (+4.5) 67.9 (+1.7)
(+122))
7.5 (+3.9)
86.0 (46.0)
87.6
(+2.3)
87.7 (+1.5)
Intern- IBsenn-VL-13B
53.2
82.8
86.4
+ LLcoRRE
(4.19
(+1.7)
50.1 (+2.7) 70.4 (40.))
43.6 (+1.8)
12.5 (40.2)
86.3 (43.5)
87.2 (40.) 87.3
SGRS + LocoRE
56.8 (+3.6)
(+3.6)
$2.0 (+4.6) 71.0 (+0.9)
45.2 (+3.4)
14.0 (+2.7)
87.0 (+4.2)
88.1
(+1.7) (+18
Owen2-VL-7B
75.6
63.2
57.3
74.1
25.0
7.3
79.1
86.6
87.6
LLocoRE
(+2.2)
64.8
(+2.)1
74.2 (40.1)
23.5
6.8 (40.5)
81.3 (+2.2)
87.5 (40.9)
(40.))
SGRS + LocoRE.
79.7 (+4.1)
67.7 (+4.5)
60.3 (+3.0) (+1.2)
19.3 (+5.7)
5.1 (+2.2)
82.6 (+3.5)
(+104)
89.0 (+1.4)
Qwen2.5-VL-7B
7688
62.2
60.9
79.0
27.2
9.0
80.4
87.4
88.4
+LocoRE
(+1.1)
(+2.6)
(+0.7) (+1.8)
(+4.2)
(40.5)
80.9 (40.5)
87.8 (+0.4)
88.7 (+0.3)
SGRS +LocoRE
80.0 (+3.2)
66.2
62.7 (+1.8) 82.1 (43.1)
21.0 (+6.2)
6.5 (+2.5)
81.5 (+0.5)
88.3 (40.9)
89,5 (+1.))
Qwen2.5-
81.2
72,2
70.8
89.0
43.6
9.5
79.1
86.7
87.8
+LoocoRE
82.7 (+0.5)
73.1 (+0.9)
(+0.4 (40.))
(+1.8)
(+1.0)
79.5 (+0.4)
86.9 (40.2)
88.0 (+0.2)
of directly replacing the attention head, LocoRE has a higher recall because it does not change the internal representation of the model, and therefore does not affect the diversity of the model output.
Compared to Vissink Kang et al. (025) and TAME Tang et al. (2025)), which also allocate attention, LocoRE's CHAIR performance is more prominent. TAME allocates the attention on the system token to other tokens, but still ignores the visual information, while Vissink only intervenes with the visual attention sink and ignores the contextual association of the text output. As a result, both of them perform not that well on long text output datasets such as CHAIR, while this also proves the effectiveness of our approach, which is able address the shortcomings of both of them, i..., enhancing the visual information as well as enhancing the contextual dependencies between text outputs.
7

---

Published as a conference paper at ICLR 2026
Cballaaation
Cerrrt
Malliratioo
correct
LoCoRE
WaTel
Sandnn
T 1j "oe
There are two bicycles one locatedipn the left sideand the other on the right side of the imago. Block caatchicnn also be seen.....
The man is wearing skis while the child is standing nearby, possibly learning how to ski. The man appears to be leaching (standing on the ground
Figure 3: Effect of LocoRE on output token saliency map (Qwen2-VI.-77) Without LocoRE: When generating an incorrect token(clock), saliency scores assigned to prior output tokens are low indicating weak contextual grounding. With LocoRE: The same position now generates a correct (oken(watch) accompanied by significantly higher saliency scores to recent outputs demonstrating LocoRE's ability restore contextual coherence and prevent hallucination via attention reinforcement.
4.3 EVALUATION RESULTS ON GENERATION BENCHMARK
MME and Other Benchmarks Evaluations. As shown in Table and Table we tested on several popular LVLMs' general ability benchmarks. MME comprises ten subtasks to evaluate models' perceptual capabilities and four subtasks for assessing recognitive abilities in the form of yes/no questions. LocoRE can maintain and improve the multimodal capability on LVLMs benchmarks. Our method achieve much higher score (corresponds to less hallucination) across all categories. This underscores its effectiveness in addressing a broader range of multimodal hallucination challenges beyond objects. Combining SGRS with LocoRE further improves reasoning intensive tasks, as demonstrated by the cognitive categories of MME. This performance is particularly pronounced on the "Existence" and "Position" tasks, as SGRS directly suppresses hallucinations while LocoRE focuses solely on contextual coherence.
reasoning-intensive
4.4 ABLATION STUDY
Effect of LocoRE other LVLMs As shown in Table 2 the integration of LocoRE plug-in into -LL.VA--,5-7B// Liu et al. (2024a), Owwn2-VVL-BB113B/333
Wang et al. (2024) and Intern- VL-7/13B Chen et (024dd) was effective improving results in both integrated and generalized VQA tasks. In addition, it achieved a significant improvement in hallucination metries. These results indicate that LocoRE is effective in reducing hallucinations in both structured and unstructured environments.
Saliency map Visualization with LocoRE. shown in Figure 3 which visualizes the LVLMs- Saliency maps from prior output tokens to the current token, applying LocoRE significantly increases the saliency scores assigned recently generated context tokens particularly those within the local coherence window. This demonsirates that LocoRE effectively strengthens the model's dependency on its immediate output history, counteracting the "forgetting" behavior observed in the baseline. The saliency boost under LocoRE confirms design principle: by explicitly reinforcing attention to recent outputs, the model maintains stronger contextual links during autoregressive generation. This prevents the decay of intra-output saliency that leads to hallucinations, ensuring that each new token remains grounded its textual predecessors.
4.5 ABLATION STUDY ON KEY HYPERPARAMETERS
We evaluate (SGRS) and 3 (LocoRE) on both CHAIR and POPE benchmarks. As shown in Table 3 and Figure 4 our full method (a = 0.6, 3 = 0.15) reduces CHAIR hallucination rate by 28.3% (AL.)) and 22.8% (Qwen2-VID compared to baseline. SGRS alone (a 0.6, 3 = 0.0) contributes most of the improvement, but LocoRE adds further gains (e.g.. POPE F1 score from 85.4% to 86.9% in LLa VA. Increasing to 0.9 yields marginal improvement at high latency cost (+33%). We recommend a = 0.6, 3 = 1.2 as the optimal balance. While increasing a to 0.9 further reduces hallucination rates (CHAIR, s: 35.6% 30.0%; POPE: 87.0% 87.1 %), it incurs 33%
8

---

Published as a conference paper at ICLR 2026
90
LLaVA-L5
Qwen2-VL-78 VL-7B
Recall iccearage
CHAIR
POPE
CHAIR
POPE
00
S4
11
FLT
Ace
54
FIT
AccT
48.0
85.4
84.0
25.0
7.3
86.6
87.6
0.0
0.15
38.4
10.2
86.9
87.3
0.0
0.20
23.5
6.8
87.5
88.2
30-
0.6
86.9
87.4
20.5
5.6
87.9
88.9
0.6
0.15
35,6
8.2
87.0
87.5
10
0.6
1.0
50.2
20.9
60.3
57.8
37.5
18.5
55.3
54.6
0.5
scaling)
Table 3: Ablation study on a (SGRS) and / (LocoRE). Best in bold. 33: 0.15 (LLaVA-1.5), 0.20 (Qwen2-VL).
Figure Ablation study of ot trade-offs be- tween hallucination rate, recall, and latency.
higher latency cost (30.8 ms/token - 41.2 ms/token) and risks degrading generation fluency due to over-rejection. In extreme cases, correct but moderately salient tokens may be rejected, leading to fallback- generated outputs that are less diverse or natural. We thus recommend a 0.6 as the optimal trade-off it suppresses 2 of hallucinations while maintaining practical inference speed and output quality.
RELATED WORK
5.1 NEXT TOKEN PREDICTION
After obtaining the next token probability, different decoding strategies proposed predict the next token. The decoded token HHuang et al. (2024); Chuang et al. (2023); Chen et al. (2024a) is concatenated with the last of the original input text for the next-token generation until the generation ends.
5.2 INFORMATION FLOW OF IN LVLMS
Some research Huang et al. (2025): Zhang et (2026) Zhao et (2026) uses Crad-CAM and attention maps to visualize the interaction between images and text in complex reasoning tasks. Attention scores highlight relevant areas through forward propagation. The EAH Zhang et al. (2024a) identifies that most hallucinations stem from the attention sink pattern marked by images in the attention matrix. Based on this insight, EAH proposes method that enhances attention heads without additional training. TAME Tang et al. (2025a) and Farsight Tang al.) (2025b) investigate the causes of hallucinations by analyzing local self-attention patterns "anchor tokens" and defines the degree attentional localization as the probability of token propagation.
2024 Wei & Zhang 2024); Zhang et al. (2024b) 2025c by: Zhao et al.
CONCLUSION
In this work, we revisit the conventional explanations linking attention sinks to hallucinations and propose saliency-based framework complement existing analyses. Our findings reveal that hallucinations frequently correlate with weak saliency in prior output tokens. To this end, we introduce SGRS and LocoRE, aalg-andd-llay intervention that dynamically boosts visual attention and reinforces local coherence during text generation. Experiments confirm that LocoRE consistently improves output accuracy across various benchmarks without requiring model retraining.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation NO. 62273235, National Major Scientific Research Instrument Development Project (62227811).

---

Published as a conference paper at ICLR 2026
REFERENCES
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qian Ying Wang, Guang Dai, Ping Chen, and Shijian Lu, Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local attention. arXiv preprint arXiv:2406. 12718, 2024.
Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, Yichi Zhang, Ruoyu Wu, Qingxiu Dong, Ge Zhang, Jian Yang, Lingwei Meng, Shujie Hu, Yulong Chen, Junyang Lin, Shuai Bai, Andreas Vlachos, Xu Tan, Minjia Zhang. Wen Xiao, Aaron Yee, Tianyu Liu, and Baobao Chang. Next token prediction towards multimodal intelligence: comprehensive survey, 2024a. URL https://aara brq//bs.. / 2412 18619
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and and-play inference acceleration for large vision- language models. 18th European Conference on Computer Vision ECCV 2024, 2024b.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425. 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24199, 2024d.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Vidcollama 2: Advancing ppatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.07476. 2024.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint trXii:2309.033883, 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Hao Fang. Changle Zhou, Jiawei Kong, Kuofeng Gao, Bin Chen, Tao Liang, Guojun Ma, and Shu- Tao Xia. Grounding language with vision: A conditional mutual information calibrated decoding strategy for reducing hallucinations in Ivlms. arXiv preprint arXiv:2505.19677 2025.
Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. Damro: Dive into the attention mechanism Ivlm to reduce object hallucination. arXiv preprint arXiv:2410.04514 2024.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617, 2018
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection cnooppectionaallocation allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspecttvve decoding: Alleviating hallucinations for large languaee models. In The Thirteenth International Conference Learning Representations, 2025. URL https:
7 /ooenneview. wwt.ieewwweeeeeeee..m.m. http://t/oooommm/////////////
(HHDD
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large dision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411. 16724, 2024.
10

---

Published as a conference paper at ICLR 2026
Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. Why language models hallucinate. arXiv preprint arXiv:2509.0465 2025.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. 2025.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decod- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13872-13882, 2024.
Jiaming Li, Jiacheng Zhang, Zequn Jie, Lin Ma, and Guanbin Li. Mitigating hallucination for large vision language model by inter-modality correlation calibration decoding, arXiv preprint arXiv:2501.01926. 2025a.
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal Ilms follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprini arXiv:2305. 10355, 2023b.
Zhuowei Li, Haizhou Shi, Yunhe Gao, Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang. and Dimitris N Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. arXiv preprint arXiv:2502.03622 2025b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. 2023.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in vision-language models via latent
space steering. arXiv preprint arXiv:2410. /5778, 2024b.
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in Ivlms. In European Conference Computer Vision, pp. 140-140. Springer, 2024c.
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-fiee free method for alleviating hallucination in Iylms. arXiv preprint arXiv:2407.21771, 2024d.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chur -Chn Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, et al. :ellow-your-emoji: Fine-controllable and expressive freestyle portrait animation. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-12, 2024.
Yue Ma, Kunyu Feng, Zhongyuan Hu, Xinyu Wang, Yucheng Wang, Mingzhe Zheng, Xuanhua He, Chenyang Zhu, Hongyu Liu. Yingqing He, et Controllable video generation: A survey, arXiv preprint 1XX.:22007.16699 2025a.
Yue Ma, Kunyu Feng, Xinhua Zhang, Hongyu Liu, David Junhao Zhang, Jinbo Xing, Yinhan Zhang, Ayden Yang. Zeyu Wang. and Qifeng Chen. Follow-your-creation. Empowering 4d creation through video inpainting. arXiv preprint arXiv:2500.04459. 2025b.

---

Published as a conference paper at ICLR 2026
Yue Ma. Yingqing He, Hongfa Wang, Andong Wang, Leqi Shen, Chenyang Qi. Jixuan Ying, Chengfei Cai, Zhifeng Li, Heung- Yeung Shum, et al. Follow-your-che Open-domait regional image animation via motion prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 6018-6026, 2025c.
Yue Ma, Yulong Liu, Qiyuan Zhu, Ayden Yang, Kunyu Feng, Xinhua Zhang, Zhifeng Li, Sirui Han, Chenyang Qi, and Qifeng Chen. Follow-your-motee mootio:: Video motion transfer via efficient spatial-temporal decoupled finetuning. arXiv preprint arXiv:2506.05207, 2025d.
Yue Ma, Zexuan Yan, Hongyu Liu, Hongfa Wang. Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, et al. coneroontoll,ale, and expressive freestyle portrait animation. arXiv preprint arX1v:2509.16630, 2025c.
your-emoji-fasterr Towards efficient,
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 2023.
Kyungmin Min, Minbeom Kim, Kang-il Lee, Dongryeol Lee, and Kyomin Jung. Mitigating hallucinations in large vision-language models via summary-guided decoding. arXiv preprint arXiv:2410.13322 13321, 2024.
Dexter Neo and Tsuhan Chen. Vord: Visual ordinal calibration for mitigating object hallucinations in large vision-language models. arXiv preprint arXiv:2412. 2024.
Dongmin Park, Zhaofang Qian, Guangxing Han, and Ser-Nam Lim. Mitigating dialogue hallucination for large multi-modal models via adversarial instruction tuning, arXiv preprint arXiv:2403. 2024.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint 1xXi::8809002550 2018.
Yuying Shang, Xinyi Zeng, Yutao Zhu, Xiao Yang, Zhengwei Fang, Jingyuan Zhang, Jiawei Chen, Zinan Liu, and Yu Tian. From pixels to tokens: Revisiting object hallucinations large vision- language models. arXiv preprint arXiv:2410.06795, 2024.
Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXhv:240..17662, 2024.
Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. Intervening anchor token: Decoding strategy in alleviating hallucinations for MILMs. In The Thirteenth International Conference on Learning Representations, 2025a. URL htttp:://opereet//// met/foruue id zGb 4WCW51
Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, al. Seeing far and clearly: Mitigating hallucinations in milms with attention causal decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26147-26159, 2025b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023.
Zifu Wan, Ce Zhang, Silong Yong, Martin Q Ma, Simon Stepputtis, Louis-Philippe Morency, Deva Ramanan, Katia Sycara, and Yaqi Xie. Only: One-layer intervention sufficiently mitigates hallucinations in large vision-language models. arXiv preprint arXiv:2507.00896 2025.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305. 14160, 2023.
12

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Jinfeng Wei and Xiaofeng Zhang. Dopra: Decoding over-accumulation penalization and re-allocation in specific weighting layer. Proceedings of the 32nd ACM International Conference on Multimedia, 2024.
Sangmin Woo, Jaehyuk Jang, Donguk Kim, Yubin Choi, and Changick Kim. Ritual: Random image transformations as a universal anti-hallucination lever in Ivlms. arXiv preprint arXiv:2405.1782. 2024.
Jiarui Wu, Zhuo Liu, and Hangfeng He. Mitigating hallucinations in multimodal spatial relations through constraint-aware prompting. arXiv preprint arXiv:2502.083/1 08317, 2025a.
Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Generate, but verify: Reducing hallucination in vision-language models with retrospective resam- pling. arXiv preprint arXiv:25044 13169, 2025b.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.1592 2024.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jiannin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Jumyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2.306. 13549, 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 024900, 2023.
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406. 15765, 2024.
Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective. The 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhut- dinov, Louis-Philippe Morency, Katia Sycara, and Yaqi Xie. Self-correcting decoding with generative feedback for mitigating hallucinations in large vision-language models. arXiv preprint arXiv:2502.061300 2025a.
Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in Ivims. arXiv preprint arXiv:241 I. 1.09668, 2024a.
Xiaofeng Zhang, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie. Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Enhancing explainability in multimodal large language models. arXiv preprint 2666:92400..6557, 2024h.
13

---

Published as a conference paper at ICLR 2026
Xiaofeng Zhang, Fanshuo Zeng. and Chaochen Gu. Simignore: Exploring and enhancing multimodal large model complex reasoning via similarity computation. Neural Networks, 10e,,,8844 184:107059, 2025b.
Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. Enhancing multimodal large language models complex reason via similarity computation. AAAI, 2025c.
Xiaofeng Zhang, Yuanchao Zhu, Chaochen Gu, Jiawei Cao, Hao Cheng, and Kaijie Wu. What drives attention sinks? study of massive activations and rotational positional encoding in large vision-language models. Information Processing & Management, 63(2): 104431, 2026.
Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:2312.15710, 2023.
Qiyan Zhao, Xiaofeng Zhang, yiheng Li, Yun Xing, Yuan Xiaosong, Feilong Tang, Sinan Fan, Xuhang Chen, Xuyao Zhang, and Dahan Wang. Mca-llava: Manhattan causal attention for reducing hallucination in large laiiunl models. Proceedings of the 32nd ACM International Conference Multimedia, 2025.
Qiyan Zhao, Xiaofeng zhang, Shuochen chang, Qianyu chen, Xiaosong yuan, Xuhang Chen, Luoqi Liu, Jiajun zhang, Xu- Yao zhang, and Da-Han Wang. Context tokens are anchors: Understanding the repetition curse in diffusion MLLMs from information flow perspective. In The Fourteenth International Conference on Learning Representations, 2026. hR net/Eorum? id==m JVYx
Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. Mitigating modality prior- induced hallucinations in multimodal large language models via deciphering attention causality. arXiv preprint arXiv:2410.0445 044700, 2024.
Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. Mitigating modality prior- induced hallucinations in multimodal large language models via deciphering attention causality. In The Thirteenth International Conference on Learning Representations, 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Xin Zou, Yizhou Wang. Yibo Yan, Yuanhuiyi Lyu, Kening Zheng, Sirui Huang, Junkai Chen, Peijie Jiang. Jia Liu, Chang Tang, et al. Look twice before you answer: Memory-space visual retracing for hallucination mitigation multimodal large language models. arXiv preprint orXiv:2410.03577. 2024.
A RELATED WORK
A.0.1 INFERENCE- EFFICIENCY
While our full framework (SGRS + LocoRE) achieves the strongest hallucination suppres- sion. it incurs higher latency due to the backward pass required for saliency computation in SGRS - typically adding 30-40% overhead per token compared to standard greedy decoding. While the full SGRS+LocORE framework achieves the strongest hallucination suppression, its reliance on gradient computation introduces non-negligible latency overhead - making it less suitable for real-time applications. In prac- tice, however, LocoRE alone serves highly effective compromise: as forward-only mod- ule that manipulates attention weights in- in-place, it incurs <2% latency increase while still signif- icantly mitigating context-drift hallucinations.
Conerstion Thme Sample Acmes Methode
As shown in Figure 6 compared to prior plug-and-play methods such VCD Leng
Neteed
Figure Generation time of a single response.
14

---

Published as a conference paper at ICLR 2026
The image features.. The bus Is... street appears residential area with a few "people"/ " ""houses"
Attention map people
Attention map "houses"
Saliency socre g people"
Saliency socre "houses"
pattor
The image features...Thor The bus Is... There are several other "benches"/ "pprson"
Attention map "benches"
Attention map "person"
Saliency socre "benches"
Saliency socre "person"
Mo puttor
pattirn
Patters
Pathond
Nih
Figure 5: Attention map and saliency map of LLaVAI.5-7B
et al. 2024). OPERA Huang et al.
2024 Far-
sight Tang et al. 2025b), HALC Chen et al.
(2024c), and EAH Zhang et al. (2024)) LocoRE requires no auxiliary models, external detec-
tors, and no multi-pass decoding. By operating entirely within the standard autoregressive loop,
achieves superior -peed-efficiency trade-offs.
A.1 SALIENCY SCORE
To reveal why the MLLM produces hallucination token, it is necessary to elucidate the infor- mation flow. In this section, we use saliency score to analyze the information flow across the different tokens(system, image. prompt, and output). In this section, we examine types of to- kentsystem//mage//rompp///
We can use a Taylor expansion to compute the saliency score for each element of the attention matrix:
OC(x) An. C
S An
(II)
DAi,
where As, denotes the value of the attention matrix for the Ab, attention head in layer 1, and denotes the input. L(z) is the loss function of the task, e.g., the cross-entropy of the quiz task
objective. The saliency matrix St for layer is obtained by averaging all heads of attention. More saliency maps and attention maps of LLaVA 1.5/Qwen2- VL are shown in Figure 5 and Figure 13 and Figure 14
A.2 INFORMATION FLOW OF IN LLMS
Information flow provides an intuitive method of understanding the internal mechanisms of the black-box models of LVLM. Label words Wang et al. (2023). and ACT Yu et al. (024)) are early
works that explore the mechanism of LLM&Zhu et (2023 : Devlin al. (2018) Touvron al.]
(2023) by rving information flow patterns. By calculating saliency scores, it is possible to visualize information flow.
StreamingL.MM Xiao et al. (2023) introduces the concept of attention sink, observing an intriguing phenomenon: Initial tokens, despite their seemingly minor role in content generation, consistently receive high attention scores. This is visualized in the attention map columns with notably high attention scores, which is counterintuitive. Due to the autoregressive nature of generative models, these initial tokens continue attract attention from subsequent tokens, amplifying their impact on the generation process. To address this, StreamingLLM leverages attention sink tokens during the pre-training phase to enhance the model's performance.
15

---

Published as a conference paper at ICLR 2026
Generate time siep
gray
The roors feature a slepal wailing
with striped wallgaper pallern
is shades Max
X
Decoder layer
Local Coherence Reinforcement
Progresitv
Aigh saleegy
room
slepal cilling striped wallpaper paitern
gray
The
window
Next token prediction (B)
(A)
Figure 7: The structure of Local Coherence Reinforcement (LocoRe): attention from the next token recent outputs is enhanced to preserve contextual coherence.
Massive activation$Sun et al. (2024) highlights that, while there are approximately 40,000 activations per hidden state, only four are recognized. In the feature dimension of language models, large activations consistently occur in very small number of fixed dimensions. LLMs are categorized into three types based on the location of massive activations: (a) occurring only at the onset, (b) occurring at the onset of lexical elements and the first "strong separator" word (e.g. ".", "/n"), (c) occurring at the onset, separator words (e.g., "" "/n"), and the first "strong separator" word, as well as some semantically weaker words (e.g.. "and", "from", "of").
A.3 INFORMATION FLOW OF LVLMS
LLaVA-CAM Zhang et al. (2024b) 2025c b); Wei & Zhang (2024) utilizes Grad-CAM and attention maps visualize the interaction between images and text in complex reasoning tasks. Attention scores highlight relevant areas through forward propagation, while Grad-CAM captures gradient changes through backpropagation, revealing the salience of image features. These complementary approaches provide comprehensive understanding of the dynamics of information flow by assessing the importance of input and demonstrating their specific impact on model predictions.
The EAH study Zhang et al. (2024)) identifies that most hallucinations stem from the attention sink pattern marked by images in the attention matrix. To address this, EAH proposes a method that enhances attention heads without additional training. By strengthening attention heads with visual depression characteristics in shallow layers, the method improves attention distribution for image tokens, effectively reducing ballucinations across various LVLMs.
TAME Tang al. (2025a) investigates the causes of hallucinations by analyzing local self-attention patterns of ancchr points" and defines the degree of attentional localization as the probability of token propagation. The analysis reveals that over-vverooeeeoo propaaatonnn of anchor tokens occurs when the eigenvalue distributions of the query and key matrices exhibit non-zero mean and polarized variance, leading to an over-reliance on anchor tokens while ignoring visual information, resulting in hallucinations.
As illustrated in Figure 8 in summary, Zhangaan et al. (2024a) differs from existing methods while remaining non-confiiting and even complementary. Existing methods primarily adjust decoding strategies by modifying logits. OPERA Huang et al. 20024 and DOPRA Wei & Zhang (2024) identify that anchor output tokens can lead to hallucinated token generation and try to penalize anchor tokens' logits. TAME Tang et al. (2025a) focuses on the propagation of the anchor token in all layers, dynamically adjusting these anchor tokens.
A.4 LIMITATIONS
The primary limitation of our Saliency-Gudded Rejection Sampling (SGRS) framework lies in its computational demands: computing token-level saliency requires storing intermediate activations and performing backward passes during inference, which consumes significant GPU memory. As result, we are currently unable to deploy SGRS on very large models such as 72B 2B--aarameter LLMs, where gradient computation exceeds the memory capacity of even high-end GPUs (e.g., A100 80GB).
16

---

Published as a conference paper at ICLR 2026
(1) OPERA
(2) TAME
(3) EAH
enhance vision sink
Attention map
hend attention
Output tokens
Img
Out
Image
Image
System Tokens
(4) Vissink
(5) Farsight
ImageTokens
attention
reallocation
reallocation
Prompt Tokens
Output tokens
Img
Img
Image
Image
Figure 8: The information flow of different models.
This restricts evaluation to models up 1BB13B parameters (e.g., LLaVA-1.5, Qwen2-VL-7B), limiting the generalizability of our full framework to the largest-scale architectures.
However, emphasize that LocoRE module remains fully applicable to any model size, as it operates purely in the forward pass and requires no gradient computation. For large models where SGRS is infeasible, LocoRE alone provides lightweight, plug-and-play solution that still significantly mitigates context-drift ballucinations.
A.5 VIDEO BENCHMARKS EVALUTION
In Zero-Shot Video Question Answering Tasks, LocoRE achieves significant improvements over video MILMM such Video-LLaVA [in et al. (2023) and Video-LLaMA2 Cheng et al. (2024) in three key benchmark datasets. As shown in Table on the MSRVTT-QA dataset, our method delivers average accuracy gain.
Table 4: Comparison of different Video LVLMs and LocoRE across all video benchmarks. In the Video-Based Text Generation Benchmark, five scores are assessed: Cr. (Correctness Information), Cs. (Consistency). De. (Detail Orientation). Ct. (Contextual Understanding) and Te. (Temporal Understanding). Following Maaz et al Maaz et al. (2023), we use the GPT-3.5 Turbo model to assign relative to the model outputs, with scores ranging from to 5.
Method
MSVD-QA
MSRVIT-QA
ActivityNet-QA Accuracy
Video-Based
Generation
Accuracy
Scoret
Accuracyt
Score?
Score
Cr.
Cs.l
De.j
Ct.
To..
Video LLaVA
64.8
3.7
59.0
3.5
41.5
3.3
2.32
2.34
2.65
2.75
2.09
LocooRE (Ours)
(+1.))
(42.))
(40.4)
Video LLaMA2
70.9
3.8
67.2
3.6
49.9
3.3
3.13
3.23
2.70
3.42
2.45
(Ours)
(+0.9)
(+2.7)
52.2 (42.3)
3.36
Differences with other fusion strategies: We compared the other three fusion strategies:
After the attention and gradient are added, the visualized image shows an overall high state, and the pattern cannot distinguished at all. It meaningless pattern. The fusion scores of many tokens are concentrated in the middle range, and the overall appearance is "gray", making difficult to distinguish the key tokens.
Addition:
Maximum value (Max): Take the larger of the attention and gradient. Although it can amplify individual high values, the visualization result still cannot distinguish the effective pattern.
Concat + MLP: Gradient and attention are spliced and then adaptively fused through the neural network. The score distribution is rich. The visualization similar the addition.
17

---

Published as a conference paper at ICLR 2026
Max-value "bea"
Max-value "chil"
Pieose describe the inoge
Mmb are
room
near the bes", onocher
arowddogg
additionaa sloaping space
15 fusion ateataatttiooonrdieen "bod"
Attention-gradlent
Attention-gradient 1 "Chid""
fusion-attentoon-rrddiie
@ "bod"
W "Chad"
Reputtve
pattem
patternn
Figure 9: The saliency map of output token. We compared the other three fusion strategy including Addition, Maximum value (Max) and concat attention+ by mip, there is no pattern between normal and hallucination token, in contrast, attention * gradient fusion is clearer in distinguishing important from unimportant tokens.
The lower triangle shows color close to the same, and the effective pattern cannot be distinguished.
In contrast, iitention&gradien fusion is: clearer in distinguishing important from unimportant tokens.
USE OF LLM
The authors used generative Al tools (e.g., Grammarly, ChatGPT) solely for grammar checking and language polishing of the manuscript. All technical content, experimental design, data analysis, and conclusions were generated and verified exclusively by the human authors. The use of AI tools does not affect the originality or authorship of this work.
B ETHICS STATEMENT
This work focuses on improving the reliability of large Dision-languaan models (LVLMs) by mitigating hallucination through inference-time interventions. Our method, SGRS+LocoRE, operates solely on publicly available models (e.g.. LLaVA-1.5, (wwn2-VL) and benchmark datasets (e.g., CHAIR, POPE, MME), without collecting using any private, sensitive, or human-subject data. The proposed techniques do not introduce new biases beyond those already present in the base models, and they are designed enhance that while our method reduces hallucination, it does not eliminate all risks of harmful or mislcading outputs. Users should exercise caution when deploying LVLMs in high-stakes scenarios such as medical diagnosis, legal advice, or autonomous decision-making.
replace human oversight in critical applications. We acknowledge
C REPRODUCIBILITY STATEMENT
To ensure full reproducibility, we provide the following resources: (1) Code: Complete implemen- tation of SGRS and LocoRE, including saliency computation, rejection sampling, and attention reinforcement modules, will be released publicly on GitHub upon publication. (2) Hyperparameters:
All key hyperparameters (a = 0.6, 3 = 0.15 for LLaVA-1.5; 3 0.20 for Qwen2- (wee2--L-7BB) and
training-free inference protocols are detailed in Section 4.5
(3) Evaluation: We standard,
18

---

Published as a conference paper at ICLR 2026
publicly available benchmarks (CHAIR, POPE, MME) with official evaluation scripts. All results averaged over 500 samples with fixed random seeds. (4) Compute: Experiments are conducted on NVIDIA A100 80GB GPUs; average latency is reported in ms/token (Figure 4.. (5) Models: We evaluate on open-source LVLMs: LLaVA-1.5 (13B) and (wwe2--VL-770 using official checkpoints from Hugging Face Model Hub. No proprietary data models used in this work.
D MORE EXPERIMENTS IN REBUTAL
D.1 STATISTICAL VALIDATION OF THE SAJIECY-HALLUCIMTIIN RELATIONSIIIP
hype
Cookiaes Negaiih ty 15
Faber phap
(A)
(B)
Figure 10: Statistical analysis of output-token saliency vs, hallucination. (a) Mean saliency for correct vs. hallucinated tokens across three models. (b) Hallucination probability as function of saliency bin (per model average).
To rigorously test our core hypothesis: "The saliency score of hallucination tokens is often relatively low:" conduct three complementary quantitative analyses the token level across three diverse VLMs: LLaVA-v1.5-7B Qwen2-VL. VL-7B, and InterVV--7.. All experiments are performed on the POPE and CHAIR benchmarks, with hallucination labels assigned via human annotation.
Token-level saliency distribution: hallucinated vs. correct tokens. For each generated token yt in dataset (~~ 12,00 tokens total), compute the saliency score from the immediately preceding output token to the current token. We then group tokens by label (correct or hallucinated) and report mean standard deviation.
As shown in Figure 10 and Table 5 a consistent and statistically significant pattern emerges across all models:
Table 5: Mean saliency scores for correct vs. hallucinated tokens across models.
Model
Correct Tokens
Hallucinated Tokens
LLa VA-v1.5-7
0.472 + 0.136
0.193 + 0.087
Owen2-VL-771
0.664 0.158
0.355 0.103
Intern VL--BB
0.508 + 0.124
0.224 + 0.095
These results confirm that the significantly lower saliency scores of hallucinated tokens, compared with correct tokens, is phenomenon that generalizes across different model architectures.
(2) Saliency score and negative correlation with hallucination: As shown in Figure 10 we divide the saliency score of the previous output token into 10 equally wide intervals and calculate the conditional probability of ballucination in each interval. All three models (LLaVA-v1.5- 7B, DW-72- and InternVL-7B) showed strong negative correlation: the hallucination rate systematically decreased with increasing saliency. A clear, smooth, and monotonic negative correlation is evident: <1> In the lowest saliency (0.0,0.1). hallucination rates reach 68 18%%766% %:
19

---

Published as a conference paper at ICLR 2026
Table 6: Hallucination experiments that artificially lower saliency scores
Decay rate
CHAIRs 4
POPE-FI
POPE-A 1
1.0
35.6
87.0
87.5
0.8 (decay 20%)
37.9
86.5
86.8
0.6 (decay 40%)
42.1
85.4
85.6
0.4 (decay 60%)
47.8
84.8
84.0
0.2 (decay 80%)
56.0
83.0
83.8
<2> In the highest [0.9, 1.0], rates drop to 18%-28% The trend holds across all models, with no non-monotonic jumps or plateaus.
(3) Saliency Intervention Experiment: As shown in Table we also conducted an intervention experiment on the L.LVAA-v -.--7B model. For each sample in the POPE and CHAIR datasets, highly significant correct tokens were selected for intervention (these tokens came from the correct tokens with saliency > 0.45 Step 1). The intervention method was as follows: after generating the target token, its saliency output in the decoder was scaled (multiplied by factor e 11.0,0.8,0. 0.00..0.. to simulate the process its saliency being weakened. The results showed that after the saliency value was artificially reduced, the hallucination rate increased significantly."
Conclusion. These findings support our claim that hallucinations are not triggered by a single threshold event, but rather emerge gradually as contextual saliency decays. This gradient nature suggests that saliency can serve as continuous diagnostic signal.
D.2 FAILURE CASE: HIGH-SALIENC HALLUCINATION
bench
conversation
high saliency
high saliency
sentencel The image depicts a man and a woman sitting on a sidewalk near a street. sentence2 The man is sitting on a chair, while bench.
sentence3 They appear to be engaged in conversation or enjoying each other's company,
Figure 11: Failure example. Even though it's an hallucination token, the output saliency is still high.
Regarding our core claim that hallucinated tokens overwhelmingly exhibit low saliency, although this is strongly supported by extensive statistical evidence, we also identify several failure cases in which hallucinated tokens instead display relatively high saliency scores. Figure TTaa) illustrates such an instance: on Dwwe2--VI-77. the ground-truth answer is "a traffic cone".This contradicts the low-saliency hypothesis and reveals two fundamental limitations:
(1) Context-independder generated content: The effectiveness of the method may decrease when the content generated by the model deviates significantly from or is inconsistent with the current context. Specifically, when the saliency of a candidate token is low, indicating that the currently generated content lacks relevance to the previously generated content, SGRS will reject these tokens. However, in some cases. if the context itself is ambiguous or the input information is insufficient, the model may generate irrelevant content, which may not pass the SGRS filter even if conforms to the rules of language generation.
20

---

Published as a conference paper at ICLR 2026
(2) Some incorrect tokens may have high saliency because the model believes that the token it outputs at this time correct (high confidence). This observation is consistent with the conclusion proposed by Adam et al. of Openai Kalai al. (2025): "The model will make mistakes with confidence", The reason for this problem that </> the model is trained to output seemingly reasonable answers (high confidence) instead of expressing "I don't know". <2> after human RLHF, the model becomes overconfident.
D.3 LONG SEQUENCE HALLUCINATION TOKEN AND LAYER EXPERIMENT
As show in Figure 12 tion case:
we performed token-level magnified visualization of the following hallucina-
Lo)g sequence experiments: As shown Figure 12 in the generated sequence, a ballucination token (e.g.. "few") appears in the third sentence, while the first sentence (e.g., "preparing") and the fourth sentence (e.g.. "significant") are both correct outputs. This shows that even in different sentences and adjacent positions, the saliency of hallucination tokens is significantly lower than that of the correct tokens preceding and following them.
preparing
few
significant
saliency
salien
sentence1 A man is standing next to the train, holding a green trash bag, possibly preparing to throw it the train's trash can,
sentence2 Another person is standing in the train, looking out the window. sentence3 They are a few other people in the scene, one of them holding handbag. sentence4 The train occuplies significant portion of the image.
Long sequence example. comparison of the saliency of the correct tokens before and after the hallucination token shows that the saliency of the correct tokens before and after the hallucination token is still greater than that of the original token.
Figure 12:
(2) Layer experiment:As shown in Figure 13 and Figure 14 show the saliency distribution of correct and hallucination tokens across different layers. We observe that the saliency of correct tokens is relatively high across both shallow and deep layers, while the saliency of hallucination tokens is relatively low across all layers. Regarding attention heads, they serve only as intermediate computational units and are individually dependent. The LVLMs-Saliency calculation logic in our paper is as follows: first, calculate the saliency matrix for each attention head (Equation 4), then average it across all heads (Equation 5), finally outputting the layer- saliency. Our entire method does not bind to any specific attention head, nor does it claim that any particular head/type of head key to the connection between saliency and hallucination. We only use attention heads as the basic carrier of attention weights, and after aggregation, they are no longer considered individually.
21

---

Published as a conference paper at ICLR 2026
Layer1
Layer4
QD
Layer8
DO
Layer12
LOW saliency
Low sallency
Low salieny
Low salency
vin
Layer16
Layer20
Layer24
Layer32
saliency
Low saliency
Figure 13: Saliency map LLaVA1.5 from layerl layer32 (hallucination pattern..
Layer1
Layer4
Layer8
Layer12
High vallency
High sallency
salency
High saliency
Layer16
Layer20
Layer24
Layer32
High sallecy
High soliency
High sallency
High sallency
Figure 14: Saliency map of LLaVAI.5 from layerl to layer32 (correct pattern).
22


Reviews:
[{"review_id": "8gmOY3tDOI", "reviewer": "Reviewer_rJHs", "summary": "Many previous studies have focused on reducing hallucination by leveraging attention-based information. In contrast, this paper innovatively employs saliency (i.e., gradient information) to mitigate hallucination. By incorporating the saliency of previous tokens to guide the prediction of the next token, the authors propose SGRS and LocoRE modules to reduce hallucination.", "questions": "see Weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method in this paper is highly novel. While most previous studies have focused on leveraging attention mechanisms to reduce hallucination, this work explores the use of saliency, which represents a promising and valuable direction for further research.", "weaknesses": "1. The key finding and the basis for the proposed method in this paper is that “Hallucinations occur when prior output tokens show low saliency to the next token prediction, indicating a failure of contextual memory.” However, after reading the entire manuscript, I could not find any statistically significant validation of this claim. Figure 1 appears to be merely a case study and does not provide statistical evidence to support the finding.\n\n2. From Table 1, the proposed method does not appear to demonstrate a statistically significant improvement over previous approaches. The performance differences are relatively minor and may fall within the margin of experimental variability.\n\n3. The proposed method relies heavily on a large number of hyperparameters, which significantly limits its practical applicability. Moreover, the paper does not provide a comprehensive analysis of the impact of these hyperparameters—only a partial examination is presented. For instance, the parameters required in Algorithm 1 and Algorithm 2 are not thoroughly analyzed or discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rIFBo4j1sB", "reviewer": "Reviewer_EfHa", "summary": "This paper proposes \"LVLMs-Saliency,\" a diagnostic tool for quantifying token-level grounding in large vision-language models (LVLMs), aiming to better distinguish hallucinated from correct outputs by combining attention weights and their gradients. From the identified pattern—hallucinations arising where saliency from previous output tokens drops—the authors introduce a dual mechanism: Saliency-Guided Rejection Sampling (SGRS) for filtering low-saliency tokens during decoding, and Local Coherence Reinforcement (LocoRE) to strengthen attention among recent outputs, both at inference-time. Experimental results on major benchmarks and several popular LVLMs demonstrate significant hallucination mitigation and competitive or improved performance metrics.", "questions": "1. Given the memory-intensive nature of SGRS due to backward passes, which limits testing to models up to 13B parameters, could the authors explore or implement optimizations?\n\n2. The paper claims a \"direct causal link\" between low saliency and hallucinations, but the supporting evidence appears largely correlational. Could the authors provide controlled experiments, such as synthetic interventions where saliency is artificially boosted or reduced in specific output tokens, to demonstrate causality (e.g., measuring resulting changes in hallucination rates on a subset of benchmarks like CHAIR or POPE)? \n\n3. The absence of error analysis leaves unclear when and why the method fails. Could the authors include a qualitative breakdown of failure modes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a compelling, interpretable explanation of LVLM hallucinations, showing through both empirical results and visualizations that attention-alone is insufficient and that joint attention-gradient saliency captures key failure modes (see Figures 1 and 2). \n\n2. The proposed SGRS (Algorithm 1) and LocoRE (Algorithm 2) do not require retraining, operate at inference, and effectively use saliency to dynamically control and reinforce generation. This “plug-and-play” aspect increases their practical value.\n\n3. Tables 1 and 2 demonstrate consistently strong gains in hallucination reduction (e.g., substantial CHAIR-S and POPE improvements), with Table 3 and Figure 5 providing ablation studies on key hyperparameters ($\\alpha$, $\\beta$), establishing the contribution of each module.\n\n4. The latency/efficiency trade-off discussion (Section 4.4.1, Figure 4) is practical and relevant for deployment scenarios, and the LocoRE-only alternative appears effective for large models or latency-critical settings.", "weaknesses": "1. The SGRS component's reliance on backward passes during inference imposes significant memory constraints, limiting its applicability to models up to 13B parameters and preventing scalability to larger LVLMs like Qwen2.5-VL-32B or 72B, which undermines the method's claimed broad generalizability and real-world deployment feasibility.\n\n2. While the paper asserts a direct causal link between low saliency and hallucinations, the evidence is primarily correlational from observational patterns in figures and benchmarks, lacking rigorous controlled experiments or ablations to demonstrate causality, such as interventions that artificially manipulate saliency levels to observe corresponding changes in hallucination rates.\n\n3. Despite strong aggregate benchmark results, the paper provides virtually no error analysis or qualitative examination of failure cases, such as specific hallucination types (e.g., knowledge-based or relationship errors) that the method fails to mitigate, domain-specific limitations like medical imaging, or edge cases involving rare objects or noisy inputs, which obscures the true robustness and variability of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KnqxjjMflt", "reviewer": "Reviewer_YszH", "summary": "This paper identifies an interesting pattern in LVLMs: Hallucinations occur when prior output tokens shows low saliency to the next token. Building on this, the authors propose two mechanism for mitigating halluciantion: (1) Saliency-Guided Rejection Sampling (SGRS), which  filters candidate tokens based on their saliency with respect to prior output context. (2)  Local Coherence Reinforcement (LocoRE),  strengthens attention from the current token to its most recent predecessors.  With extensive experiments, the proposed demonstrates\nsignificant hallucination-mitigating performance across different LVLMs on image hallucination and generation benchmarks.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper offers a novel and well-motivated perspective on hallucination detection: leveraging gradients of attention weights to localize hallucination-prone tokens. To the best of my knowledge, this is the first systematic use of this signal, underscoring the work’s novelty. The two inference-time interventions—SGRS and LocoRE—are tightly coupled to this insight and translate it into practical, training-free improvements with minimal changes to the base model. The experimental study is thorough, and consistently shows gains over strong baselines, lending credibility to the approach. The paper is clearly written and easy to follow (I enjoy reading this paper); overall, the contribution is both original and useful.", "weaknesses": "My main concern is that the claimed “hallucination pattern” is supported largely by a few curated cases (Figs. 1–2), which risks selection bias. To make the claim compelling, the paper should provide population-level, statistically significant evidence to substantiate this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JQbS2cKmN5", "reviewer": "Reviewer_xsTD", "summary": "## Overview\n\nThis paper introduces LVLMs-Saliency, a gradient-aware diagnostic that fuses attention weights with their gradients to quantify how strongly the next token prediction is grounded in prior output tokens. The central empirical finding is that hallucinations emerge when saliency from recent outputs collapses. Building on this, the authors propose two inference-time mechanisms: Saliency-Guided Rejection Sampling (SGRS) to block low-saliency candidate tokens during decoding, and Local Coherence Reinforcement (LocoRE) to strengthen attention from the current token to its most recent outputs. Experiments across multiple LVLMs show reduced hallucinations and improved reliability. \n\n## Motivation\n\nExisting mitigation methods often rely on forward-pass attention alone, making it difficult to reliably distinguish correct from hallucinated outputs or to explain *why* hallucinations arise in autoregressive generation. Recent “attention sink” analyses add perspective but still lack a token-level, causally informative signal tied to the next prediction. The paper frames this gap and motivates a diagnostic that incorporates gradient information to reveal how token influence propagates, thereby connecting low output-token saliency with hallucination.  \n\n## Methodology\n\nLVLMs-Saliency: For each layer/head, the method computes a saliency matrix as the Hadamard product of the attention matrix and its loss gradient (with causal masking), then aggregates across heads and layers to read saliency from previous outputs to current position. This provides a token-level grounding score for the next token. \n\nSGRS: During decoding, sample candidates for the next token and accept only those whose grounding saliency exceeds an adaptive threshold tied to recent outputs, thereby preventing low-saliency tokens from entering the sequence. \n\nLocoRE: After a token is accepted, multiplicatively reinforce attention from the next query to a short window of the most recent output tokens, helping the model maintain local coherence in subsequent steps. \n\n## Experimental results\n\nAcross LLaVA-1.5, Qwen2-VL, and Intern-VL variants, LocoRE and SGRS+LocoRE generally reduce hallucination metrics (lower $CHAIR(_S)$/$CHAIR(_I)$; higher POPE-Recall/F1/Acc) relative to baselines and several decoding-time methods. Representative tables show consistent gains on hallucination benchmarks and competitive performance on a series of VQA suites.  \n\n## Analysis\n\nAblations on α (SGRS sensitivity) and β (LocoRE gain) indicate that (i) SGRS accounts for most hallucination reduction, (ii) modest LocoRE gains further improve POPE/CHAIR, and (iii) overly large β degrades results; the paper highlights workable settings. The authors also note that LocoRE is forward-only and thus lightweight compared with multi-pass or detector-based methods.", "questions": "Please refer to Weaknesses section. I would consider raising my score if authors can address my concerns.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* 1. Clear, mechanistic diagnostic for hallucination.\n   The paper isolates a simple, testable pattern—hallucination coincides with a collapse of saliency from prior output tokens to the next token—that standard attention maps miss. The gradient-aware *LVLMs-Saliency* definition is mathematically explicit (Hadamard product of attention and its gradient with causal masking) and tied directly to the next-token prediction.      \n\n* 2. Strong empirical coverage across models and benchmarks and SOTA comparisons to strong baselines.\n   Results span LLaVA-1.5 (7B/13B), Qwen2-VL (7B/13B/32B), and Intern-VL (7B/13B), with evaluations on MME (Hallucination), MM-Vet, VizWiz, ScienceQA, POPE, and CHAIR. Gains are consistent on hallucination metrics (lower $CHAIR(_S)$/$CHAIR(_I)$, higher POPE Recall/F1/Acc) and often maintain or improve general VQA ability.  \n   On LLaVA-1.5-7B, LocoRE and especially SGRS+LocoRE outperform or match recent training-free methods on POPE/CHAIR while remaining competitive on MME; the tables include recent ICLR/NeurIPS/CVPR baselines.   \n\n* 3. Ablations and hyperparameter guidance.\n   The work analyzes the effect of α (SGRS sensitivity) and β (LocoRE gain), and discusses trade-offs among hallucination rate, recall, and latency, giving practical guidance for tuning. \n\n* 4. Compelling qualitative evidence aligned with the mechanism.\n   Visualizations show cases where attention maps look similar for correct vs. hallucinated tokens, but saliency sharply diverges; with LocoRE, the same position flips from an incorrect token to a correct one alongside restored output-token saliency.", "weaknesses": "* 1. Quantitative validation of the saliency–hallucination link is missing.\n  The saliency-guided hallucination detection has been shown through cases, but not quantitive analysis. Does such phenomenon always appear in no matter short or long generated content? and does low saliency always suggest hallucinated content, or hallucination content are more likely to be with low saliency? Authors are suggested to conduct quantitive analysis for their saliency-guided motivation.\n\n* 2. SGRS behavior during inference is under-specified.\n   Report per-benchmark stats: percentage of steps with at least one rejection, average resamples per token, distribution tails, and a false-rejection rate for correct tokens. Break out short QA vs. long captioning.\n\n* 3. Hyperparameter portability is unclear.\n   Seems that the hyper-parameters alpha and beta need to be adjusted for each new MLLM. Does it have to take several rounds of tests till an optimal setting comes out? \n\n* 4. Causality vs. correlation.\n   Can authors demonstrate that low output-token saliency is not merely correlated with position/length, as late tokens naturally have lower influence?\n\n* 5. Layer/head dependence.\n   SGRS aggregates over “target layers”. Can authors provide sensitivity to layer/head selection and show whether the diagnostic holds with earlier vs. deeper layers?\n\n* 6. Unclear Computation Cost.\n    Many production stacks don’t expose backward passes. Authors are suggested to provide full memory cost analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)