Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation
Authors: Xingyu Zhu
Kesen Zhao
Liang Yi
Shuo Wang
Zhicai Wang
Beier Zhu
Hanwang Zhang
Xiangnan He
Venue: ICLR 2026 Poster (2026)
Keywords: MLLMs
Alignment
LVLM
Hallucination
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Multimodal large language models (MLLMs) have achieved remarkable progress in vision–language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from the visual evidence. Existing mitigation strategies either demand costly supervision during training or introduce additional latency at inference. Recent vision-enhancement methods attempt to address this by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, leading to interference from background regions and distracting the model from critical cues.  To overcome this challenge, we propose an **A**daptive v**I**sual **R**einforcement framework for MLLMs, dubbed as **AIR**. AIR consists of two main components: prototype-based token reduction, which condenses the large pool of visual tokens into a compact subset to suppress redundancy, and OT-guided patch reinforcement, which quantifies the alignment between hidden state and patch embeddings to selectively integrate the most consistent patches into the feed-forward layers.  As a result, AIR enhances the model’s reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective and independent solution for building reliable MLLMs.

Published as a conference paper at ICLR 2026
LOOK CAREFULLY: ADAPTIVE VISUAL REINFORCE- MENTS IN MULTIMODAL LARGE LANGUAGE MODELS FOR HALLUCINATION MITIGATION
Xingyu Zhu Kesen Zhao", Liang Yi', Shuo Wang', Zhicai Wang',
Beier Zhu-; Hanwang Zhang", Xiangnan He
1 MoE Key Lab of BIPC, University of Science and Technology of China
2 Nanyang Technological University
xyzhuxyz&mail 1. ustc.edu. en, beier. zhu@ustc.edu.co
ABSTRACT
Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from the visual evidence. Existing mitigation strategies either demand costly supervision during training or introduce additional latency at inference. Recent vision-enhanceemen methods attempt to address this by re- inforcing visual tokens during decoding, but they typically inject all tokens indis- criminately, leading to interference from background regions and distracting the model from critical cues. To overcome this challenge, we propose vlsual Reinforcement framework for MILMs, dubbed as ATR. AIR consists token reduction, which condenses the large pool of visual tokens into a compact subset to suppress redundancy, and OT- guided patch reinforcement, which quantifies the alignment between hidden state and patch embeddings to selectively integrate the most consistent patches into the feed-forward layers. As a result, AIR enhances the model's ' reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MILLMs demonstrate that AIR substantially reduces hallu- cination while preserving general capabilities, establishing it as an effective and independent solution for building reliable MLLMs.
Adaptive
main components: prototype-basee
INTRODUCTION
Multimodal large language models (MLLMs) (Chen et al., 2023; Zhu al., 2024a;e; Liu et al., 2024a; Li et al., 2025; Liu et al., 2024b; Han et al., 2025; Wu et al., 2025ega;b;d) have achieved re- markable progress by unifying vision and language, enabling reasoning over interleaved text-image inputs. They have been widely applied in tasks (Wu & Yang, 2024; Wu et al., 2025c) such as vi- sual question answering (Wu et al., 2025a) and image captioning (Yang et al., 2023; Zhao et al., 2025). Despite these advances, MILMMs remain prone to hallucination (Jiang et al., 2025; Zheng et al., 2025; Yang et al., 2025a), where generated content inconsistent with the visual input, e.g., describing non-existent objects producing contradictory interpretations. This vulnerability poses a barrier to deployment in real-world scenarios.
Existing hallucination mitigation strategies can be broadly divided into training-time and inference- time methods. Training time approaches (Gunjal et al., 2024; Lyu et al., 2024; Fu et al., 2025; Yang al., 2025b) rely on additional annotations to fine-tune MILMs,, while inference-time approaches typically adopt contrastive decoding or reranking. Although effective, they either require costly supervision or introduce extra latency. Recent efforts (Fazli et al., 2025; Zheng & Zhang, 2025; Yin et al., 2025; Zou et al., 2024) have strengthened the contribution of image tokens during decoding. They are annotation-free and incur little additional overhead, making them broadly applicable in practice. Concretely, these methods improve grounding by re- e-iineccting visual tokens into the feed- forward network (FFN) (Zou et al., 2024; Yuan et al., 2025; Wan et al., 2025; Zhou al., 2025).
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Original Image rjjetiin
The painting shows a bear, fox, and a squirrel gathered around a picnic blanket, playing musical instruments in the forest.
Answer:
Question: 'What are the animals in the painting and what are they doing?"
Look with whole image
Look relevant patches
The painting shows at |baar, a cat, and rabbit sitting: around a tree stump, having tea, iand eating cookies together.
Answer:
Selected Injection (AIR)
(a)
Patch
Patch 2
Originallaae
Patch
Patch 1
LLAV4-15
Maniiit
VAF
AIl
0.6
Siri ariy Score
190.0
MiI-Denchh
23.8 CHA
Eestenne
0.4
Original Image
B0.37
Count
Patch
135.0
120,0
0.2
Tarslate
10
20
Layer Index
70.
(b)
(d)
Figure 1: Analysis of existing hallucination mitigation strategies in multimodal large language mod- els (MLLMs). (a) existing strategies hallucinate non-existent objects, while AIR produces faithful, image-grounded answers. (b) Similarity across decoder layers between hidden states and differ- ent visual tokens, showing that salient patches consistently achieve higher alignment than irrelevant ones. (c) Attention heatmaps comparing existing re-injection with AIR: prior methods spread at- tention irrelevant regions, whereas AIR focuses semantically critical areas. (d) AIR reduces hallucination across benchmarks with minimal impact on general multimodal performance.
Despite their success, we observe that visual inputs often contain substantial interference, such as background regions, which may include redundant objects or distracting semantics. Simply fusing all visual tokens into the decoding process can distract the model's 's attention from critical regions. As illustrated in Fig. 1 (a), we compare two injection strategies, The first strategy, original token injection (Zou et al., 2024), leads to hallucinations as the model attends to irrelevant background regions. In contrast, with relevant token re-injection, the model effectively mitigates hallucinations by focusing on the critical visual content. This also demonstrated by the heatmap depicted in Fig. (c). To explore this phenomenon in more depth, analyze the similarity between hidden states and visual tokens across decoding layers using -LLVA-115575 et al., 2014). As illustrated in Fig. 1 (b), effective target regions (e.g., patch and patch 2) consis- tently yield higher similarity, whereas background regions (patch 3) remain low. The similarity of the original image tokens lies between, suggesting that they dilute the importance of salient cues.
the MSCOCO dataset (Lin
Motivated by these findings, we propose AIR, an adaptive visual reinforcement framework that amplifies critical evidence and suppresses redundancy. Instead of re-injecting the full of visual tokens, AIR is built on two key components. The first, prototype-based token reduction, com- presses image tokens into a compact subset, filtering out repetitive background signals and reducing computation. The second, OT-guided patch reinforcement, leverages entropically regularized opti- mal transport evaluate alignment between hidden states and patch embeddings, ensuring that only well-aligned regions are integrated into the decoder's 's FFN. Together, these designs enable the model to focus on salient regions and mitigate hallucination in training- and efficient manner.
To validate the effectiveness of our framework, we conduct extensive experiments on multiple rep- resentative MLLMs, including VAL (Liu et al., 2024a), Qwen-VL Bai et al. (2023), and GLM-4V-9B Zeng et al. (2024). The results demonstrate that AIR consistently lowers hallucination rates on benchmarks such as CHAIR (Rohrbach et al., 2018) and POPE (Li et al., 2023b), while maintaining strong performance lation, as illustrated in Fig. I (d). These results highlight that AIR is a training- ffree framework that generalizes across diverse MI.LMs, providing an effective solution for hallucination mitigation.
complementary tasks including existence, counting, and trans-
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Hallucination in MLLMs. Object hallucination occurs when multimodal L.LMs generate fluent but visually inconsistent outputs, compromising reliability in multimodal reasoning. Existing mit- igation strategies fall into three categories. Training-bate -baeed methods fine-tune models with curated datasets (Gunjal al., 2024), enforce cross-modal alignment (Fu et al., 2025), apply preference optimization (Yang et al., 2025b), but these approaches require costly annotations and heavy com- putation. Post-processing approaches revise or filter responses with external models lightweight revisers, such as LURE (Zhou et al., 2024; Yin et al., 2024; Wu et al., 2024; Yang et al., 2025a; Liu et al.. 2025), which increase flexibility but add system complexity. Inference-time interventions modify decoding without retraining, for example logit shifting (Zhao et al., 2024) or contrastive decoding (Leng al., 2024; Wang et al., 2025), providing efficiency but sometimes reducing stabil- ity. Different from these paradigms, we introduce training-free approach that adaptively calibrates attention and selectively reinforces critical visual patches via optimal transport, achieving effective hallucination mitigation without fine-tuning or auxiliary models.
Optimal transport. Optimal transport (OT) provides principled framework to measure discrepan- cies between distributions by explicitly modeling the cost of transporting probability mass (Monge, 1781). Unlike pointwise similarity metrics (e.g., cosine distance), OT captures the global geometric structure of two distributions, yielding more faithful measure of semantic alignment. Although the exact solution is computationally demanding, entropic regularization with the Sinkhorn algo- rithm (Cuturi, 2013) has made OT scalable to high-dimensional problems. These advances have enabled a broad range of applications, including domain adaptation (Turrisi et al., 2022; Chang al., 2022), distribution calibration (Guo al., 2022), and image recognittion/clustering (Wang et al., 2023; Li et al., 2023a; Zhu et al., 2026). In vangune modeling, OT has been adopted to align modality distributions in few-shot learning (Zhou et al., 2022: Lazarou al., 2021; Zhu et al., 2025abb), refine prompts for cross-mode transfer (Chen et al., 2022), and improve zero-shot generalization (Zhu al., 2024d; Fang et al., 2025; Zhu et al., 2024h; Zhou et al., 2023). In contrast, our approach employs OT directly at inference: we compute the transport distance between origi- nal image tokens and patch embeddings, and use it as a fine-grained criterion to select patches that preserve critical visual semantics. The selected patches are then fused into the decoder, providing lightweight yet distribution-awant reinforcement of visual evidence.
METHOD
3.1 PRELIMINARIES
Multimodal large language models. Multimodal large language models (MLLMs) extend con- ventional large language models (LLMs) to jointly process text and images. An MLLM typically consists of a vision encoder, a text encoder, and an autoregressive decoder. Given textual query
I [11..... .2........ .I.] _z]] and an image v, the vision encoder extracts visual features and then transforms them into aligned visual tokens Z = |zz Z2, [ Let [x,,... xr.] denote the embeddings of x produced by the text encoder, and define the multimodal input sequence:
X Z. XI,...,XI..
(1)
At decoding step f, the transformer- bassed decoder produces unnormalized logits fo(. | X, Vet), from which the next-token distribution is obtained via the softmax:
p- X, ver) = softmax (f X, vc+)).
Yt p. X, vet), t= I....,, T.
(2)
Here, l denotes all model parameters and yer is the previously generated tokens.
Optimal transport. Optimal Transport (OT) (Monge, 1781; Wang et al., 2023) offers a principled framework for quantifying the discrepancy between two probability distributions. Consider two discrete measures in the feature space: TP = ΣM a,5(v; - v) and Q = Σ byō(u, - uu. where ó denotes the Dirac delta function, and [ and [U] are the number of support points in F and Q. respectively. Here, a [a1...., v and b
bicz) are probability vectors that sum to one. Given a cost matrix C E TR [ where C(i.j) is the element in C, denoting the cost of transporting unit mass from v, to uj. and the OT distance between IP and Q is formulated as:
dor (P, Q C) min C)2 s.t. Tl a, T vi = b,
(3)
T
3

---

Published as a conference paper at ICLR 2026
Transformer Layer x N
Question
What are the painting and they daing?
Textual tncoddr
2
Response:
In the painting, there is bear, cat, and rabbit sitting around tree stump that is used as table. They appear to having a meal together.
Dp
Input Image
Encoder
Visual
Topo
+ Text Taken
Image Toisnn
Hidden State
Patch Token
Figure 2: Overview our proposed AIR framework. Given a multimodal input (image and ques- tion), visual features are extracted by the visual encoder and aligned with textual embeddings via the projector. Inside the Transformer layers, visual tokens are first compressed through prototype- based selection remove redundancy, and then reinforced by patch-level alignment using optimal transport. These selective reinforcement strategies enrich the hidden states with salient visual cues, enabling AIR to produce safer, more faithful, and visually grounded responses.
where T I V|x is the transport plan specifying how mass is moved between the two distri- butions. The notation ()-)) denotes the Frobenius inner product, and | is an all-ones vector of dimension [V]. Directly solving Eq. (3) is computationally intensive. Prior work (Lazarou et al., 2021; Chen al., 2022) addresses this by employing the Sinkhorn algorithm (Cuturi, 2013), which introduces entropic regularization to achieve efficient optimization:
doT Q: C) mTn C) eh(T),
where l(()) denotes the entropy and 2 0 controls the strength of the regularization.
3.2 ADAPTIVE VISUAL REINFORCEMENT
Prototype-based token reduction. Our visual reinforcement strategy operates in the feed-forward network (FFN) of each Transformer block, which consists of two fully connected layers with non-linear activation:
FF) = (H W1) WT
(5)
where ((-) denotes the activation function and H represents the hidden states. As the Transformer goes deeper, attention tends become progressively biased toward textual tokens, which diminishes the contribution of visual tokens in later layers. A common remedy is to re-inject visual tokens into the FFN (Zou et al., 2024):
ZF = <HHZ' )z
where Z denotes the visual tokens aligned by the projector. However, since visual tokens are derived from the entire image, their length is typically large (c.g., = 576 in LLaVA), which introduces redundancy and noise. Directly re-injecting all tokens not only incurs unnecessary computational overhead but also prevents the model from focusing on the most informative regions. To address this issue, we first condense H into a compact subset. We compute a prototype h, = R Σx= h as coarse summary of the visual semanties, and rank tokens by their distance this prototype:
d(bg, hp) = l hpl12, A: == 1,....,, K.
(7)
Tokens with larger distances encode more distinctive cues not captured by the global prototype. We therefore retain only the Topq tokens:
H' + {he k E (opo((d(hs, hp))),
(
ensuring that subsequent reinforcement operates on a compact set of visual representations.
OT-guided patch reinforcement. While prototype selection reduces global redundancy, different image regions may still vary in importance. To further emphasize critical details, we crop the image into multiple patches { with corresponding embeddings mmm) 1m=1 where each T (z 2m x encodes fine-grained visual details. We model the original hidden states and patch-level tokens as discrete distributions:
P(h)
agd agō (hg h), 2m = Σ
5Tth 5(2
2).
(9)
n=1
4

---

Published as a conference paper at ICLR 2026
where 6.) denotes the Dirac delta function, and ak, by are normalized importance weights. The alignment between and Qm is quantified using the OT distance:
dot (I, Qmi Cm) = min ( m. Cm), Tm 20
s.t. Tmlo= a, T 1N = bm
(10)
where a [ ......... bm = I L and Cn (k, n)
cos (2t; 2 The transport
plan Tm is efficiently obtained via the Sinkhorn- Sinkhor-Knoop algorithm (Zhu et al., 2024d; Chen et al., 2022). For each patch m, we compute aggregated OT distance:
Q
N
doT(m) =
ΣΣ k=
Tm n) (m,,
(11)
A lower OT distance indicates stronger alignment with the original image and thus suggests that the patch retains more critical visual information. We therefore select patches with threshodling 7:
M {m dor(m) S T.
(12)
The embeddings of these selected patches are then fused with the original image tokens:
Z + concat (2 m mmm MM).
(13)
This selective fusion enhances the representation with critical visual details, strengthening the role of image information in the hidden states. The final re-injection into the FFN formulated as:
FFN(H ((WW,) W + HH'2 Z
(14)
3.3 THEORETICAL ANALYSIS
To validate the effectiveness of our OT-based patch selection in Dynamic Token Infusion, compare the sensitivity our OT distance metric dor(m) with the baseline cosine distance (m)) = KN Ek= Σ n= nk, In our framework, patch m is selected as useful if dor (m) S т. as lower OT distances indicate stronger alignment with critical visual semantics. Conversely, for the cosine baseline, a patch is selected if (m) I Teoe reflecting misalignment. We prove that the OT-based method achieves strictly higher sensitivity in distinguishing patches:
(dor(m)) - dor(m2)) > Ideox (m1) deas (m2))
(15)
except in the degenerate case where Cmn = C.mg-
Why OT enhances patch selection sensitivity. The OT-based metric employs an adaptive trans- port plan Tm,, computed via Sinkhorn- Knopp, prioritizing low-cost alignments (high cosine similar- ities) between original and patch tokens. This adaptive weighting amplifies differences in semantic alignment, resulting in a larger separation [dor(m)) dor(m2)) compared to the uniform weight- ing of deos. which averages all pairwise costs and dilutes discrininative features. The increased sensitivity ensures more precise identification of patches in M that capture critical visual informa- tion. proof is given in Appendix C.
4 EXPERIMENTS
In this section, present the experimental results of our method across hallucination and general benchmarks, including performance comparisons, ablation studies, and visualization analyses.
4.1 EXPERIMENTAL SETUP
Base model and baselines. We validate our method on three MLLMs, including LLaVA-1.55 7B (Liu et al., 2024a), Qwen-VL--Cat VL-Chat (Bai et al., 2023), and GLM-4V-9B (Zeng al., 2024). For comparison, we include several state-of-the-an object hallucination mitigation methods: VCD (Leng al., 2024), Me (Zou et al,, 2024), and VAF (Yin al., 2025).
5

---

Published as a conference paper at ICLR 2026
Table CHAIR evaluation results on MSCOCO dataset of MLLMs with different methods. We use 64 as the maximum token in this experiment. Bold indicates the best result of all methods.
LLaV7-L.5-7B
Qwen-VL-CHn VL-Chat
GLM-4V-9B
Method
C , 4 JBLEU1 5 &. CHAIR, CHAII , BBLEU
Vanilla
22.0
6.7 10.0
14.5 10.0
20.0 10.0
6.2 20.0
13.5 no.
13.0
5.6
9.8
VCD
24.6 026
7.3 10.6
13.9 1o.6
19.2 jos
5.7 J0.5
13.4 no.1
14.8 118
6.5 10.
9.5 jn.3
MemVR
21.6 104
6.4 1o.3
14.4 1o.1
20.0 10.0
6.1 Jn.1
13.3 .0.2
13.0
5.6
9.8 70.0
VAF
20.4 11.6
6.5 1o2
14.6 10.1
20.6 10.6
6.6 10.4
13.4 Lo.1
11.6 114
5.3 1o.3
9.7 10.1
AIR
18.4 13.6
5.7 L1.0
14.4 1o.1
18.6 p1.4
5.9 J0.3
13.6 n0.1
11.6 .14
5.3 po.3
9.7 J0.1
Table 2: Performance POPE benchmark using LLa Bold numbers indicate the best results. We report accuracy and Fl-score under three settings, i.e., Random, Popular, and Adversar- ial, show the robustness of different methods.
Random
Popular
Adversarial
Datasets
Methods
Accuracy
F1-score f
Accuracy f
Fi-score f
Accuracy 1
Fi-score f
Vanilla
83.7
70.0
83.0 10.0
78.2
70.0
78.4
10.0
75.0
70.0
76.0
10.0
VCD
85.4 11.7
83.7 10.7
84,3
16.1
83.0
14.6
81.8
16.8
80.9
14.9
MSCOCO
Mem VR
87.6
86.2
13.2
86.0
84.7
16.3
83.5
18.5
82.5
16.5
VAF
87.6 13.9
86.2 13.2
86.2 18.0
85.0
16.6
83.9
18.9
82.8
16.8
AIR
89.0 05.3
88.2 n5.2
78.9
86.4
78.0
83.9
78.9
83.6
17.6
Vanilla
83.4 70.0
82.6
79.9
70.0
79.6
70.0
74.0
70.0
75.1
10.0
VCD
85.9 92.5
85.4 12.8
81.9
72.0
82.0
12.4
76.7
+2.7
78.4
13.3
A-OKVQA
Mem
89.0 75.6
88.5 15.9
84.6
14.8
84.6
15.1
78.3
14.3
79.6
14.5
VAF
88.7 15.3
88.3 15.7
84.1
14.2
84.3
14.7
76.9 12.9
78.7
13.6
AIR
89.2
88.9
84.4
84.7
78.0
79.7
14.6
Vanilla
83.7 10.0
83.0
10.0
78.2
10.0
78.4
10.0
75.1 10.0
76.1
10.0
VCD
86.3 62.6
85.8
12.8
78.4
10.2
79.0
76.2 (1.1
77.4
11.3
GQA
Mcm VR
89.3 15.6
88.9
15.9
82.9
14.7
83.4
15.0
80.3
15.2
81.4
15.3
VAF
88.1 14.4
87.7
14.7
79.4
11.2
80.6
12.2
78.2
13.1
79.7 13.6
AIR
89.7 16.0
89.5
16.5
83.0
14.8
83.8
15.4
80.4
15.3
81.7
15.6
Evaluation benchmarks and metrics. We evaluate our method on a variety of benchmarks. For hallucination assessment, use CHAIR (Rohrbach et al., 2018) 500 randomly sam- pled MSCOCO (Lin et al., 2014) images, and POPE (Li et al., 2023b) on MSCOCO, A- OKVQA (Schwenk et al., 2022), and GQA (Hudson & Manning, 2019). For general-purpose eval- uation, we employ LLaVA-Bench (Liu et al., 2023), MME (Fu et al., 2023), and MMBench (Liu al., 2024c). As evaluation metrics, report CHAIRs, CHAIR,, BLEU, accuracy, and Fl-score. Detailed descriptions and implementations are provided in the Appendix B. and B.2.
4.2 PERFORMANCE ON HALLUCINATION BENCHMARKS
As shown in Table 1, AIR consistently achieves the lowest CHAIR, and CHAIR, across three rep- resentative MLLMs, demonstrating its effectiveness in suppressing hallucinations. For example, on LLLVA-1.5-7BB AIR reduces CHAIRs from 22.0 to 18.4 and CHAIR, from 6.7 to 5.7, while main- taining comparable BLEU scores. This confirms that selectively reinforcing salient tokens mitigates hallucinations more reliably than indiscriminate re-injection. Table further validates the robustness of AIR on the POPE benchmark. Across MSCOCO, A-OKVQA, and GQA, AIR achieves the best or near-best accuracy and Fl-score under Random, Popular, and Adversarial settings. Notably, AIR sustains strong performance even under adversarial prompts, outperforming prior defenses such as Mem More experimental results are reported Appendix B.3, Tables 8 and
4.3 PERFORMANCE ON GENERAL- PURPOSE BENCHMARKS
As shown in Table 3, AIR preserves strong performance on MME and MMBench, achieving results comparable better than existing methods across object- and attribute-level tasks. This indicates

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of evaluation results on the MME Hallucination subset and MMBench.
MME-Hall
Object-Level
Attribute-Leve
Cognition
MMBench
Methods
Total
Existence 1
Count
Position 1
Color 1
Score
Accuracy t
LLaVA-1.5
Vanilla
643.3 1oo0
190.0 yoo
155.0
70.0
128.3 pao
170.0 too
357.8
70.0
64.6 10.0
VCD
613.3
130.
190.0 poo
140.0
118.3
Lto.
165.0
337.1
J2n.
61.6
MemVR
648.3
15.0
190.0
155.0
133.3
150
170.0
378.6
64.6
VAF
603.3
140.
190.0
10.0
135.0
L20.
108.3
120.
170.0
100
322.8
135.
14.8
149.
AIR
638.3
15.0
190.0
10.0
155.0
10.0
123.3
L5.0
170.0
100
372.5
114.
64.7
10.1
Vanilla
631.7
10.0
185.0
10.0
145.0
10.0
126.7
10.0
175.0
10.0
342.8
10.0
59.9
10.0
Qwen-Vi VL
VCD
626.7
15.0
180.0
15.0
145.0
20.0
126.7
10.0
175.0
10.0
348.9
16.1
60.3
10.1
MemVR
636.7
15.0
185.0
10.0
145,0
-0.0
131.7
15.0
175.0
10.0
337.5
J5.3
60.0
10.1
VAF
631.7 10o
185.0
145.0
126.7
1o.
175.0
329.3
J14.
60.0
10.1
AIR
636.7 15.0
185.0
10.0
145.0
126.7
10.0
180.0
15.0
352.1
19.3
60.0
10.1
GLM-4V-9B
Vanilla
703.3
10.0
200.0
10.0
168.3
156.7
10.0
178.3
479.6
81.3
VCD
696.7
1666
200.0
10.0
170.0
11/
151.7
15.0
175.0
133
485.0
1544
80.2 11.1
MemVR
703.3
10.0
200.0
10.0
168.3
10.0
156.7
10.0
178.3
100
479.6
10.0
81.3 10.1
VAF
703.3
10.0
200.0
10.0
173.3
75.0
151.7
15.0
178.3
10.0
479.6
70.0
81.3
152
AIR
703.3
10.0
200.0
173.3
151.7
4500
178.3
479.6
70.3
81.3
Table 4: Results of GPT-4V-aided evaluation on LLaVA-Bench following the setting in (Leng et al., 2024). Both metrics are on scale of 10.
Table 5: Performance comparison of LLaVA- 1.5-7B when operating different decoding layer ranges with our method.
Model
Method Accuracy?
Detailednesst
{8}
CHAIRs 4
CHAIR, 4
BLEU
Vanilla
5.59
4.72
16-32
19.6
6.0
14.5
LLaVA-1.5
AIR
5.83
5.12
18-32
19.2
6.0
14,4
20-32
19.0
5.5
14.4
Vanilla
5.85
4.98
22-32
19.4
5.8
14.5
AIR
6.18
5.12
24-32
18.8
14.4
26-32
18.4
5.7
14.4
GLM-4V-9B
Vanilla
6.76
5.32
28-32
19.4
5.7
14.5
AIR
6.93
5.52
30-32
22.0
6.7
14.5
that selective reinforcement does not compromise general reasoning ability. Table 4 further shows that AIR consistently improves GPT-4V-aided evaluation on LLaVA-Bench. Across all models, both Accuracy and Detailedness scores increase, confirming that AIR enhances output quality while maintaining broad multimodal capability. The detailed results on MME and MMBench are provided in Appendix B.3, Table 10 and 11.
4.4 ABLATION STUDIES
Impact of operating layers. To evaluate how the choice operating layers influences performance, we conduct an ablation study on the CHAIR dataset with 1--77- as reported in Table 5. Following prior work (Zou et al., 2024; Yin et al., 2025; Yang al., 2025a) that emphasizes the role of mid-to-deep layers in vision-language fusion, we begin our analysis from layer 16 onward. The results indicate that reinforcing visual tokens in the range 24-32 yields the most favorable trade- off, achieving the lowest CHAIR, (18.8) while keeping CHAIR, (6.0) and BLEU (14.4) stable. In comparison, applying reinforcement too late (e.g., 30-32) or across overly broad spans (e.g., 18-32) leads to higher hallucination scores, suggesting that very late layers overly text-biased and wide ranges dilute the effect. These findings confirm that mid-to-deep layers are the most effective operating region for enhancing visual grounding while preserving generation quality.
Effectiveness different components. Table 6 presents the ablation results on CHAIR. Removing both components yields the highest hallucination rates, highlighting the importance of visual rein- forcement. Incorporating only prototype- prototype-based based token reduction reduces CHAIR, from 22.7 to 22.3, showing that condensing redundant tokens provides modest gains. OT-baaee patch reinforcement

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study of our method on the CHAIR using LLaVA-1.5-7B. where we ablate two
alignment components: prototype-based based token reduction and OT-based patch reinforcement.
CHAIR
Model
Prototype-basee
OT- Patch Reinforcement
Token Reduction
CHAIRs
CHAIR
X
22.7
6.7
22.3
6.5
LLaVA-1.5-7B
X
20.2
6.2
/
19.8
5.8
23
23
23
22
22
CHAIRs
CHAIRs
-13
CHAIRS
21
21-
21
20-
-11
20
100
200
500 Topa
400
500
19
0.02
0.04
0.06
0.08
0.10
19
10
15
number patches
Figure 3: Performance under different numbers of retained visual tokens TopQ
Figure 4: Performance with varying OT-based distance threshold
Figure 5: the number of selected image patches increases.
Performance as
alone achieves larger improvement (22.7 to 20.2), confirming the effectiveness of distribution- aware patch selection. When both modules are combined, CHAIRs further decreases to 19.8 and CHAIR, drops to 5.8, demonstrating that the two components are complementary and jointly con- tribute to mitigating hallucinations.
Impact of Topq selection. As shown in Fig. 3, increasing the number of retained visual tokens Topo leads to a steady reduction in hallucinations, evidenced by the decline CHAIR,- This result highlights that prototype-based selection successfully preserves the most discriminative cues while filtering redundancy. Moreover, BLEU scores remain nearly unchanged, indicating that semantic fidelity is maintained without compromising language generation quality.
Impact of threshold т. Fig. illustrates that varying the OT threshold produces a U-shaped performance trend. An overly strict threshold removes informative patches, which weakens visual grounding and increases hallucinations, while an overly loose threshold admits irrelevant regions and introduces noise. A moderate value of achieves the best balance between selectivity and coverage. resulting in reduced CHAIR, and stable BLEU, thereby confirming the effectiveness of OT-based patch selection.
Impact of augmentated patches. Fig. 5 quantifies the effect of enlarging the patch pool for OT- based selection: with more candidate patches, the transport plan can match lower-cost, better-aligned regions, yielding stronger visual evidence. Correspondingly. CHAIR, decreases steadily, while BLEU remains essentially unchanged, indicating that increased candidate coverage improves se- lected visual information without harming fluency.
4.5 ANALYSIS OF OT-BASED PATCH REINFORCEMENT
We evaluate based patch reinforcement the CHAIR benchmark using LLaVA the backbone model. Fig. 6 provides both quantitative and qualitative evidence of its superiority over cosine- based selection. In (a), the distribution of margin differentials shows that OT consistently produces larger gaps between safe and unsafe patches, confirming its stronger discriminative ability. The scatter plot in (b) further supports this observation, as the majority of points above the = æ line, indicating that OT achieves greater separation across patch-level comparisons. Importantly, Qualitative examples in (c), drawn from LLaVA-Bench, further illustrate that OT-selected patches concentrate visually salient regions aligned with the image semantics. Together, these results validate that the adaptive transport plan in OT accentuates meaningful cues and reduces redundancy, thereby improving the model's ability to mitigate hallucinations.
8

---

Published as a conference paper at ICLR 2026
60
(dmemm) (my)l (dar(m:) dorimall
[darlma] I darlmsll
0.08
60
Density
0.06-
40
20
0 0.00
0.0.00
0.02
0.04
0.06
0.08
0.02
0.04
0.06
0.08
Differencials
[deas (m)) des
Original Image
Selected Patches
(a) Distribution of margin differen- tials for OT and cosine distances.
(b) Comparison of margins be-
(c) Examples of original images and OT OT-selected patches.
tween OT and cosine distance.
Figure 6: Analysis of OT-based versus sssine-based patch selection. (a) OT produces consistently larger margin differentials than cosine. (b) Most patch-level results lie above the y line, showing that OT provides clearer separation between patches, (c) Qualitative evidence shows that OT focuses on visually informative regions aligned with the original image.
Table 7: Comparison of inference speed, GPU memory usage, and hallucination performance on the CHAIR using single A100 GPU.
Model
Avg. Latency +
GPU Memory +
CHAIRs
CHAIR;
LLaVA-1.5-7B
1.68s
13.5G
22.0
6.7
VAF
1.69s
13.5G
20.4
6.5
MemVR
2.05s
13.6G
21.6
6.4
AIR
2.07s
13.7G
18.4
5.7
4.6 COMPARISON OF EFFICIENCY
The comparison in Table 7 indicates that our method substantially improves safety while preserv- ing efficiency. LLaVA-1.557B yields the highest hallucination rates (CHAIR,=22.0, CHAIR;=6.77 7). VAF and Mem VR achieve moderate reductions, lowering CHAIR, to 20.4 and 21.6, and CHAIR; to 6.5 and 6.4, respectively, but with only limited improvements relative to the baseline. In contrast, our framework achieves the lowest hallucination rates CCHAIR,=18.. CHAIR;=5.7), representing with
clear gain in both sentence-level and object-level accuracy. These safery improvements a slight increase in latency (2.07s vs. 1.68s for LLaVA) and GPU memory (13.7G vs. 13.5G), but the overhead remains marginal compared to the robustness benefits. Overall, the results demonstrate the effectiveness of our approach in suppressing ballucinations while maintaining efficiency.
5 LIMITATIONS & FUTURE DISCUSSION
Although AIR demonstrates clear effectiveness in mitigating hallucinations, its application to rea- soning multimodal models and agents has not yet been explored. Future work can extend AIR to these broader settings, where adaptive reinforcement may further enhance robustness under complex reasoning tasks. In addition, the strength of OT capturing distributional discrepancies suggests wider potential in multimodal alignment problems. Applying OT-based reinforcement to cross- modal grounding and alignment remains a promising direction for future research.
CONCLUSION
In this work, we introduced AIR, an Adaptive vIsual Reinforcement framework to mitigate hallu- cinations in MLLMs. AIR combines prototype-based token reduction with OT-based patch rein- forcement to selectively strengthen salient visual cues while suppressing redundancy. Experiments representative MLLMs, including LLaVA-115--BB, Qwen-VL, and GLM-4V-9B, show that AIR substantially reduces hallucination while maintaining strong multimodal performance. Generally, AIR provides training-free and effective solution for building reliable MILMs..

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our method reduces hallucinations in MLLMs by directing the model's attention toward critical visual regions while suppressing background interference. This improves grounding in visual ev- idence and enhances the reliability of generated outputs. However, since MLLMs are trained on large-scale web data, risks such as inherited biases and harmful content remain. We therefore rec- ommend responsible use and continuous monitoring in practical applications.
REPRODUCIBILITY STATEMENT
We have taken several steps reproducibility. Detailed descriptions of the datasets, data processing, and inference procedures are provided the main paper (Sections 3 and 4) and the Appendix B. These materials enable other researchers reliably replicate our results and further build upon our work.
ACKNOWLEDGEMENT
This research is supported by the National Natural Science Foundation of China (U24B20180, No. 62576330), and the National Natural Science Foundation of Anhui (No.2508085MF143))
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Wanxing Chang, Ye Shi, Hoang Tuan, and Jingya Wang. Unified optimal transport framework for universal domain adaptation. In NeurlPS, 2022.
Guangyi Chen, Weiran Yao, Xiangehen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Promptlearning learning with optimal transport for vision-language models. 2022.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. CoRR, abs/2312. 1..13228,, 2023.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurIPS, 2013.
Xiang Fang, Wanlong Fang, and Changshuo Wang. Hierarchical semantic-ane augmetee navigation: Optimal transport and graph-driven reasoning for vision-language navigation. In Advances in Neural Information Processing Systems, 2025.
Mehrdad Fazli, Bowen Wei, and Ziwei Zhu. Mitigating hallucination in large vasion-language mod- els via adaptive attention calibration. CoRR, abs/2505.21472 2025.
Chaoyou Fu, Peixian Chen. Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306. abs/2306.13394, 2023.
Jinlan Fu, Shenzhen Huangfu. Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, and See-Kiong Ng. Chip: Cross-modal hicrarchical direct preference optimization for multimodal 1lms. In ICLR, 2025.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In AAAI, 2024.
Dandan Guo, Long Tian, He Zhao, Mingyuan Zhou, and Hongyuan Zha. Adaptive distribution calibration for few-shot learning with hierarchical optimal transport. In NewrIPS, 2022.
10

---

Published as a conference paper at ICLR 2026
Zhiyuan Han, Beier Zhu, Yanlong Xu, Peipei Song, and Xun Yang. Benchmarking and bridging emotion conflicts for multimodal emotion reasoning. In ACM MM, 2025.
Drew A. Hudson and Christopher D. Manning, GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucina- tions via attention lens. In CVPR, 2025.
Michalis Lazarou, Tania Stathaki, and Yannis Avrithis. Iterative label cleaning for transductive and semi-supervised few-shot learning. In ICCV, 2021.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. In CVPR, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. Trans. Mach. Learn. Res., 2025, 2025.
Miaoge Li, Dongsheng Wang, Xinyang Liu, Zequn Zeng, Ruiying Lu, Bo Chen, and Mingyuan Zhou. Patchet: Aligming patch and label set with conditional transport for multi-label image classification. In ICCV, 2023a.
Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. EMNLP, 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV (5), 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In CVPR, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoming, ocr, and world knowledge, 2024b. URL https: wttphh/......ooo/////o//e/
30--00--1---- -a1v-----lava-ext.... next/.
Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision-language models via latent space steering. In ICLR, 2025.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In ECCV (6), 2024c.
Xinyu Lyu, Beitao Chen, Lianli Gao, Hengtao Shen, and Jingkuan Song. Alleviating hallucinations in large vasion-language models through mallucination-induee induced optimization. In NeurIPS, 2024.
Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Mem. Math. Phys. Acad. Royale Sci., pp. 666-704, 1781.
Shangpin Peng, Senqiao Yang, Li Jiang, and Zhuotao Tian. Mitigating object hallucinations via sentence-level early intervention. In ICCV, 2025.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In EMNLP. 2018.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. In ECCV (8), 2022.
11

---

Published as a conference paper at ICLR 2026
Rosanna Turrisi, Rémi Flamary, Alain Rakotomamonjy. and Massimiliano Pontil. Multi-source domain adaptation via weighted joint distributions optimal transport. In UAI, 2022.
Zifu Wan, Ce Zhang, Silong Yong, Martin Q. Ma, Simon Stepputtis, Louis-Philippe Morency, Deva Ramanan, Katia P. Sycara, and Yaqi Xie. ONLY: one-layer intervention sufficiently mitigates hallucinations in large vision-language models. CoRR, abs/2507.00 000898 2025.
Chao Wang, Xuancheng Zhou, Weiwei Fu, and Yang Zhou. Mitigating hallucinations in large vision- language models with internal fact-based contrastive decoding. CoRR, abs/2502.01056, 2025.
Dongsheng Wang. Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen, and Hanwang Zhang. Tuning multi-mode token-level prompt alignment across modalities. 2023.
Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed loop: Uncovering object hallucinations in large vision-language models. In ACI. (Find- ings), 2024.
Yongliang Wu and Xu Yang. A glance in-context learning. Frontiers of Computer Science, 18(5): 185347, 2024.
Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. In CVPR, 2025a.
Yongliang Wu, Zonghui Li, Xinting Hu, Xinyu Ye, Xianfang Zeng, Gang Yu, Wenbo Zhu, Bernt Schiele, Hiuan Yang, and Xu Yang. Kris-bench: Benchmarking next-level intelligent image editing models. CoRR, abs/2505. 16707, 2025b.
Yongliang Wu, Shiji Zhou, Mingzhuo Yang, Lianzhe Wang. Heng Chang, Wenbo Zhu, Xinting Hu, Xiao Zhou, and Xu Yang. Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 8496-8504, 2025c.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Miua-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint urXiv:2508.05622 2025d.
Yongliang WU, Wenbo Zhu, Jiawang Cao, Yi Lu, Bozheng Li, Weiheng Chi, Zihan Qiu, Lirian Su, Haolin Zheng, Jay Wu, et al. Video repurposing from generated content: A large-scale dataset and benchmark. AAAI, volume 39, pp. 8487-8495, 2025e.
Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. Nullu: Mitigating object hallucinations large vision-languse models via halluspace projection. In CVPR, 2025a.
Xu Yang. Yongliang Wu, Mingzhuo Yang. Haokun Chen, and Xin Geng. Exploring diverse in- context configurations for image captioning. In NeurlPS, 2023.
Zhihe Yang. Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. Mitigating hallucinations in large laniun-l models via DPO: on-policy data hold the key. In CVPR, 2025b.
Hao Yin, Guangzong Si, and Zilci Wang. Clearsight: Visual signal enhancement for object halluci- nation mitigation in multimodal large language models. In CVPR, 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: hallucination correction for multimodal large lan- guage models. Sci. China Inf. Sci., 67()),, 2024.
Bowen Yuan, Sisi You, and Bing- Kun Bao. Dtoma: Training-fre dynamic token manipulation for long video understanding. In IJCAI, 2025.
Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng. Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang. Jiadai Sun, Jiajie Zhang. Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li. Lei Zhao, Lindong Wu. Lucen Zhong, Mingdao Liu, Minlie Huang. Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lx,
12

---

Published as a conference paper at ICLR 2026
Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song. Xunkai Zhang. Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang. Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from GL.M-130B to GLM-4 all tools. CoRR, abs/2406. 12793, 2024.
Kesen Zhao, Beier Zhu, Qianru Sun, and Hanwang Zhang. Unsupervised visual chain-of-thought reasoning via preference optimization. In ICCV, 2025.
Linxi Zhao, Yihe Deng. Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via classifier-free guidance, 2024. URL /ttp///aavvv//.a///av//eee/eaaea org/abs/ 2402. 08680.
Haohan Zheng and Zhenguo Zhang. Modality bias in Ivlms: Analyzing and mitigating object hallu- cination via attention lens. CoRR, hass/2508.024199 2025.
Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, Huiyu Zhou, and Xuming Hu, Reefknot: A com- prehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. In ACL (Findings), 2025.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination large vision-language models. In ICLR, 2024.
Yuan Zhou, Yanrong Guo, Shijie Hao, and Richang Hong. Hierarchical prototype refinement with progressive inter-categorical discrimination maximization for few-shot learning. IEEE Transac- tions on Image Processing, 31:3414-3429, 2022.
Yuan Zhou, Yanrong Guo, Shijie Hao, Richang Hong, and Jiebo Luo. Few-shot partial multi-view learning. IEFE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 1824-11841) 2023.
Yuan Zhou, Qingshan Xu, Jiequan Cui, Junbao Zhou, Jing Zhang. Richang Hong, and Hanwang Zhang. Care transformer: Mobile friinl-freenlly linear visual transformer via decoupled dual interac- tion. In CVPR, 2025.
Xingyu Zhu, Shuo Wang, Jinda Lu, Yanbin Hao, Haifeng Liu, and Xiangnan He. Boosting few-shot learning via attentive feature regularization. In AAAI, 2024a.
Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang. Yanbin Hao, and Hanwang Zhang. Enhancing zero- shot vision models by label-free prompt distribution learning and bias correcting. In NeurIPS, 2024b.
Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Selective vision- language subspace projection for few-shot clip. In ACM Multimedia, 2024c.
Xingyu Zhu, Shuo Wang, Beier Zhu, Miaoge Li, Yunfan Li, Junfeng Fang. Zhicai Wang, Dongsheng Wang, and Hanwang Zhang. Dynamic multimodal prototype learning in vision-language models. In ICCV, 2025a.
Xingyu Zhu, Beier Zhu, Shuo Wang, Kesen Zhao, and Hanwang Zhang. Enhancing clip robustness via modssty alignment. In NeurIPS, 2025b.
Xingyu Zhu, Beier Zhu, Yunfan Li, Junfeng Fang, Shuo Wang, Kesen Zhao, and Hanwang Zhang. Hierarchical semantic alignment for image clustering. In AAAI, 2026.
Yuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu, and Limin Wang. Awt: Transferring vision- language models via augmentation, weighting, and transportation. In NeurlPS, 2024d.
Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. CoRR, abs/2410.03577,, 2024.
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Related Works
3
Method
3.1
Preliminaries
3.2
Adaptive Visual Reinforcement
3.3
Theoretical Analysis
Experiments
4.1
Experimental Setup
4.2
Performance Hallucination Benchmarks
4.3
Performance on General-purpose Benchmarks
4.4
Ablation Studies
4.5
Analysis of OT-based Patch Reinforcement
4.6
Comparison of Efficiency
5
Limitations & Future Discussion
Conclusion
A LLM Usage Statement
15
Implementations, Benchmarks, and Additional Results
15
B.1 Implementations details
15
B.2 Benchmarks
15
B.3 Addintionl Results
15
C Proof of Theorem on OT-Based Patch Selection
20
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
We used ChatGPT only for minor language editing to improve clarity and conciseness. No part of the research idea, methodology, analysis was generated by LLMs.
B IMPLEMENTATIONS, BENCHMARKS, AND ADDITIONAL RESULTS
We provide additional details on the implementation, benchmarks, and results referenced the main paper. To assess ballucinations and general multimodal ability,
B.I IMPLEMENTATIONS DETAILS
For fair comparison, we follow the settings prior work (Yin al.,, 2025; Zou et al., 2024), adopting greedy decoding with so-sample=False, temperature set O, threshold 0.75, and beam size 1. All basclines (MemVR (Zou et al., 2024), VAF (Yin et al., 2025), VCD (Leng et al., 2024)) with their official recommended hyperparameters under the same decoding constraints and token caps to ensure consistent evaluation. In our method, we set TopØ=100, т = 0.06, and patch number= 12. Unless otherwise specified, all experiments conducted on single NVIDIA A40 GPU.
B.2 BENCHMARKS
CHAIR (Rohrbach al., 2018) measures how well generated captions align with the visual content. It includes two variants: CHAIR,, which reports the proportion of captions containing hallucinated objects, and CHAIR; which quantifies the proportion of hallucinated objects among all mentioned objects. Following prior practice, we use the MSCOCO val2014 split (Lin et al., 2014) with anno- tations for 80 categories, and randomly sample 500 images. The query prompt fixed as: "Please describe this image in detail." The CHAIR metric includes per-instance evaluation (CHAIR,) and per-sentence evaluation 5C defined as follows:
| [hallucinated objects||
CHAIR,
[all objects mentioned]
CHAIR [sentences with hallucinated object]
K all sentences]|
POPE (Li et al., 2023b) evaluates hallucination through a binary VQA setting. Given an image and an object name, the model is asked: "Is [object] in this image? Please answer yes no." Three sampling strategies are used to select the object queries: random, popular, and adversarial. Performance is reported under all three settings.
MME (Fu et al., 2023) is a comprehensive benchmark covering perception and cognition. The perception part includes tasks such as existence, counting, location, color, scene, landmark, artwork, and OCR. The cognition part includes commonsense reasoning, numerical calculation. translation, and code reasoning. All questions are framed as yes/no to standardize evaluation across tasks.
MMBench (Liu et al., 2024c) is large-scale benchmark for evaluating LVLMs, focusing on both perception and reasoning across multimodal inputs. adopts a hierarchical taxonomy with Level-1, Level-2, and Level-3 dimensions, enabling fine-grained analysis of model performance in diverse scenarios.
-LaVA-Bende (Liu et al., 2023) is used to assess whether our method maintains general multimodal performance. consists of 60 situational questions, including dialogue, description, and reasoning, posed randomly sampled MSCOCO val2014 images. Generated answers compared against GPT-4 text-only responses to evaluate consistency and instruction-followii ability.
B.3 ADDINTIONL RESULTS
Results CHAIR. To further corroborate the main results reported in the paper, include sup- plementary evaluation on the CHAIR benchmark in the appendix. As shown in Table 8, AIR consis- tently achieves the lowest hallucination rates across both
and WW-Cha.. For
15

---

Published as a conference paper at ICLR 2026
example, on LLaVA-1.5-78 AIR reduces CHAIR, and CHAIR; to 6.8 and 2.9, respectively, out- performing MemVR (7.0 / 3.2) and the vanilla baseline (9.2 / 4.0). Similar trends are observed on Qwen- VL-Chat, where AIR improves grounding by lowering ballucination while maintaining com- petitive BLEU and Recall. These findings reinforce that selective reinforcement is more effective than indiscriminate token re-injection.
Table CHAIR evaluation results on the MSCOCO dataset MLLMs with different methods. We use 32 as the maximum token in this experiment. Bold indicates the best result of all methods.
Qwen-VLe VL-Chat
Method
CHAIRs 1 CHAIR, J. BLEUt Recall |CHAIRs CHAIR,
BLEU1 Recallf
Vanilla
6.8 1oo
2.9 10.0
22.9
100 52.4 too
6.6 700
2.9 10.
21.4
52.11 10.0
VCD
8.2 f14
4.0 11.1
21.7
12.2
50.0 L2.4
6.6 70.0
2.9 10.0
21.6 10.2
52.0
.0.1
Mem VR
6.6 102
2.9 10.0
22.8
jo.1
52.3 jo.
7.0 n0.4
3.2 to.
21.3 jo.1
52.3 -0.2
VAF
7.0 foz
3.1 10.2
23.0
F0.1
52.4 ruo
6.6
3.0 to.1
to.
J0.
AIR
6.8 too
2.9
23.0
y0.1 52.8 r0.
5.8 U1.2
2.8 1o:1
21.7
10.3
52.0 .0.1
Table 9: Performance on POPE benchmark using Qwen- Wwen-VV-CChat. Bold numbers indicate the best results. We report accuracy and Fl-score under three settings, i.e., Random, Popular, and Adversar- ial, to show the robustness of different methods.
Random
Popular
Adversarial
Datasets
Methods
Accuracy f
Fl-score f
Accuracy f
Fl-score t
Accuracy 1
Fl-score t
Vanilla
84.5 70.0
81.9
n0.0
84.0
70.0
81.4 10.0
83.0
70.0
80.5
10.0
MSCOCO
MemVR
84.7 70.2
82.1 20.2
84.2
70.2
81.6 70.2
83.1
70.1
80.6
AIR
84.7 70.2
82.1 70.2
84.2
70.2
81.6
70.2
83.2
70.2
80.6 10.1
Vanilla
86.2 70.2
84.5
20.2
86.2
70.2
84.5
70.2
81.2
70.0
80.0 10.0
A-OKVQA
McmVR
86.8 10.6
85.3 10.8
86.8
10.6
85.3
10.8
81.6
10.4
80.6
10.6
AIR
86.8 10.6
85.3
10.8
86.8
10.6
85.3
10.8
81.7
10.5
80.7
10.7
Vanilla
86.1 10.0
84.3
10.0
85.1
10.0
83.3
10.0
82.1
10.0
80.5
10.0
GQA
Mem
86.3
10.2
84.5
10.2
85.2
10.1
83.4
10.1
82.2
10.1
80.7
10.2
AIR
86.5
70.4
84.7
10.4
85.3
10.2
83.6
10.3
82.4
10.3
81.0 10.5
We also report extended results on POPE to examine robustness under different perturbation settings. As presented in Table 9, AIR consistently outperforms baselines across Ran- dom, Popular, and Adversarial splits on MSCOCO, A-OKVQA, and GQA. In particular, on GQA adversarial evaluation, AIR achieves 86.5 accuracy and 85.7 F1, clearly surpassing Mem VR (83.7 / 83.5) and the vanilla model (80.5 / 80.5). These consistent improvements across datasets and con- ditions demonstrate that AIR not only mitigates hallucinations but also enhances robustness against adversarial distractors.
Results on POPE.
Detailed results on MME. As reported in Table 10, AIR achieves competitive performance across different LVLMs. For LLaVA-I.5-7B, it reaches an overall score of 1876.67, close to the best MemVR result, while delivering higher cognition (372.50) than the baseline, indicating stronger reasoning ability. For Qwen- VL-Chat, AIR obtains the best overall score (1829.01) and consistently improves perception and cognition, showing that our method effectively balances both dimensions. On GLM-4V-9B, AIR matches the baseline across all metrics, confirming that our approach intro- duces no degradation even on stronger models. These results highlight that AIR enhances cognition- oriented performance while maintaining robust overall accuracy across model scales.
Detailed results on MMBench. As shown in Table 11, AIR achieves consistent improvements on MMBench. For LLaVA-1.5-7B. it obtains the highest overall score (64.69), with clear gains in FP-S (+6.0) and LR (+0.9), demonstrating stronger fine- gre-graindd perception and reasoning. For VL-Cha,, AIR boosts AR, FP-C, and LR simultaneously, yielding a balanced improvement across both perception and reasoning dimensions. On the stronger GLM-4V-99, AIR matches the baseline without degradation, confirming the robustness of our framework across different model
16

---

Published as a conference paper at ICLR 2026
Table 10: Results on the MME dataset. Bold indicates the best result of all methods.
MME
Method
Overall 1
Perception
Cognition t
LLaVA-1.5-7B
1863.89
1506.03
357.86
VCD
1822.04
14185
1484,90
U21.3
337.14
420.72
MemVR
1890.95
127.06
1512.38
16.35
378.57
120.71
VAF
1746.55
1117.34
1423.69
JR2.44
322.86
135.00
AIR
1876.67
112.78
1504.17
L1.86
372.50
114.68
Qwen- VL-Chat
1818.06
1475.20
342.86
VCD
1814.87
13.19
1465, 949 19:26
348.93
16.07
MemVR
1802. 14
415.92
1464.64
310.56
337.50
15.36
VAF
1801.6 61
L1645
1472.33
U2.87
329.28
(13.58
AIR
1829.01
110.5
1476.87
11.67
352.14
19.28
GLM-4V-9B
2161.28
1681.64
479.64
VCD
2153.72 L7.56
1668.72 U12.92
485.00
15.36
MemVR
2161.28
10.00
1681.64
10.00
479.64
10.00
VAF
2161.53 10.25
168 1.89.89 10.25
479.64
10.00
AIR
2161.53 10.25
1681.89 10.25
479.64
10.00
Table 11: Results on the MMBench dataset. Abbreviations adopted: AR for Attribute Reasoning: CP for Coarse Perception; FP-C for Fine-grained Perception (Cross Instance); FP-S for Fine-grained Perception (Single Instance); LR for Logical Reasoning, RR for Relation Reasoning. Bold indicates the best result all methods.
MMBench
Method
AR t
CP
FP-C
FP-S t
LR
RR
Overall t
LLaVA-1.5-7B
73.37
77.03
57.34
61.92
30.51
53.04
64.60
VCD
68.34
5,00
75.34
55.24
62.80
0.88
28.81
4
53.04
61.60
1.300
MemVR
73.37
+ 0.00
77.03
1.000
57.34
0.000
61.92
0.000
30.51
1 0.00
53.04
0.000
64.60
0.00
VAF
17.08
56.29
21.28
1.5577
18.88
38.46
10.92
51.00
5.08
2/24.43
8.70
14.3
14.78
49.8.
AIR
72.36
1:01
77.03
0.000
58.04
0:77
67.92
6.000
31.36
1 0.85
53.91
0.87
64.69
0.09
Owen-VL-Chal
61.31
75.00
51.05
65.53
30,51
45.22
59.88
VCD
60.80 0.051
75,68
0.0.8
51.75
0.00
66.21
0.68
31.36
1 0.85
45.22
0.00
60.31
0.43
McmVR
62.31 t
73.65
1.335
53.15
65.53
0.00
32.20
1
44.35
60.05
VAF
61.31 t 0.000
75.68
6.68
52.45
1.40
64.85
0.68
31.36
1 0.85
44.35
1.0.8
60.05
0.17
AIR
62.31
t 1.00
73.31
1.669
53.15
1:2.10
65.53
0.00
32.20
1 1.69
45.22
1 1.000
60.05
0.1.7
GLM-4V-9B
85.93
86.15
69.93
84.64
64.41
83.48
81.27
VCD
85.93
1 0.00
85.14
1.1.1
68.53
1.440
82.94
1.70
61.5 1,66
1.2.5
83.48
0.0.0
80.15
L.122
MemVR
85.93
0.000
86.15
1.000
69,93
0.000
84,64
0.000
64,41
1 0.00
83,48
0.000
81.27
0.00
VAF
85.93
0.00
86.15
69.93
7:000
84.64
0.00
64.41
0.00
83.48
81.27
AIR
85.93
86.15
1.000
69.93
0.000
84.64
0.00
64.41
0.000
83.48
1.000
81.27
0:00
scales. These results indicate that AIR enhances both perception oriented (AR, CP, FP-S, FP-C) and reasoning-orient (LR, RR) abilities, particularly benefiting mid-scale MLLMs.
-orientte-
Results on Comprehensive Hallucination Benchmarks. Across the five benchmarks in Tables 12- 16, AIR shows consistent improvements. For HallusionBench (Table 12), it achieves the strongest fACC together with the best easyA and hardA scores. On V* Bench (Table 13), AIR provides the highest Attribute, Spatial, and Overall results for both LLaVA-1.5 and Qwen-VL-Chi
MMHal- Bench (Table 14) further shows that AIR attains the top average score while also yielding the lowest hallucination rate across all categories. For MM- Vet (Table 15). AIR enhances both reasoning- oriented and OCR-related metrics, achieving the highest total score. Finally, LLaVA-Bench (In- (ie-Wil) (Table 16), AIR performs best on conversational, detailed, and complex queries. Together, these results highlight ATR's robustness across diverse hallucination types and evaluation settings.
17

---

Published as a conference paper at ICLR 2026
Table 12: Results on HallusionBench.
Model
FACC1
qACCt
casyAt
hardAt
aACC1
LLaVA-1.5
17.9
8.1
36.0
36.7
41.5
VCD
13.9
11.4
33.00
34.7
41.1
MemVR
17.9
36.9
37.7
42.5
AIR
19.9
9.3
37.5
38.3
43.2
Table 13: Performance on V* Bench.
Model
Attribute
Spatial
Overall
LLLVA-115
43.47
56.57
48.68
MemVR
45.38
57.82
49.35
AIR
48.23
59.31
51.26
Qwen-VI.-Che VI.-Chat
74.78
68.42
72.25
McmVR
75.31
69.08
73.58
AIR
76.02
69.46
76.23
Table 14: Results MMHal-Bench.
Method
Average score" Hallucination rate Attribute Compaaariial Counting Relation Environment Holistic
LLaVA-155
1.99
2.58
2.58
1.67
2.00
3.00
1.17
1.83
VCD
2.69
0.58
3.25
2.17
3,00
2,42
2.58
3.25
2.42
2.42
MemVR
3.60
2.35
3.30
2.40
2.85
3.40
2.40
2.30
AIR
3.05
3,90
2.55
3,70
2.50
3,10
3.70
2.55
2.40
Table 15: Results MM-Vet.
Model
Rt
OCR_St
OCR.K.R
OCR_G.ST
Total
LLaVA-1.5
67.6
17.7
21.2
10.0
31.1
VCD
62.2
15.8
17.5
60.0
30.2
MemVR
70.3
23.8
21.2
32.2
AIR
72.1
24.6
21.5
30.0
34.7
Table 16: Results LLaVA-Bench (In-the-Wild).
Model
Convst
Detailf
Complext
Allf
Averaget
L..
58.8
52.1
74.6
63.4
64,8
VCD
57.8
50.8
77.9
59.1
63.2
MemVR
63.8
52.6
77.9
64.0
65.2
65.3
52.7
79.1
64.3
65.8
23
22
14
22-
-14
22
CHAIR,
CHAIRs
-13
CHAIR
81 BLEU
21
20
11
20
100
19
0.02
0.04
0.06
0.08
0.10
19
10
15
Topo
number patches
Figure 7: Performance under different numbers of retained visual tokens Topo
Figure 8: Performance with varying OT-based distance threshold
Figure 9:
Performance the number of selected image patches increases.
Ablation study Qwen-VL-Chal To verify that the hyperparameter behaviors observed in the main text generalize across architectures, we conduct the same ablations on Qwen- VL-Chat. As shown in Figure 7, varying the number of retained visual tokens (Top-Q) produces clear U-shaped
18

---

Published as a conference paper at ICLR 2026
15
15
22-
14
CHAIRs
13
CHAIRs
13
20-
20-
12
12
18
11
18
11
10
0.01
0.05
0.10
0.01
0.05
0.10
E
E
(a) LLaVA-1.5-BB.
(b) Qwen-VL-Chat VL-Chat.
Figure 10: Effect of the OT regularization strength
trend: very small or very large token sets increase hallucination, while moderate range yields the best performance. Figure 8 shows a similar pattern when sweeping the OT distance threshold т, where intermediate values achieve the most favorable balance between selective reinforcement and visual coverage. Finally, in Figure 9, increasing the number of selected patches improves perfor- mance until reaching a stable region, after which the gains saturate. These observations align with the trends reported in the main experiments, indicating that AIR exhibits stable and consistent hy- perparameter sensitivity across different model families.
Effect of the OT regularization strength. We further investigate the influence of the OT regular- ization strength с on both LLaVA-1.5-78 and Qwen-VI VL-Chat, as shown in Figure 10. Across the tested range, the CHAIRs and BLEU curves remain stable, with only minor variations in halluci- nation and fluency. This consistency indicates that AIR is insensitive to the precise choice of e and maintains its effectiveness under different regularization strengths. The similar behavior across the two models further suggests that the effect is not architecture-speciff and that the reinforcement mechanism is robust to changes in entropic regularization.
Adversarial Stress Test Results. To examine AIR under more challenging visual conditions, we evaluate both VA-V1.5--77 and Qwen-VL-Chat VL-Chat with adversarial perturbations of 5 16/255, as summarized in Table 17. On both models, AIR consistently lowers CHAIRs and CHAIR, com- pared with the original outputs, indicating reduced hallucination under noisy crops and cluttered backgrounds. Although BLEU decreases under strong perturbations for all methods, AIR maintains competitive fluency while achieving substantially lower hallucination scores. These results show that AIR remains effective even when visual inputs are degraded by adversarial noise.
Table 17: Results on CHAIR and BLEU under adversarial scenarios.
Model
CHAIRs 4
CHAIR, +
BLEU
LLaVA-LL5-BB
25.3
11.5
17.3
AIR
22.1
9.5
14.2
Qwen-VL--Cat VL--Cat
23.5
11.9
16.8
AIR
21.4
8.9
13.7
Comparison with the random patch baseline. To examine whether AIR's improvement can be reproduced without alignment-guided patch selection, we include control variant that injects the same number of visual patches but selects them at random. As shown in Table 18, random patch injection does not reduce hallucination and slightly worsens both CHAIR and CHAIR, compared with the base model. In contrast, AIR achieves clear reduction on both metrics while maintaining BLEU. This result shows that the improvement does not arise from merely adding visual patches, but from selecting regions that are semantically aligned with the hidden states, which is the key factor behind the observed gains.
Effect of generation length. We further examine the behavior of AIR under different generation lengths, as shown in Table 19. Increasing the maximum output length from 64 to 128 and 256 tokens substantially raises hallucination levels for all models, confirming that longer captions introduce more opportunities for drift. Nonetheless, AIR consistently yields the lowest CHAIR, and CHAIR, scores across all lengths. In particular, AIR maintains clear margin of improvement at 128 and
19

---

Published as a conference paper at ICLR 2026
Table 18: Comparison with the random patch baseline.
Model
CHAIRs
CHAIR;
BLEU1
LLaVA-I.S... 1.5-BB
22.0
6.7
14.5
random patch
22.3
6.9
13.1
AIR
18.4
5.7
14.4
Table 19; Results different generation lengths.
CHAIRs +
CHAIR, +
Recall
Model
64
256
128
256
128
256
LLaVA-1.5
22.0
47.5
47.8
6.7
13.1
13.4
66.2
73.1
80.6
MemVR
21.6
47.2
6.5
13.0
13.2
66.5
73.5
81.0
AIR
18.4
38.1
38.8
5.7
9.3
10.0
66.7
73.9
81.4
256 tokens, where hallucination becomes more pronounced for the baselines. These results indicate that the benefit of reinforcement limited short captions and extends longer outputs where hallucination pressure is higher.
Comparison with fine-tuned models. To examine whether AIR remains effective when combined with stronger base models produced by supervised fine-tuning, we further evaluate its integration with SENTINEL across multiple hallucination benchmarks, as shown in Table 20. For both LLaVA- 1.5-7B and LLLaVA-1.5-13B adding AIR yields consistent reductions in hallucination metrics on Object HallBench, including both response and mention errors. AMBER, AIR further decreases CHAIR, hallucination, and cognitive scores beyond those obtained by SENTINEL alone. Similar improvements are observed on HallusionBench, where AIR maintains or increases question accu- racy. These results indicate that AIR complements fine-tuned models and provides additional gains across diverse hallucination types and dataset conditions.
Effect of noised and averaged visual inputs. To examine the stability of AIR under perturbed or degraded visual conditions, we further compare its behavior with two variants that modify the injected visual features. The first variant adds random noise to the selected patches, and the second replaces the reinforcement term with the average of all visual features. As shown Table 21, both variants lead higher hallucination scores than the baseline, indicating that simply altering or smoothing visual features does not improve robustness and may even weaken grounding. In contrast, AIR continues yield the lowest CHAIR and CHAIR, values while maintaining comparable BLEU. These results demonstrate that AIR's improvement is not attributable to noise injection or feature averaging, but selectively reinforcing visual regions that remain semantically meaningful under distribution variations.
C PROOF OF THEOREM ON OT-BASED PATCH SELECTION
Theorem 1. For two distinct patches my and ma with cost matrices Cm, * Cmgy the optimal transport (OT) distance is strictly more sensitive than the cosine distance:
[dor(m1) - dor(mm)) > (m1)) deos (mz))
(16)
Proof. To prove that the optimal transport (OT) distance (or(m) more sensitive than the cosine distance deas (m) in distinguishing patches my and m2, compare their differences when the cost matrices satisfy Cm, + Omg- Let Cm, E R x denote the cost matrix for patch m, with entries Cm, (k, n))
cos(Zk, 2
The OT distance defined as:
K N
dor(m:) = -TT Cmú) =
Σ
T
Th (k, n) C) (k,
(17)
k= n=
20

---

Published as a conference paper at ICLR 2026
Table 20: Comparison with fine-tuned basclines.
Method
Object HallBench
AMBER
HallusionBench
Resp.t
Ment.p
CHAIR
Hal.
Cog
Question Acc.1
LLaVA-y1.5-7B
52.7
28.0
8.4
35.5
4.0
46.86
SENTINEL (Peng al., 2025)
2.6
2.9
14.6
1.2
47.56
+ AIR
3.9
2.2
2.1
13.3
1.1
47.25
LLaVA-v1.5-13B
46.0
23.0
6.9
31,9
3.3
46.43
SENTINEL (Peng al., 2025)
3.3
2.7
11.7
0.9
46.77
+ AIR
2.8
1.6
2.5
10.9
0.7
46.77
Table 21: Comparison of noisc/averaging baselines.
Model
CHAIRs
CHAIR, + BLEU
LLaVA-L.5-7B
22.0
6.7
14.5
AIR (noise)
24.5
13.2
AIR (averaged)
20.8
6.4
14.5
AIR
18.4
5.7
14.4
where T is the optimal transport plan computed via the Sinkhorn-Knopp algorithm, satisfying
the marginal constraints T 1K = a ...... cosine distance is given by:
1
and
T IN = bm. =
[
11 The N
K
1 KN
deos (m4) (U, Cm.)
Σ Cm, (k,
(18)
where the uniform transport plan with U(k: n) = KN. Since T minimizes the OT objective (T, Cm.) eT)... where h(T) Ek. T(k, n)) log T n) is the entropy and 2 0 controls regularization, it follows that:
1
TT,, Cm.) S (U, Cm.) ) = doos (m:))
(19)
This inequality is strict unless T = U, which occurs only when Cm, is constant, a degenerate case not applicable here since Cmm # Cma-
To compare the sensitivity, consider the difference in OT distances:
doT(m:) - ( (T
m1 Cmy) - (Tmyy Cmz) (TT my Cm, C.ma) (T mi m2? Cmg); T
(20)
where AC Cm, Cmz- For the cosine distance, the difference is:
(m deas (m2) AU).C).
(21)
The OT distance difference includes two terms: the cost difference under the optimized plan T mi
and the effect of differing transport plans applied Cm2- Since Ty, is optimal for Cmgy, we have:
T m2? Cmz) I (T mm Cmz),
(22)
implying:
T m22 Cmm)) 0. Because Cm, * Cmzy the optimal plans typically differ (T+ my * Tmn)) making this term strictly positive in general.
(T m2
(23)
The key to the OT distance's greater sensitivity lies in the first term, (T AC). Decompose it as:
(T AC) (U,AC) (T U,AC).
(24)
Since mi assigns higher weights to pairs (k. n) where Cm,, (k, n) is small (indicating high similar- ity), if C., (k, n) < Cung (ks n) for some pairs, then AC(k; n) < 0. and T (k,n) n) > U(k,n)
21

---

Published as a conference paper at ICLR 2026
KN- This correlation makes (Tis, U, A,AC) O, amplifying the magnitude of (Tm, AC) com- pared to (U, AC).
To establish the strict incquality, assume without loss of generality that (mo)) < dor so:
(mr(m1) dor(m2)= = (T AC) + (T -T Cmz) < 0.
(25)
Since (T mg
T ma? Cmg) 2 0, follows that:
(T AC) S doT(m)) dOT(m2) 0.
(26)
Thus:
[do[(m)) dm)( = ((T mi AC) + (T -Tmy Cmz)) Given (T m1² AC) (U,AC) + (T m1 U, U, and (T mi - U. AC) < O, we have:
(27)
(T mı
AC) < (U, AC),
(28)
implying:
(T my AC (U,AC)
(29)
The non-negative term (T T 1n ) 2 0 further increases the magnitude, so:
Idor(m,) - doT (m²)) 2 ((T my AC > (U Ides (m,) deos (m2))
(30)
The strict inequality holds when Cm * Cmg+ as the adaptive weighting of T m. Cm then T in transport plans amplify the cost differences. In the degenerate case, if C.., T so dot ( = dot (mz) and doos (m1) = deos (m2), making both differences zero, consistent with the theorem's condition.
and the difference
22


Reviews:
[{"review_id": "OsGDu3l9dE", "reviewer": "Reviewer_wmbu", "summary": "The paper proposes AIR, a training-free inference intervention to reduce hallucinations in MLLMs. AIR has two components: (i) prototype-based token reduction that retains the Top-Q most distinctive visual tokens before re-injection, and (ii) OT-guided patch reinforcement that computes an entropically-regularized optimal transport distance between hidden states and patch embeddings to select well-aligned image patches for FFN re-injection.", "questions": "Please see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Efficiency-aware design. Prototype reduction + selective patch fusion yields small overhead, acceptable for many deployments. \n\nClarity. Method is easy to implement in existing FFN-reinjection pipelines", "weaknesses": "Benchmark coverage is narrow. Evaluation centers on CHAIR (captioning) and POPE (binary VQA). Absent are harder hallucination suites probing language bias and visual illusions, such as HallusionBench, RLHF-v, and MMHal-Bench, V* etc; including them would strengthen claims of robustness.\n\nPure performance: The performance in incremental compared to VAF. May be provide curves for ε (entropic regularization), τ, Top-Q, and #patches on at least two models, and adversarial stress tests (noisy crops, cluttered backgrounds) for better explanation\n\n\nLack of novelty: While the paper positions AIR as a new training-free approach, its core idea—adjusting visual token utilization during inference to mitigate hallucination—is conceptually aligned with recent decoding- or selection-based techniques (e.g., Visual Contrastive Decoding, CLIP-guided decoding, or entropy-based token filtering). The novelty of AIR mainly lies in adopting optimal-transport–based patch selection combined with prototype-based token reduction, rather than introducing a fundamentally new mechanism. The contribution is thus incremental", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GVKmzhozWX", "reviewer": "Reviewer_q4Bt", "summary": "This paper proposes a training-free framework to mitigate hallucination in MLLMs. The method combines (1) prototype-based token reduction, which compresses visual tokens to remove redundancy, and (2) optimal-transport (OT)–guided patch reinforcement, which selectively re-injects patches that are most semantically aligned with the hidden states. The goal is to emphasize salient visual cues while suppressing background noise during decoding.", "questions": "How does AIR perform when generating longer outputs (e.g., 256–512 tokens) or in multi-turn dialogues, where hallucination typically escalates? It would be valuable to analyze how hallucination rates vary with generation length and whether the advantage of salient-patch reinforcement persists or diminishes as outputs become longer and less directly grounded in the visual context.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method operates purely at inference, making it broadly applicable to existing MLLMs without retraining.\n2. Integrating optimal transport to quantify alignment between hidden states and patch embeddings is a creative idea.", "weaknesses": "1. The central claim, that reinforcing salient patches directly causes lower hallucination, is not rigorously demonstrated. The supporting evidence is purely descriptive and does not establish a causal relationship between visual emphasis and reduced hallucination. The observed gains could equally arise from reduced visual redundancy or implicit regularization rather than genuine enhancement of visual grounding.\n2. All experiments restrict generation to 64 tokens (Table 1), whereas hallucination severity is known to grow with output length. As a result, the evidence from short captioning tasks is insufficient to claim robustness in realistic scenarios involving long-form or multi-turn reasoning. Moreover, shorter captions are inherently more correlated with salient regions in the image, which may exaggerate the apparent benefit of saliency-based reinforcement, suggesting that the chosen generation length is tuned for conditions where the proposed mechanism performs best.\n3. The paper compares only to a limited set of inference-time baselines.\n4. The method depends heavily on empirically chosen parameters—such as the OT regularization strength, threshold, and the number of retained tokens—yet no principled analysis or sensitivity study is provided. Additionally, there is no discussion of computational scalability under higher-resolution inputs or longer textual outputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WolC2HWfZn", "reviewer": "Reviewer_MiAJ", "summary": "This paper proposes AIR (Adaptive Visual Reinforcement) — a training-free framework to reduce hallucinations in multimodal large language models (MLLMs).\nIt combines prototype-based token reduction to remove redundant visual signals and optimal transport (OT)-guided patch reinforcement to re-inject only salient image regions during decoding. Experiments across LLaVA, Qwen-VL, and GLM-4V show consistent hallucination reduction (CHAIR, POPE) while preserving general performance (MME, MMBench).", "questions": "1.Have the authors tested AIR on reasoning-oriented hallucination benchmarks (e.g., MM-SafetyBench, SPA-VL)?\n\n2.Does prototype-based reduction influence the OT alignment results, and could the two be adaptively coupled?\n\n3.Is the OT transport computation reused across layers, or recomputed at each reinforcement step?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The method achieves notable improvements across multiple models and benchmarks, showing robustness under both standard and adversarial conditions\n\n2.The OT-based analysis is well-motivated and supported by proof and visualization, providing a clear justification for the proposed selection mechanism.\n\n3.The paper is clearly written, visually well-presented, and includes detailed experimental settings for replication.", "weaknesses": "1.AIR assumes well-aligned hidden and visual spaces; if this alignment is weak, OT distance may emphasize irrelevant correlations, limiting reliability on misaligned models.\n\n2.Despite its name, AIR uses fixed thresholds and token counts. Introducing data- or entropy-driven adaptation could further enhance robustness across tasks.\n\n3.Experiments focus on standard datasets with clean imagery; robustness under distribution shifts or noisy visuals remains untested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "itPTsueQ1X", "reviewer": "Reviewer_9UNK", "summary": "This paper introduces AIR, a training-free, selective reinforcement mechanism that (i) prunes redundant visual tokens and (ii) injects only OT-aligned patches during decoding, yielding reduced hallucination while maintaining general multimodal ability across representative MLLMs.  \n\n## Motivation\n\nMultimodal LLMs (MLLMs) still hallucinate—producing text that contradicts the image. Existing fixes either require costly fine-tuning or add inference overhead. “Re-inject all visual tokens” defenses help but indiscriminately amplify background noise. The paper proposes making visual reinforcement selective and adaptive, not blanket, and doing so training-free during decoding.  \n\n## Methodology (AIR: Adaptive vIsual Reinforcement)\n\n* A training-free decoding framework that reinforces only image regions most aligned with the current hidden states, rather than all tokens. \n* Visual evidence is integrated inside transformer feed-forward layers while generating, without auxiliary models. \n* Prototype-based token reduction: From the full set of visual tokens (Z), keep a compact subset by ranking tokens against a global prototype to suppress redundancy before reinforcement. \n* OT-guided patch reinforcement: Crop the image into patches, embed them, and compute optimal transport (OT) distances between selected hidden states and patch embeddings (Sinkhorn-regularized). Patches with stronger alignment (lower OT distance) are selectively injected to emphasize salient evidence.  \n* Rationale for OT: Unlike pointwise similarity, OT compares distributions and captures global geometric structure, providing a finer criterion for semantic alignment at inference time.  \n\n## Experimental Evaluation\n\n* Benchmarks: POPE, CHAIR, MME and LLaVA-Bench.  \n* POPE (LLaVA-1.5-7B): AIR achieves the best or near-best Accuracy/F1 across MSCOCO, A-OKVQA and GQA.\n* LLaVA-Bench: GPT-4V-aided scores increase with AIR for all three models.  \n\n## Analysis & Ablations\n\n* Operating layers: Reinforcing in mid-to-deep layers is most effective; for LLaVA-1.5-7B, layers 26–32 yield the best CHAIR trade-off (CHAIR(_S)=18.4, CHAIR(_I)=5.7, BLEU=14.4). Very late or overly wide spans degrade hallucination metrics.  \n* Component study: Both modules contribute; combining prototype reduction + OT patch reinforcement gives the lowest hallucination scores in CHAIR ablations.", "questions": "Please refer to the weakness section. I would be glad to raise the score once authors have addressed my concerns.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Clear motivation with empirical evidence. The paper diagnoses why 'inject all image tokens' can hurt—background tokens dilute salient cues—and supports this with similarity/attention analyses and a motivating figure.   \n* Training-free, selective reinforcement that fits standard hooks. AIR stays within the FFN re-injection interface, but i) prunes redundant visual tokens via a prototype and ii) reinforces only OT-aligned patches—no fine-tuning or auxiliary models required.    \n* Theoretically and empirically grounded use of OT. The paper argues OT is more sensitive than cosine for patch selection and shows larger patch-level separation and qualitative focus on salient regions.   \n* Consistent gains on hallucination benchmarks across models. AIR reduces CHAIR$_S$/CHAIR$_I$ on three backbones and improves/near-improves POPE under Random/Popular/Adversarial.  \n* General ability maintained. MME/MMBench remain competitive and GPT-4V–aided LLaVA-Bench scores rise.  \n* Efficiency profile is reasonable. On A100, latency and memory rise slightly vs. baseline while hallucinations drop the most among compared methods.  \n* Transparent ablations. Component ablation and hyperparameter sweeps clarify contributions and sensitivities; operating-layer study identifies effective mid-to-deep layers.", "weaknesses": "* 1. The evaluation benchmarks are limited. Adding more MLLM evaluation benchmark results is preferred (MM-Vet, HallusionBench, LLaVA Bench (in-the-wild)).\n* 2. This novel approach seems to be hyperparameter sensitive. Performance depends on choices like TopQ and τ (showing a U-shaped trend), implying tuning is needed per setting/model. Do all models have the same performance variation when tuning the value of $Top_Q$, τ and the number of selected image patches, as show in Figure 3,4,5?\n* 3. Non-zero inference overhead and added computation. Although modest, AIR introduces extra latency/memory and requires per-patch Sinkhorn OT plans during selection. Could authors provide theoretical analysis on the computation cost and inference latency of AIR?\n* 4. Dataset/protocol breadth. In Table 1, CHAIR uses 500 MSCOCO images with a 64-token cap, why does the authors limit the max generation token? \n* 5. The FFN injection adds a term from selected visual tokens. Will this operation introduce disturbance to the model? how are feature scales matched to avoid over/under-amplifying the visual branch across layers?\n* 6. Were MemVR/VAF/VCD run with author-recommended settings and the same decoding constraints and token caps across backbones? If yes, it is recommended that the authors should make a clearer statement on their settings in Appendix B.1.\n* 7. Additional ablation study on the model's performance when injecting noised/averaged visual feature is recommended, to prove the effectiveness of AIR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)