Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
Authors: Hao Fang
Changle Zhou
Jiawei Kong
Kuofeng Gao
Bin Chen
Tao Liang
Guojun Ma
Shu-Tao Xia
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision-Language Models
Hallucination Mitigation
Decoding Strategy
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Large Vision-Language Models (LVLMs) are susceptible to hallucinations, where generated responses seem semantically plausible yet exhibit little or no relevance to the input image. Previous studies reveal that this issue primarily stems from LVLMs' over-reliance on language priors while disregarding the visual information during decoding. To alleviate this issue, we introduce a novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy, which adaptively strengthens the mutual dependency between generated texts and input images to mitigate hallucinations. Unlike existing methods solely focusing on text token sampling, we propose to jointly model the contributions of visual and textual tokens to C-PMI, formulating hallucination mitigation as a bi-level optimization problem aimed at maximizing mutual information. To solve it, we design a token purification mechanism that dynamically regulates the decoding process by sampling text tokens remaining maximally relevant to the given image, while simultaneously refining image tokens most pertinent to the generated response. Extensive experiments across various benchmarks reveal that the proposed method significantly reduces hallucinations in LVLMs while preserving decoding efficiency.

Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
Hao Fang"1, Changle Zhou"!, Jiawei Jiawei Kong"),
Kuofeng Gao',
Bin Chen², Tao Liang', Guojun Mat Shu-Tao Xia
Tsinghua Shenzhen International Graduate School, Tsinghua University.
Harrbi Institute of Technology, Shenzhen, (fangh25, kjw25, 20aiisl www.nhuusttiinanaa.a...aamaaaa letinghua.ddu..com cn, leliuzhe@165 com, chenbin2021@hit.c edu. en, taoliangdpg@126. com,
magjhaha@126.com com, xiast@az. slastdez.teinghua. edu.. cn
Abstract
Large Vision- Lisguu-Languae Models (LVLMs) are susceptible to hallucinations, where generated responses seem semantically plausible yet exhibit little relevance to the input image. Previous studies reveal that this issue primarily stems from LVLMs' over-reliance language priors while disregarding the visual infor- mation during decoding. To alleviate this issue, we introduce a novel Con- ditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy, which adaptively strengthens the mutual dependency between generated texts and input images to mitigate hallucinations. Unlike existing methods solely focusing text token sampling, we propose to jointly model the contribu- tions of visual and textual tokens C-PMI, formulating hallucination mitiga- tion as a bi-level optimization problem aimed at maximizing mutual informa- tion. To solve it, we design a token purification mechanism that dynamically regulates the decoding process by sampling text tokens remaining maximally relevant to the given image, while simultaneously refining image tokens most pertinent to the generated response. Extensive experiments across various bench- marks reveal that the proposed method significantly reduces hallucinations LVLMs while preserving decoding efficiency. The code is available at: https
//github. wwwoo//fffieeeeeeeeee VLD Htt.ee/M.MMWW.aaaaaaaaaCC...CCC Hallucinaannaaeeaa on Mitigiiion
Introduction
4 5 have The unprecedented breakthroughs in large vision-language models (LVLMs) 1 3 expanded their applicability across various vision-language (V+L) tasks such as autonomous driving 7. Benefiting from advanced designs of model architectures and training algorithms, IVL.Ms trained on high-quality image-text pairs have exhibited outstanding capabilities in cross-modal alignment and complex V+L understanding. Despite the remarkable success, the issue of hallucination continues to pose challenges to LVLMs. Concretely, LVLMs may generate semantically coherent yet factually incorrect contents that are entirely inconsistent with the input image [ 9 [0] E.g., describe non-existent objects or misinterpret the attributes and relationships of visual entities within the image. This raises serious concerns regarding the deployment of LVLMs in real-world applications, particularly in high-risk scenarios such as medical diagnosis and financial systems [12]
Equual Contribution Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sysee Prorpe
Attention Score
Please descriée this image in detoiil
Irage
User Tremuation
06
84
0.3
00
System
Image
User
Generation
The image festures large yellow (dog)
The image features lorge yollow dog [kittenn]
(a) Attention allocation.
(b) Our visual token refinement.
Figure 1: (a) Illustration of the attention bias of MVL.ss While image tokens constitute the majority of the input tokens, they receive significantly less cumulative attention scores compared to text tokens. (b) The proposed purification mechanism when the masking ratio is 50%. Our method promotes more reliable generation by adaptively retaining image tokens with high relevance to the ongoing response.
To address this issue, one line of research explores fine-tuning for more fine-grained alignment or post-hoc analysis to correct hallucinated elements within the generated responses [15] Another research stream focuses on directly modifying token distributions during decoding
13
14]
16
9 10 19]. These methods employ various techniques to penalize the probabilities of irnpiiinatio-innuuinn tokens, thereby encouraging the generation of more faithful and reliable responses. While these decoding-based strategies have shown practical effectiveness, their designs are typically grounded empirical findings and lack convincing theoretical foundations. Moreover, they generally fail to explicitly quantify and control the dynamic mutual relevance between the visual input and the progressively generated text, leading to insufficient effectiveness in certain scenarios.
18 g
In this work, we build upon the line of decoding- based methods and investigate the issue from an information -teeorrttti perspective, based on which we propose novel Conditional Mutual Information eawre adaptive Vision-Languaa
Decoding strategy (CMI- VLD). Specifically, previous studies (20 21 have revealed that a key factor contributing to hallucination is LVLM's ' tendency to overly depend on text tokens during the autoregressive generation, with limited attention paid to the critical visual input (see Fig. (a)). As a result, the generated text guided more by the language priors inherent in the LLM backbone, rather than grounded in the actual visual content of the input image. This eventually leads to low mutual dependency between the input images and the final responses, hence exacerbating the occurrence of hallucinations in LVLMs.
To mitigate this issue, we introduce conditional pointwise mutual information (C-PMI) quantify the mutual correlation between the visual inputs and the generated texts during generation. Cor- respondingly, we reformulate the hallucination mitigation objective as vision-language mutual information maximization problem. which is further decomposed into two complementary sub-tasks that capture the respective contributions of visual and textual tokens. Based on the analysis, we derive bi-level optimization formulation and design an effective solution that adaptively calibrates each decoding step during the generation process. To optimize the inner sub- sub-proble problem, we calibrate the token distribution using the derived formula to prioritize tokens that exhibit strong relevance to the visual input. For the outer sub-problem, we propose an efficient visual token purifier parameterized as a learnable network, to dynamically refine image tokens that are most pertinent to the current textual context. By filtering out redundant image tokens that impair mutual information with the generated content, the proposed strategy directs the model to focus more on the key visual tokens most relevant to the ongoing response (see Fig. (b)), further enhancing the dependence of the generated text on the input image. To summarize, main contributions are threefold:
We revisit the hallucination mitigation problem in LVLMS from an information-theoree perspective, where we reformulate it as conditional mutual information maximization problem and introduce a novel bi-level optimization- solution framework.
To implement this optimization, language decoding strategy that dynamically refines the most informative visual and textual tokens maximize the C-PMI throughout the generation process.
propose effective and efficient adaptive vision-
Extensive experimental results on multiple LVLMMs such as VLa across five evalua- tion benchmarks demonstrate the exceptional effectiveness of the proposed CMI-VL. in mitigating ballucination, significantly outperforming competitive baselines.

---

2 Related Work
Large Vision- LanguaLe Models. Built upon advanced pre-trained LLMs
24
LVLMS
successfully bridge the gap between visual perception and linguistic reasoning 226
27 3
28
29
achieving impressive performance in generating diverse responses and tackling complex visual un- derstanding tasks. To incorporate visual information into the LLM backbone, LVLMs like LLaVA
2 4 and Shikra employ linear projection layers trained by instruction fine-tuning to directly map visual features into the L.M embedding space. Meanwhile, the BLIP series 30 introduces Q-former to integrate visual tokens dynamically through gated cross-attention layers, thereby re- ducing redundancy in image token representations. Benefiting from better training data, improved algorithms, and increasingly powerful LLM backbones, recent LVLMs such as LLaVA-Nex: [32] have demonstrated stronger multimodal understanding capabilities. Despite the progress, LVLMs still suffer from serious hallucination problems, where the generated responses are plausible yet unfaithful or factually incorrect. Our work aims mitigate this issue and enhance the reliability of LVLMs.
Mitigating Hallucinations in LVLMs. To address the critical issue, various strategies have been proposed to alleviate ballucinations from different perspectives. Early efforts focused on improving the multimodal alignment by training LVL.Ms with higher-quality data or more advanced algo- However, they often require additional datasets and incur substantial computa- tional overhead, primarily due the exhaustive instruction-un ttnnnn procedures. In parallel, post-hoc correction methods based auxiliary models have been explored [6] to filter revise halluci- nated content in the output responses. Nevertheless, these methods heavily rely on the performance of the auxiliary model and introduce extra inference overhead.
rithms 13 14 (3
Another research line focuses on decoding- based hallucination mitigation. These methods primarily seek to construct token distributions that adaptively suppress the probabilities of hallucinated to- kens 19 8 35]. By sampling from carefully crafted distributions, these methods significantly reduce hallucinated concepts in generated responses. In addition, OPERA 19 identifies strong correlation between hallucinations and summary tokens, and proposes to penalize the over-trust logits along with rollback strategy. (20] conducts a modular analysis and empirically reveals that certain attention heads overly focus on textual tokens while neglecting the pivotal visual information, based on which they introduce two correction algorithms to penalize text attentions. Among these methods, only M3ID [21] considers theoretical aspects, yet it introduces mutual information solely to justify its vision-promg dependency metric in contrastive decoding, without delving deeper into the key factors influencing C-PMI or exploring an effective optimization paradigm. In contrast, this paper proposes a novel multimodal adaptive decoding algorithm grounded in C-PMI. which dynamically amplifies the mutual relevance between image and text and effectively reduces hallucinations in LVLM outputs.
3 Methodology
This section first introduces the basic generative paradigm of LVL.ss. Building on this, we propose our adaptive decoding algorithm for hallucination mitigation, i.e., CMI-VLD. Finally, we present the detailed design of a learnable predictor for visual token purification in our method.
3.1 Preliminary
Before delving into the proposed adaptive decoding algorithm, i.e., CMI-VLD, we revisit the autore- gressive generation paradigm of LVLMs, which serves as the foundation for subsequent derivations.
Given user prompt and image v as input, pre-trained LVLM fo(.) first processes the image through vision encoder, followed by a cross-modal projection module, to generate a set of visual tokens e = {vo, .I,..., uN J. At decoding step t, the visual tokens are concatenated with the textual tokens from the instruction x and the previously generated token sequence yea. The resulting sequence is fed into the LLM backbone of the LVLM to autoregressively predict the next token:
Ve ~ p- | U. x. Vet) softmax(fol | u. Bct)),
(1)
3

---

where is the token being sampled at current generation step t. In particular, the probability of generated sentence y of length I can be factorized a product of conditional probabilities:
go u,
II (v v,2, t=
II softmax (Jo ( U. v.e)) 1=0
(2)
where do denotes the sentence-level conditional probability distribution characterized by the LVLM fo(.). This yields an appealing property for the subsequent expansion of mutual information, as the likelihood of given text under specific LVLM can be accurately computed by Eq. (
3.2 The Proposed CMI-VLD
To reduce hallucinntiion-related content in the output response, we propose to strengthen the bidi- rectional dependency between the input image and the generated sentence by maximizing their conditional mutual information measured by the target LVLM fo()). However, standard CMI com- putation requires estimating the full conditional distributions of image variable V and text variable Y given the user instruction variable X, which is intractable in practice due to challenges such as dimension explosion or data sparsity. To overcome this challenge, we adopt its pointwise formulation [36.. which balances theoretical rigor with practical feasibility, to quantify the local dependency between specific image-text pair (V U, Y = y). conditioned on given instruction X =
max C-PMI&(V 0, Y y x) max (log
po y x) po x)) Po
v.g
x:)
To achieve more effective optimization, we carefully analyze this objective from the perspectives of both visual and textual data points involved in C-PMI calculation. Given an input image v, the algorithm should encourage the generation
text y that is highly aligned with the visual input v to strengthen their mutual dependency. Simultaneously, for the given text y, the visual input can be refined to exhibit strong relevance to 1/. hence further amplifying the mutual information between two modalities. As result, the bidirectional dependency between the two variables in maximizing C-PMI naturally induces a bi-level optimization framework, which can be effectively addressed by alternately optimizing the derived inner and outer subproblems. However, the conditional distributions in Eq. (3) can not yet be directly calculated. Based on Eq. (2) and Bayes' Theorem, we then further expand the optimization objective as follows:
max C-PMIa(V, V x max Σ log pa (We U, u, (, - log Po (Ve Vct)].
(4)
Detailed proof is in Appendix A This formula decomposes the original objective over individual decoding steps, enabling each term be explicitly computed using the token-level probabilities provided by the LVLM. An interesting observation is that the token distributions used in existing contrastive decoding studies [8
9 I0 can be viewed as specific variants of the optimization goal in Eq. (4), and thus can be naturally regarded as special cases of our framework when only the text's influence on C-PMI is considered. Next, we concretize the solution of interdependent subproblems from text and visual modalities to form our alternating optimization procedure:
17
(1) Calibrated Distribution Sampling for Text Modality. To optimize the text sequence y. Eq. encourages us to construct an improved distribution P. to prioritize text tokens that maximize the difference between probabilities predicted with and without the visual input. However, directly applying this formulation yields unsatisfactory results since it can excessively penalize reasonable tokens in certain contexts. Inspired by [17 9. introduce hyperparameter fine-grained control over the strength of the subtraction, which can be formally expressed as:
provide a more
Vt Pe(- (, softmax (1 (1fo (- (. Xfo (: [.,
(5)
This strategy calibrates the token distribution by urging the generation toward tokens that are more informative of the image and hence enhancing its reliance on visual content. To ensure the quality of generated sentences, we also incorporate the adaptive token truncation mechanism [3] to prune the sampling space of Eq. into more reliable token candidate pool.

---

Visual Token Purifier
Visual Input
VIT Block
Please deseribe this image detail.
Basebail
throwing a...
MLP
Softmax())
Layero-
Sumple
GumbelSoftmax
G (0.1) 2
Layer
LLM backbone the LVLM
G
M,
(1 (1+ X) log Pa v", log Pa x,yxt)
Calibrated Distribution Sampling
Figure 2: Overview of the proposed CMI-VLD decoding. At each timestep I, CMI-VLD mitigates hallucination by maximizing mutual dependency between the visual input and the ongoing response through the proposed vision-language purification. Specifically, the visual token purifier first incorpo- rates current input tokens to predict an image mask MM. which filters out irrelevant visual tokens to enhance C-PMI. Based on the refined visual input, text token distribution is correspondingly constructed to penalize hallucination-relaae further strengthen the dependency the visual input.
text tokens and hence guide the next-token sampling to
(2) Visual Token Refinement for Visual Modality. Motivated by recent findings many image tokens in LVLMs are redundant, we propose visual token purification mechanism that enhances C-PMI by evicting tokens considered non-informative with respect way, the LVLM can focus more the most critical visual tokens for improved generation. Moreover, we also incorporate the model's attention scores over the visual input identify tokens that exert stronger influence on the LVLM's decisions. Given the query vectors Q E TR. H Xnxx2 and key vectors K; E R xnxd4 at the i-th LVLM layer, where H is the head number, n is the current token number, and dę is the latent dimension, the total attention scores of an image v is calculated as:
that
given text. this
(utn,(u) =
Σ
H-1
[-1][ush where A; = softmax
(QiK) Vdl
+ MMMe).
(6)
H
By Ev k=0
where Mr is the causal attention mask and A, E Hxxx denotes the attention matrix at the i-th layer. This design enables the optimizer to select visual tokens that are not only text-relevant but also highly impactful in guiding the model's predictions, boosting the effectiveness of our visual purification. Formally, the overall bi-level optimization objective can expressed as:
(1 A) log po (ya v. ,,,,,, Vea) - Allog po (yi x. v<ee)
arg max [a (ttn;()) log (yt u, Met) log (ye xx V<t)))
where À is the aforementioned correction factor for distribution calibration and a non-negative hyperparameter balancing the influence of C-PMI and attention scores on visual token selection.
At decoding step we first optimize the upper subproblem by sampling ye from the distribution adjusted based on Eq. (5). To solve the lower subproblem, we then adaptively retain a proportion the current textual context. Motivated by findings in (38 that token sparsification at the second layer of LVLMs yields optimal performance, we utilize attention scores from this layer (i the refinement accordingly. By alternately solving the two subproblems at each decoding step, the optimizer simultaneously samples text tokens that tightly align with eurrent visual content and purifies informative visual tokens with strong relevance to the ongoing textual context, hence effectively amplifying C-PMI reducing hallucinatioo--elated elements the final response. For stable performance, we also incorporate the feature stecring mechanism [39 into implementation.
y of image tokens as the purified input promote its relevance
for visual purification and start
5

---

3.3 Visual Token Purifier for Visual Refinement
To address the outer subproblem in Eq. (7). an intuitive solution is to manually select image tokens that maximize the defined score for every decoding step. However, this requires repeatedly calculating token-wise scores and would incur substantial computational burdens compared existing decoding- based approaches [8]. To overcome this challenge, we propose lightweight visual token purifier PC), which consists of only few transformer blocks and ML.P layers (see Appendix Dfoor details) 37 to automatically filter visual tokens that benefit C-PMI maximization.
As illustrated in Fig. 2 the purifier P(-) incorporates the concatenated embeddings [Z., 22, Zycel of the image and the current text (æ, Ver) to output a probability distribution श = max (P (z)) E [0, where N is the number of visual tokens. Here, 7,,0 repre- sents the probability of discarding the i-th token, and 7,1 represents the probability of retaining The final visual token mask My E {0, 13.² can be then extracted via:
arg max Tij i 1, 10. -11 jE(0,1)
M
Model Training. The principle challenge in training the purifier lies in the non-differentiabiiiyy of the arg max operation used for discrete token selection. To address this, we employ the Gumbel-Softma technique with a temperature parameter to enable differentiable sampling:
G Gumbel--Softmax(77777
where the sampling output G E {0, 1] containing N one-hot vectors. Since the retention probability of a visual token corresponds to the second column in π, the differentiable mask M, E (0, 1) can be extracted M2 = G[: 1]. This approach introduces stochasticity via noise from a fixed Gumbel distribution, which enables gradients to propagate back through the probability parameters. Moreover, the temperature factor r helps soften the sampling distribution, thereby improving gradient stability and facilitating convergence during training.
To specify the retaining ratio y, introduce Frobenius norm-based regularization term that penalizes incorrect retention rate. The overall training objective at decoding step is defined as:
Liotal = ( log pe (y/ | U, x.Her) - log Po(: x. vcc))
(9)
+ + 3 [|sum(M.)/N - >11p:
where B is a weight coefficient controlling the regularization strength of the reduction ratio, Il-l denotes the F-norm of matrix, and sum(-) represents the summation operation.
By iteratively updating the network using the loss function in Eq. (9) paired image-text data, the purifier learns to dynamically identify visual tokens that effectively contribute to mutual information maximization, which further enhances the informativeness of the visual input while discarding those redundant and distracting visual tokens. Furthermore, our method preserves the decoding efficiency despite introducing an additional network, the purifier module lightweight and the removal of non-essential visual tokens helps reduce the overall inference cost (see Sec
4 Experiments
4.1 Experimental Setup
Models and Baselines. We align with (9] and choose four representative LVLMs for evaluation, including InstructBLIP (3T], Shikra B. LL. on the 7B scale 4. and LLaVA-NeXT [32
on the 8B scale. We conduct a comprehensive evaluation of the proposed CMI- VLD on range of of-te-of-the-as (SOTA) baselines, including Sampling (Top-p=1), Greedy, VTI 39), VCD [7].
ICD 18]. HALC [8), OPERA 19] SID (9. and VASparse [10] Following SID, we implement the proposed method under both sampling and greedy decoding settings. It is worth noting that HALC, OPERA, and VASparse adopt the more flexible and stronger beam search strategy, which may raise fairness concerns as it retains a broader set of promising candidate paths during decoding. Nevertheless, our CMI-VLD still consistently outperforms these methods.

---

Table 1: Comparison of the proposed CMI-VLD with SOTA baselines on the CHAIR metric. We evaluate the performance on MSCOCO. The indicates decoding strategies based on beam search.
LLa -1
InstructBLIP
Shikra
LLa VA-Next
Method
Cs +
C₁
Cs
C₁ 4
Cs
C1
Cs
C1 4
Sampling
52.2
15.8
55.0
25.3
56.2
15.8
34.8
9.4
ICD
51.0
15.2
64.0
20.2
56.6
15.5
33.4
8.7
VCD
50.4
15.6
57.6
19.2
56.2
15.5
36.0
9.3
VTI
37.2
11.4
49.2
21.9
47.0
14.1
32.2
7.8
SID
49.2
14.5
58.0
18.7
54.4
14.4
39.4
9.9
CMI-VLD
30.2
9.3
51.0
16.1
38.2
10.1
30.6
7.6
Greedy
45.0
13.5
52.2
21.8
54.8
15.8
31.6
8.2
ICD
44.8
12.8
48.8
14.1
55.0
14.0
32.8
9.1
VCD
49.4
14.0
46.6
13.3
55.8
15.3
36.8
9,4
HALC
33.2
10.3
61.4
20.0
55.4
14.7
36.7
9.5
OPERA
39.4
10.3
48.2
13.8
36.8
11.7
33.6
VTI
30.6
10.1
48.3
20.7
44.6
13.7
30.1
7,6
SID
42.8
12.1
56.2
15.8
51.2
13.6
38.0
8.9
VASparse
49.6
14.2
53.6
14.9
51.6
14.8
33.6
9.1
CMI-VLD
29.9
8.9
43.2
12.9
30.6
8.9
27.2
6.8
LLAV--L5
TestructB1.11
Shilkra
LLLVA-NeXT
SB1
911
W71
Sampling
OPERA
VASparse
Figure 3: GPT-4o assisted benchmark. We calculate the Sentence-level Hallucination Ratio (SHR) as the major metric for hallucination degree, along with 1&2- 202mm,mm, the number of sentences per image (SPI), and the number of words per image (WPI). A larger radar area indicates better performance.
Evaluation Benchmarks. Following the evaluation protocol in 35 we analyze our CMI-VLD across five widely-used benchmarks: (1) the CHAIR metric 40 on the MSCOCO dataset [4] that measures object hallucinations; (2) a GPT-4 assisted evaluation [33], where we adopt the advanced GPT-4o [2] to detect more fine-grained hallucinations and compute the Sentence-level Hallucination
Ratio (SHR); (3) Polling-based Object Probing Evaluation (POPE) 43] another object hallucination
evaluation also conducted on MSCOCO; (4) Multimodal Large Language Model Evaluation (MME)
[44] a general-purreee purpose benchmark for assessing multimodal capabilities; and (5) MMBench [45), which includes multiple-choie questions designed to evaluate visual perception and reasoning.
Implementation Details. We set a = 102 and B3 = 102 in Eq. (9) during the training of the for LLLVA-1..5 and 4 for InstructBLIIP In Sec. 4.3] we explore the contrastive strength
purifier for all LVLMs. To initiate the refinement process, we adopt Layer Shikra, and A ranging from to 0.9. For all experiments, we set max new tokens 512 for evaluation. Note that CMI-VLD is compatible with the feature steering mechanism in VTI [39], which then incorporated in our implementation for stable and enhanced performance. More details are in Appendix
4.2 Performance Evaluation
we query LVLMs with the input prompt "Please describe this image in detail." using 500 images randomly sampled from the validation set of MSCOCO. By dynamically amplifying the mutual relevance between visual inputs and generated texts, the proposed method achieves remarkable improvements over SOTA baselines
CHAIR Evaluation. Following previous studies [19
7

---

Table Comparison of the proposed CMI-VLD with SOTA baselines on the POPE metric. The indicates decoding strätegies based on beam search.
Random
Popular
Adversarial
Model
Method
Accuracy
Fl score
Accuracy
Fl score
Accuracy
Fl score
Sampling
82.53%
79.19%
81.57%
78.31%
80.30%
77.16%
ICD
82.77%
79.57%
81.97%
78.81% %%
81.03%
77.95%
VCD
83.67%
80.80%
82.17%
79.40%
80.90%
78.25%
VTI
82.70%
79.45%
81.43%
78.25%
80.10%
77.05%
SID
84.67%
82.20%
83.57%
81.16%
81.60%
80.27%
CMI-VLD
85.17% %
83.02%
84.10%
82.03%
82.30%
80.40%
Greedy
83.40%
80.32%
82.60%
79.55%
81.77%
78.77%
LLaee-Next
ICD
83.47%
80.41%
82.60%
79.56%
81.90%
78.91%
VCD
84.43%
81.85%
83.30%
80.77%
82.33%
79.88%
HALC
83.34%
80.36%
82.33%
79.48%
81.40%
78.92 %
OPERA
83.50%
80.46%
82.70%
79.69%
81.87%
78.91%
VTI
84.70%
82.09%
83.67%
81.11%
82.90%
80.40%
SID
84.97%
82.53%
83.93%
81.56%
82.97%
80.67%
VASparse
83.47%
80.52 %
82.24 %
79.69%
81.33%
78.88 %
CMI-VLD
86.43 %
84.52%
85.07%
83.22%
83.90%
82.14%
Sampling
82.03%
81.30%
78.77%
78.66%
76.37%
76.81% %
VTI
83.50%
82.01%
80.83%
79.70%
79.13%
78.29%
ICD
83.20%
82.29%
79.87%
79.51%
77.63%
77.74%
VCD
83.43%
82.49%
79.70%
79.36%
77.53%
77.65%
SID
85,43%
84.81%
82.43%
82.24%
79.47%
79.84%
CMI-VLD
86.33 %
85.41
84.60%
83.87
81.57%
81.29
Greedy
87.27%
85.91%
84.87%
83.72%
82.97%
82.04%
InstructBLLIP
ICD
87.23%
85.82%
84.90%
83.68%
83.13%
82.11%
VCD
86.73%
85.30%
84.37%
83.16%
82.47%
81.49%
HALC
87.30%
85.96%
84.83%
83.70%
83.00%
82.08%
OPERA
87.53%
86.26%
85.07%
84.00%
83.07%
82.24%
VTI
85.73%
83.86%
84.13%
82.36%
82.50%
80.89%
SID
88.10%
87.15%
85.87%
85.10%
82.90%
82.52%
VASparsef 87.33%
86.00%
84.87%
83.74%
83.00%
82.09%
CMI-VLD
88.37%
87.50
86.10%
85.40%
82.87%
82.64% %
on different LVLMs, as observed Table E.g., a notable improvement of 7% and 2.1% Cs and C7 for Sampling the LLaVA-1.5 model. Notably, some baselines exacerbate hallucination content compared to standard decoding strategies in some cases. In contrast, the proposed CMI-VLD consistently reduces both sentence-level and instance-level object hallucinations in the final responses.
GPT-4o Assisted Evaluation. While CHAIR reliable evaluation metric widely adopted in previous studies, it is limited within the scope of object hallucinations and fails to identify other types, such as attribute, relational, and positional hallucinations. To more comprehensively evaluate the effectiveness of our method, we introduce the GPT-assisted benchmark (3), which uses the object- level descriptions in the Visual Genome dataset [46) as ground-truth, to judge more fine-grained hallucinations assisted by the advanced GPT-4o. Figure demonstrates that the proposed CMI-VI.D significantly outperforms SOTA basclines across four LVLMs. Compared with competitive basclines, achieve a relative improvement of 15.89% for VA-1.5 in the hallucination metric SHR while maintaining text fluency. We also note that our method reduces the length of generated texts to some extent, which can be caused by the removal of hallucinated sentences
[9]
POPE Evaluation. The POPE metric also focuses on object hallucinations by using a prompt "Is there <object> in the image?" to query LVLMs for answering yes/no question. We report results of the accuracy and score in Table The quantitative results reveal that our method generally performs best across the three split datasets. Notably, in the POPE evaluation, where responses are typically short and follow fixed patterns such as "Yes, there is <object> the image." the evaluation primarily hinges on the first one few tokens (i.e., Yes or No) [19

---

Table 3: Comparison of the proposed CMI-VLD with SOTA baselines on LVLM benchmarks. The
indicates beam search-based methods, while other methods adopt the Greedy decoding.
Benchmarks
Greedy
ICD
VCD
HALC
OPERA!
VTI
SID
VASparse
(MIVLLD
MME
1465.11
1432.43
1472.57
1473.43
1471.37
1435.35
1467.05
1466.61 64.78
1481.17
MMBench
64.86
64.26
64.26
$7.90
64.78
64.43
64.26
65.12
202
Assrape Thase
33
Cenoonaioo Time per Sumole
10:
33
31
32
30
31
30
7.5
29
0.1
0.3
0.4
0.6
0.7
Ø valuoss (x100)
values
Mithod
(6)
Figure Generation time per sam- ple of different methods.
Figure 5:
CHAIRs results of the proposed CMI-VLD under vary- ing values of hyperparameters or and X.
Consequently, CMI-VLD may not fully exhibit its effectiveness in this constrained setup, it is designed to dynamically adjust decoding over the entire generation rather than concentrating solely the initial tokens. Nevertheless, method still achieves notable improvements baselines. Due to the page limit, results on more LVLMs are provided Appendix
competitive
MME and MMBench Evaluations. Apart from the above benchmarks tailored for hallucination evaluation, we additionally test on two popular LVL.M benchmarks, i.e. MME 7 and MMBench [45, to systematically analyze their various capability dimensions. MME provides suite of fine- grained, saauge-grounde multiple-choice questions across various categories. We follow SID and report the overall perception score covering 10 sub-tasks, such object existence and fine-grained recognition. MMBench is another large-scale bilingual benchmark consisting of over 3,000 curated multiple-choice questions. We compute the LVLM's average as attributes, logical reasoning, and coarse/fine- ggaiieddandd perception, to comprehensively evaluate its capabilities. As observed, CMI-VLD not only reduces the hallucinated contents but also enhances diverse capabilities of MLLMs, bringing remarkable improvements over the default decoding methods. These results underscore CMI-VL. reliable and practical strategy for ballucination mitigation.
across 20 multimodal tasks, such
Inference Time Analysis. Since our method introduces an additional visual token purifier for effective visual refinement, it crucial to assess its influence the overall computational efficiency. Following 200). we calculate the generation time per response based on LLaVA-1.5 to assess computing burdens. Figure reveals that the proposed CMI-VLD achieves satisfactory decoding efficiency, introducing negligible computational overhead. This is primarily attributed to the lightweight architecture of the visual purifier and the removal of redundant visual tokens that would incur significant computational overhead, demonstrating that CMI- VLD effectively balances performance and efficiency.
4.3 Ablation Study
Next, we provide ablation analysis regarding several critical hyperparameters. More ablation analysis about the retaining ratio and the effectiveness of the proposed techniques is presented in Appendix C
The effect of varying loss parameter a. The value of a a critical factor as adjusts the contribution of the attention scores during purifier training. We then evaluate the performance under various values of a in Figure (a). The performance gains observed when comparing to a = suggest that incorporating Attn,()) enhances the effectiveness of the visual purifier. Moreover, the results indicate that & x 102 yields the best performance, and is therefore adopted in our main experiments.
The effect of calibration intensity X. During decoding, the hyperparameter À plays pivotal role in regulating the strength distribution calibration. We present the CHAIR results under varying À in Figure 5 (b). The proposed method reaches optimal performance when À 0.5, hence we adopt it as the default setup. Moreover, we emphasize that a properly selected range of positive values yields significant improvements over the X 0 setup, validating distribution calibration strategy.

---

5
Conclusion
In this work, we first revisit the key reason for hallucination in LVLMs, based on which we introduce conditional mutual information theoretical foundation to enhance the mutual dependency between visual input and generated text. To strengthen this cross-modal association, we propose a novel CMI-aware bi-level optimization framework, which is efficiently and effectively solved via carefully designed vision- language decoding strategy. Through extensive experiments across multiple LVLMs and evaluation benchmarks, we demonstrate the superiority of the proposed approach in mitigating hallucinations and improving the recognition capability of LVLMs in diverse scenarios.
Acknowledgement
This work is supported in part by the National Natural Science Foundation of China under grant 62171248, 62301 189, 62576122, and Shenzhen Science and Technology Program under Grant 01000244000000000000000000000200
GXWD2020881
References
[1] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny. "Minigpt-4: Enhancing vision-language under- standing with advanced large language models," in The Twelfth International Conference on Learning Representations.
[2] H. Liu, C. Li, Q. Wu, and Lee, "Visual instruction tuning," Advances in neural information processing systems, vol. 36, pp. 34 892-34916, 2023.
[3] K. Chen, 7. Zhang, W. Zeng, Zhang. F. Zhu, and R. Zhao, "Shikra: Unleashing multimodal Ilm's referential dialogue magic." arXiv preprini rrXiv:2306.15199 2023.
[4] H. Liu, C. Y.I,Y,L,, and J. Lee, "Improved baselines with visual instruction tuning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024, pp. 26 296-26 306.
[5] H. Fang. J. Kong. Yu, B. Chen, J. Li, H. Wu, S. Xia, and Xu, "One perturbation is enough: On generating universal adversarial perturbations against vision-language pre-training models," arXiv preprint arXiv:2406.00 05491, 2024.
[6] W. Wang, J. Xie, C. Hu, H. Zou, J. Fan, W. Tong, Y. Wen, S. Wu, H. Deng, Z. Li et al., "Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving." arXiv preprint Miix:2312.09244 2023.
[7] C. Cui, Y. Ma. X. Cao. W. Ye. Y. Zhou, K. Liang, J. Chen. J. Lu, Z. Yang. K.-D. Liao et al., "A survey multimodal large language models for autonomous driving." in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 958-979.
[8] Z. Chen, Z. Zhao, H. Luo, H. Yao, B. Li, and J. Zhou, "Hale: Object hallucination reduction via adaptive focal-contrast decoding." " arXiv preprint arXiv:2403. 0044555 2024.
F. Huo. W. Xu, Z. Zhang, H. Wang, Z. Chen, and P. Zhao, "Self-introspective decoding: Alleviating hallucinations for large rision-language models," " arXiv preprint arXiv:2408. 02033,, 2024.
[10] X. Zhuang, Z. Zhu, Y. Xie, L. Liang, and Y. Zou, "Vasparse: Towards efficient visual hallucination mitiga- tion for large vision-language model visual-aware aware sparsification," arXiv preprint arXiv:2501.066 2025.
[11] F. Liu. T. Zhu. Wu, B. Yang, You, Wang. L. Lu, Liu, Y. Zheng. Sun et al., "A medical multimodal large language model for future pandemics," NPJ Digital Medicine, vol. 6, no. 1, 226, 2023.
[12] Huang, M. Xiao, D. Li, Jiang, Yang. Y. Zhang, L. Qian, Wang. X. Peng. Ren et al. "Open-finllms Open multimodal large language models for financial applications," arXiv preprint arXiv:2408. / /878, 2024.
[13] F. Liu, K. Lin, L. J. Wang, Y. Yacoob, and L. Wang, "Mitigating hallucination in large multi-modal models via robust instruction tuning," arXiv preprint ar ar:iv:2306. 230., 2023.
[14] T. Yu, Y. Yao, H. Zhang. T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H.-T. Zheng, M. Sun et al., "Rlhf-v: Towards trustworthy milms via behavior alignment from fine-graines grained correctional human feedback." in Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2024, pp. 13 807-13 137--3800 816.
[15] Y. Zhou, C. Cui, J. Yoon. L. Zhang, Z. Deng, C. Finn, M. Bansal, and H. Yao, "Analyzing and mitigating object ballucination in large vision-language models," arXiv preprint 2rXiv:2310.0075.0 2023.
10

---

[16] S. Yin. Fu, S. Zhao. T. Xu, H. Wang, D. Sui. Y. Shen. K. Li, X. Sun. and E. Chen, "Woodpecker: Hallucination correction for multimodal large language models," Science China Information Sciences. vol. 67, no. 12. p. 220105, 2024.
[17] S. Leng, H. Zhang. G. Chen, X. Li. S. Lu, C. Miao, and L. Bing, "Mitigating object hallucinations in large vision-language models through visual contrastive decoding," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 872-13 882.
[18] X. Wang. J. Pan, L. Ding, and C. Biemann, "Mitigating hallucinations in large vision-language models with instruction contrastive decoding," " in ACL (Findings), 2024.
[19] Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang. D. Lin, W. Zhang, and N. Yu, "Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allcatii in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp- 13418-13427.
[20] T. Yang. Li, Cao, and Xu. "Mitigating ballucination in large vision- language models via modular attribution and intervention," in Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning.
[21] A. Favero, I.. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and Soatto, "Multi-modal hallucination control by visual information grounding," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp.
[22] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., "Llama: Open and efficient foundation language models," arXiv preprint 23Xxv:302..39771 2023.
[23] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava,
S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint 2r37: 092888 2023.
[24] J. Kong, H. Fang, X. Yang, K. Gao, B. Chen, S.-T. Xia, Y. Wang, and M. Zhang, "Wolf hidden in sheep's conversations: Toward harmless data-based backdoor attacks for jailbreaking large language models," arXiv preprint arXiv:2505. 17601, 2025.
[25] H. Fang, J. Kong. T. Zhuang, Y. Qiu, K. Gao, B. Chen, S.-T. Xia, Y. Wang, and M. Zhang, "Your language model can secretly write like humans: Contrastive paraphrase attacks Ilm-generated text detectors," " arXiv preprint arXiv:2505. 15337, 2025.
[26] Q. Ye. H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi al. "mplug-on owll: Modularization empowers large language models with multimodality." arXiv preprint arXiv:2300. 14178, 2023.
[27] B. Li, Y. Zhang, L. Chen, J. Wang, F. Pu, J. Yang. C. Li. and Z. Liu, "Mimic-it: Multi-modal in-context instruction tuning." " arXiv preprint arXiv:2306.00 05425,, 2023.
[28] H. Fang, J. Kong, B. Chen, T. Dai, H. Wu, and S.-T. Xia, "Clip-guided generative networks for transferable targeted adversarial attacks," " in European Conference on Computer Vision. Springer, 2024, pp. 1-19.
[29] T. Zhang, K. Gao, J. Bai, L. Y. Zhang, X. Yin, Z. Wang, S. Ji, and W. Chen, "Pre-training clip against data poisoning with optimal transport- -ased matching and alignment," arXiv preprint arXiv:2509.1811 18717, 2025.
[30] J. Li. D. Li. S. Savarese, and S. Hoi, "Blip-2: Bootstrapping language -magge---maeeeeee pre-training with frozen image eneoders and large language models," in International conference machine learning. PMLR, 2023, pp. 19 730-19742. 742.
[31] W. Dai. J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, "Instructblip: Towards general- perpral-purpose vision-language models with instruction tuning." 2023. [Online]. Available: https:/arriv.or//hss22.....000
purpose
[32] B. Li and F. Huo, "Reqa: Coarse-to-fin assessment of image quality to alleviate the range effect," Journal of Visual Communication and Image Representation, vol. 98, p. 104043, 2024.
[33] 7. 7hao, B. Wang. I.. Ouyang, X. Dong, J. Wang, and C. He, "Beyond hallucinations: Enhancing Ivims through ballucination-wwant direct preference optimization." arXiv preprini arXi:22311.1683,, 2023.
[34] Y.S. Chuang, Y. Xic, H. Luo, Y. Kim, J. Glass, and P. Hc, "Dola: Decoding by contrasting layers improves factuality in large language models," arXiv preprint 2rXivv2309.03888 2023.
[35] C. Wang, X. Chen, N. Zhang, B. Tian, H. Xu, S. Deng, and H. Chen, "Milm can see? dynamic correction decoding for hallucination mitigation." International Conference on Learning Representations, 2025.
[36] L. Van Der Poel, R. Cotterell, and C. Meister, "Mutual information alleviates hallucinations in abstractive summarization." in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 5956-5965.
11

---

[37] Y. Rao. W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, "Dynamicvit: Efficient vision transformers with dynamic token sparsification." Advances in neural information processing systems, vol. 34, pp. 13 13 2021.
[38] L. Chen, H. Zhao, T. Liu, S. Bai, Lin, C. Zhou, and B. Chang, "An image is worth 1/2 tokens after layer 2: Plug and play inference accelcration for large vision lainual models," in European Conference Computer Vision. Springer, 2024, pp. 19-35.
[39] S. Liu, H. Yc, and J. Zou, "Reducing hallucinations in large vision-language models via latent space steering," in The Thirteenth International Conference on Learning Representations, 2025.
[40] A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Sacnko, "Object hallucination in image captioning," arXiv preprint
2018.
[41] T.-Y. Lin, M. Maire, S. Belongie, Hays, Perona, D. Ramanan, Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in Computer vision-F.CCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13. Springer, 2014, pp. 740-755.
[42] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat ef al., "Gpt-4 technical report," arXiv preprint arXiv:2303.08772 2023.
[43] Y. Li, Y. Du, K. Zhou, J. Wang. W. X. Zhao, and J.RR. Wen, "Evaluating object hallucination in large rision-language models," arXiv preprint arXiv:2305.
2023.
[44] Z. Liang. Xu, Y. Hong, P. Shang. Q. Wang, Q. Fu, and K. Liu, "A survey of multimodel large language models," in Proceedings of the 3rd International Conference on Computer. Artificial Intelligence and Control Engineering, 2024, pp. 405-409.
[45] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang. W. Zhao, Y. Yuan, J. Wang. C. He, Z. Liu et al. "Mmbench: your
model an all-around player?" in European conference on computer vision. Springer, 2024. pp. 216 233.
[46] R. Krishna, Zhu, O. Groth, J. Johnson, K. Hata, Kravitz, S. Chen, Kalantidis, L..J. Li, D. A. Shamma et al. "Visual genome: Connecting language and vision using crowdsourced dense image annotations," International journal of computer vision, vol. 123, pp. 32-73, 2017.
[47] L. Chen, Li, X. Dong, Zhang, C. He, Wang, F. Zhao, and D. Lin, "Sharegpt4v: Improving large multi-modal models with better captions," in Luropean Conference on Computer Vision. Springer, 2024, pp. 370-387.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state our contributions and research scope in the claims presented in the abstract and introduction, all of which are supported by theoretical foundation or extensive experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We present the limitations of our paper section of the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[Yes]
Justification:
We have provided full proof for the formulas in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Detailed information of experiments provided in the Experimental section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We submit our code in the supplementary material and will open-source the code, models, and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccppblle Sttpc///uuidees/deeuuuubiiis
(CodeSubiisss wwtplubmmsssionii ssiiiioiiccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide exhaustive training and test details in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide hyperparameter analysis in different settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide them the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///eeuipp.cccppppe///ttbbaa
Answer:
[Yes]
Justification:
The research conform every respect of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We highlight that our work can reduce hallucinations LVLMS and enhance their reliability in real-world applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The license and terms of use are properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
No new assets introduced except code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not utilize LLMs for our core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
19

---

A
Complete Derivation of Equation (4)
Based on Eq. 2 and Bayes' theorem, provide the detailed derivation of Eq. 4 follows:
po(u, y x))
max C-PMIo(v,) (v,y x)) max log
(10)
Pe x)) po x)
Po (v, n, ())
max log
(11)
( x)) (o x))
Pe (u, 00 x.,)) y)
max log e.y
(12)
(2:) po ( x)) Po x))
(u 22, y)
max log v.y
(13)
po (v, X)) pe x)
po [y v,
max log v.y
(14)
Po x))
- ( 0,2,yec)
max log v.y
(15)
t=0 Po (x,))))))
max log w.y
Po (V: [,,, log II
Po(Be 2, ve)))))
(16)
x=
[log (v 0, Vet) log Po(ue xr,
(17)
B Experimental Details
B.1 Implementation Details
Throughout our experiments, we retain 80% of the visual input for LLaVA and LLaVA-NeXT, and 90% for Shikra and InstructBLIP. To guide the training of the purifier, we utilize image-text pairs from ShareGPT4V 47 high-qualits image question answering dataset constructed using images from the MSCOCO dataset. Specifically, we use 2,000 samples for training the purifiers of LLaVA and VAN. and 4,000 samples for InstructBLIIP and Shikra. The learning rate set to 1 x 10-6 across all models for the decoding hyperparameters of LLMss and the purifier is trained for 5 epochs.
B.2 Evaluation Model
As mentioned above, adopt InstructBLIP 31.. Shikra B], LLa on the 7B scale [4] and LLaVA-NeXT [32] on the 8B scale. InstructBLIP employs Q-former as a cross- modal connector, leveraging 32 learned query tokens to extract and align visual features with text representations in an efficient manner. Other models adopt a simpler architecture of linear projection layers, which directly map visual features into the language model's embedding space, typically using larger number of image tokens (256 or even 576) as input.
B.3 Evaluation Benchmarks
CHAIR Evaluations. The Caption Hallucination Assessment with Image Relevance (CHAIR) metric is specifically designed evaluate object hallucination image captioning tasks. quantifies the extent to which generated caption includes references to objects that are not present in the corresponding ground-truth annotations. Specifically, CHAIR computes the proportion of hallucinated objects, those mentioned in the generated caption but absent from the reference object set, providing a direct measure of hallucination severity. CHAIR comprises two commonly used variants: CHAIR; (Cr) and CHAIR, (Cs), which evaluate the degree of object hallucination at the instance and sentence level, respectively. The lower values of C₁ and C's correspond to a lower degree of object hallucination, indicating greater factual consistency. The two variants can be formulated as follows:
Ihallucinated objects]
captions with hallucinated objects)
C all mentioned objects
Cg
lall captions|
20

---

POPE Evaluations. The Polling-base based Object Probing Evaluation (POPE) benchmark is also proposed to evaluate object hallucination in LVLMs. It adopts a discriminative approach by prompting models with binary questions such as "Is there <object> in the image?" to assess whether the model can correctly identify the presence or absence of specific objects. To ensure balanced evaluation, POPE includes 50%/50% ratio of queries about present and absent objects. POPE further categorizes the queries into three negative sampling settings: (1) random, where absent objects are sampled randomly; (2) popular, where negative objects are selected from the most frequent categories; (3) adversarial, where negative objects are chosen based on their high co-occurrence likelihood with present ones to increase difficulty. Evaluation is conducted using Accuracy and F1 score, with higher scores indicating stronger performance in mitigating object hallucinations. Due to the concise format of POPE responses, which are typically short declarative sentences, the benchmark primarily reflects the visual grounding ability of a model rather than its long-form generation capacity.
GPT-4 Assisted Evaluations. In addition to object-level hallucinations via CHAIR and POPE, we adopt the GPT-4 assisted benchmark (33, which leverages fine-grained object-level annotations from the Visual Genome (VG) dataset [46] as ground truth. In our implementation,
employ the advanced GPT-40 to identify detailed hallucinations, such as positional, relational, and attribute-based errors, and compute the Sentence-level Hallucination Ratio (SHR) as evaluation results. Given the generated captions and manually annotated facts, GPT-4o is prompted by a template to assess hallucinations for every sentence. Following previous studies [99 9 we evaluate on 200 VG images with a maximum output length of 512 tokens based on the prompt: "Please describe this image in detail."
MME and MMBench Evaluations. MLLM Evaluation (MME) benchmark is designed to rigorously assess hallucination in MILMs. provides a suite of fine-grained, amage-groundder multiple-choice questions across various categories, such as object recognition, OCR, counting, and commonsense reasoning, each requiring accurate visual understanding. By offering carefully controlled distractors and consistent answer format, MME allows for precise evaluation of model's ability to generate responses. MMBench is a large-scale, bilingual, multimodal benchmark designed to comprehensively evaluate the capabilities of lisgun-lagguage models (VLMs). It consists of over 3,000 carefully curated multiple-choice questions covering 20 fine- ane-grained ability dimensions, ranging from perception to reasoning. To ensure robustness and fairesss, MMBench introduces the CireularEval strategy, where models must consistently answer question across multiple permutations of choices. MME and MMBench provide rigorous and scalable frameworks for evaluating multimodal understanding and citructtionfoolowinn capabilities across a wide spectrum of models.
faithful,
B.4 Principles of Hyperparameter Choices and Adaptation to New LVLMs.
The byprrparameters are chosen based on our empirical analysis and relevant literature, which are further supported through ablation studies. Specifically, we summarize our choice strategy behind several critical hyperparameters for LLaVA-115 in our algorithm as follows:
Contrast strength A balances the difference between the vision-conditioned and vision-free distributions. Large values may favor casual and incorrect tokens, while a small X can fail to sufficiently penalize hallucination prone tokens. We aim to preserve correct distributions while penalizing hallucinated predictions, and thus select a moderate value of À = 0.5, which further validated by ablations (see Fig. (b)) and prior studies 9 I7].
Visual token retention ratio y is a sensitive and critical parameter. high y may retain noisy tokens and weaken the C-PMI maximization, while low can discard important visual information. Hence, we adopt an adaptive strategy: for models with many visual tokens (e.g., LLaVA-1.55,
set relatively smaller 7 80%; for models with fewer, already refined tokens (e.g., InstructBLIIP), we use a higher y = 90%. Ablation studies on each model validate the rationality, with results on VA- 1.5 shown in Fig. 7 as an illustration.
Coefficient of attention loss a balances the importance between attention loss and C-PMI loss during purifier training. Empirically, we find the attention loss to be ~ 10000 smaller in magnitude compared to C-PMI loss, so we set a = 100 to adequately amplify its impact while preserving the dominance of the C-PMI objective.
Purification starting layer i is chosen based on existing well-established studies on to- ken selection (9 38) which have been empirically validated to yield strong task-spec performance while preserving its general capability.
21

---

Table Comparison of the proposed CMI-VLD with SOTA baselines on the POPE metric. To make a fair comparison, all the methods are based on the sampling decoding.
Random
Popular
Adversarial
Model
Method
Accuracy
Fl score
Accuracy
F1 score
Accuracy
F1 score
Default
85.20%
85,42%
81.67%
82.50%
76.20%
78.40%
ICD
85.73%
85.84%
81.90%
82.61%
76.70%
78.68%
VCD
83.77%
84.24%
80.77%
81.84%
76.10%
78.38%
VTI
85.23%
85.33%
82.77%
83.24%
76.63%
78.56%
SID
87.93%
87.65%
84.57%
84.69%
79.43%
80.59%
Ours 88.63%
87.83%
86.37 %
85.71%
82.27% %
82.18%
Default 85.07%
83.44%
83.13%
81.68%
81.63%
80.37%
ICD
85.27%
83.87%
83.13%
81.94%
81.73%
80.73%
VCD
85.17%
83.77%
83.27%
82.03%
82.03%
80.96%
Shikra
VTI
84.03%
81.94%
82.43%
80.49%
81.07%
79.29%
SID
85.53%
84.26%
83.47%
82.39%
81.43%
80.64%
Ours
86.23%
85.15%
83.83% %
82.96%
81.63%
81.08 %
82
1480
score
MME scores
1470
80
1460
F1
79
1450
78
1440
Creedy CMI-VLD; CMI-VLD, CMI-VDD
60%
70%
80%
90%
100%
Method
Retaining Ratio y
Figure 6: Ablation analysis of the proposed two techniques the POPE metric.
Figure 7: MME results of the proposed CMI- VLD under varying values of retaining ratio 7.
For the adaptation of our method, we empirically observe that the above hyperparameters can transfer new L.V.Ms and achieve effectiveness. necessary adaptation involves slightly tuning according to the characteristies of the new models. We also recommend adjusting the retention ratio based on the number of input visual tokens, as previously suggested. addition, it is also necessary to correspondingly adjust the hyperparameters in latent feature steering 339 for different LVLMs.
B.5 Explanation about the Inference Time Comparison
we evict the selected visual tokens by applying masks to the attention matrices for implementation convenience, rather than physically removing them. The inference time of CMI-VLD reported in Fig. is measured under this implementation and demonstrates that our method maintains inference efficiency. In practice, performing actual eviction of visual tokens would further accelerate inference, implying that the real efficiency advantage of CMI-VLD is even greater.
Notably, following SID
C More Experimental Results
POPE evaluation more LVLMs. We supplement the results of POPE metrics on more LVLMs, including LLaVA-1.5 and Shikra. The quantitative results in Table agaain confirm the effectiveness of our method in mitigating object hallucinations.
Ablation study of vision-language decoding. We then conduct an ablation analysis to validate contributions of the proposed two techniques, i.e., Calibraied Distribution Sampling and Visual Token Refinement, which interact with each other to fully maximize the C-PMI. Specifically, we design two variants CMI-VLD, and CMI-VLD,, which retain only the Calibrated Distribution Sampling
22

---

Table Inference costs under varying sequence lengths. The number of visual tokens is fixed to 576.
Sequence Length
Metric
Method
633 (prefilling)
850
1000
1250
1500
w/o purifier
1.9214
2.5814
3.0391
3.8044
4.5731
FLOPs (le14)
CMI- VLD
1.5671
2.4582
3.1226
4.3182
5,6241
w/o purifier
0.37
17.99
30.26
50.43
70.72
Inference Latency (s)
CMI-VLD 0.35
18.53
31.17
52.2
73.22
Table 6: Inference under varying numbers of input visual tokens. We fixed text query from the CHAIR evaluation, where the number of text tokens is 56.
Visual Token Count
Metric
Method
49
256
576
1024
w/o purifier
0.3188
0.9448
1.9214
3.3067
FLOPs (1c14)
CMI-VLD
0.2888
0.7876
1.5671
2.6718
w/o purifier
0.24
0.28
0.37
0.60
Inference Latency (s)
CMI-VLD
0.23
0.28
0.35
0.53
and Visual Token Refinement, respectively. Results in Fig. reveal that both the removal of the two components degrade the performance our algorithm, validating their considerable contributions to guarantee successful approach for hallucination mitigation.
Ablation study of varying retaining ratio. The retaining ratio y a sensitive hyperparameter that should be carefully tuned. A high retaining ratio may fail sufficiently enhance C-PMI, whereas an excessively low value can degrade performance due to information loss. We evaluate the influence under varying 7 to confirm the optimal value. Fig. 7 indicates that 80% is optimal choice.
Detailed Analysis of the Computational Costs. To reduce computational overhead, we design the purifier as lightweight network with only 0.1% of the parameters of the LVLM. Its effectiveness in mitigating hallucination has been thoroughly validated by extensive experiments in the main text. Next, we present detailed computational cost analysis of the visual purifier using L.LVV--Nex 8B.
As shown in Table and thanks to its lightweight design and visual token reduction, purifier introduces negligible overhead and generally maintains computational efficiency comparable to the purifier-free variant. Furthermore, the number of visual tokens increases, the benefits of visual token reduction become more pronounced, further reducing the computational complexity.
Table 7: CHAIR metrics and Token- seeeeod (TPS) of CMI-VLD and its learning free variant CMI-VLD, four LVLMs using greedy decoding. We present the results of the existing SOTA method, SID 9 as a reference.
Metric
Method
LLaVA-1.5
InstructBLIP
Shikra
LLaVA-NEXT
SID
42.8
56.2
$1.2
38.0
Cs
CMI-VLDif
30.0
40.4
36.2
26.6
CMI-VLD
29.9
43.2
30.6
27.2
SID
12.1
15.8
13.6
8.9
C7
CMI-VLD
9.0
10.2
CMI-VLD
8.9
8.9
6.8
SID CMI-VLDI5
8.76
11.70
3.85
15.71
TPS 1
2.45
2.41
1.05
2.35
CMI-VLD
8.96
11.86
4.29
16.52
Investigation the learning-free variant. Initially, we proposed learning a purifier to reduce the computational overhead incurred by manual token selection. To validate this design, we implement learning-free variant CMI-VL.D5 which selects tokens by manually computing our derived score in Eq. (7) each step, with all other settings unchanged.
23

---

Table 7 shows that both variants of our method significantly mitigate hallucination compared to the SOTA baseline, validating the effectiveness of our objective function derived from C-PMI. However, manual token selection incurs substantial latency due to repeated score computations at each decoding step, limiting its practicality in real-world applications. In contrast, our learned purifier efficiently selects informative tokens with nearly 4x faster inference than CMI-VLD;; while preserving strong effectiveness, exhibiting an excellent trade-off between performance and efficiency.
Model Architecture of Visual Token Purifier
We provide the detailed purifier architecture follows. Notably, this learnable network contains fewer than 1% of the LVLM's total parameters, hence introducing only marginal computation overheads.
Self- Attention
Self- Attention
LayerNorm
LayerNorm
LayerNorm
LayerNorm
LayerNorm
Linear
GeLU
MLP
MLP
Linear
GeLU
Linear
GeLU
Transformer EncoderBlock
Transformer EncoderBlock
Figure 8: Overview of the architecture of the visual token purifier.
E
Limitations
Despite the promising performance of LVLMs, the proposed method still faces several limitations. First, the introduced visual purifier introduces additional computational overhead for purifier training. Second, when the LVLM generates very long responses, the efficiency gain from the removal of visual tokens may become less significant. Besides, the performance the LVL.M is highly sensitive the retaining ratio, and the optimal ratio may vary at each decoding step. However, our method adopts fixed mask rate throughout the generation process. Future work could better address this limitation by exploring more advanced purification strategies with adaptive retaining ratios.
F Visualization Results
To visually demonstrate the effectiveness of our approach, we present generated captions using the prompt "Please describe this image in detail. from our method and baseline approaches on the MSCOCO dataset in Fig. 9 and 10 Qualitative results reveal the effectiveness of our method.
24

---

Instruction
(max new tokens-5512 512)
Please describe this
image in detail.
Greedy: image
close-up
delicious
a
top.
half. revealing the cucumber and other ingredients inside. The sandwich placed dining table, accompanted by bowl and cup, The bowl is located on the left side of the table, while the cup is situated on the right side. The sandwich appears to be delightful and bealthy meal option.
STD: image features dining table two sandwiches on a piece cardboard. Each sandwich filled tuna sandwiches. knifé can also be seen on the table, possibly used for spreading the tunn salad, The table setting is simple, with only the two sandwiches and the jur of greens occupying the space.
salad and topped with green sprouts or spinach leaves. jar greens, possibly parsley. placed
VASparse: The image features close-up view of delicious-looking sandwich, cut in half and placed on dining table. The sandwich is filled with meat and accompanied by leafy green garnish, adding touch of freshness to the dish. The dining table various items, including wine glass, a cup, lork, spoon. wine glass positioned table, while the cup located near the center. The fork and spoon placed the right side of the table.
of
a sprouting wooden table, and there nearby, possibly containing spread The sandwiches half, revealing the filling inside. The overall scene uppears to be casual and uppetizing meal.
image
two sundwiches,
lop.
ure placed
Instruction
(max new 512)
Please describe this image in detail.
Greedy: The image features delicious break fast plate with blueberry, strawberry, and bunana pancake lupped with whipped cream. The plate placed on dining table, and cup of coffee is positioned next to it. The table surounneed by chairs, with another right side. arrangement creates cozy inviling atmosphere enjoying
chair the
morning meal.
image showeases plate breakfast pastry topped strawberries blucberrics, sitting on dining table. Beside the plate, there cup of coffee, enhancing the overall breakfast setting. There two chairs visible in the seene; located to the right side of the table, while the other is situnted towards the top-right corner. The breakfast setting appears to be well-prepared and inviting.
VASpurse:
The image features deficious breakfast plate with blueberry, strawberry, and banana pancake topped with whipped cream. The plate is placod on dining table, accompanied by a cup of coffee. There are two chairs visible in scene, right side plate, ready used
the left side and another right side the table fork be to enjoy the scrumptious meal.
plate with delicious strawberry blueberry topped pancake. plate is placed dining table, and coffee situated next it. The pancake topped with fresh strawberries and blacberries, making it visually appealing and uppetizing breakfast or snack.
CMI-VLD: image
Figure 9: Visualization results comparing our CMI-VLD and other methods with LLaVA-1.5 back- bone, Hallucinations are marked in red.
25

---

Instruction
(max new tokens=5122
Please describe this image in detail.
The image features a train traveling over bridge that spans river. The train is positioned in the middle of the scene, with its length visible as it crosses the bridge. The bridge itself is made of concrete and appears to be a part of larger infrastructure. In addition to the train, there are several birds flying in the sky, adding sense of liveliness to the scene. The birds are scattered throughout the image, with some closer to the train and others further away. The combination bridge, and birds creates dynamic and lively atmosphere.
Greedy:
train,
SID: The scene features train crossing silver and blue cars, moving on the bridge. The hridge itself has large pipe on top, adding distinctive architectural element to the scene. In addition to the train, there are three birds visible in the sky. possibly flying or soaring above the water. The overall setting gives a sense of tranquility, with the train geatly traveling over the waterway.
red brick bridge that spans body of water. The train is composed of two long
VASparse:
The image depicts train traveling over a bridge that spans river. The train positioned in the middle of the scene, with its length visible as crosses the bridge. The bridge itself is made concrete and appears to be part larger infrastructure.
few people seattered throughout scene, possibly enjoying waiting for train pass, Some them closer to the while others further away, the edges of the image.
to the train,
CMI-VLD: The image features truin traveling over red bridge that spans body of water. The train is blue and white, and appears to be a commuter train. The bridge is made of concrete and provides clear path for the train to cross the water. In the background, there are trees and a gas tower, adding to the scenery of the image. The combination the train, the bridge, and water creates picturesque scene.
Instruction
(max new tokens=512)
Please describe this image in detail.
The image features a kitchen with sink and refrigerator on the left side. Above the sink, there is window with projection screen, which is displaying a picture of house. The kitchen is equipped with a microwave and an oven, both located on the countertop. There are several bottles placed around the kitchen, with one near the sink, another on the countertop, and third one further away. A bowl can also soen on countertop, a focus on projection screen and displayed image of the house.
Greedy:
vase is placed near sink. The kitchen appears to be
well-equipped organized,
SID: The image shows kitchen with stainless steel sink. black refrigerator, and black cabinets. There various items in the kitchen, including several bottles and cups placed around the countertop. mounted in the kitchen, likely showing the view of the house outside. The room is brightly lit, giving warm atmosphere to the space. addition to the main appliances, there is potted plant placed in the kitchen, adding touch of greenery to the room.
bowl, and spoon. A TV screen is
The image depicts kitchen with stainless steel sink, a refrigerator on the left side, and microwave on the countertop. Above the sink, there is a projection screen displaying an image of a house. The kitchen equipped with various appliances, including toaster, bottle, bowl, spoon. There are two people the scene, one standing near
VASparse:
other closer the refrigerator. The kitchen appears be well-organized and clean, with focus functionality and aesthetics.
CMI-VLD: The image features a kitchen with black cabinets and sink, Above the sink, there is window with a projection screen, which is displaying a house image. The kitchen also has a refrigerator on the left side and a bottle placed on the countertop. The overall atmosphere of the kitchen is cozy and inviting.
Figure 10: Visualization results comparing our CMI-VLD and other methods with LLaVA-1.5 backbone. Hallucinations are marked in red.
26


Reviews:
[{"review_id": "Nb6qLPJiKl", "reviewer": "Reviewer_bTG8", "summary": "This paper addresses the hallucination problem in LVLMs. The authors argue that hallucinations largely stem from LVLMs’ over-reliance on textual priors and underutilization of visual information during decoding. To address this, they propose a novel decoding strategy, which maximizes Conditional Pointwise Mutual Information (C-PMI) between images and generated text.\n\nThe approach consists of two complementary components:\n\n1. Calibrated Distribution Sampling: Prioritizes text tokens whose generation is strongly influenced by the visual input.\n2. Visual Token Purifier: A lightweight trainable module that prunes irrelevant image tokens based on relevance to the textual context and attention scores.\n\nBy formulating hallucination mitigation as a bi-level optimization problem, the method dynamically refines both text and image tokens during generation. Extensive experiments on multiple LVLMs and benchmarks show that the method significantly reduces hallucinations.", "questions": "1. **Evaluation on Recent LVLMs**: Why did the authors not include models like Qwen-VL or InternVL2.5 in their evaluation? If these models have lower hallucination rates, what is the practical relevance of hallucination mitigation for them?\n2. **Purifier Design and Cost**: The visual token purifier adds complexity. Have the authors considered or compared against training-free token reduce mechanisms (e.g., attention scores, cls-token)? How about the generalizability of the trained purifier across tasks or domains?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\n- **Theoretical Soundness**: The reformulation of hallucination mitigation as a C-PMI maximization problem offers a principled and novel perspective, contrasting with prior heuristic decoding approaches.\n- **Methodological Effectiveness**: The bi-level optimization strategy that alternately calibrates text sampling and visual token refinement is well-motivated and effectively operationalized. The method achieves state-of-the-art hallucination mitigation across diverse models and metrics.\n- The paper is clearly written, and the methodology is carefully derived and supported with detailed experiments and ablations.\n\n### **Weaknesses**\n\n- **Limited Methodological Novelty**: The proposed approach primarily combines two established ideas—text token prioritization (common in hallucination mitigation literature) and visual token reduction (widely used in efficient inference). As such, the paper's main contribution lies in the integration and formalization of these components into a unified bi-level optimization framework, rather than in architectural or algorithmic breakthroughs.\n- **Computational Overhead and Generalizability**: Unlike prior visual token reduction methods that often adopt training-free method (e.g., attention scores, cls-token), this work introduces a learnable visual token purifier, which adds training overhead. The necessity of this additional training is questionable, and the generalizability of the trained purifier across tasks or domains also remains uncertain.\n- **Experimental Models**: The evaluation is conducted on a select set of mid-scale LVLMs (e.g., LLaVA, Shikra, InstructBLIP), while more capable and recent models such as Qwen-VL and InternVL are not considered. This raises the question of whether hallucination is still a prominent issue in stronger or more recent models.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns by providing empirical evidence supporting the benefits of the proposed learnable token purifier and demonstrating the method’s generalization to more capable MLLMs. Overall, this is a solid paper that presents a promising direction for reducing hallucination in MLLMs and has the potential to attract attention within the community.", "rebuttal": "We express our sincere gratitude for dedicating your valuable time to providing insightful comments. We greatly appreciate your positive feedback on our motivation, writing quality, theoretical soundness, algorithmic effectiveness, experiments, and the principled and novel perspective introduced by CMI-VLD. Our detailed responses to all of your concerns are presented below.\n\n> `[Q1]` Methodological Novelty.\n\nThank you for this valuable question! We would like to present the following points to address your concerns.\n\n1. General Directions vs. Concrete Innovations:\n\n    We would like to respectfully point out that, similar to other research topics, text token prioritization and visual token reduction are **two general research directions**, which can be instantiated by different algorithms [1, 2, 3, 4], each with its specific design and implementation. Therefore, we would kindly suggest that the key to assessing methodological novelty might lie in **how the algorithms are concretely formulated, designed, and implemented** in specific tasks or scenarios. \n\n2. C-PMI-Grounded Bi-level Optimization with Task-Specific Techniques: \n\n    We highlight that our methodological innovation involves introducing C-PMI to regulate the decoding process for hallucination mitigation, where we **derive a computable objective formula and propose a novel bi-level optimization framework**. Building upon this formulation, we **correspondingly design and implement two specific token purification strategies**, which are conceptually and algorithmically distinct from prior heuristic approaches. \n\n3. Learnable Visual Purifier with Custom Design: \n\n    In addition, for the visual subproblem, the proposed method goes beyond simple selection mechanisms and proposes to learn a visual purifier, where we accordingly **design its tailored loss function, training procedure, and inference paradigm**. \n\nAs outlined above, our work makes methodological contributions in problem reformulation, a novel optimization framework, and the design of tailored techniques and implementations. \n\nWe will revise the Sec. 3 to better highlight these contributions.\n\n> `[Q2]` Evaluation on Recent LVLMs.\n\nThank you for the thoughtful suggestion! Initially, our evaluation aligns with the well-recognized SID [1] and includes four representative LVLMs. To incorporate your valuable feedback, we consider more advanced models Qwen-VL 2.5 7B and InternVL 2.5 8B. \n\nTable 1. CHAIR results of our CMI-VLD with two SOTA baselines on two advanced LVLMs. \n| Method  | QwenVL-2.5 7B | QwenVL-2.5 7B   | InternVL-2.5 8B |  InternVL-2.5 8B    |\n|-|-|-|-|-|\n| $\\Downarrow$  | $C_{S}\\downarrow$ |  $C_{I}\\downarrow$  |      $C_{S}\\downarrow$ |  $C_{I}\\downarrow$  | $C_{S}\\downarrow$ |  $C_{I}\\downarrow$  |\n| Vanilla | 34.2|  10.1  | 35.8  | 10.3 | \n| VTI    |      32.4|9.7|37.4|  10.5|\n| SID    |  34.6  | 10.6| 38.0            | 10.9 |\n| Ours   |  **29.2**|**7.7**| **32.4**| **9.8** |\n\n\nThe results show that our method continues to effectively mitigate hallucinations on these SOTA models. Notably, the observed $C_S$ exceeding 30% and $C_I$ over 10% indicate that **even advanced LVLMs still suffer from serious hallucination**, consistent with recent findings in [5, 6]. This further underscores that hallucination remains a practically relevant challenge in LVLMs, highlighting the necessity of continued research in this area.\n\nWe will supplement more detailed results in Appendix D of the revision.\n\n> `[Q3]` Necessity of Purifier and its Generalizability.\n\nVery constructive question! \n\n1. Necessity of purifier training: \n\n    To address your concerns, we implement a training-free variant that selects visual tokens based on attention scores, with all other setups unchanged. \n\n    Table 2. CHAIR results of CMI-VLD and a training-free attention-based variant.\n    | Metric | Method        | LLaVA-1.5 | InstructBLIP | Shikra | LLaVA-NEXT |\n    |--------|---------------|-----------|--------------|--------|------------|\n    | $C_S\\downarrow$     | Training-free | 44.4          |    52.6          |   51.6     |    30.4        |\n    |        | Ours          |  **29.9**         | **43.2**              | **30.6**       |  **27.2**          |\n    | $C_I\\downarrow$     | Training-free |  11.3         |   16.0           |   13.8     |   10.2         |\n    |        | Ours          |  **8.9**         |      **12.9** |  **8.9**      | **6.8** |\n\n   Trained with our well-designed loss function derived from C-PMI, the purifier brings remarkable performance gains over the attention-based selection strategy, validating both the effectiveness of our proposed technique and the necessity of training a purifier.\n\n2. Generalizability across tasks: \n\n    We fully understand your concern and would like to address it with the following points:\n\n    i. A plug-in purifier with only a single training: \n\n    Note that the purifier is **trained only once** using the proposed paradigm and then **directly evaluated across multiple benchmarks**, i.e., CHIAR, POPE, GPT-4 assisted Evaluation, MMBench [7], and MME [8], which have included various data domains and multimodal tasks. \n\n    ii. Excellent cross-task and cross-domain generalization: \n\n    In particular, MME and MMBench are two **general-purpose LVLM benchmarks**. For MME, we follow SID [1] and report the **overall perception score covering 10 sub-tasks** such as object existence, counting, OCR, and fine-grained recognition. For MMBench, we report **the average score across 20 multimodal tasks**, such as attributes, logical reasoning, and coarse/fine-grained perception. The results in Table 3 of the original paper demonstrate that the proposed purifier can **generalize well to various tasks**.\n \n    iii. More results on 4 complex tasks:\n\n    To further address your concerns, we supplement the analysis on four recognition tasks from MME's cognitive evaluation.\n    \n    Table 3. Comparison of our CMI-VLD with SOTA baselines on four complex tasks from the recognition evaluation part of MME. \n    | Method | Commonsense reasoning | Numerical calculation | Text translation | Code reasoning | Overall\n    |-|-|-|-|-|-|\n    | VTI    | 117.86             | **47.5**                  | 72.5             | 57.5           | 295.36|\n    | OPERA  | 115.71             | **47.5**                  | 87.5             | 60             |310.71 |\n    | SID    | 113.57            | 45                    | 75               | 65.5           | 299.07|\n    | Ours   | **118.57**              | 45                    | **90**              | **67.5**           |**321.07** |\n\n   Table 3 again confirms the generalization ability of our visual purifier across diverse tasks and domains, revealing its practicality and broad utility.\n\nThanks again for this precious comment. We will supplement these in Appendix D to better clarify these aspects.\n\nFinally, we would like to express our gratitude once again for your valuable feedback! We would be glad to further discuss any remaining concerns you might have.\n\n[1] F Huo et al. Self-introspective decoding: Alleviating hallucinations for large vision-language models, In ICLR 2025.\n\n[2] Z Chen et al. HALC: Object hallucination reduction via adaptive focal-contrast decoding, In ICML 2024.\n\n[3] S Leng et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding, In CVPR 2024.\n\n[4] X Zhuang et al. OPERA: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In CVPR 2024.\n\n[5] Y Wu et al. Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception. In CVPR 2025.\n\n[6] J Duan et al. TruthPrInt: Mitigating LVLM Object Hallucination Via Latent Truthful-Guided Pre-Intervention. \n\n[7] Y Liu et al. MMBench: Is your multi-modal model an all-around player? In ECCV 2024.\n\n[8] Z Liang et al. A survey of multimodel large language models. In CAICE 2024."}, {"review_id": "7kuuQoVo57", "reviewer": "Reviewer_KxNK", "summary": "The paper addresses the issue of hallucinations in LVLMs. The authors propose a Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy to maximize the dependency between generated text and image inputs. Specifically, the paper introduces a bi-level optimization framework, involving a learnable visual token purifier and distribution calibration, aiming to dynamically promote visually grounded and faithful text generation. Extensive experiments on several benchmarks and models demonstrate reduced hallucinations and competitive computational efficiency.", "questions": "Please see my comments in the Strengths and Weakness section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Reducing the hallucination of LVMs is a critical problem for LVMs and has significant implications for practical use cases.\n\n2. The idea of strengthening the mutual dependency between generated texts and input images is interesting and reasonable.\n\n3. Extensive experiments are conducted on diverse LVMs (LLaVA-1.5, Shikra, InstructBLIP, LLaVA-Next) across multiple hallucination benchmarks.\n\n**Weakness**\n\n1. Missing ablations. Most of the ablation focuses on hyperparameters, but the ablation can be expanded to more architectural variations. For instance, comparing the learning-free visual token selection vs. the proposed purifier.\n\n2.  There is limited analysis of why C-PMI is less effective on some datasets (e.g., POPE results, Table 2, are more marginal compared to benchmarks like CHAIR).", "was_revised": true, "final_justification": "My concerns have been well-addressed. I will keep my positive score.", "rebuttal": "Thank you for dedicating your time and effort to reviewing our paper. We are deeply encouraged by your positive comments on the novelty, soundness, and experiments of our work! Below, we provide point-by-point responses to address your concerns.\n\n> `[Q1]` Ablation of the learning-free variant.\n\nThank you for the insightful suggestion!\n\nInitially, we proposed learning a purifier to reduce the computational overhead incurred by manual token selection. To validate this design, we implement **a learning-free variant CMI-VLD$_{lf}$, which selects tokens by manually computing our derived score in Eq. (7) at each step**, with all other settings unchanged.\n\nTable 1. CHAIR metrics and Token-per-second (TPS) of CMI-VLD and its learning-free variant CMI-VLD$_{lf}$ on four LVLMs using greedy decoding. We present the results of the existing SOTA method, SID [1], as a reference.\n| Metric | Method        | LLaVA-1.5 | InstructBLIP | Shikra | LLaVA-NEXT |\n|-|-|-|-|-|-|\n| $C_S\\downarrow$   | SID       |  42.8         |56.2 |51.2|38.0| \n| | CMI-VLD$_{lf}$ |30.0|**40.4**|36.2|  **26.6**|\n|        | CMI-VLD       | **29.9**     |   43.2 | **30.6**  | 27.2         |\n| $C_I\\downarrow$    | SID       |  12.1 |15.8              | 13.6       |    8.9|\n| | CMI-VLD$_{lf}$ |   9.0  | **11.8** |10.2|    **6.5**    |\n|        | CMI-VLD       |  **8.9**         |  12.9            |   **8.9**     |    6.8        |\n| TPS$\\uparrow$    | SID       |  8.76  |   11.70           |     3.85   | 15.71| \n|   | CMI-VLD$_{lf}$ |2.45|2.41|1.05  |     2.35     |\n|   |CMI-VLD|  **8.96** |**11.86**|**4.29**|**16.52**|\n\n\nTable 1 shows that both variants of our method significantly mitigate hallucination compared to the SOTA baseline, validating the effectiveness of our objective function derived from C-PMI.  However, manual token selection **incurs substantial latency due to repeated score computations at each decoding step**, limiting its practicality in real-world applications. \n\nIn contrast, our learned purifier **efficiently selects informative tokens with nearly 4× faster inference than CMI-VLD$_{lf}$ while preserving strong effectiveness**, exhibiting an excellent trade-off between performance and efficiency.\n\nWe will include this experiment as a new ablation in Sec. 4.3 to better highlight the significance of the purifier. Thank you again for your thoughtful recommendation.\n\n> `[Q2]` Analysis of difference in performance gain.\n\nThis is a very insightful question！\n\n1. The key reason lies in the nature of the POPE benchmark, which typically involves binary yes/no questions, with responses typically short and following fixed patterns, such as \"Yes, there is a [object] in the image.\" As a result, the evaluation primarily **hinges on the first one or a few tokens** (i.e., Yes or No) [2]. \n2. Consequently, our CMI-VLD may **not fully demonstrate its potential in such a constrained setup**, as it is designed to dynamically adjust the decoding process throughout the entire generation rather than concentrating solely on the initial tokens.\n3. In contrast, the CHAIR benchmark is **a more general benchmark and evaluates hallucinations based on the entire generated sentence**, allowing our strategy to take full effect during the whole decoding process. This results in stronger performance gains (e.g., a notable average improvement of 7.5 in $C_S$ for Shikra than SOTA baselines), confirming the effectiveness of our method **in more expressive and practical generation scenarios**.\n\nWe will revise Sec. 3.2 to better highlight and explain this distinction. Thank you again for the thoughtful suggestion.\n\nFinally, we would like to express our gratitude once again for your perceptive and valuable feedback! It would be our pleasure to engage in further discussion with you if there are any remaining concerns.\n\n[1] F Huo et al. Self-introspective decoding: Alleviating hallucinations for large vision-language models, In ICLR 2025.\n\n[2] X Zhuang et al. OPERA: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In CVPR 2024."}, {"review_id": "gntSeLBW1k", "reviewer": "Reviewer_BpZA", "summary": "This paper addresses the hallucination problem in LVLMs with an inference-time strategy named CMI-VLD. The core idea is to maximize the mutual information between visual inputs and the generated text. Generally, this paper shares the same insight with many of other works in this domain: LVLM hallucinations mainly stem from the over-reliance on language priors while neglecting visual information as the generated text length grows.\n\nSpecifically, the optimization problem turns into two components: (i) fine-grained controlled maximization of the distribution gap with against without visual inputs; and (ii) a visual token purification mechanism which evicts tokens that are not informative enough depending on the attention scores.\n\nThe proposed method is evaluated across multiple LVLM backbones on several benchmarks.", "questions": "* What principles guide the choices of different hyperparameter settings? How would you adapt the method to new LVLM architectures?\n* Can you provide a more detailed analysis of the computational costs, including training time, memory usage, and inference latency? How does the method scale with sequence length and visual token count?\n* Most evaluations focus on descriptive image captioning tasks. How does the method generalize to more complex tasks such as multimodal reasoning?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* The reformulation of hallucination mitigation as conditional mutual information maximization is well-motivated and intuitively reasonable. \n* Although there are several hyperparameters required to stabilize the alternating optimization procedure, the problem formulation and target is mathematically sound. They also systematically ablate the hyperparameters to show the effect of the main components.\n* The authors conducted comprehensive experiments on multiple models and benchmarks. The performance gains are consistent across different settings.\n\n### Weaknesses\n* The method requires careful tuning of multiple hyperparameters and different layer selections depending on the empirical results. This could limit the generalizability and cause additional cost to find the appropriate settings.\n* As an inference-time method, the main concern would be the trade-off between model performance and inference efficiency. For example, the calculation of distribution gaps and the maintenance of the purification mechanism may need additional computational and memory resources. It is important to discuss this further for users to better understand the quality--cost trade-off of CMI-VLD.\n* Since the decoding decisions are made depending on $p{theta}$, it is possible that the proposed strategy over-penalizes or ignores potentially correct tokens due to the inherent deficiency (imperfections) of $p{theta}$. How would the authors determine the upper bound of the model performance given a fixed $p_{theta}$ and the proposed pipeline of CMI-VLD?", "was_revised": true, "final_justification": "The preliminary empirical approach proposed by the authors would be an insightful experiment for understanding the capabilities and stability of CMI-VLD, and has the potential to provide meaningful insights for future improvements to the method. The complexity analysis is also well-executed and convincing.\n\nBased on these contributions, I am pleased to maintain my current positive rating.", "rebuttal": "We would like to express our gratitude for your valuable suggestions. Your positive feedback on our problem reformulation, algorithmic effectiveness, mathematical soundness, and comprehensive experiments has greatly encouraged us! Detailed point-by-point responses are listed as follows.\n\n> `[Q1]` Principles behind the choices of hyperparameters and adaptation to new LVLM architectures.\n\nThank you for the constructive question! The hyperparameters are chosen based on our empirical analysis and relevant literature, which are further supported through ablation studies. \n\nSpecifically, we summarize our choice strategy behind several critical hyperparameters in our algorithm as follows:\n\n1. **Contrast strength $\\lambda$** balances the difference between the vision-conditioned and vision-free distributions. Large values may favor casual and incorrect tokens, while a small $\\lambda$ can fail to sufficiently penalize hallucination-prone tokens. \n\n    We aim to **preserve correct distributions while penalizing hallucinated predictions**, and thus select a moderate value of $\\lambda=0.5$, which is further validated by ablations (see Fig. 5(b)) and prior studies [1, 2].\n\n2. **Visual token retention ratio $\\gamma$** is a sensitive and critical parameter. A high $\\gamma$ may retain noisy tokens and weaken the C-PMI maximization, while a low $\\gamma$ can discard important visual information. \n\n    Hence, we adopt **an adaptive strategy**: for models with many visual input tokens (e.g., LLaVA-1.5), we set a relatively smaller $\\gamma = 80$%; for models with fewer, already refined tokens (e.g., InstructBLIP), we use a higher $\\gamma = 90$%. Ablation studies on each model validate the rationality, with results on LLaVA-1.5 shown in Fig. 8 as an illustration.\n\n3. **Coefficient of attention loss $\\alpha$** balances the importance between attention loss and C-PMI loss during purifier training. Empirically, we find the attention loss to be ~1000x smaller in magnitude compared to C-PMI loss, so we set $\\alpha = 100$ to **adequately amplify its impact while preserving the dominance of the C-PMI objective**. This choice is validated in Fig. 5(a).\n\n4. **Purification starting layer $i$** is chosen based on existing well-established studies on token selection [1, 3], which have been empirically validated to yield strong task-specific performance while preserving its general capability.\n\nFor the adaptation of our method, we empirically observe that the above hyperparameters can directly transfer to new LVLMs and already achieve significant effectiveness. For further improvement, a practical adaptation may involve **slightly tuning $\\lambda$ according to the characteristics of the new model’s $p_\\theta$**. In addition, we recommend adjusting the retention ratio $\\gamma$ based on the number of input visual tokens, as previously suggested.\n\nWe will include these in Appendix B.1 to support hyperparameter selection and facilitate the broader application of our method.\n\n> `[Q2]` Detailed Analysis of Computational Costs.\n\nThis is a very valuable suggestion! \n\nTo reduce computational overhead, we design the purifier as **a lightweight network with only ~0.1% parameters of the LVLM**. Its effectiveness in mitigating hallucination has been thoroughly validated by extensive experiments in the main paper. Below, we present a detailed computational cost analysis based on LLaVA-Next 8B on a single NVIDIA A6000 GPU.\n\n1. Analysis of Training Costs.\n\n    Table 1. Training costs for the purifier under our configuration. \n    | Memory of Purifier| Training Time | Purifier Memory Usage| LVLM Memory Usage| Overall Memory Usage\n    |-|-|-|-|-|\n    | 32.7 MB | 6.5h | 0.64 GB | 39.20 GB| 43.96 GB|\n\n    Due to its lightweight design, the purifier introduces minimal computational burdens and can be trained in just 6.5 hours on a single A6000 GPU. Importantly, it requires **only a one-time training** and can **generalize well across diverse multimodal tasks**.\n\n2. Analysis of Inference Costs. \n    We further analyze inference efficiency with and without the purifier, scaling with sequence length and visual token count.\n    \n    Table 2. Inference costs scale with input sequence lengths. The number of visual tokens is fixed to 576.\n    |Metric| Method|633 (prefilling)|850|1000|1250| 1500\n    |-|-|-|-|-|-|-|\n    |FLOPs (1e14)| w/o purifier|1.9214|2.5814|3.0391|3.8044|4.5731|\n    ||w/ purifier (Ours)|1.5671|2.4582|3.1226|4.3182|5.6241|\n    | Inference Latency (s) | w/o purifier| 0.37 | 17.99 | 30.26 | 50.43 | 70.72 |\n    | |  w/ purifier (Ours) | 0.35 | 18.53 | 31.17  | 52.20 | 73.22 |\n\n    Table 3. Inference costs under varying numbers of input visual tokens. We use a fixed text query from the CHAIR evaluation, where the number of text tokens is 56.\n    |Metric| Method|49|256|576|1024|\n    |-|-|-|-|-|-|\n    |FLOPs (1e14)|w/o purifier|0.3188| 0.9448|1.9214|3.3067|  \n    | |  w/ purifier (Ours) |0.2888|0.7876|1.5671|2.6718|\n    |Inference Latency (s)|w/o purifier| 0.24 | 0.28 | 0.37 | 0.60  |\n    |       |w/ purifier (Ours)| 0.23 | 0.28  | 0.35 | 0.53 |\n     \n   Thanks to its lightweight design and visual token reduction, our purifier **introduces negligible overhead and generally maintains computational efficiency** comparable to the purifier-free variant. Furthermore, as the number of visual tokens increases, the **benefits of visual token reduction become more pronounced, further reducing the computational complexity**.\n\nWe will include these analyses in a new appendix section to demonstrate that our method achieves an **excellent trade-off between effectiveness and efficiency**.\n\n> `[Q3]` Generalize to more complex tasks such as multimodal reasoning.\n\nThank you for the insightful question! \n\nWe would like to respectfully clarify that we have evaluated on the MME's perception analysis [4] that **covers 10 sub-tasks including counting, OCR, and fine-grained recognition**, and MMBench [5] that **covers 20 complex subtasks, such as multimodal logical/relational reasoning**. The results in Tab. 3 of the original paper reveal that our method generalizes well on various complex tasks.\n\nTo further address your concern, we additionally provide results on four complex sub-tasks from the cognitive part of the MME evaluation:\n\nTable 4. Comparison of our CMI-VLD with SOTA baselines on four complex tasks from the cognitive evaluation part of MME. \n|Method|Commonsense reasoning|Numerical calculation|Text translation|Code reasoning|Overall\n|-|-|-|-|-|-|\n|VTI|117.86| **47.5**|72.5|57.5| 295.36|\n|OPERA|115.71|**47.5**| 87.5|60 |310.71|\n|SID|113.57|45.0|75.0|65.5|299.07|\n|Ours|**118.57**|45.0|**90.0**|**67.5**|**321.07**|\n\nTable 3 again validates the superior generalizability of CMI-VLD across a variety of sub-tasks. We will add more detailed results in Appendix D.\n\n> `[Q4]` Determine the upper bound of model performance given a fixed $p_{\\theta}$ and our CMI-VLD.\n\nVery insightful question! Indeed, CMI-VLD calibrates the LVLM's output token distribution by maximizing C-PMI. However, the final output distribution is still parameterized by the original LVLM, i.e., $p_\\theta$, and hence the ultimate performance is inherently bounded by the expressive capacity of $p_\\theta$.\n\nTo estimate this upper bound, we propose a preliminary empirical approach as follows.\n\nLet strategy $\\mathcal{S}$ denote a set of hyperparameters in our algorithm (e.g., contrast strength $\\lambda$ and retention ratio $\\gamma$). Then, the calibrated distribution at timestep $t$ is denoted as $q_t(\\cdot \\mid y_{<t}, v^*; \\theta, \\mathcal{S})$.\n\nWe define a **“golden sampler”**, which ideally performs the following decoding procedure:\n\n1. At each decoding step $t$, **if the probability assigned by $q_t$ to the ground-truth token is greater than zero** (or above a predefined small threshold), the sampler directly selects the ground-truth token;\n2. Otherwise, it selects the token with the highest probability under $q_t$;\n3. Repeatedly apply this process to generate sufficient sentences, and compute average hallucination scores (e.g., CHAIR) based on ground-truth sentences.\n\nBy adopting a variety of strategies $\\mathcal{S}$ to influence $q_t$ and computing the corresponding hallucination score, we can empirically **approximate the best achievable performance under a fixed $p_\\theta$**.\n\nThis idealized setup, which directly leverages the ground-truth token whenever its model-assigned probability is non-zero, **minimizes hallucination as much as possible** under the given $p_\\theta$. Therefore, the resulting performance can serve as a practical estimate of the upper bound for hallucination mitigation under our framework.\n\nWe will include this procedure in a new appendix section and explore more rigorous theoretical bounds in future work.\n\nOnce again, we deeply appreciate your valuable suggestions for improving our work and would be delighted to further discuss with you about any remaining concerns!\n\n[1] F Huo et al. Self-introspective decoding: Alleviating hallucinations for large vision-language models, In ICLR 2025.\n\n[2] S Leng et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding, In CVPR 2024.\n\n[3] L Chen et al. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV 2024.\n\n[4] Z Liang et al. A survey of multimodel large language models. In CAICE 2024.\n\n[5] Y Liu et al. MMBench: Is your multi-modal model an all-around player? In ECCV 2024."}, {"review_id": "v7rMXRvTkw", "reviewer": "Reviewer_wD9a", "summary": "This paper looks at hallucination mitigation in large vision language models (LVLMs) from an information-theoretic perspective, motivated by the fact that previous strategies are i) mostly empirically-derived and ii) lacking an explicit quantification of the relevance between the visual input and the generated text. Then, it links the intuition of previous work, i.e., LVLMs overly depend on text tokens, to a low mutual dependency between the input images and the generated responses. \nFrom this, it proposes to mitigate hallucinations as a mutual information maximization problem, solved with a bi-level optimization formulation. The proposed method is tested on multiple models and benchmarks.", "questions": "**Q1:** I assume $\\beta$ (L169) should be an $\\alpha$, correct?  \n\n**Q2:** Could you elaborate more on what you mean by *... adopt the more flexible and stronger beam search strategy, which may raise concerns regarding fairness* (L224-225)? Do you refer to reproducibility and fair model comparison?\n\n**Q3:** How did you set the hyperparameters listed in the implementation details?\n\n**Q4:** I wonder if this method can be effective on smaller-scale models as well, which also tend to hallucinate more. Did the authors try their method on some more recent LVLMs (InternVL, QwenVL, LLaVA-OneVision) in their 0.5/1/3B scale version? Could this work?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:** The motivation is clear and conveyed effectively. The methodology is interesting, novel, and overall well-explained. I appreciate that the authors attached the code in the supplementary already, supporting open research and reproducibility. \n\n**Weaknesses:** Some things should be more self-contained, e.g., when talking about MME and MMBench evaluation, it should be clear to which metrics those numbers refer. I see an explanation has been added to the appendix, but something should be clearer from the Table/main paper as well in my opinion.\nMinor things on the figures should be improved as well. I find the caption of Figure 2 is not very informative, and Figure 5/6 could show directly the actual value used for both hyperparameters.", "was_revised": true, "final_justification": "My judgment was already positive. The authors addressed the few concerns I had, and with the proposed modifications, I believe this work meets the required standard and should be accepted.", "rebuttal": "We sincerely thank you for taking the time to thoroughly review our paper and provide precious suggestions. We are very grateful for your recognition of our motivation, novelty, and soundness, as well as your positive feedback on our work! Point-by-point responses to your comments are summarized as follows.\n\n> `[Q1]` Make Tables/Figures more self-contained.\n\nWe sincerely appreciate your thoughtful suggestion. Following your advice, we will make the following revisions to improve the clarity of the paper:\n\n1. Provide details about the evaluation protocol of MME and MMBench in Sec. 4.3.\n\n2. Revise the caption of Fig. 2 to provide a detailed workflow of the proposed two techniques, along with a brief explanation of their underlying mechanisms.\n\n3. Update two subplots of Fig. 5 with the actual numerical values used in our experiments.\n\nThank you again for your careful review and constructive feedback.\n \n> `[Q2]` A Typo in L169.\n\nThank you for your careful review. We apologize for this typo and have corrected it accordingly in the revision.\n\n> `[Q3]` Fairness concern raised by baselines with beam search decoding.\n\nSorry for the missing details! This statement refers to methods such as OPERA [1] and HALC [2] are designed based on the *beam search* decoding strategy, which **retains a broader set of promising candidate paths during decoding**, potentially offering advantages over standard *sampling* or *greedy* decoding strategies used in our method and other baselines.\n\nDespite this, our approach still achieves remarkable improvements, as shown in Table 1-3 of the original paper, confirming the effectiveness of our proposed techniques. \n\nWe will add more details in Sec. 4.1 to better explain this statement.\n\n> `[Q4]` Choice of Hyperparameters.\n\nVery insightful question! The hyperparameters are chosen based on our research experience and guided by existing relevant studies, which are further supported via ablation studies. We take LLaVA-1.5 as a representative backbone and report ablation studies for key hyperparameters in Sec. 4.3 and Appendix D. \n\nBelow, we summarize key principles guiding the selection of several critical hyperparameters that significantly influence the effectiveness of our algorithm.\n\n1. **Contrast strength $\\lambda$** controls the amplification of differences between the vision-conditioned and the vision-free (language prior) distribution. A larger $\\lambda$ may suppress reasonable tokens, while a small value may fail to sufficiently penalize hallucination-related tokens. \n\n    Therefore, we choose a moderate $\\lambda$ value of $0.5$ to preserve reasonable tokens while penalizing hallucinated predictions. Further adjustment around this value (see Fig. 5(b)) confirms its effectiveness. The rationality is also validated by previous studies [3, 4].\n\n2. **Visual token retention ratio $\\gamma$** is another crucial hyperparameter. An overly high retention ratio may retain irrelevant tokens and fail to sufficiently amplify C-PMI, while an overly low ratio could result in the loss of essential visual information. \n\n    Our strategy is **tailored to the nature of different LVLM backbones**: for models with sufficient input visual tokens (e.g., LLaVA-1.5), we set a relatively lower $\\gamma$ of 80% to filter out redundant tokens. In contrast, for models with fewer and already refined visual tokens (e.g., InstructBLIP), we adopt a more conservative $\\gamma$ at 90%. Ablation studies for each model validate the rationality, with LLaVA-1.5 presented in Fig. 8 as an illustration.\n\n3. **Attention coefficient $\\alpha$** balances the influence of the attention loss and the C-PMI loss during purifier training. We observe that the attention loss is ~1000× smaller in magnitude compared to the C-PMI loss. We set $\\alpha = 10^2$ to **ensure its contribution remains significant while preserving the dominance of the main C-PMI objective**,. This choice is empirically validated in Fig. 5(a).\n\n4. **Purification starting layer $i$** is directly adopted from existing well-established token selection research [3, 5], which has been empirically validated to yield strong task-specific performance while preserving LVLM generation capability.\n\nWe will summarize the above principles in Appendix B.1 as a reference for future research to facilitate broader applications of our method in future work.\n\n\n> `[Q5]` Effectiveness on Smaller-models.\n\nThank you for the valuable suggestion. We conduct a preliminary study on smaller versions of two more advanced LVLMs Qwen-VL 2.5 3B and Intern-VL 2.5 2B.\n\n| Metric | Qwen2.5-VL 3B |  Qwen2.5-VL 3B   | InternVL-2.5 2B |   InternVL-2.5 2B   |\n|-|-|-|-|-|\n|  $\\Downarrow$ |$C_S$|$C_I$| $C_S$|$C_I$|\n| Sample        | 43.4          | 9.0   | 38.2            | 10.6 |\n| VTI           |    36.2           |  8.2   | 34.4            | 9.3  |\n| SID           | 44.0            | 9.8 | 34              | 8.5  |\n| Ours          |  **32.6**             |  **7.9**   | **32.4**    | **8.2**  |\n\n\nAs expected, these smaller models initially exhibit higher hallucination rates than their larger-size versions, and our method continues to help calibrate the output distribution and effectively mitigate the hallucination.\n\nWe will include detailed results on more smaller models in Appendix D to reveal the effectiveness of our CMI-VLD on lightweight models.\n\nThank you again for your thoughtful suggestion! If you have any further concerns, feel free to reach out to us. :)\n\n[1] X Zhuang et al. OPERA: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In CVPR 2024.\n\n[2] Z Chen et al. HALC: Object hallucination reduction via adaptive focal-contrast decoding, In ICML 2024.\n\n[3] F Huo et al. Self-introspective decoding: Alleviating hallucinations for large vision-language models, In ICLR 2025.\n\n[4] S Leng et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding, In CVPR 2024.\n\n[5] L Chen et al. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV 2024."}, {"review_id": "KUkNraLREt", "reviewer": "Reviewer_bZfZ", "summary": "This paper introduce conditional mutual information as a theoretical foundation to enhance the mutual dependency between visual input and generated text to mitigate hallucination in MLLMs. The method and evaluation are reasonable and comprehensive. However the core idea is not innovative enough and the comparison lacks fairness.", "questions": "please refer to weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The experiments are rigorous and comprehensive, including several MLLMs and benchmarks.\n\n2. The method and figure is clear and easy to follow.\n\nWeaknesses\n\n1. The propose method CMI is training-based and the baseline methods are training-free, so the comparison is unfair.\n\n2. The core idea of Calibrated Distribution Sampling that contractive decoding has been widely explored in many existing method like VCD[1], RBD[2].\n\n3. The most difference of the method  between prior works is the Gumbel-Softmax, which is used to select critical visual tokens, but it is not a innovative technique. \n\n[1] Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding.\n\n[2] Mitigating Hallucination in Visual-Language Models via Re-balancing Contrastive Decoding.", "was_revised": true, "final_justification": "My concerns are resolved and I have raised my score.", "rebuttal": "We sincerely thank you for your precious efforts in providing constructive suggestions and appreciate your praise on the comprehensive experiments and clear presentation of our work! We have carefully read the comments and provided detailed responses as follows.\n\n> `[Q1]` Clarifications about Comparison Fairness.\n\nWe fully understand your concern and would like to respectfully clarify the following points to address your concerns.\n\n1. Training Only for efficient Visual Token Selection:\n\n   The training is exclusively used to **learn a plug-in module that automatically purifies visual tokens for reducing decoding overhead**. Crucially, the **LVLM backbone remains completely frozen**, and our method essentially operates purely at inference time via distributional calibration, fully aligned with prior decoding-based approaches [1, 2, 3, 4].\n2. Clarifications about baselines: \n\n    Similarly, the baseline method HALC [1] also employs a trained auxiliary network for distributional calibration, and VTI [4] conducts training to learn steering latent features. However, both our method and these baselines are **decoding-based hallucination mitigation approaches and follow the same comparison protocol**, as established in existing well-acknowledged studies [1, 2, 3, 4].\n3. Minimal Cost with Significant Gains:  \n\n    In addition, we respectfully suggest that this might **not be a matter of fairness, but rather a practical trade-off between computation and effectiveness**, especially given that hallucination remains a serious problem in existing LVLMs. \n    Our purifier is extremely lightweight (~32.7 MB, i.e., ~0.1% the size of the LVLM) and the training can be finished in just ~6.5 hours on a single A6000 GPU, yet it significantly mitigates the hallucination while preserving decoding efficiency.\n    From the perspective of cost-effectiveness in algorithm design, we believe this minor investment is **a worthwhile and practical trade-off for substantially reducing hallucination rates and enhancing decoding efficiency**.\n\nWe appreciate your valuable feedback and will add these to Sec. 3 of the revision to clarify this.\n\n> `[Q2]` The difference from previous contrastive decoding approaches.\n\nWe sincerely appreciate your insightful question and provide the following aspects to address your concerns. \n\n1. General Directions vs. Specific Innovations:\n\n    We would like to clarify that, similar to other research directions, distribution calibration is **a general research direction** for hallucination mitigation, which can indeed be implemented by various algorithms [1 ,2, 3], each with its own unique design and implementation.\n    Therefore, we respectfully suggest that the key to assessing the contribution might lie in **how the algorithms are concretely formulated, designed, and implemented** in specific tasks or scenarios.  \n\n2. C-PMI-Grounded Distributional Calibration：\n\n   In contrast to prior works that focus on empirical contrastive decoding designs, our work **contributes a unified and information-theoretically grounded hallucination mitigation framwork**, where we derive an explicit and computable objective based on C-PMI, and accordingly design a principled text token purification strategy, which are both **conceptually and algorithmically dinstinct from prior heuristic approaches** such as VCD and RBD.\n\n3. A Unified Framework that Generalizes Prior Methods:\n\n   We highlight that, as discussed in lines 142–145, prior heuristic methods, despite their diverse forms, can **be viewed as special cases or variants** of our established framework when considering only the influence of text tokens. This not only enhances interpretability but also presents a unified theoretical lens to generalize existing approaches.\n\nWe will include these in Sec. 3 to clarify our difference from previous methods and better highlight our contributions in both **problem formulation, theoretical framework, and algorithmic design**.\n\n> `[Q3]` Difference and technique innovation compared with prior works.\n\nThanks for the question! We respectfully clarify our contributions and differences from previous studies as follows.\n1. A Well-designed Visual Purifier Beyond Gumbel-Softmax:\n\n    We would like to clarify that the **Gumbel-Softmax is merely used as a differentiable technique tool** within the visual purifier. The core innovation in solving the visual subproblem lies in the learnable Visual Token Purifier, for which we design **a specialized loss function, training paradigm, and efficient inference mechanism**.\n2. Contribution in a Theoretically Grounded Framework Beyond Heuristic Decoding:\n\n    Unlike previous studies [1, 2, 3] that focus on heuristic contrastive decoding strategies, a significant difference of our work lies in a **novel, theoretically grounded bi-level hallucination mitigation framework**, which introduces C-PMI as a principled objective to enhance the mutual dependence of visual and textual modalities. \n3. Explicit Objective and Coordinated Optimization:\n\n    To solve the proposed bi-level problem, we **derive an explicitly computable objective from C-PMI**, based on which we accordingly design **two coordinated purification strategies** for text and visual tokens. To the best of our knowledge, this is the first decoding-based hallucination mitigation method to introduce a theoretically grounded and explicitly computable objective, enhancing both interpretability and rigor of the algorithm.\n4. Validation by Strong Empirical Results:\n\n    As shown in the paper, extensive experiments across multiple LVLMs and benchmarks consistently demonstrate significant reductions in hallucination, validating the effectiveness of our theoretically motivated framework.\n    \nWe sincerely thank you for raising this point. We will revise Sec. 3 to better highlight our contributions in the reformulation of hallucination mitigation, the novel design of purification algorithms, and the theoretical guarantees that go beyond existing heuristic approaches.\n \nThank you again for your valuable feedback! We're more than glad to have more discussions with you if you have any further questions.\n \n[1] Z Chen et al. HALC: Object hallucination reduction via adaptive focal-contrast decoding, In ICML 2024.\n\n[2] F Huo et al. Self-introspective decoding: Alleviating hallucinations for large vision-language models, In ICLR 2025.\n\n[3] X Zhuang et al. VASparse: Towards Efficient Visual Hallucination Mitigation for Large Vision-Language Model via Visual-Aware Sparsification. In CVPR 2025.\n\n[4] S Liu et al. Reducing hallucinations in large vision-language models via latent space steering. In ICLR 2025."}]

Author rebuttal:

Meta-review: The paper introduces CMI-VLD, a novel decoding strategy to mitigate hallucinations in Large Vision-Language Models (LVLMs). The core idea is to reframe the problem from an information-theoretic perspective, maximizing the Conditional Pointwise Mutual Information (C-PMI) between visual inputs and generated text. This is operationalized through a bi-level optimization framework that concurrently calibrates text token sampling and prunes irrelevant visual tokens via a lightweight, learnable purifier.

Reviewers found the paper to be well-motivated, clearly written, and technically sound. Initial discussions raised valid and critical concerns regarding methodological novelty (vs. prior work in contrastive decoding and token pruning), the computational overhead of the trainable purifier, and the evaluation's scope on the most recent SOTA models. However, the authors' comprehensive rebuttal effectively addressed these points by providing substantial new experiments and detailed analyses. This exemplary engagement successfully convinced all reviewers, leading to a strong and unanimous consensus for acceptance.

# Summary Of Reasons To Publish:
1) The paper's main strength lies in grounding hallucination mitigation in a theoretically sound C-PMI maximization framework. This provides a unified and principled perspective that moves beyond the more heuristic nature of many prior decoding strategies.
2) The authors' rebuttal significantly strengthened the paper's claims. They demonstrated the method's effectiveness on more advanced LVLMs (Qwen-VL, InternVL), proving its relevance. Crucially, new ablation studies validated that the learnable purifier outperforms training-free alternatives and is highly efficient (~4x faster than a manual-computation variant).
3) Concerns about the purifier's overhead were effectively dispelled. The authors provided a detailed cost analysis showing that the module is extremely lightweight, introduces negligible latency, and can even improve efficiency by reducing the number of visual tokens processed.

# Summary Of Suggested Revisions:
The authors have already agreed to most of the necessary revisions during the discussion phase. To further strengthen the camera-ready version, it is essential to incorporate these promised changes clearly:
1) Integrate the new experimental results on advanced LVLMs (Qwen-VL, InternVL) and the comparative analysis against the training-free/manual-computation variants into the main paper or appendix to fully substantiate the claims of effectiveness and efficiency.
2) Add the detailed computational cost analysis (FLOPs, latency, memory) to an appendix to provide a transparent view of the method's practical trade-offs.
3) Formalize the principles for hyperparameter selection and guidance for adapting the method to new LVLMs in the appendix, as discussed in the rebuttal, to enhance reproducibility and broader adoption.
4) Refine the main text and figure captions to improve clarity, particularly regarding the evaluation protocols for MME/MMBench and the "fairness" discussion concerning beam search, as committed to in the author-reviewer discussion.
Decision: Accept (poster)