Title: Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Vision-Language Models

Hallucination Mitigation

Decoding Strategy
Abstract: Large Vision-Language Models (LVLMs) are susceptible to hallucinations, where generated responses seem semantically plausible yet exhibit little or no relevance to the input image. Previous studies reveal that this issue primarily stems from LVLMs' over-reliance on language priors while disregarding the visual information during decoding. To alleviate this issue, we introduce a novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy, which adaptively strengthens the mutual dependency between generated texts and input images to mitigate hallucinations. Unlike existing methods solely focusing on text token sampling, we propose to jointly model the contributions of visual and textual tokens to C-PMI, formulating hallucination mitigation as a bi-level optimization problem aimed at maximizing mutual information. To solve it, we design a token purification mechanism that dynamically regulates the decoding process by sampling text tokens remaining maximally relevant to the given image, while simultaneously refining image tokens most pertinent to the generated response. Extensive experiments across various benchmarks reveal that the proposed method significantly reduces hallucinations in LVLMs while preserving decoding efficiency.


Peer reviews:
comments: SUMMARY:
This paper introduce conditional mutual information as a theoretical foundation to enhance the mutual dependency between visual input and generated text to mitigate hallucination in MLLMs. The method and evaluation are reasonable and comprehensive. However the core idea is not innovative enough and the comparison lacks fairness.

STRENGTHS AND WEAKNESSES:
Strengths

1. The experiments are rigorous and comprehensive, including several MLLMs and benchmarks.

2. The method and figure is clear and easy to follow.

Weaknesses

1. The propose method CMI is training-based and the baseline methods are training-free, so the comparison is unfair.

2. The core idea of Calibrated Distribution Sampling that contractive decoding has been widely explored in many existing method like VCD[1], RBD[2].

3. The most difference of the method  between prior works is the Gumbel-Softmax, which is used to select critical visual tokens, but it is not a innovative technique. 

[1] Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding.

[2] Mitigating Hallucination in Visual-Language Models via Re-balancing Contrastive Decoding.

QUESTIONS:
please refer to weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at hallucination mitigation in large vision language models (LVLMs) from an information-theoretic perspective, motivated by the fact that previous strategies are i) mostly empirically-derived and ii) lacking an explicit quantification of the relevance between the visual input and the generated text. Then, it links the intuition of previous work, i.e., LVLMs overly depend on text tokens, to a low mutual dependency between the input images and the generated responses. \nFrom this, it proposes to mitigate hallucinations as a mutual information maximization problem, solved with a bi-level optimization formulation. The proposed method is tested on multiple models and benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** The motivation is clear and conveyed effectively. The methodology is interesting, novel, and overall well-explained. I appreciate that the authors attached the code in the supplementary already, supporting open research and reproducibility. \n\n**Weaknesses:** Some things should be more self-contained, e.g., when talking about MME and MMBench evaluation, it should be clear to which metrics those numbers refer. I see an explanation has been added to the appendix, but something should be clearer from the Table/main paper as well in my opinion.\nMinor things on the figures should be improved as well. I find the caption of Figure 2 is not very informative, and Figure 5/6 could show directly the actual value used for both hyperparameters.\n\nQUESTIONS:\n**Q1:** I assume $\\beta$ (L169) should be an $\\alpha$, correct?  \n\n**Q2:** Could you elaborate more on what you mean by *... adopt the more flexible and stronger beam search strategy, which may raise concerns regarding fairness* (L224-225)? Do you refer to reproducibility and fair model comparison?\n\n**Q3:** How did you set the hyperparameters listed in the implementation details?\n\n**Q4:** I wonder if this method can be effective on smaller-scale models as well, which also tend to hallucinate more. Did the authors try their method on some more recent LVLMs (InternVL, QwenVL, LLaVA-OneVision) in their 0.5/1/3B scale version? Could this work?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the hallucination problem in LVLMs with an inference-time strategy named CMI-VLD. The core idea is to maximize the mutual information between visual inputs and the generated text. Generally, this paper shares the same insight with many of other works in this domain: LVLM hallucinations mainly stem from the over-reliance on language priors while neglecting visual information as the generated text length grows.\n\nSpecifically, the optimization problem turns into two components: (i) fine-grained controlled maximization of the distribution gap with against without visual inputs; and (ii) a visual token purification mechanism which evicts tokens that are not informative enough depending on the attention scores.\n\nThe proposed method is evaluated across multiple LVLM backbones on several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* The reformulation of hallucination mitigation as conditional mutual information maximization is well-motivated and intuitively reasonable. \n* Although there are several hyperparameters required to stabilize the alternating optimization procedure, the problem formulation and target is mathematically sound. They also systematically ablate the hyperparameters to show the effect of the main components.\n* The authors conducted comprehensive experiments on multiple models and benchmarks. The performance gains are consistent across different settings.\n\n### Weaknesses\n* The method requires careful tuning of multiple hyperparameters and different layer selections depending on the empirical results. This could limit the generalizability and cause additional cost to find the appropriate settings.\n* As an inference-time method, the main concern would be the trade-off between model performance and inference efficiency. For example, the calculation of distribution gaps and the maintenance of the purification mechanism may need additional computational and memory resources. It is important to discuss this further for users to better understand the quality--cost trade-off of CMI-VLD.\n* Since the decoding decisions are made depending on $p{theta}$, it is possible that the proposed strategy over-penalizes or ignores potentially correct tokens due to the inherent deficiency (imperfections) of $p{theta}$. How would the authors determine the upper bound of the model performance given a fixed $p_{theta}$ and the proposed pipeline of CMI-VLD?\n\nQUESTIONS:\n* What principles guide the choices of different hyperparameter settings? How would you adapt the method to new LVLM architectures?\n* Can you provide a more detailed analysis of the computational costs, including training time, memory usage, and inference latency? How does the method scale with sequence length and visual token count?\n* Most evaluations focus on descriptive image captioning tasks. How does the method generalize to more complex tasks such as multimodal reasoning?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper addresses the issue of hallucinations in LVLMs. The authors propose a Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy to maximize the dependency between generated text and image inputs. Specifically, the paper introduces a bi-level optimization framework, involving a learnable visual token purifier and distribution calibration, aiming to dynamically promote visually grounded and faithful text generation. Extensive experiments on several benchmarks and models demonstrate reduced hallucinations and competitive computational efficiency.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. Reducing the hallucination of LVMs is a critical problem for LVMs and has significant implications for practical use cases.

2. The idea of strengthening the mutual dependency between generated texts and input images is interesting and reasonable.

3. Extensive experiments are conducted on diverse LVMs (LLaVA-1.5, Shikra, InstructBLIP, LLaVA-Next) across multiple hallucination benchmarks.

**Weakness**

1. Missing ablations. Most of the ablation focuses on hyperparameters, but the ablation can be expanded to more architectural variations. For instance, comparing the learning-free visual token selection vs. the proposed purifier.

2.  There is limited analysis of why C-PMI is less effective on some datasets (e.g., POPE results, Table 2, are more marginal compared to benchmarks like CHAIR).

QUESTIONS:
Please see my comments in the Strengths and Weakness section.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the hallucination problem in LVLMs. The authors argue that hallucinations largely stem from LVLMs’ over-reliance on textual priors and underutilization of visual information during decoding. To address this, they propose a novel decoding strategy, which maximizes Conditional Pointwise Mutual Information (C-PMI) between images and generated text.\n\nThe approach consists of two complementary components:\n\n1. Calibrated Distribution Sampling: Prioritizes text tokens whose generation is strongly influenced by the visual input.\n2. Visual Token Purifier: A lightweight trainable module that prunes irrelevant image tokens based on relevance to the textual context and attention scores.\n\nBy formulating hallucination mitigation as a bi-level optimization problem, the method dynamically refines both text and image tokens during generation. Extensive experiments on multiple LVLMs and benchmarks show that the method significantly reduces hallucinations.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- **Theoretical Soundness**: The reformulation of hallucination mitigation as a C-PMI maximization problem offers a principled and novel perspective, contrasting with prior heuristic decoding approaches.\n- **Methodological Effectiveness**: The bi-level optimization strategy that alternately calibrates text sampling and visual token refinement is well-motivated and effectively operationalized. The method achieves state-of-the-art hallucination mitigation across diverse models and metrics.\n- The paper is clearly written, and the methodology is carefully derived and supported with detailed experiments and ablations.\n\n### **Weaknesses**\n\n- **Limited Methodological Novelty**: The proposed approach primarily combines two established ideas—text token prioritization (common in hallucination mitigation literature) and visual token reduction (widely used in efficient inference). As such, the paper's main contribution lies in the integration and formalization of these components into a unified bi-level optimization framework, rather than in architectural or algorithmic breakthroughs.\n- **Computational Overhead and Generalizability**: Unlike prior visual token reduction methods that often adopt training-free method (e.g., attention scores, cls-token), this work introduces a learnable visual token purifier, which adds training overhead. The necessity of this additional training is questionable, and the generalizability of the trained purifier across tasks or domains also remains uncertain.\n- **Experimental Models**: The evaluation is conducted on a select set of mid-scale LVLMs (e.g., LLaVA, Shikra, InstructBLIP), while more capable and recent models such as Qwen-VL and InternVL are not considered. This raises the question of whether hallucination is still a prominent issue in stronger or more recent models.\n\nQUESTIONS:\n1. **Evaluation on Recent LVLMs**: Why did the authors not include models like Qwen-VL or InternVL2.5 in their evaluation? If these models have lower hallucination rates, what is the practical relevance of hallucination mitigation for them?\n2. **Purifier Design and Cost**: The visual token purifier adds complexity. Have the authors considered or compared against training-free token reduce mechanisms (e.g., attention scores, cls-token)? How about the generalizability of the trained purifier across tasks or domains?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-12"}