Title: Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision Language Models (VLMs)

Preference learning

Hallucination mitigation

Reinforcement Learning from AI Feedback (RLAIF)
Abstract: The success of Direct Preference Optimization (DPO) in mitigating hallucinations in Vision Language Models (VLMs) critically hinges on the true reward gaps within preference pairs. However, current methods, typically relying on ranking or rewriting strategies, often struggle to optimize these reward gaps in a systematic way during data curation. A core difficulty lies in precisely characterizing and strategically manipulating the overall reward gap configuration, that is, the deliberate design of how to shape these reward gaps within each preference pair across the data. To address this, we introduce Topic-level Preference Rewriting (TPR), a novel framework designed for the systematic optimization of reward gap configuration. Through selectively replacing semantic topics within VLM responses with model’s own resampled candidates for targeted rewriting, TPR can provide topic-level control over fine-grained semantic details. This precise control enables advanced data curation strategies, such as progressively adjusting the difficulty of rejected responses, thereby sculpting an effective reward gap configuration that guides the model to overcome challenging hallucinations. Comprehensive experiments demonstrate TPR achieves state-of-the-art performance on multiple hallucination benchmarks, outperforming previous methods by an average of $\sim$20%. Notably, it significantly reduces hallucinations by up to 93% on ObjectHal-Bench, and also exhibits superior data efficiency towards robust and cost-effective VLM alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel DPO-based framework to mitigate hallucinations in VLMs, referred to as Topic-level Preference Rewriting (TPR). The methodology aligns well with the stated motivations, and the experimental results demonstrate the effectiveness of the proposed approach. Overall, I find this to be a solid piece of work that meets the acceptance threshold.\n\nHowever, I have some questions and suggestions that I hope the authors will take into consideration. I am open to raising my score if these concerns are addressed satisfactorily.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper is well-organized and easy to follow, with good writing quality.\n2. The experiments demonstrate the effectiveness of the proposed method.\n3. The concept of the \"reward gap\" is a valuable contribution, as it is often overlooked by DPO-based methods. The derivative curriculum learning approach makes a lot of sense and is well-justified.\n4. The ablation studies are thorough and provide sufficient insights.\n\n**Weaknesses:**\n1. The proposed method involves multiple steps. Introducing them together in the form of a pseudo code or flowchart would greatly enhance readability and help readers better understand the methodology.\n2. Some relevant references and baselines are missing, such as HA-DPO [1], OPA-DPO [2], and mDPO [3]. These works are closely related and should be properly cited and compared.\n3. The experiments are conducted using only one base model (LLaVA-Instruct-1.5-7B). Extending the evaluation to larger models (e.g., 13B) would further strengthen the claims of the proposed method. However, I understand that time constraints during the rebuttal period may limit this. I hope the authors can include more results if the paper is accepted.\n\n**Some minor issues:**\n1. The paper does not include any equations. At the very least, the training loss for DPO should be introduced, as not all readers may be familiar with it. Additionally, in lines 128–130, it should be noted that DPO does not only align $\\pi_\\theta$ with the BT reward function; the KL divergence constraint is crucial, and the practical training loss of DPO cannot be derived without this term.\n2. Please pay attention to the capitalization of article titles in the references. Only the first letter of the first word in a title and special abbreviations should be capitalized.\n\n[1] Zhao et al. Beyond hallucinations: Enhancing LVLMs through hallucination-aware direct preference optimization.\n\n[2] Yang et al. Mitigating hallucinations in large vision-language models via DPO: On-policy data hold the key\n\n[3] Wang et al. mDPO: Conditional preference optimization for multi-modal large language models.\n\nQUESTIONS:\n1. In Figure 1(b), what metric is utilized to quantify \"hallucination\"?\n2. Suppose a response $y$ consists of two topics, $y_1$ and $y_2$, generated in sequence. The conditional probabilities $p(y_2|x, I, y_1)$ and $p(y_1|x, I, y_2)$ are generally not equivalent. How do the authors control the topic sequence in the proposed method to ensure consistency and coherence?\n3. Do \"Intra-Topic Self-Resampling\" and \"In-Context Rewriting\" utilize prompts that are different (denoted as $x_1$ and $x_2$, respectively) from the prompt $x$ used during response generation? If so, $p(y|x, I)$ would likely differ from $p(y|x_1, I)$ and $p(y|x_2, I)$. Have the authors considered the impact of such differences on the proposed method?\n\nLIMITATIONS:\nThe limitations and broader impacts are discussed within the paper. \n(I think the complexity of the proposed algorithm is also a limitation and should be included.)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of mitigating hallucinations in vision-language models (VLMs) through reward-based methods, such as Direct Preference Optimization (DPO). To address the difficulty in optimizing the reward gap during dataa curation, the authors propose **T**opic-level **P**reference **R**ewriting (**TPR**), which selectively replaces topic-level content using model's internally resampled candidates. Addtionally, a curriculum learning strategy is employed to further improve TPR's effectiveness by gradually increasing the learning difficulty of by contructing hallucinated responses with progressively higher scores. Extensive experiments demonstrate that TPR achives the state-of-the-art performance across five hallucination benchmarks and two general benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. **New Prospective**. \nThe proposed method TPR enhances the hallucination reduction performance from the prospectives of topic-level and progessively increasing the difficulty of hallucinated responses, which is different from previously published methods.\n\n2. **Good Performance**.\nTPR achives the state-of-the-art performance of hallucination reduction across popular hallucination benchmarks and general benchmarks. Furthermore, the results of ablation studies highlight the effectiveness of proposed method.\n\nWeaknesses:\n\n1. **Limited Technical Novelty**. \nThe technical contributions of the paper appear limited. The optimization method (TPR), rewriting strategies, and curriculum learning are all based on existing techniques. Additionally, the authors do not provide sufficient convincing evidence demonstrating why TPR outperforms other fine-grained approaches. It would strengthen the paper to include detailed comparisons or ablation studies to show whether the observed gains stem from the topic-level rewriting itself or simply from using a stronger rewriter model.\n\n2. **Lack of Computational Cost or Time Cost Analysis**. The paper does not include any comparison of computational costs between the proposed TPR method and the other baseline approaches.\n\n3. **Vague Definition of \"Topic**. Another major concern is the vague definition of the term \"topic\" as proposed in this paper. To my knowledge, many fine-grained approaches have already been introduced to mitigate hallucination issues in VLMs, focusing on elements such as attributes, relations, and objects. However, the paper does not clearly articulate how \"topic\" differs from these existing fine-grained strategies.\n\nQUESTIONS:\n1. **Clarification on Candidate Sampling Strategy**.\nCould the authors clarify how TPR samples multiple candidate responses prior to topic-level decomposition? Specifically, what sampling strategy is used, and what temperature setting is employed during this process?\n\n2. **Missing Comparisons on Data Efficiency**. \nThe paper appears to omit comparisons in data efficiency between TPR and certain baseline methods listed in Table 1, such as FGAIF and HSA-DPO. Could the authors explain the rationale behind this omission?\n\n3. **Motivation for Topic-Level Rewriting**.\nIf I understand correctly, TPR is fundamentally a topic-level rewriting approach, which can be interpreted as a fine-grained method for optimizing hallucination reduction. Given this design choice, could the authors clarify why the \"topic\" level was selected over other potential levels of granularity (e.g., sentence-level or entity-level)? While topic-level decomposition is indeed common in preference learning, the definition of \"topic\" in this paper remains vague. For instance, in Figure 2, the elements labeled as \"topics\", such as weather, tower time, and tower style, do not clearly align with the conventional understanding of a \"topic.\" It would be helpful if the authors could provide a more precise definition or criteria for what constitutes a \"topic\" in their framework.\n\n4. **Difficulty Score used in Curriculum Learning**.\nWhat 's the difficulty score used in TPR-CL in practice? Is it the reward gap?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Topic-level Preference Rewriting (TPR), a novel framework designed to systematically optimize the reward gap configuration within preference data to mitigate hallucinations in Vision-Language Models (VLMs). The core idea is to achieve fine-grained control over semantic details in VLM responses by selectively replacing semantic topics with the model's own resampled candidates. This precise control enables advanced data curation strategies, such as progressively adjusting the difficulty of rejected responses, which helps in sculpting an effective reward gap configuration to guide the model against challenging hallucinations. Comprehensive experiments demonstrate that TPR achieves state-of-the-art performance on multiple hallucination benchmarks, outperforming previous methods and exhibiting superior data efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n---\n\n* **Novel and Systematic Reward Gap Optimization**: The paper introduces a novel paradigm, Topic-level Preference Ranking (TPR), which systematically optimizes true reward gaps within preference pairs for Direct Preference Optimization (DPO). This approach offers fine-grained, topic-level control over the semantic details in Vision-Language Model (VLM) responses, representing a significant advancement over existing ranking or rewriting methods that frequently yield suboptimal reward gap configurations. Such systematic control is critical for achieving robust VLM alignment, particularly in mitigating hallucinations.\n\n* **Significant Performance Improvement**: TPR demonstrates state-of-the-art performance across multiple hallucination benchmarks. Notably, the TPR-CL variant achieves an approximate 93% reduction in hallucinations on ObjectHal-Bench and around 41% on MMHal-Bench, alongside substantial improvements on AMBER and RefoMB. These empirical results underscore the efficacy of the proposed design and methodology.\n\n* **Effective Curriculum Learning Strategy**: The authors leverage TPR's fine-grained control to implement TPR-CL, a simple yet highly effective curriculum learning strategy. This strategy progressively adjusts the difficulty of hallucinations present in rejected responses ($y_l$), akin to hard negative mining. This mechanism compels the model to discern subtle inconsistencies and fine-grained details, thereby enhancing its robustness against challenging hallucinations and consistently yielding superior performance.\n\n* **Superior Data Efficiency**: TPR exhibits high data efficiency, a direct consequence of its capacity to curate high-quality preference pairs. While human-annotated data, such as RLHF-V, demonstrates initial efficiency, its inherent cost limits scalability. TPR's automated process enables a rapid reduction in hallucination levels with increasing data scale, ultimately surpassing human-annotated data in both performance and cost-effectiveness. Furthermore, other automated methods, such as RLAIF-V, often incur significantly higher computational overhead.\n\n---\n\n## Weakness\n\n---\n\nA primary concern pertains to the selection of LLaVA-NeXT-34B as the labeler model ($\\pi_{label}$) for intra-topic ranking, particularly given that the policy model ($\\pi_{\\theta}$) and reference model ($\\pi_{ref}$) undergoing fine-tuning are LLaVA-1.5-7B. The utilization of a substantially larger and presumably more capable model (34B parameters versus 7B parameters) to ascertain factual accuracy and non-hallucinatory content for generating preference pairs raises a significant question regarding the degree to which this constitutes a form of **knowledge distillation** from a larger model to a smaller one.\n\nThe paper posits that \"evaluating fine-grained units tends to yield more reliable assessments, allowing even moderately capable models (e.g., $\\pi_{ref}$ itself) to serve effectively as $\\pi_{label}$ in TPR.\" However, the presented research lacks a direct **ablation study** empirically demonstrating how the claims would hold if a smaller, equivalently capable model (e.g., $\\pi_{ref}$ itself, i.e., LLaVA-1.5-7B) were consistently employed as $\\pi_{label}$ instead of LLaVA-NeXT-34B. This absence of a direct comparative analysis impedes the ability to fully disentangle the inherent benefits of the TPR framework from the potential advantages conferred by leveraging a significantly larger and presumably less hallucinatory model for the labeling process. Consequently, the extent to which TPR's observed performance gains are attributable to its architectural design versus the superior capabilities of the $\\pi_{label}$ remains underexplored.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of hallucination in vision-language models (VLMs) trained via preference learning. It argues that the effectiveness of Direct Preference Optimization (DPO) depends on the reward gaps between preferred and rejected responses in curated preference data, but current ranking- or rewriting-based curation strategies do not allow systematic control of these gaps.\nThe authors propose Topic-level Preference Rewriting (TPR): a data curation framework that decomposes responses into semantic topics, generates topic-level alternatives via self-resampling, and then constructs preference pairs by selective replacement of topics to control semantic differences deliberately. They further introduce a curriculum learning variant (TPR-CL) that gradually increases negative example difficulty.\nExtensive experiments show that TPR (especially TPR-CL) achieves state-of-the-art hallucination mitigation on multiple benchmarks (reducing hallucinations by up to 93% on ObjectHal-Bench), while maintaining or improving general VLM capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Topic-level decomposition and self-resampling for systematic reward gap control is an original and well-motivated idea.\n2. Outperforms ranking-based and rewriting-based baselines across multiple benchmarks. Curriculum learning variant (TPR-CL) shows particularly impressive gains.\n3. Includes detailed ablations, comparisons with human-annotated data, and discussion of data efficiency.\n\nWeakness:\n1. Experiments target typical hallucination benchmarks, with complex multimodal reasoning tasks left for future exploration. It remains uncertain whether topic-level decomposition will work on more complex tasks.\n2. The notion of reward gap optimization is compelling but under-defined mathematically. There's no theoretical grounding for why these gaps generalize better.\n3. While TPR improves data efficiency per sample, its complex multi-step generation and scoring pipeline may limit practical scalability for truly large-scale training regimes or for models serving many languages and domains.\n\nQUESTIONS:\nCan you quantify the total compute cost (e.g., GPU hours or forward passes per preference pair) and discuss feasibility when scaling to millions of pairs or much larger models?\nHow well does TPR handle more complex multimodal reasoning hallucinations, e.g., involving multi-step reasoning, ambiguous contexts, or commonsense knowledge?\n\nLIMITATIONS:\nThe authors discuss technical limitations well (e.g., focus on simpler hallucinations, compute cost). However, they should explicitly address potential negative societal impacts, such as misuse of generated “wrong” responses for disinformation or the environmental cost of large-scale data curation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}