Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization

Authors: beitao  chen
xinyu  lyu
lianli  gao
heng tao shen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:47:27

Abstract: Although Large Visual Language Models (LVLMs) have demonstrated exceptional abilities in understanding multimodal data, they invariably suffer from hallucinations, leading to a disconnection between the generated text and the corresponding images. Almost all  current visual contrastive decoding methods attempt to mitigate these hallucinations by introducing visual uncertainty information that appropriately widens the contrastive logits gap between hallucinatory and targeted ones.
    However, due to uncontrollable nature of the global visual uncertainty, they struggle to precisely induce the hallucinatory tokens, which severely limits their effectiveness in mitigating hallucinations and may even lead to the generation of undesired hallucinations.
    To tackle this issue, we conducted the theoretical analysis to promote the effectiveness of contrast decoding. Building on this insight, we introduce a novel optimization strategy named Hallucination-Induced Optimization (HIO). This strategy seeks to amplify the contrast between hallucinatory and targeted tokens relying on a fine-tuned theoretical preference model (i.e., Contrary Bradley-Terry Model), thereby facilitating efficient contrast decoding to alleviate hallucinations in LVLMs.
    Extensive experimental research demonstrates that our HIO strategy can effectively reduce hallucinations in LVLMs, outperforming state-of-the-art methods across various benchmarks.

Main paper: 
 introduction: 
 The recent success of Large Vision-Language Models (LVLMs) marks a major milestone in artificial intelligence research [OpenAI, 2023 , Alayrac et al., 2022 , Li et al., 2023a , Liu et al., 2023c , Zhu et al., 2023 , Bai et al., 2023 , Dai et al., 2023 , Wang et al., 2023b , Driess et al., 2023] . By seamlessly integrating visual cues with Large Language Models (LLMs), LVLMs have demonstrated unparalleled expertise in multimodal comprehension, logical reasoning, and interactive engagement. This The performance of the state-of-the-art methods compared with Ours. integration has ushered in a new era in AI, breaking through traditional limitations and enabling a more holistic understanding of complex information OpenAI [2023], Yang et al. [2023] , Lu et al. [2023] , Zhang et al. [2022] , Sun et al. [2024] . Despite these advancements, certain challenges remain, particularly the issue of hallucination Li et al. [2023b] , Gunjal et al. [2023] , Liu et al. [2023b] , Lovenia et al. [2023] . Hallucination occurs when the language model generates content that deviates from the image's actual content, including imagined objects, fabricated scenes, incorrect spatial relationships, and misidentified categories.Substantial research efforts have been directed towards mitigating hallucinations in Large Vision-Language Models (LVLMs). These efforts include post-hoc correction methods that refine LVLM outputs after the fact Zhou et al. [2023] and self-correcting frameworks specifically designed to reduce object hallucinations Yin et al. [2023] . Additionally, numerous decoding strategies have been developed to minimize hallucinations through the enhanced use of textual and visual priors Leng et al. [2023] , Zhang et al. [2024] , Favero et al. [2024] , Zhu et al. [2024] , Wang et al. [2024] , Chen et al. [2024] . These methods aim to alleviate hallucinatory tendencies by integrating visual uncertainty, thereby increasing the contrastive disparity between hallucinatory and target logits. For example, Leng et al. [2023] augment the hallucinatory effect by introducing Gaussian noise into the images. Similar approaches by Zhang et al. [2024] and Favero et al. [2024] introduce substantial image noise, effectively reducing the original image to pure noise or unrecognizable content. Zhu et al. [2024] use instructional bias to enable the model to amplify its own hallucinations, while Wang et al. [2024] focus on deliberately amplifying the inherent image bias in LVLMs.However, the inherent uncontrollable nature of global visual uncertainty challenges the precise induction of hallucinatory tokens. This limitation significantly undermines the effectiveness of these methods in reducing hallucinations and may inadvertently lead to undesired hallucinatory outputs.As shown in the left portion of the Fig. 1 Spoon, Table , and Fork are identified as hallucinated words, while People being the accurate term. For Greedy Decoding method shown in Fig. 1 (a), Table is selected as the final output based on the logits distribution. Moreover, although Visual Contrastive Decoding introduces perturbations to images to enhance hallucinations in Fig. 1 (b), it fails to widen the logits gaps between hallucinatory (Spoon, Table , and Fork) and targeted tokens (People), yielding a new hallucination as Fork.To tackle this issue, we conducted the theoretical analysis to explore mechanisms for more effective contrast decoding (refer to Section 5 for detailed information on the process). Theoretically, a clear distinction between hallucinatory and target tokens can significantly enhance the effectiveness of contrast decoding methods in mitigating hallucinations. Based on this crucial insight, we introduce a novel optimization strategy called Hallucination-Induced Optimization (HIO). This strategy enhances the distinction between hallucinatory and targeted tokens by utilizing a refined theoretical preference model(as shown in the Fig. 1 on the left, section (c)), accurately outputting the correct result, People.Consequently, this improves the efficiency of contrast decoding, thereby mitigating hallucinations in Large Vision-Language Models (LVLMs). Furthermore, our proposed method significantly reduces hallucinations in LVLMs compared to existing contrast decoding methods(as shown in the Fig. 1 on the right). To sum up, our main contributions are as follows:1. We conducted a comprehensive theoretical analysis to explore mechanisms that enhance the effectiveness of the contrast decoding strategy.2. We introduce Hallucination-Induced Optimization (HIO), an innovative strategy that utilizes a finely-tuned theoretical preference model to intensify the contrast between hallucinatory and target tokens. This enhancement strengthens the effectiveness of contrast decoding and effectively reduces hallucinations in Large Visual Language Models (LVLMs).3. Extensive experimental research demonstrates that our Hallucination-Induced Optimization (HIO) strategy effectively reduces hallucinations in Large Visual Language Models (LVLMs), surpassing state-of-the-art methods across various benchmarks. 
related work: 
 Hallucination in LVLMs. Before the advent of Large Language Models (LLMs), "hallucination" in natural language processing (NLP) primarily referred to generating nonsensical or source-deviating content Lee et al. [2018] , Zhou et al. [2020] , Lin et al. [2021] , Ji et al. [2023] , Zhang et al. [2023] , Shi et al. [2023] . Recent studies have tackled the complexities of object hallucination in Large Vision-Language Models (LVLMs), focusing on evaluation and detection methods Wang et al. [2023a] , Liu et al. [2023a] , Li et al. [2023b] , Lovenia et al. [2023] . The CHAIR metric Rohrbach et al. [2018] evaluates the exact match between generated and ground-truth image captions, while POPE Li et al. [2023b] assesses the model's awareness of object existence through binary classification.Decoding Method. The decoding method determines the generation of text tokens at each time step within language models. Traditional decoding strategies such as beam search Boulanger-Lewandowski et al. [2013] , top-k decoding Fan et al. [2018] , and sampling methods Holtzman et al.[2019], despite their widespread use, are prone to producing hallucinatory content. Recent research Li et al. [2022] , Chuang et al. [2023] , Leng et al. [2023] , Huang et al. [2023] has made attempts to address this issue by proposing better decoding methods. For instance, Leng et al. [2023] uses contrastive decoding in LVLMs; However, global visual uncertainty poses challenges to the precise induction of hallucinatory tokens, limiting the effectiveness of mitigation strategies and risking unwanted hallucinations. To address this, we developed Hallucination-Induced Optimization (HIO), a novel strategy that enhances the contrast between hallucinatory and targeted tokens. Fig. 1 presents the comparison results, where our approach demonstrates superior performance than other decoding methods.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We have listed our contributions in dicussion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 We first review the Contrast Decoding pipeline in Leng et al. [2023] (and later Zhang et al. [2024] , Favero et al. [2024] ). Then take a close look at the Bradley-Terry model Bradley and Terry [1952] and its application such as Direct Preference Optimization Rafailov et al. [2024] . Inspired by these studies, we propose our Hallucination-Induced Optimization.Visual Contrastive Decoding. We consider an LVLM parameterized by θ. The model takes a textual query input x and a visual input v, where v provides contextual visual information to assist the model in generating a relevant response y to the textual query. The response y is sampled auto-regressively from the probability distribution conditioned on the query x and the visual context v. Mathematically, this can be formulated as:y t ∼ p θ (y t | v, x, y <t ) ∝ exp logit θ (y t | v, x, y <t ) (1)where y t denotes the token at time step t, and y <t represents the sequence of generated tokens up to the time step t -1. Specifically, given a textual query x and a visual input v, the model generates two distinct output distributions: one conditioned on the original v and the other on the distorted visual input v ′ , which is derived by applying pre-defined distortions (i.e., Gaussian noise mask) to the original v. Then, a new contrastive probability distribution is computed by exploiting the differences between the two initially obtained distributions. The new contrastive distribution p vcd is formulated as:p vcd (y | v, v ′ , x) = softmax[(1 + α) logit θ (y | v, x) -α logit θ (y | v ′ , x)](2) where larger value of α indicate a stronger amplification of differences between the two distributions (α = 0 reduces to regular decoding). Direct Preference Optimization. Reinforcement learning (RL) effectively fine-tunes Large Language Models (LLMs) to align with human behavior. Given an input x and a response y, a language model policy π θ generates a conditional distribution π θ (y | x). RL aims to maximize the average reward of outputs, with the reward function r(x, y). To prevent overoptimization Gao et al. [2023] , the objective loss includes a KL-divergence term, controlling the divergence between the language model policy and its reference policy π ref (y | x), typically derived from supervised fine-tuning. Thus, the overall objective is formulated as:max π θ E x∼D,y∼π θ (y|x) r(x, y) -α log π θ (y | x) π ref (y | x) (3)where D is a dataset of prompts and α is a coefficient to control KL-divergence term. However, optimizing the above loss term with common strategies like proximal policy optimization (PPO) Schulman et al. [2017] is complex to tune. Recently, direct preference optimization (DPO) Rafailov et al. [2024] simplifies the above process by leveraging preference data for optimization. Here, the preference data is defined as D = {x (i) , yw , y(i) l } N i=1, where y(i)w and y(i)l represent preferred and dispreferred responses given an input prompt x. These are then presented to human labelers who express preferences for one answer, denoted as y w ≻ y l | x where y w and y l denote the preferred and dispreferred respectively. Following a Bradley-Terry model [Bradley and Terry, 1952] , the probability of obtaining each preference pair is:EQUATIONwhere the superscript i is omitted for simplicity. In DPO, the optimization of Eqn.(3) can be formulated as classification loss over the preference data as:EQUATION)DPO enables learning π θ from a fixed dataset of preferences, which is lightweight. However, the challenge arises because the direct application of DPO does not reliably induce hallucinations in a manner that meets the criteria specified in Eqn. (17).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: The research conducted in the paper conform, in every respect with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
method: 
 An overview of the proposed HIO method is shown in Fig. 2 . It constructs a more-hallucinated LVLM by inducing hallucinations from the original LVLM to amplify the contrast between hallucinatory and targeted tokens, thereby enhancing the efficiency of contrast decoding and mitigating hallucinations in LVLMs. In Section 4.1, we harness a fine-tuned theoretical preference model to amplify the contrast between hallucinatory and targeted tokens. Furthermore, to induce more potential hallucinations for effective contrast decoding, we propose to amplify multiple hallucination tokens based on a theoretical foundation presented in Eqn. 17 of Section 5. This theory demonstrates that effective contrastive decoding requires a consistent difference between the logits of potential hallucinated tokens and the correct token. And Section 4.3 introduces additional constraints to overcome the limitations of existing classification loss in amplifying the contrast between hallucinatory and targeted tokens.We harness a fine-tuned theoretical preference model (i.e., Contrary Bradley-Terry Model [Bradley and Terry, 1952] ) to amplify the contrast between hallucinatory and targeted tokens. The studies on hallucination mitigation Zhao et al. [2023] , Yu et al. [2023] , Zhou et al. [2024] utilize BT model by defining the non-hallucinatory output as y w and the hallucinatory output as y l . Subsequently, they employ BT model training to incentivize the model to prioritize outputs without hallucinations over those containing them. However, within the context of contrast decoding, inducing hallucinations is crucial, and the resulting model output must satisfy the criteria outlined in Eqn. ( 17). (The detailed derivation of this formula is provided in Section5). To meet the requirements specified in Eqn. ( 17), the logits associated with hallucinated tokens li {v,x,y<t} need amplification, while at least one of the logits for the correct token lj {v,x,y<t} must be reduced. In contrast to the prevailing research efforts focused on alleviating hallucinations, our approach enables the model to learn to fit the distribution containing hallucinations while avoiding convergence with the distribution of correct outputs. The details are outlined as follows. To regulate li {v,x,y<t} and lj {v,x,y<t} , we utilize the dataset introduced by Yu et al. [2023] . This dataset is notable for providing a pair of outputs per input, with the output paragraphs being mostly identical except for differences in certain words or short phrases. By leveraging this dataset, we approximate the conditions outlined in Eqn. ( 17) within a unified statement. Different from Eqn. (5), we apply the Bradley-Terry (BT) [Bradley and Terry, 1952] model in a reversed way, the objective is:EQUATION)where σ(•) is defined as a sigmoid function and the reference model π ref (y|x) is usually implemented by an instruction-tuned base model we want to improve, and is kept fixed during DPO training. Only the policy model π θ (y|x) is updated.The methodology delineated in Eqn. ( 6), along with the conventional application of Direct Preference Optimization (DPO) for mitigating hallucinations, is limited to highlight the difference between a single hallucination token and the target token. Consequently, these approaches fall short in enhancing the distinctions among other hallucinations relative to the target tokens, which is critical as shown in Eqn. ( 17). In this section, we will explain how to amplify the differences between multiple hallucination tokens and target tokens through modifications at both the loss function and data levels.Multiple Hallucination-Induced Optimization. Achieving the desired distribution through single positive and negative sample fitting preference training is not feasible, leading conventional Direct Preference Optimization (DPO) applications Zhao et al. [2023] , Yu et al. [2023] , Zhou et al. [2024] to overlook a significant number of hallucinations. Thus, drawing inspiration from the implications of Eqn. ( 17), our approach strategically induces multiple hallucinations to increase the probability of producing a correct word in the output. As demonstrated in Eqn. ( 17), effective contrast decoding necessitates not only the amplification of one hallucination but also the consideration of a diverse set of potential hallucinations. We propose the simultaneous fitting of multiple pairs of preference data when modeling distributions for the same input preference, treating all pairs of preference data with equal importance. Based on Eqn. ( 6), we apply the Bradley-Terry (BT) [Bradley and Terry, 1952] model in a multi-pair way, the objective is:EQUATION)where {y li }, i ∈ {1, 2, . . . , k} represent the multiple potential hallucination tokens. Assuming access to a static dataset of comparisons D = x (i) , yw , {y(i) li } N i=1sampled from p, we can parametrize a reward model r(x, y) and estimate the parameters via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss:EQUATIONAcquisition of Multiple Candidate Hallucinations. While numerous hallucination datasets exist Yu et al. [2023] , Zhao et al. [2023] , Zhou et al. [2024] , they are either generated by GPT or manually rewritten, and thus do not accurately represent the model's potential for multiple hallucinations. Therefore, we propose a novel approach: allowing the model to directly output tokens with high confidence as negative samples. While this approach may incorrectly classify some correct tokens as hallucinations, it compensates by providing true value-labeled data for correction and supplementation. Consequently, this method effectively amplifies multiple hallucinations while reducing the target token. The detailed training process of our method is outlined in Algorithm 1.To overcome the limitations of existing classification loss in amplifying the contrast between hallucinatory and targeted tokens, we introduces additional constraints. The preference optimization strategy outlined in Eqn. (8) allows the model to accommodate a specific range of preference distributions through the cross-entropy in the classification loss function. The precise formulation is as follows:EQUATIONwhere m represents the length of the sentence y l and {k T } is token of each word, and the definition of li {v,x,y<t} is shown in Section 5. While the use of cross-entropy to minimize encoding length helps the model align with the desired output sentence, it does not consistently ensure that the logits of induced hallucinations meet the conditions specified in Eqn. (17).For example, the goal of Eqn. ( 8) is to increase π θ (y l |v, x), but both increasing exp l can also allow π θ (y w |v, x) to meet the optimization criteria. As shown in Fig. 3 , the blue curve, representing the disparity between the logits of the hallucinatory and targeted tokens, typically exhibits a positive trend. Nevertheless, it's important to note occasional segments where this value dips below zero. To tackle this issue, we further add restrictions based on Eqn. (8):EQUATION+ γ 1 m m t=1 l kt {v,x,y<t} -li {v,x,y<t}By implementing this constraint, the model can be fitted to the distribution of preference statements, thereby further expanding the difference between hallucination tokens and target tokens. 
fundamental conditions for contrast decoding: 
 Contrast decoding is capable of mitigating hallucinations when specific conditions are met. This section delves into a comprehensive discussion and analysis of these conditions. Definition. Let l {v,x,y<t} i represent the probability of the i-th token in the model's vocabulary given the query x, the visual context v and the sequence of generated tokens up to the time step (t -1). The logits can be formulated as:EQUATION)where N denotes the vocabulary length.Definition. Let L{v,x,y<t} represents the ideal logits for contrast decoding, L ′ {v,x,y<t} represents the logits with hallucination and L * {v,x,y<t} represents the logits of correct token, where {L ′ {v,x,y<t} , L * {v,x,y<t} } ∈ L {v,x,y<t} . The results of contrast decoding of logits can be formulated as:δ {v,x,y<t} = (1 + α)L {v,x,y<t} -α L{v,x,y<t} (13) where larger α values indicate a stronger amplification of differences between the two distributions (α = 0 reduces to regular decoding). The condition for the absence of hallucination in the logits subsequent to subtraction is that the values of the logits corresponding to all hallucinatory tokens are less than the magnitudes of the logits corresponding to the correct lexical tokens. The aforementioned condition is articulated mathematically as follows:EQUATIONwhere δ ′ {v,x,y<t} denotes the result of the subtraction between the logits of all hallucinated vocabulary tokens and the logits after their ideal amplification. δ * {v,x,y<t} represents the outcome of the subtraction between the logits corresponding to all correct vocabulary tokens and the logits under the ideal scenario. Eqn. 14 represents a theoretical upper bound, which guides us in enhancing the effectiveness of Contrast Decoding method for hallucination elimination by ensuring that the logits of all hallucinated words are lower than those of the correct words. Upon expansion of the left side of the equation, the following result is obtained:EQUATIONwhere m denotes the total number of hallucinated vocabulary items, and k j represents the subscript position of the i-th hallucinated vocabulary within the set L {v,x,y<t} . For the right side of the equation, one of the correct lexical items is selected as the subject for amplification.min δ * {v,x,y<t} = min{(1 + α)L * {v,x,y<t} -α L * {v,x,y<t} } ≤ (1 + α)l {v,x,y<t} j -α lj {v,x,y<t} , j ∈ {k * 1 , k * 2 , . . . , k * n } (16)where n denotes the total number of correct lexical items. Based on Eqn. (15) and Eqn. ( 16), Eqn. ( 14) can be simplified to the form presented as follows:EQUATIONwhere J represents (1+α)EQUATION). In the context of the contrast decoding method, given that the parameters of the original model remain invariant, the output can be characterized as a constant. Eqn. 17 delineates the logits for all hallucinated tokens li {v,x,y<t} and contrasts these with the logits of a single correct token lj {v,x,y<t} . It postulates that, for an optimal logits output, a pronounced divergence must be maintained between the logits of hallucinated tokens and the logit of the correct token.Eqn. 17 illustrates that hallucinations can be effectively eliminated through contrastive decoding if the difference between the logits of the hallucinatory token and the correct token in the 'Evil' LVLM's output (Left part of Eqn.17) exceeds that in the original LVLM output (J in Eqn.17). For example, as depicted in the lower part of Fig. 2 , where "Dogs" is a hallucination and "Benches" is the correct label, the hallucination of "Dogs" is removed when the difference between the logits for "Dogs" and "Benches" in the 'Evil' LVLM output surpasses the difference in the original LVLM output. When this condition is met for all potential hallucinations, all hallucinations are effectively eliminated. 
experimental settings: 
 Benchmarks. We evaluate HIO on three benchmarks including: (1) Quantitative metrics POPE Li et al. [2023b] on MSCOCO Lin et al. [2014] dataset. The Polling-based Object Probing Evaluation Li et al. [2023b] offers a streamlined approach to assessing object hallucination. In this benchmark, LVLMs are queried about the existence of specific objects in a given image. (2) CHAIR Rohrbach et al. [2018] , Caption Hallucination Assessment with Image Relevance, is a specialized tool designed to evaluate the occurrence of object hallucination in image captioning tasks. (3) General-purposed Multimodal Large Language Model Evaluation (MME) Fu et al. [2023] benchmark, which provides an extensive benchmark designed to evaluate LVLMs across multiple dimensions, including ten perception-related subtasks and four cognition-focused ones. Implementation Description We evaluate our model across three Large Vision-Language Models (LVLMs): LLaVA 1.5, InstructBLIP, and MiniGPT-4. For decoding, we use Llama-7B and Vicuna-7B as the linguistic decoder for LLaVA and InstructBLIP/MiniGPT-4, respectively. Our model's performance is compared against three leading models in the field: OPERA Huang et al. [2023] , VCD Leng et al. [2023] , and VDD Zhang et al. [2024] . To ensure a fair and rigorous comparison, we adhere to the configurations and guidelines from the original works and codebases of the compared models. The training is conducted on a robust computational setup: 4x RTX 3090 GPUs for LLaVA 1.5, 8x V100 GPUs for MiniGPT-4, and 4x A6000 GPUs for InstructBLIP. Each training session lasts approximately 2-4 hours. Hyperparameters including alpha and beta are set to 1.0 and 0.1, respectively, in accordance with the VCD model's specifications.To evaluate HIO's capability on object hallucination, we compare it with several state-of-theart Decoding methods on POPE. The results are shown in Tab. 1, which presents the experimental results on the POPE dataset across random, popular, and adversarial settings. Our method consistently outperforms the standard decoding strategy, with average improvements of 6.2% in accuracy and 7.3% in F1 score across all LVLMs. Additionally, our approach clearly surpasses state-of-the-art decoding methods, demonstrating its effectiveness in mitigating object hallucinations. The improved performance across random, popular, and adversarial settings further confirms that our HIO method effectively reduces hallucinations in diverse scenario. generalization capability of our proposed components to unseen categories, as detailed in Table 4 . For the Unseen-P dataset, we collected data from MSCOCO, A-OKVQA, and GQA, ensuring no overlap with the training set, resulting in 495 samples across 10 distinct classes. These experiments show that our components generalize effectively to unseen data. Finally, we have integrated the ablation study into the experimental results section, rather than presenting it separately.  
discussion: 
 In this study, we conduct an in-depth examination of the principles governing contrast decoding and the prerequisites for its efficacy. Based on our findings, we introduce HIO, an innovative model optimization approach designed to induce hallucinations. This method significantly amplifies hallucinatory elements within the model, thereby effectively mitigating them through contrast decoding. Extensive experimentation across various datasets has demonstrated that HIO effectively reduces hallucinations and achieves state-of-the-art performance.Limitations & Future Work. Our findings establish a necessary, but not sufficient, condition for the successful operation of contrast decoding. Further exploration of more effective conditions could significantly enhance the efficiency of contrast decoding in mitigating hallucinations. Additionally, exploring training-free methods to induce hallucinations could reduce the computational costs associated with decoding. 
acknowledgments and disclosure of funding: 
 This study is supported by grants from the National Natural Science Foundation of China (Grant No. 62122018, No. 62020106008 , No. U22A2097, No. U23A20315), and Kuaishou.The algorithm outlines the process by which the model generates its own series of potential hallucinations. Using the sample pairs produced by the model, we apply our proposed Hallucination-Induced Optimization (HIO) to enhance the distinction between hallucinated and target labels. Ultimately, hallucinations are mitigated through contrastive decoding.Using paired hallucination and non-hallucination annotations from the RLHF-V dataset, we apply beam search to generate multiple outputs where hallucination token annotations occur. These outputs include both correct and hallucinated results, which we use as hallucination samples to reinforce the model's confidence in its outputs. The correct annotations from RLHF-V serve as ground truth, helping the model avoid hallucinations by differentiating between hallucinated and target tokens. This approach expands the contrast between hallucinated and target tokens, effectively reducing hallucinations. I ′ = {i ′ 1 , i ′ 2 , . . . , i ′ m } 5: for i ′ t ∈ I ′ do 6: y ′ <i ′ trepresents the sequence of generated tokens up to the time step (i <i ′ t } j 1 ≥ l {v,x,y <i ′ t } j 2 ≥ • • • ≥ l {v,x,y <i ′ t } j k 9: for jt ∈ J {v,x,y <i ′ t } do 10: y ′ <(i ′ t +1) = y ′ <i ′ t ∪ jt 11: δ = 1 12: while y ′ (i ′ t +δ) is not period do 13: L {v,x,y <i ′ t +δ } = M(v, x, y ′ <i ′ t +δ ) 14: y ′ <(i ′ t +δ+1) = y ′ <i ′ t +δ ∪ arg min j L {v,x,y <i ′ t +δ+1 } 15: δ = δ + 1 16:end while 17:end for 18:end for 19: end forIn this appendix, we present a comprehensive verification of Eqn. ( 17), which is elucidated through the following detailed procedure:m × ((1 + α)l {v,x,y<t} j -α lj {v,x,y<t} ) - km i=k1 ((1 + α)l {v,x,y<t} i -α li {v,x,y<t} ) > 0 α km i=k1 ( li {v,x,y<t} -lj {v,x,y<t} ) -(1 + α) km i=k1 (l {v,x,y<t} i -l {v,x,y<t} j ) > 0 α (1 + α) km i=k1 ( li {v,x,y<t} -lj {v,x,y<t} ) > km i=k1 (l {v,x,y<t} i -l {v,x,y<t} j ) km i=k1 ( li {v,x,y<t} -lj {v,x,y<t} ) > J (18)This figure demonstrates the effectiveness of our ACI method (described in Section 4.3). The y-axis shows the difference between the hallucination token and the target token. The blue curve represents this difference without ACI, while the orange curve represents it with our proposed ACI. Clearly, our method accurately induces hallucinations, further amplifies the difference between the hallucination token and the target token, and thus effectively reduces hallucinations. Justification: We have listed our contributions in both abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: As demonstrated in Limitations and Future Works section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Yes, the paper describe safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, the creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: This work deals with hallucination in large visual language models, which is a common and important problem. It provides a novel perspective on the overall uncertainty of the visual input and the exploitation of multiple hallucinatory targets. The authors have observed and mitigated the limitations of existing optimisation frameworks and have provided an effective solution. Due to the strengths of the work, three reviewers recommend acceptance. Meanwhile, one reviewer has doubts about the novelty of this work, and for this, the authors have provided detailed response showing the contributions of this work. AC thus recommends acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a method for mitigating hallucinations by training an "evil" LLM to provide logits for contrastive decoding. This "evil" LLM is trained with a dataset that prefers hallucinated samples over true ones during fine-tuning. The logits from this "evil" LLM are then used for contrastive decoding. Experimental results validate the effectiveness of this approach.
strength_weakness: strength_weakness: 1. The writing is sometimes confusing and makes it difficult to understand details. For example, Sec 4.1 and 4.2  use Eqn 17 in Sec 5 as the motivation of how they design the current method and refer to it frequently. However, Eqn 17 is not explained and introduced previously, making it very difficult for the reader to understand why you design this method like this. It would be much better to explain the high level idea of eqn 17 first.
2. Besides, the figure 1 itself is also confusing. The picture provided is vague and even a human finds it difficult to identify the people and output the true answer.
3. When reading the Sec 5, it is also very confusing. How do you define the ideal logits for contras decoding? Based on my understanding, it should not equal to the hallucinatory token's logits. However, at the later part line 243, you are referring to it as the hallucinatory tokens. Besides, how do you interpret the second line of Eqn 17? I interpret J as the average logit difference between the hallucinatory token and the true token. However, I find it difficult to interpret the left part of the second line of Eq 17. What is the logic behind line 242? The whole explanation does not make sense to me.
4. For Eqn 14, is it too strong to require the minimal logit of correct tokens to be larger than the maximum logit of hallucinatory tokens? For greedy docoding, if one of the correct tokens after contrast is larger than all the hallucinatory tokens, then it will be output?
questions: questions: Sea the weakness part.
1. Typo: 
- line 127 should be "probability" instead of "probably". 
- line 233 it should be $\delta^{*\\{v,x,y_{<t}\\}}$ instead of $\delta^{\prime\\{v,x,y_{<t}\\}}$
limitations: limitations: Yes, the author has pointed out the computation cost for this method.
paper_summary: paper_summary: This paper focuses on alleviating hallucinations in Large Vision-Language Models. Specifically, the authors introduce a novel optimization strategy named Hallucination-Induced Optimization (HIO). This method amplifies the contrast between hallucinatory and targeted tokens relying on a fine-tuned preference model. Finally, the extensive experimental results verify the effectiveness of HIO, which outperforms state-of-the-art methods across various benchmarks.
strength_weakness: strength_weakness: 1. I mainly doubt the novelty. The motivation of this paper is very similar to [1]. Please describe the difference between this work and [1].
2. This paper is not complete, lacks a Conclusion section, and the ablation study section is too short. Moreover, the implementation details are missing in the paper. Thus, I think the writing of this paper is not ready. 

[1] Alleviating Hallucinations of Large Language Models through Induced Hallucinations.
questions: questions: Please see the weaknesses.
limitations: limitations: n/a
paper_summary: paper_summary: This manuscript presents a novel perspective on implementing contrastive decoding for hallucination mitigation in large visual language models. In contrast to methods such as image perturbation, it achieves a quasi-min-max optimization by enhancing model hallucination followed by contrastive decoding for multiple hallucination targets. The proposed framework and its improved components, i.e., CBTM, AMTH, and ACI, have solid theoretical and empirical derivations. The overall writing of the paper is commendable, and it has achieved state-of-the-art performance.
strength_weakness: strength_weakness: 1. The description of experimental details is weak. For instance, the experimental setups for Figure 1, Table 2, and Table 5 are not clearly stated, yet the results vary significantly.
2. The manuscript fails to provide the results of the three models purportedly studied.
3. The training details, time and hardware expenditure, and the selection of the alpha parameter, among many other experimental details for the Evil Model, have not been provided.
4. While the overall writing is satisfactory, many details betray a rather hasty preparation of the article, such as the repeated occurrences of acronym definitions, entirely duplicated result descriptions, missing spaces before parentheses in several instances, inconsistent terminologies used for metrics, etc.
5. There appears to be an error in Figure 2: 'benches' should be the correct category?
6. Inappropriate caption for Table 4.
7. Inappropriate bolding for Tables 1 and 3.
8. A Conclusion section is missing from the main text.
questions: questions: 1. In analogy to adversarial training, the utilization of hallucinatory information during the training dynamics is unclear. The subsection 'Acquisition of Multiple Candidate Hallucinations' seems relevant to this aspect, yet the discussion could be further enriched. 
2. In comparison to the proposed method, is it feasible to directly calibrate the hallucinatory and targeted logits and achieve both efficiency and effectiveness? 
3. Please carefully revise the writing details and enrich the experimental descriptions. I believe this will significantly enhance the clarity and soundness of the paper.
limitations: limitations: The authors have noted the additional overhead introduced by the proposed method during model training and inference. The potential social impact is positive, yet the discussion on this is lacking. The reviewer thinks that the current limitations in relevant datasets may also constrain the method's development. How to seek the annotation of hallucination tokens during dynamic optimization merits consideration.
paper_summary: paper_summary: This paper addresses object hallucination in large visual language models (LVLMs), a common phenomenon that such models generate texts not consistent with images. A viable approach for this issue is contrastive decoding. By comparing logits derived from images and distorted images, visual contrastive decoding reduces statistical bias and uni-modal priors thus alleviates hallucination. This paper points out that distorting images induces uncertainty which leads to unexpected results. To overcome this issue, this paper proposes to train an evil LVLM first by direct preference optimization, as illustrated in Figure 2, and contrast outputs from this model with LVLM responses. Theoretical foundations have been given and experimental results show that such strategy alleviates object hallucinations on multiple benchmarks, including POPE and CHAIR, which are commonly used metrics in this field.
strength_weakness: strength_weakness: - **Generalization issue**. In Figure 2, if I understand correctly, this paper induces an LVLM to generate hallucinated outputs (which is referred to as Hallucination Induction Optimization). Both hallucinated and corrected responses are involved in this Figure. If thinking more over objects (e.g., people, tv, dog, clock in Figure 2), a potential issue for this is, trained model in such manner can only be applied to these objects. In proposed hallucination-induced optimization, an LVLM is taught to hallucinate on some objects, these objects can work for later contrastive decoding. But I doubt that such capability can generalize to other objects which are not seen during training, making this a close-vocabulary approach, just like post-hoc correction methods in this field.
- Given that proposed hallucination-induced optimization resembles RLHF methods [A] in technical pipeline. Authors are suggested to include some results comparing these methods. Whether an amateur model can help an RLHF model seems an interesting question and may improve this paper.

[A] Aligning Large Multimodal Models with Factually Augmented RLHF.
questions: questions: - For Table 1 where POPE is evaluated, I notice that previous approach VCD has three evaluation setups, including MSCOCO, GQA and A-OKVQA. Only MSCOCO is given. Authors are suggested to include the other two as well for more comprehensive evaluations.
- Typos. In line 135, An overview of the proposed HIO method is shown in Fig. 3 (3 should be 2).
limitations: limitations: Authors have indicated limitations in Appendix D Discussion.

Comments: comment: I think the manuscript has reached the standard for publication after revision, and I intend to raise my score to 7.
Thanks for authors detailed reply and more evaluation experiments that address my concerns. I tend to accept this paper thus keep my earlier rating.