Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization
Authors: Xinyu Lyu
Beitao Chen
Lianli Gao
Heng Tao Shen
Jingkuan Song
Venue: NeurIPS 2024 poster (2024)
Keywords: Hallucination
Contrast Decoding
Primary area: optimization_for_deep_networks
TL;DR: We conducted the theoretical analysis to promote the effectiveness of contrast decoding. Building on this insight, we introduce a novel optimization strategy named Hallucination-Induced Optimization (HIO).

Abstract: Although Large Visual Language Models (LVLMs) have demonstrated exceptional abilities in understanding multimodal data, they invariably suffer from hallucinations, leading to a disconnection between the generated text and the corresponding images. Almost all  current visual contrastive decoding methods attempt to mitigate these hallucinations by introducing visual uncertainty information that appropriately widens the contrastive logits gap between hallucinatory and targeted ones.
    However, due to uncontrollable nature of the global visual uncertainty, they struggle to precisely induce the hallucinatory tokens, which severely limits their effectiveness in mitigating hallucinations and may even lead to the generation of undesired hallucinations.
    To tackle this issue, we conducted the theoretical analysis to promote the effectiveness of contrast decoding. Building on this insight, we introduce a novel optimization strategy named Hallucination-Induced Optimization (HIO). This strategy seeks to amplify the contrast between hallucinatory and targeted tokens relying on a fine-tuned theoretical preference model (i.e., Contrary Bradley-Terry Model), thereby facilitating efficient contrast decoding to alleviate hallucinations in LVLMs.
    Extensive experimental research demonstrates that our HIO strategy can effectively reduce hallucinations in LVLMs, outperforming state-of-the-art methods across various benchmarks.

Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induee Optimization
Xinyu 15+ xinyulyu66@gaill.cmm com
Beitao Chen Mlenbeitao@gmaill com
Lianli Gao
lianli, gao gaotuestc....... edu. cn
Jingkuan Song*
Heng Tao Shen
j jinkuann.. song@gmail.... com
shenhengtaoGhotmaiin com
Southwestern University of Finance and Economics, Chengdu, China Shenzhen Institute for Advanced Study,
University of Electronic Science and Technology of China Center for Future Media, University of Electronic Science and Technology of China 'Tongji University
Engineering Research Center of Intelligent Finance, Ministry of Education
http://gittub.com/BTTcccCC
Abstract
Although Large Visual Language Models (LVLMs) have demonstrated exceptional abilities understanding multimodal data, they invariably suffer from hallucina- tions, leading to disconnection between the generated text and the correspond- ing images. Almost all current visual contrastive decoding methods attempt to mitigate these hallucinations by introducing visual uncertainty information that appropriately widens the contrastive logits gap between hallucinatory and targeted ones. However, due uncontrollable nature of the global visual uncertainty, they struggle to precisely induce the ballucinatory tokens, which severely limits their effectiveness mitigating hallucinations and may even lead the generation of undesired hallucinations. To tackle this issue, we conducted the theoretical analysis to promote the effectiveness of contrast decoding. Building on this in- sight, we introduce a novel optimization strategy named HallucinationnIIduuee Optimization (HIO). This strategy seeks to amplify the contrast between hallucina- tory and targeted tokens relying
fine-tuned theoretical preference model (i.c., Contrary Bradley-Terry Model), thereby facilitating efficient contrast decoding to alleviate hallucinations in LVLMs. Extensive experimental research demonstrates that our HIO strategy can effectively reduce hallucinations in LVLMs, outper- forming state-of-the the-trr methods across various benchmarks. Code is released at https:// //gg::///ittan //tgtttttbb.////////////// com Baam/BT-CHHLE
Introduction
The recent success of Large Vision-Lanuuage Models (LVLMs) marks major milestone in artificial
intelligence research OOpenAl 2023 Alayrac et al., 2002 Li et al 2023a Liu et al., 2023c Zhu
2023 Wang al. 2023b Driess al. 2023 By seamlessly integrating visual cues with Large Language Models (LLMs), LLMS have demonstrated umparal- leled expertise in multimodal comprehension, logical reasoning, and interactive engagement. This
al. 2023 Bai al. 2023 Dai al.
"Equal contribution.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Visuall Input y
WLMB
(1 (olgitss/lk..o ef1g u)
The performancc of the the
methods compared with Ours.
IHE
Table
Cupiing
An lie mad Mene
of
Hallucinatory
Greedy Decoding Strateyy
Token
Irepul
(1 a)logits,(1x.
Howppt
III
11
Fork Hallasimatory
of S/
Decoding Strategy
Token
10
Visual Input D
LVLM
(1 a"Nogits aiogiss (y/x,,
hand Powey
Heapy smat
Kite
dugang
ls mad Aure
Tirgeted
(c) Hallucination
Optimization
Taken
Orealy
VED
OPERA
HALC
Oua
Figure I: (Left) Challenges and Solutions of Contrast Decoding Strategy. Visual Contrastive Decoding, despite introducing perturbations induce hallucinations, fails to effectively enlarge the logits gap between hallucinatory and targeted tokens, resulting in unsatisfactory outputs. On the contrary, our method addresses the issue by significantly amplifying the logits gap between hallucinatory and targeted tokens. (Right) The performance of various methods CHAIR metrics. Our HIIO generates descriptions with fewer hallucination tokens compared to other visual contrastive decoding methods, achieving lower scores on the CHAIRs and CHAIRi metrics.
integration has ushered in a new era AI, breaking through traditional limitations and enabling more holistic understanding of complex information OpenAl [2033] Yang et al. [2023] Lu et al.
2023 Yuan et al. [2022]. [Sun et al. 2024 Despite these advanceennss, certain challenges remain, particularly the issue of hallucination Li et al. (023b). Gunjal et al.) [2023), [iu et al. (203b)].
Lovenia et al. (2023). Hallucination occurs when the language model generates content that deviates from the image's actual content, including imagined objects, fabricated scenes, incorrect spatial relationships, and misidentified categories.
Substantial research efforts have been directed towards mitigating hallucinations in Large Vision- Language Models (LVLMs). These efforts include post-hoc correction methods that refine LVLM outputs after the fact Zhou et al. [2023 and self-correcting frameworks specifically designed to reduce object ballucinations Yin al. 22023. Additionally, numerous decoding strategies have been developed to minimize hallucinations through the enhanced use of textual and visual priors Leng et al. 2023 Zhang et al. [2024), Faveero al.) (222),, Zhu al. [2024), Wang al.] (2024), Chen et al.
[024]] These methods aim to alleviate hallucinatory tendencies by integrating visual uncertainty,
thereby increasing the contrastive disparity between hallucinatory and target logits. For example, 2023 augment the hallucinatory effect by introducing Gaussian noise into the images. Similar approaches by iZhang al.) [2024) and Favero et al.) (2024) introduce substantial image noise, effectively reducing the original image pure noise unrecognizable content. [huu et al.
Leng et al.
2024
use instructional bias to enable the model to amplify its own hallucinations, while Wang et al.
2024
focus on deliberately amplifying the inherent image bias in LVLMs.
However, the inherent uncontrollable nature of global visual uncertainty challenges the precise induction of hallucinatory tokens. This limitation significantly undermines the effectiveness of these methods in reducing hallucinations and may inadvertently lead to undesired hallucinatory outputs. As shown in the left portion of the Fig. Spoon, Table, and Fork are identified as hallucinated words, while People being the accurate term. For Greedy Decoding method shown in Fig. (a), Table is selected as the final output based on the logits distribution. Moreover, although Visual Contrastive Decoding introduces perturbations to images to enhance hallucinations in Fig. (b), it fails to widen the logits gaps between hallucinatory (Spoon, Table, and Fork) and targeted tokens (People), yielding a new hallucination as Fork.
To tackle this issue, we conducted the theoretical analysis to explore mechanisms for more effective contrast decoding (refer to Section (attions]oor detailed information on the process). Theoretically, clear distinction between hallucinatory and target tokens can significantly enhance the effectiveness of contrast decoding methods mitigating hallucinations. Based on this crucial insight, we introduce a novel optimization strategy called Hallucination-Indueee the distinction between hallucinatory and targeted tokens by utilizing a refined theoretical preference model(as shown in the Fig. on the left, section (c)), accurately outputting the correct result, People.
Optimization (HIO). This strategy enhances

---

Consequently, this improves the efficiency of contrast decoding, thereby mitigating hallucinations in Large Vision-Language Models (LVLMs). Furthermore, our proposed method significantly reduces hallucinations in LVI.Ms compared to existing contrast decoding methods(as shown in the Fig. D on the right). To sum up, our main contributions are as follows:
1. We conducted comprehensive theoretical analysis to explore mechanisms that enhance the effectiveness of the contrast decoding strategy.
2. We introduce Hallucination-Indun a finely-tuned theoretical preference model to intensify the contrast between hallucinatory and target tokens. This enhancement strengthens the effectiveness of contrast decoding and effectively reduces ballucinations in Large Visual Language Models (LVLMs).
Optimization (HIO), an innovative strategy that utilizes
3. Extensive experimental research demonstrates that our Hallucination-Indueee Optimization (HIO) strategy effectively reduces hallucinations in Large Visual Language Models (LVLMs), surpassing state-of-the methods across various benchmarks.
2 Related Work
Hallucination in LVLMs. Before the advent of Large Language Models (LLMs), "hallucination"
in natural language processing (NLP) primarily referred to generating nonsensical or source-deviating
content Lee et al. 2018 Zhou et al. 2020 Lin et al. 2021] Ji et al. [2033] Zhang et al. 2023 Shi
et al. 2023] Recent studies have tackled the complexities of object hallucination in Large Vision-
Language Models (LVLMs), focusing on evaluation and detection methods Wang et al. [2023 Liu
al. [2023a], Li al. (023b), Lovenia al. 2023). The CHAIR metric Rohrbach al. 2018
evaluates the exact match between generated and ground-truth image captions, while POPE Li et al.
(003b assesses the model's awareness of object existence through binary classification.
Decoding Method. The decoding method determines the generation of text tokens at each time
step within language models. Traditional decoding strategies such beam search Boulanger Lewandowski et al. [2013). top-k decoding Fan et al. 2018). and sampling methods Holtzman et al. [2019], J. despite their widespread use, are prone to producing hallucinatory content. Recent research Li et al. [2022], Chuang et al. (2023), Leng et al. [2023]. Huang et al. (2023) has made attempts to address this issue by proposing better decoding methods. For instance, Leng et al. [2023) uses contrastive decoding in LVLMs; However, global visual uncertainty poses challenges to the precise induction of hallucinatory tokens, limiting the effectiveness of mitigation strategies and risking unwanted hallucinations. To address this, we developed Hallucination- Induced Optimization (HIO), a novel strategy that enhances the contrast between hallucinatory and targeted tokens. Fig[ presents the comparison results, where our approach demonstrates superior performance than other decoding methods.
inlluiiinationIndueedd
Preliminaries
We first review the Contrast Decoding pipeline in Leng al. 2023 (and later Zhang al. [2024]
Favero et al. 2024)). ). Then take a close look at the Bradley- -errry model |Brddley and Terry 1952
and its application such as Direct Preference Optimization Rafailov et al. [024 Inspired by these studies, we propose our Hallucination-Induee Visual Contrastive Decoding. We consider LVLM parameterized by 0. The model takes textual query input æ and a visual input v, where v provides contextual visual information to assist the model in generating a relevant response y to the textual query. The response y is sampled auto-regressively from the probability distribution conditioned on the query x and the visual context v. Mathematically, this can be formulated as:
Optimization.
Vo pa (ye v. Mer) exp logita (ye Ver))
(I)
where y denotes the token at time step t, and yer represents the sequence of generated tokens up to the time step
Specifically, given textual query & and a visual input v, the model generates two distinct output distributions: one conditioned on the original and the other on the distorted visual input v', which derived by applying pre-defined distortions (i.e., Gaussian noise mask) to the original e. Then, new contrastive probability distribution is computed by exploiting the differences
3

---

between the two initially obtained distributions. The new contrastive distribution Poed is formulated as:
Pund (y 0', = softmaxi + a) logito (y v, 2) - a logit, (v [x,]]]
where larger value of a indicate a stronger amplification of differences between the two distributions
(a- 0 reduces to regular decoding).
Direct Preference Optimization. Reinforcement learning (RL) effectivvv fine-tunes Large Language Models (LLMS)) to align with human behavior. Given input and response y, language model policy тh generates a conditional distribution T r).. RL aims to maximize the average reward of outputs, with the reward function r(z, y). To prevent overoptimization Gataaaiinnnaaooaao et al. [2023], the objective
loss includes a KL-divergence term, controlling the divergence between the language model policy and its reference policy (y 2), typically derived from supervised fine-tuning. Thus, the overall S
objective is formulated as:
To x x)
max B Daw [r(z, v) - a log
Tet (3
where D is a dataset of prompts and is coefficient control KI--dveegennce term. However, opti- mizing the above loss term with common strategies like proximal policy optimization (PPO) Schulman al. [2017] complex to tune. Recently, direct preference optimization (DPO) Rafailov [2024]
simplifies simp the above process by leveraging preference data for optimization. Here, the preference data is defined D {x:(9, yw Vi J Ji=1 where you and yi represent preferred and dispreferred responses given an input prompt x. These are then presented to human labelers who express prefer- ences for one answer, denoted as Yo > V1 where Ywu and y denote the preferred and dispreferred respectively. Following Bradley-Terr Terry model [Bradley and Terry 1952]. the probability of obtaining respecer
(c) (4)
each preference pair is:
exp (r(x, yw))
Plu 22)= exp (r(z, Bww) + exp (r(z,
(4)
where the superscript i is omitted for simplicity. In DPO, the optimization of Eqn. formulated as classification loss over the preference data as:
can be
CDpo(Tes Teet) -E (u. log (
(<c log
To Tae (r) (ye
log Te (3 (z)
Teet (yn (x) )
(5)
DPO enables learning vp from fixed dataset of preferences, which is lightweight. However, the challenge arises because the direct application of DPO does not reliably induce hallucinations in manner that meets the criteria specified in Eqn. (17).
4 Method
An overview of the proposed HIO method is shown in Fig. It constructs a more-hallucinated LVLM by inducing hallucinations from the original LVLM to amplify the contrast between hallucinatory and targeted tokens, thereby enhancing the efficiency of contrast decoding and mitigating hallucinations in LVLMs. In Section we harness a fine-tuned theoretical preference model to amplify the contrast between hallucinatory and targeted tokens. Furthermore, to induce more potential hallucinations for effective contrast decoding, we propose to amplify multiple hallucination tokens based on theoretical foundation presented in Eqn. [7 of Section 5 This theory demonstrates that effective contrastive decoding requires a consistent difference between the logits of potential hallucinated tokens and the correct token. And Section 4.3 introduces additional constraints to overcome the limitations of existing classification loss in amplifying the contrast between hallucinatory and targeted tokens.
4.1 Contrary Bradley-Terry Model (CBTM)
We harness fine-tuned theoretical preference model (i.e., Contrary Bradley-Terry Model (Bradley and Terry] [952] to amplify the contrast between hallucinatory and targeted tokens. The studies on hallucination mitigation Zhao et al. 2023 Yu et al. 2023 Zhou et al. (2024) utilize BT model by defining the hollhallucinatory output as yoe and the hallucinatory output as yr. Subsequently, they employ BT model training to incentivize the model to prioritize outputs without hallucinations over m

---

Corrections
Hallucinations
Training Stage
Iu this room, there is
Hallucination Induction Optimization (HIO)
Original LVLM
logits (y/z, v)
logits;
Evil LVLM
Deveribe this pirtore in detail
People
N
Det Clack
People
Dot Clart
logits((rlx,1)
Inference Stage
(1 + w)logits (ylx, alogitsa
this park, there are
Evil LVLM
People
Dogs
Bomuhes
Tracks
park. are benches along the rand the park everyone rest.
Original LVLM
logits (Vzz,)
People
Dop Bonches Tracks
ballucinated object "Dogs eliminated
Pecgee
Dege
Braeled
Pande
Desershee tkis detail
Figure 2: An overview Hallucination-Indueee Optimization (HIO). Our approach comprises two phases: the training stage and inference decoding. During training stage, given an input image, a query, and manually annotated correction, the Large Visual Language Model (LVLM) produces multiple instances of hallucinated content. We then apply our Daallucination-Induuc LVLM by inducing hallucinations from the original LVLM. In the inference phase, the logits from the trained 'Evil' LVLM are used contrast with those generated by the original LVLM, effectively reducing the presence of halluciaaatioss.
Optimization (HIO) method to train an 'Evil'
those containing them.
However, within the context of contrast decoding, inducing hallucinations crucial, and the resulting model output must satisfy the criteria outlined in Eqn. (17). (The detailed derivation of this formula is provided in Section To meet the requirements specified in Eqn. (17). the logits associated with
hallucinated tokens & (v.z.Bee) need amplification, while at least one of the logits for the correct token I (,,,,8<))
must be reduced. In contrast to the prevailing research efforts focused on alleviating hallucinations, approach enables the model learn fit the distribution containing hallucinations while avoiding convergence with the distribution of correct outputs. The details are outlined as
follows. To regulate i, (v,,,ee)) and I, {v,2,,cc) we utilize the dataset introduced by Yu et al. 2023 This dataset is notable for providing pair of outputs per input, with the output paragraphs being mostly identical except for differences in certain words or short phrases. By leveraging this dataset, we approximate the conditions outlined Eqn. (I7) within a unified statement. Different from Eqn. (5). we apply the Bradley-Terry (BT) [Bradley and Terry 1952 model in reversed way, the objective is:
exp (((x,,1)
pln > Yor xx)
exp (r(x, vz)) + exp (r( Uhe))
(6)
log Te
B log Te (Brel (,
=a
Teet ( xv
(ee (,,2))
where a(-) is defined as a sigmoid function and the reference model Tee(y)lr) (r) is usually implemented by instruction-tuned base model want improve, and is kept fixed during DPO training. Only the policy model my (yl (r) is updated.
4.2 Amplification of Multiple Targeted Hallucination (AMTH)
The methodology delineated in Eqn. 6 along with the conventional application of Direct Preference Optimization (DPO) for mitigating hallucinations, limited highlight the difference between single hallucination token and the target token. Consequently, these approaches fall short in enhancing

---

the distinctions among other hallucinations relative to the target tokens, which is critical as shown in Eqn. (17). In this section, we will explain how to amplify the differences between multiple hallucination tokens and target tokens through modifications at both the loss function and data levels. Multiple Celluciinatin-Indueee
Optimization. Achieving the desired distribution through single positive and negative sample fitting preference training is not feasible, leading conventional Direct Preference Optimization (DPO) applications Zhao et al. [2023). Yu et al. [023]. Zhou et al. [2024) to overlook significant number of hallucinations. Thus, drawing inspiration from the implications of Eqn. (77. our approach strategically induces multiple hallucinations to increase the probability of
producing correct word in the output. As demonstrated in Eqn. (I7), effective contrast decoding necessitates not only the amplification of one hallucination but also the consideration of diverse set of potential hallucinations. We propose the simultaneous fitting of multiple pairs of preference data when modeling distributions for the same input preference, treating all pairs of preference data with equal importance. Based Eqn. (6), we apply the Bradley-Terry (BT) (Bradley and Terry 1952 model a multi-pair way, the objective is:
cxp (r(x, mi:) II exp (r(æ, mr:) + exp (r(z, Bw))
plun Vi x))
l
Blog To (Mle (x)
Blog To ( (x)) Teat (yve xz)
II o
Tea (3152z)
where ( E 21. A)} represent the multiple potential hallucination tokens. Assuming access
to static dataset of comparisons D = (1((), ,30' {Mr
;=1 sampled from we can parametrize a reward model r(x, y) and estimate the parameters via maximum likelihood. Framing the problem as binary classification we have the negative ligellhoo loss:
(4) (8) }}
k p(u x yoo 12)
LAMTH( (To: Tret) - B(a.nnaa>D log
II i=1
(8)
4:
To (o(yslu,, B log To To |u, 12,)
2) Teet Uio v2
--E )
log (Blog
Tret (Ml v,
Acquisition of Multiple Candidate Hallucinations. While numerous hallucination datasets exist Yu et al. [2023], Zhao et al. [2023). Zhou et al. [2024), they are either generated by GPT or manually rewritten, and thus do not accurately represent the model's potential for multiple hallucinations, Therefore, we propose a novel approach: allowing the model to directly output tokens with high confidence as negative samples. While this approach may incorrectly classify some correct tokens as hallucinations, it compensates by providing true value-labeled data for correction and supplementation. Consequently, this method effectively amplifies multiple hallucinations while reducing the target token. The detailed training process of our method is outlined in Algorithm
4.3 Advanced Constraints for Inducing (ACI)
To overcome the limitations of existing classification loss in amplifying the contrast between halluci- natory and targeted tokens, we introduces additional constraints. The preference optimization strategy outlined in Eqn. (8) allows the model to accommodate specific range of preference distributions through the cross-entropy in the classification loss function. The precise formulation is as follows:
m exp tiee
Tolyi x.)
expl, (1.n (ke) y1 t = (1,2...., .....
(10)
where m represents the length of the sentence and {kr} } is token of each word, and the definition of f (v....eee) is shown in Section 5 While the use of cross-entropy to minimize encoding length helps the model align with the desired output sentence, does not consistently ensure that the logits of induced hallucinations meet the conditions specified in Eqn. (17)

---

For example, the goal of Eqn. 8 is to increase To(yilu, x), but both increasing explk, (0.2.
or decreasing Σj exply
can achieve this goal. Meanwhile, decreasing the value of
Σn expli (1,2,99c)
can also allow no (V x) to meet the optimization criteria. As shown in Bi the blue curve, representing the disparity between the logits of the hallucinatory and targeted tokens, typically exhibits positive trend. Nevertheless, it's important to note occasional segments where this value dips below zero. To tackle this issue, we further add restrictions based on Eqn. (8)
log (glog (o(ussu,a)) 3 log To(Val (er((Holo,))
(To; Tree)
(11)
1
(uiae -e (
lke
By implementing this constraint, the model can be fitted the distribution of preference statements, therehy further expanding the difference between hallucination tokens and target tokens.
Fundamental Conditions for Contrast Decoding
Contrast decoding capable of mitigating hallucinations when specific conditions met. This section delves into a comprehensive discussion and analysis of these conditions.
Definition. Let i (...2.)) represent the probability of the i-th token in the model's vocabulary given the query z, the visual context v and the sequence of generated tokens up to the time step (t 1). The logits can be formulated as:
logity (ye U, v, Met) (uaaee)) (1, (LfNzz.Be)) (oa.vee) ...., "...,,faaBee fea.eett
(12)
where denotes the vocabulary length.
Definition. Let f.(aa.see represents the ideal logits for contrast decoding, L /'tuweeer repre- sents the logits with hallucination and L [v.one)))) represents the logits of correct token, where (I. (v. (...2..))) E [(v.2.8c)) The results contrast decoding of logits can be formulated as:
g(v,a992)) = (1 +1+ (++)) (..000)))))))) af. (v,a,9e))
(13)
where larger values indicate a stronger amplification of differences between the two distributions (a 0 reduces to regular decoding). The condition for the absence of hallucination in the logits subsequent to subtraction is that the values of the logits corresponding to all hallucinatory tokens less than the magnitudes of the logits corresponding to the correct lcxical tokens. The aforementioned condition is articulated mathematically as follows:
Proposit.
max 6 (mam,ee
min ( (M,N,ee)
(14)
where (v,,,ace denotes the result of the subtraction between the logits of all hallucinated vocabulary tokens and the logits after their ideal amplification. (* represents the outcome of the subtraction between the logits corresponding all correct vocabulary tokens and the logits under the ideal scenario. Eqn. [4[represetts a theoretical upper bound, which guides in enhancing the effectiveness of Contrast Decoding method for hallucination elimination by ensuring that the logits of all hallucinated words are lower than those of the correct words. Upon expansion of the left side of the equation, the following result is obtained:
max 8' (v.2,0<r) = max ((1 + a fu, al ([u...
-max((1 + a)) a) (uz. - ali (u, j tkis Ky.,ky...,,ki
(15)
((1 (1+ (o..vel -ali (v.m.ec))
where m denotes the total number of hallucinated vocabulary items, and ky represents the subscript position of the i-th hallucinated vocabulary within the set L (v,0,80)), For the right side of the equation,
7

---

one of the correct lexical items is selected as the subject for amplification.
min 8 (m.naee)) min((1 + +.) - al a (v.am))))
(16)
(1 + a a al, (v. , (ki, ke............ kn
where denotes the total number of correct lexical items. Based on Eqn. can be simplified the form presented as follows:
15) and Eqn. (6)., Eqn.
14
km
m x ((1 + a))) a...e al, (ow.eett)
Σ
((1 + a)f.
als (u. }
im k=
(17)
km
(4 (vn - tusnwee
where J represents [1+a) Dim method, given that the parameters the original model remain invariant, the output can be character- ized as a constant. Eqn. 17 delineates the logits for all hallucinated tokens 1, (v...vce) and contrasts these with the logits of a single correct token I, [v....ee) postulates that, for an optimal logits output, pronounced divergence must be maintained between the logits of hallucinated tokens and the logit of the correct token.
((foaraee)) 1 ((...))))) ). In the context of the contrast decoding
Di=ky
Eqn. 17 illustrates that hallucinations can be effectively eliminated through contrastive decoding if the difference between the logits of the hallucinatory token and the correct token in the 'Evil' LVLM's output (Left part of Eqn. 17) exceeds that in the original LVLM output (J in Eqn. For example, as depicted in the lower part of Fig. 2 where "Dogs" is a hallucination and "Benches" is the correct label, the hallucination of "Dogs" is removed when the difference between the logits for "Dogs" and "Benches" in the 'Evil' LVLM output surpasses the difference in the original LVLM output. When this condition is met for all potential ballucinations, all hallucinations are effectively eliminated.
Experiments
6.1 Experimental Settings
Benchmarks. We evaluate HIO on three benchmarks including: (1) Quantitative metries POPE et al. [202bb] on MSCOCO Lin al. (2014) dataset. The Polling-based Object Probing Evaluation
Li al. 2023b) offers a streamlined approach to assessing object hallucination. In this benchmark, LVLMs queried about the existence of specific objects in given image. (2) CHAIR Rohrbach ct al. 20188. Caption Hallucination Assessment with Image Relevance, a specialized tool designed to evaluate the occurrence of object hallucination in image captioning tasks. (3) General-purposee Multimodal Large Language Model Evaluation (MME) Fu et al.) [2023) benchmark, which provides extensive benchmark designed evaluate LVLMMs across multiple dimensions, including ten perception-related subtasks and four cognition- Implementation Description We evaluate model across three Large Vision-Languaan Models (LVLMs): LLaVA 1.5, InstructBLIP. and MiniGPT-4. For decoding, we use Llama-7B and Vicuna- 7B the linguistic decoder for LLaVA and Instruc respectively. Our model's performance is compared against three leading models in the field: OPERA Huang et al. 2022)) VCD VCD et al. (2023) and VDD Zhang et al. [024]. To ensure a fair and rigorous comparison, adhere to the configurations and guidelines from the original works and codebases of the compared models. The training is conducted on robust computational setup: 4x RTX 3090 GPUs for LLaVA 1.5, 8x V100 GPUs for MiniGPT-4, and 4x A6000 GPUs for InstructBLIP. Each training session lasts approximately 2-4 hours. Hyperparameters including alpha and beta are set to 1.0 and 0.1, respectively, in accordance with the VCD model's specifications.
ones.
6.2 Experimental Results
POPE. To evaluate HIO's capability on object hallucination, we compare with several state-of-the- art Decoding methods on POPE. The results are shown in Tab. I which presents the experimental
8

---

results on the POPE dataset across random, popular, and adversarial settings. Our method consistently outperforms the standard decoding strategy, with average improvements of 6.2% in accuracy and 7.3% in F1 score across all LVLMs. Additionally, our approach clearly surpasses state-of-ther -frte--rrt decoding methods, demonstrating its effectiveness mitigating object hallucinations. The improved performance across random, popular, and adversarial settings further confirms that our HIO method effectively reduces hallucinations diverse scenario.
Dataset
Setting
Decoding
Accuracyt
Precision
Recall
Fl Score
Regular
83.29
92.13
72.80
81.33
VCD
87.73
91.42
72.80
87.16
Random
ICD
89.56
88.71
90.66
89.68
VDD
90.00
97.36
79.13
88.79
Ours
90.21
93.23
86.85
89.94
Regular
81,88
88.93
72.80
80.06
VCD
85,38
86.92
83.28
85.06
MSCOCO
Popular
ICD
86.16
83.18
90.66
86.76
VDD
85.91
94.33
76.33
84.40
Ours
88.12
88.96
86.83
87.84
Regular
78,96
83.06
72.75
77.37
VCD
80.88
79.45
83.29
81.33
Adversarial
ICD
79.71
74.35
90.66
81.70
VDD
83.52
89.34
76.20
82.20
Ours
84.32
84.28
84.33
84.34
Table Results on POPE. Regular decoding denotes direct sampling, whereas VCD refers to Visual Contrastive Decoding method, whereas VDD refers to Visual Debias Decoding. The best performances within each setting are bolded.
CHATR. Beyond the "Yes-or-No" discriminative evaluations conducted on the POPE and MME our model's performance in open-ended caption generation using the CHAIR
datasets, also benchmark. Tab.2 and Tab.5 display results for 500 randomly selected images from the COCO val2017 and val2014 datasets, respectively. These results show consistent improvements our model compared to other methods. Specifically, our approach significantly reduces object hallucinations in generated captions, as evidenced by lower CHAIRS and CHAIRI scores (8. (8.1% reduction in CHAIRS and 4.9% in CHAIRI). Furthermore, it enhances caption detail, as indicated by higher Recall scores. Overall, our method achieves an effective balance between accuracy and detail open-ended caption generation by widening the gap between hallucinated and correct tokens.
Row
Method
Length
CHAIRs 4
CHAIR +
Recall
100.6
50.0
15.4
77.1
VCD
100.4
48.6
14.9
77.3
OPERA
98.6
47.8
14.6
76.8
OPERA (fast)
85.3
48.6
14.5
76.7
ICD
106.3
50.8
15.0
78.5
Ours
110.3
41.4
10.5
77.4
Table 2: Hallucination performance of different methods.
MME. To evaluate HIO's capability on object-level and attribute-level hallucination, we compare with several state-of-the-at Decoding methods on MME. The results shown Tab. Consistent with the performance on POPE and CHAIR, HIO also achieves competitive results on MME compared other decoding methods. Concretely, HIO outperforms the VCD 6.4%, 21.7%, 4. 4.7% and 17.0% at Existence, Count, Position on MME, respectively. The results demonstrate the effectiveness of our method.
6.3 Ablation Study
To verify the effectiveness of each component of the proposed HIO, we conduct ablation studies on Contrary Bradley-Terry (CdMM() Amplification of Multiple Targeted Hallucination(AMMi MTTH) and Advanced Constraints for Inducing(ACI) under the MSCOCO Lin et al. [014]. The results are shown Tab. 4 when constrained by CBTM in Exp the model outperforms the baseline(i.e. Exp 1). This helps LVLM amplify hallucinations. Furthermore, after being integrate with AMTH

---

Object-level
Attribute-level
Model
Decoding
Total Scorest
Existence
Countt
Positiont
Colort
Regular
175.67
124.67
114.00
151.00
565.33
LLV
VCD
184.66
138.33
128.67
133.00
601.66
VDD
190.00
143.33
145.00
165.00
643.33
Ours 190.00
160.00
133.33
170.00
653.33
Table 3: Results on the hallucination subset of MME. Regular decoding denotes direct sampling, VCD denotes Visual Contrastive Decoding method, whereas VDD refers to Visual Debias Decoding. The best performances within each setting bolded.
in Exp 3, LVLM obtain sigificant gains on C and CHAIR,, When integrate with ACI, the LVLM achieve superior performance on CHAIRs, CHAIR, and Recall. These results demonstrate the effective of each component. Moreover, we have enriched the ablation study to analyze the
Exp
CBTM
AMTH
ACI
CHAIR5 4
CHAIR, +
Recallf
1
33.4
9.07
81.1
2
18.6
5.08
79.9
3
14.2
3.06
80.5
11.2
2.02
81.3
Table 4: Ablation study with different components of our model on CHAIR-COCO.
generalization capability of our proposed components unseen categories, as detailed in Table 4. For the Unseen-P dataset, we collected data from MSCOCO, A-OKVQA, and GQA, ensuring no overlap with the training set, resulting in 495 samples across 10 distinet classes. These experiments show that our components generalize effectively to unseen data. Finally, we have integrated the ablation study
into the experimental results section, rather than presenting separately.
Dataset
CBTM
AMTH
ACI
Accuray 1
Precision,
Recall
Fl Score
88.88
84.88
95.63
83.93
89.79
86.22
95.63
90.68
unseen-N
/
91.83
95.30
88.64
91.85
v
/
92.97
91.94
94.75
93.33
81.15
64.86
100.00
78.68
82.61
66.66
100.00
80.02
unseen-P
84.05
72.41
87.51
79.24
85.51
75.01
87.51
80.76
Table 5: Ablation study on the generalization of each component unseen datasets.
7 Discussion
In this study, we conduct an in-depth examination of the principles governing contrast decoding and the prerequisites for its efficacy. Based
findings, introduce HIO, innovative model optimization approach designed to induce hallucinations. This method significantly amplifies hallu- cinatory elements within the model, thereby effectively mitigating them through contrast decoding. Extensive experimentation across various datasets has demonstrated that HIO effectively reduces hallucinations and achieves Kate-of-the-ars performance.
Limitations Future Work.
Our findings establish necessary, but not sufficient, condition for the successful operation of contrast decoding. Further exploration of more effective conditions could significantly enhance the efficiency of contrast decoding in mitigating hallucinations. Additionally, exploring training- -fee methods to induce hallucinations could reduce the computational costs associated with decoding.
Acknowledgments and Disclosure of Funding
This study is supported by grants from the National Natural Science Foundation of China (Grant No. 62122018 8, No. 62020106008, No. U22A2097, No. U23A20315), and Kuaishou, and Naturll Science Foundation of Sichuan Province (Grant No. 20252NSFSC1463,
10

---

References
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican. M. Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-22 237366, 2022.
J. Bai, S. Bai, S. Yang. S. Wang. S. Tan, P. Wang. Lin, C. Zhou, and Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
N. Boulanger-Lewandowsii Y. Bengio, and P. Vincent. Audio chord recognition with recurrent neural networks. In ISMIR, pages 335-340. Curitiba, 2013.
R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Z. Chen, Z. Zhao, H. Luo, H. Yao, B. Li, and J. Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403. 00425, 2024.
Y.S.. Chuang, Y. Xie, H. Luo, Y. Kim, J. Glass, and P. He. Dola: Decoding by contrasting layers improves factuality large language models. arXiv preprint arXiv:2309. 03388383,3 2023.
W. Dai, J. Li, D. Li, A. M. H. Tiong, Zhao, Wang, B. Li, P. Fung, and Hoi. Instruct- blip: Towards general-purposs purpose vasion-language models with instruction tuning. arXiv preprint arXiv:2306.044 04387, 2023.
D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
A. Fan, M. Lewis, and Y. Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Association for Computational Linguistics, 2018.
A. Favero, L. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and S. Soatto. Multi-modal hallucination control by visual information grounding. arXiv preprint arXiv:2403. 14003, 2024.
C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, Z. Qiu, W. Lin, J. Yang, X. Zheng, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. 2023.
A. Gunjal, J. Yin, and E. Bas. Detecting and preventing hallucinations in large vision language models. arXiv preprint arXiv:2308.0m 06394, 2023.
A. Holtzman, Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degencration. arXiv preprint 1904::100..099
2019.
Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu. Opera: Allevi- ating hallucination multi-modal large language models via over-trust penalty and retrospection- allocation. arXiv preprint arXiv:2311. 1,79111,, 2023.
Z. Ji. N. Lee, R. Frieske, T. Yu, D. Su, Xu, Ishii, Y. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
K. Lee, O. Firat, A. Agarwal, C. Fannjiang, and D. Sussillo. Hallucinations in neural machine translation. OpenReview, 2018.
S. Leng, H. Zhang, G. Chen, Li, S. Lu, Miao, and L. Bing. Mitigating object halluci- nations in large vision-language models through visual contrastive decoding. arXiv preprint arXiv:2311.1699 2023.
Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023a.
11

---

X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto. L. Zettlemoyer, and M. Lewis. Con- trastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097 2022.
Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Lin, J. Hilton, and O. Evans, Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0795 2021.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
F. Liu, K. Lin, L. Li, J. Wang, Y. Yacoob, and L. Wang. Aligning large multi-modal model with robust instruction tniigg. arXiv preprint ar3iv:2306. 14565, 2023a.
Liu, K. Lin, L. Li, J. Wang, Y. Yacoob, and L. Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023b.
H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08855 2023c.
H. Lovenia, W. Dai, Cahyawijaya, Z. Ji, and P. Fung. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. arXiv preprint arXiv:2310.05333, 2023,
P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Math Vista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
OpenAl. GPT-4V(ision) system card. 2023.
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko. Object hallucination in image captioning. arXiv preprint 1809.02800.02.. 2018.
Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W.-t Yih. Replug: Retrieval-augmente black-box language models. arXiv preprint arXiv:2301. 2023.
Y. Sun, S. Yuan, X. Wang, L. Gao, and J. Song. Any target can be offense: Adversarial example att.:2227..299 2024.
generation via generalized latent infection. arXiv preprint
Wang, Y. Zhou, G. Xu, P. Shi, C. Zhao, H. Xu, Q Ye, M. Yan, Zhang, J. Zhu, et al. Evaluation and analysis of hallucination in large lasion-language models. arXiv preprint arXiv:2308. 15/26, 2023a.
W. Wang. Q. Lv, W. Yu, W. Hong. Qi, Wang, Ji, Yang. L. Zhao, X. Song, et al. CogVLM: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03077 2023b.
X. Wang. J. Pan, L. Ding, and C. Biemann. Mitigating hallucinations in large vision vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18775. 2024.
Yang, I.. Li, K. Lin, Wang, C.-C. Lin, Z. Liu, and L.. Wang The dawn of L.MMs: Preliminary explorations with (is--V( arXiv preprint arXiv:2309.1742). 9, 2023.
S. Yin, C. Fu, S. Zhao, T. Xu, H. Wang, D. Sui, Y. Shen, K. Li, X. Sun, and E. Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:23100 16045, 2023.
12

---

T. Yu, Y. Yao. H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H.-T. Zheng, M. Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:23/2.0084) 2023.
S. Yuan, Q. Zhang, L. Gao, Y. Cheng, and J. Song. Natural color fool: Towards boosting black-box unrestricted attacks. Advances in Neural Information Processing Systems, 35:7546 7560, 2022.
Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Chen, et al. Siren's 's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 2023.
Y.-F. Zhang, W. Yu, Q. Wen, X. Wang, Z. Zhang, L. Wang, R. Jin, and T. Tan. Debiasing large visual language models. arXiv preprint arXiv:2403.05262, 2024.
Zhao, B. Wang. Ouyang, Dong, Wang, and He. Beyond hallucinations: Enhancing Ivims through hallucination-awar direct preference optimization. arXiv preprint 1..1822311.16839, 2023.
C. Zhou, G. Neubig. J. Gu, M. Diab, P. Guzman, L. Zettlemoyer, and M. Ghazvininejad. Detecting hallucinated content in conditional neural sequence generation. arXiv preprint 2:X102201 2020.
Y. Zhou, C. Cui, J. Yoon, L. Zhang, Z. Deng, C. Finn, M. Bansal, and H. Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754. 2023.
Y. Zhou, C. Cui, R. Rafailov, C. Finn, and H. Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024.
D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision- lanoonl under- standing with advanced large language models. arXiv preprint arXiv:2304. 10592,, 2023.
L. Zhu, D. Ji, T. Chen, P. Xu, J. Ye, and J. Liu. Ibd: Alleviating hallucinations in large vision- models via image-biased decoding. arXiv preprint arXiv:2402. 18476, 2024.
13

---

Appendix
A
Algorithm
The algorithm outlines the process by which the model generates its own series of potential hallucina- tions. Using the sample pairs produced by the model, apply our proposed fallucinationnIIduuce Optimization (HIO) to enhance the distinction between hallucinated and target labels. Ultimately, hallucinations are mitigated through contrastive decoding.
Using paired hallucination and non-halline non-hallucination annotations from the RLHF-V V dataset, we apply beam search generate multiple outputs where hallucination token annotations occur. These outputs include both correct and hallucinated results, which we use as hallucination samples to reinforce the model's confidence in its outputs. The correct annotations from RLHF-V serve ground truth, helping the model avoid hallucinations by differentiating between hallucinated and target tokens. This approach expands the contrast between hallucinated and target tokens, effectively reducing hallucinations.
Algorithm 1 Training LVLM to Amplify Multiple Targeted Hallucination
Require: training image set V; user prompt set X; pair-wise groundtruth descriptions, y hallucination description and y" for correct description; IVLM MM-) with parameters
I: According to each pair's 's hallucination description y and correct description y", get starting subscripts of y' compared with y" Different subscripts denoted as Z {i
2: Initialize the LVLM's parameter 0 and an empty set Snew + {} ,., }.
3: for each image v E y, cach prmopt x E X, the correpsonding hallucinatory description y' E y and correpsonding hallucinatory description y" E y" do 4: Get starting subscripts of y' compared with y" Different subscripts denoted as I' = (11, s b
5: for : E I do
6: #zv, represents the sequence of generated tokens up to the time step (ie 1)
Generate next logits (D.E (o.
(0.2.y (0.Z. !
MMU, M(0,21924)
(4
....I
(U.2.
8: Find Top-K subscripts
arg (1.2.... RIIT IT =k Ejerl
(0,2.0
jik (i1.2. .kk where 5
1/2
2 4
for j: E (1.6:2221 do
y He((+1) -Vz Ui
11:
8 5=1
12:
while V(((t) is not period do
-M
13:
14:
Je +8+1)
Vestt U Uargmin, L
15:
8 8 ++1
16:
end while
17:
end for
18:
end for
19:
end for
14

---

B Mathematical Derivations
In this appendix, we present comprehensive verification of Eqn. (17 which is elucidated through the following detailed procedure:
Am
ai (. - alj to.. Σ ((1 + a) (....00))) - al,
m x ((1 + a)1)
1m
pm a
km
(u...ect ig (vneweet) (1 + a.)
Σ im
(1) ((ooa Ifuse.vecc) >
(18)
km
Am
(4 (1, - (1,a.eec)
(1 ((umaae))) (fuse.Bec))
(1
k1
(o. low. >
C
Visualization
This figure demonstrates the effectiveness of our ACI method (described Section 4.3) The y-axis shows the difference between the hallucination token and the target token. The blue curve represents this difference without ACI, while the orange curve represents it with our proposed ACI. Clearly, our method accurately induces hallucinations, further amplifies the difference between the hallucination token and the target token, and thus effectively reduces hallucinations.
wil ACI
ACI
Figure 3: The Difference between hallucination token and target token. The horizontal axis represents the progression of training steps, while the vertical axis quantifies the disparity in logits, calculated as the hallucination token's 's logits minus those of the target token. It is evident that ACI effectively augments the distinction between the hallucination and target tokens.
D Additional experiments
MME. To evaluate HIO's capability object-level and attribute-level hallucination, we compare it with several -ftte-of-the-art Decoding methods on MME. The results are shown Tab. [3] Consistent with the performance POPE and CHAIR, HIO also achieves competitive results MME compared to other decoding methods. Concretely, HIO outperforms the VCD at Existence, Count, Position, Color, Posters, on MME, respectively. The complete POPE evaluation is shown in the Tab 7
Model
Decoding
Existence
Povition
Color
Celebrity
Scone
Landmark
Percetion
Rogular
175.67
121.67
114.00
151.00
127 82
113.59
148.30
129.95
102.20
92.00
1279.19
LLAVAI.5
VCD
181.66
138.33
128.67
153.00
132
120.94
152.20
140.45
109.60
104.00
1363.96
Ours
190.00
160.00
133.33
170.00
145.50
138.50
158.70
165.00
121.00
142.50
1524.70
Table 6: Results on all MME percoption-related tasks. The hest performance of cach setting is bolded.
15

---

Dataset
Setting
Model
Decoding
Accuracy
Precision
Rocall
PT Soore
Regular
53.23
LLaVAL5
VCD
87.73
91.42
83.28
57. 16
90.21
93.23
86.85
89.94
Regular
67.04
69.06
06.54
67.77
Randow
miniGPT4
VCD
69.60
72.76
66.73
69.62
Ours
77.96
74.15
85,86
79.57
Regular
80.71
81.67
79.19
80.41
InstructBLIP
84.53
79.32
83.65
Ours
87.33
96.12
77.73
85.95
Regular
LLaVAL5
VCD
85.38
56.92
83 28
85, 066
Ours
88.1
86.83
87.84
Regular
60.80
61.34
65.74
63.46
MSCOCO
Popular
miniGPT4
VCD
62.91
63.69
64.81
64.24
Ours
72.51
67.75
85.86
75.74
Regular
78.22
17.87
78, 85
78. 36
InstructBL.IP
70.32
Ouns
84.83
90.59
77.72
83.67
Regular
LLaVAL5
VCD
80.83
79. 79.15
83 29
81.33
Ours
84.32
84,28
84,33
84.34
Regular
59.12
59.62 61
64.15
61.95
Adversorial
miniGPT4
VCD
62.07
02:15
66.76 76
041.37
Regular
75.84
74.30
79.03
76.59
InstructBLIP
VCD
79.33
Ours
82.96
86.82
77.70
82.00
Regular
62.55
LLaVAL5
VCD
86.15
$5.18
87 53
86.31
Ours
90.61
94,97
85.73
99.19
Regular
64.79
65.26
65.73
65, 50
Random
min:GPT4
VCD
66.68
66.47
68.21
67.53
74.74
69.46
Regular
80.91
17.97
86.16
8 .6
Ours
88.56
90.25
86.46
88.32
Regular
79.90
$0.85
78.36
79.59
LLaVAL5
VCD
$1.85
78.60)
87.54
82, 82
Ours
86.93
87.84
85.73
86.77
Regular
60.75
8-1
A-OKVQA
Popular
miniGPT4
VCD
62.22
62.23
08.55
05.24
Regular
76.19
72.16
85, 228
78.17
InstructBLIP
VCD
87.03
Ours
81.16
78.17
86.46
82.11
Regular
74.04
72.08
78.49
75.
LLaVAL5
VCD
74.97
70.01
87.36
77. 73
Ours
80.83
78.08
85.73
82.71
Regular
58.50
Adversorial
miniGPT4
VCD
60.67
60.56
68.47
64.26
Regular
70.71
65.91
85.83 83
73.56
InstructBLIP
VCD
74.33
46
Ours
74.55
69.74
86.46
77.22
Regular
83.73
57.18
70.12
82.995
LLaVAL5
VCD
86.63
84.85
89.24
86.99
Ours
89.06
93.53
83,93
88.47
Regular
65.
Random
miniGPT4
VCD
67.08
68.30
60.04
68.67
Regular
79.65
77.14
$1.29
80. 36
InstructBLIP
VCD
83.09
844
Ours
87.26
89.09
84.93
86.96
Regular
78.17
77.64
79.12
75.37
LLaVAL5
VCD
80.73
76.26
89.24
82.21
Ours
84.76
85.35
83,93
84.63
Regular
57.19
GQA
Popular
miniGPT4
VCD
62.14
61.14
72.26
24
Ours
64.74
60.72
83.28
70.21
Regular
73.87
89.63
$4.63
76.42
IndtuctBLIP
VCD
78.57
74.62
86.61
80.
Ours
77.11
73.42
84,93
78.76
Regular
75.08
13.19
79.16
76.06
ILVVVL
Ours
82.11
80.96
83.93
82.42
Regular
Adversarial
miniGPT4
VCD
57.78
57.70
69 82
18
Ours
59.09
56.11
83.23
67.02
Regular
70.56
66.12
84.33
74.12
InstructBLIP
VCD
75.08
70.59
85.19
77.53
Ours
74.86
70.69
84.93
77.16
Table 7: Results on POPE. Regular decoding denotes direct sampling. Higher accuracy and Fl score indicate better performance and fewer hallucinations. The best performances within each setting are bolded.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have listed our contributions in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have listed our contributions in dicussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
As shown in our proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All code and data are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[Yes]
Justification:
is faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
cessionnooccyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pulli///uuii wwwpc//ggue////eeeeeoooc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes)
Justification: sure
Guidelines: As demonstrated in Fig.3.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
As depicted in Implementation Description.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:/::////iiss/eeeee http://s.....blli/ccttccccsiieaa
Question:
Answer: [Yes]
Justification: The research conducted in the paper conform. in every respect with the NeurlPS Code of Ethics httppp////earrrs..cc///bbbi/eiie
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
As demonstrated in Limitations and Future Works section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, the paper describe safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the creators or original owners of assets (e.g., code, data, models), used in the paper, properly eredited and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, the assets introduced in the paper is well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: yes
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Yes, describe potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "HNbGSCAHp8", "reviewer": "Reviewer_DdM1", "summary": "This paper proposes a method for mitigating hallucinations by training an \"evil\" LLM to provide logits for contrastive decoding. This \"evil\" LLM is trained with a dataset that prefers hallucinated samples over true ones during fine-tuning. The logits from this \"evil\" LLM are then used for contrastive decoding. Experimental results validate the effectiveness of this approach.", "questions": "Sea the weakness part.\n1. Typo: \n- line 127 should be \"probability\" instead of \"probably\". \n- line 233 it should be $\\delta^{*\\\\{v,x,y_{<t}\\\\}}$ instead of $\\delta^{\\prime\\\\{v,x,y_{<t}\\\\}}$", "limitations": "Yes, the author has pointed out the computation cost for this method.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The high-level idea is easy to understand.\n2. The method itself is interesting and novel.", "weaknesses": "1. The writing is sometimes confusing and makes it difficult to understand details. For example, Sec 4.1 and 4.2  use Eqn 17 in Sec 5 as the motivation of how they design the current method and refer to it frequently. However, Eqn 17 is not explained and introduced previously, making it very difficult for the reader to understand why you design this method like this. It would be much better to explain the high level idea of eqn 17 first.\n2. Besides, the figure 1 itself is also confusing. The picture provided is vague and even a human finds it difficult to identify the people and output the true answer.\n3. When reading the Sec 5, it is also very confusing. How do you define the ideal logits for contras decoding? Based on my understanding, it should not equal to the hallucinatory token's logits. However, at the later part line 243, you are referring to it as the hallucinatory tokens. Besides, how do you interpret the second line of Eqn 17? I interpret J as the average logit difference between the hallucinatory token and the true token. However, I find it difficult to interpret the left part of the second line of Eq 17. What is the logic behind line 242? The whole explanation does not make sense to me.\n4. For Eqn 14, is it too strong to require the minimal logit of correct tokens to be larger than the maximum logit of hallucinatory tokens? For greedy docoding, if one of the correct tokens after contrast is larger than all the hallucinatory tokens, then it will be output?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer DdM1\nWe thank all the reviewers for the insightful comments and the recognition of our work   \n**\"Novel Idea\"** (Reviewers #1, #3, #4)  \n**\"Promising Results\"**(Reviewers #2, $3, #4)  \n**\"Interesting\"**(Reviewers #1, #4)  \n**\"Easy to understand\"**(Reviewers #1, #4). Now, we carefully answer your question as follows.  \n\n#### W1: The writing is sometimes confusing and makes it difficult to understand details. For example, Sec 4.1 and 4.2 use Eqn 17 in Sec 5 as the motivation of how they design the current method and refer to it frequently. However, Eqn 17 is not explained and introduced previously, making it very difficult for the reader to understand why you design this method like this. It would be much better to explain the high level idea of eqn 17 first.\n**W1-A1**:Thanks for your suggestion, we have added the descriptions of Eqn.17 in line 141 as below: \n... to induce more potential hallucinations for effective contrast decoding, we propose to amplify multiple hallucination tokens, building on the theoretical foundation presented in Eqn. 17 of Section 4.2. This theory demonstrates that **effective contrastive decoding requires a consistent difference between the logits of potential hallucinated tokens and the correct token**. And Section 4.3 introduces additional constraints ... .\n\n#### W2: Besides, the figure 1 itself is also confusing. The picture provided is vague and even a human finds it difficult to identify the people and output the true answer.\n**W2-A2**: As per your suggestion, **we have clarified the image and enhance its quality** to ensure that the details are more discernible. \n\n#### W3: When reading the Sec 5, it is also very confusing. How do you define the ideal logits for contras decoding? Based on my understanding, it should not equal to the hallucinatory token's logits. However, at the later part line 243, you are referring to it as the hallucinatory tokens. Besides, how do you interpret the second line of Eqn 17? I interpret J as the average logit difference between the hallucinatory token and the true token. However, I find it difficult to interpret the left part of the second line of Eq 17. What is the logic behind line 242? The whole explanation does not make sense to me.\n**W3-A3**:   \n**(a)** The ideal logits for effective Contrastive Decoding, as we defined, **require that the logits of all hallucinated tokens should be lower than those of the correct token**. The rationale behind this is that traditional DPO-based Contrastive Decoding methods, while reducing hallucinations on the preferred hallucinated token, often generate new hallucinated tokens, thereby diminishing the effectiveness of contrastive decoding. Therefore, we assert that the model should simultaneously reduce the logits of multiple potential hallucinated tokens so that they all remain lower than the logits of the correct token, thereby significantly enhancing the effectiveness of Contrastive Decoding in mitigating hallucinations.\n\n**(b)** Eqn.17 illustrates that hallucinations can be effectively eliminated through contrastive decoding if **the difference between the logits of the hallucinatory token and the correct token in the \"evil\" LVLM's output (Left part of Eqn.17) exceeds that in the original LVLM output (J in Eqn.17)**. For example, as depicted in the lower part of Figure 2, where \"Dogs\" is a hallucination and \"Benches\" is the correct label, the hallucination of \"Dogs\" is removed when the difference between the logits for \"Dogs\" and \"Benches\" in the \"evil\" LVLM output surpasses the difference in the original LVLM output. When this condition is met for all potential hallucinations, all hallucinations are effectively eliminated. A more detailed explanation will be provided in line 242 of the revised manuscript.\n\n\n\n#### W4: For Eqn 14, is it too strong to require the minimal logit of correct tokens to be larger than the maximum logit of hallucinatory tokens? For greedy docoding, if one of the correct tokens after contrast is larger than all the hallucinatory tokens, then it will be output?\n**W4-A4**:    \n**(a)** Eqn.14 represents a **theoretical upper bound, which guides us in enhancing the effectiveness of Contrast Decoding method for hallucination elimination** by ensuring that the logits of all hallucinated words are lower than those of the correct words. We further refine this upper bound into a more practical optimization objective, as detailed in Eq.17. Specifically, the induced Evil model must simultaneously maintain lower logits of multiple hallucinated words than that the correct token. To achieve this, we propose the Hallucination-Induced Optimization (HIO) method, which significantly improves the effectiveness of hallucination elimination in contrast decoding.  \n**(b)** If the logit of the correct token after contrast is greater than all hallucinatory tokens, **it would be selected during greedy decoding**.\n\n\n#### Q1: Line 127 should be \"probability\" instead of \"probably\"\n**Q1-A5**: Thank you for highlighting this issue, we have corrected it in our revised manscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "83vfUMADkg", "reviewer": "Reviewer_RdjK", "summary": "This paper focuses on alleviating hallucinations in Large Vision-Language Models. Specifically, the authors introduce a novel optimization strategy named Hallucination-Induced Optimization (HIO). This method amplifies the contrast between hallucinatory and targeted tokens relying on a fine-tuned preference model. Finally, the extensive experimental results verify the effectiveness of HIO, which outperforms state-of-the-art methods across various benchmarks.", "questions": "Please see the weaknesses.", "limitations": "n/a", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of this paper is reasonable. \n2. The experimental results are significant compared to previous works.", "weaknesses": "1. I mainly doubt the novelty. The motivation of this paper is very similar to [1]. Please describe the difference between this work and [1].\n2. This paper is not complete, lacks a Conclusion section, and the ablation study section is too short. Moreover, the implementation details are missing in the paper. Thus, I think the writing of this paper is not ready. \n\n[1] Alleviating Hallucinations of Large Language Models through Induced Hallucinations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### W1: Novelty: motivation is similar to [1]. [1] Alleviating Hallucinations of Large Language Models through Induced Hallucinations\n**W1-A1**: We indeed adopt the same concept of induced hallucination from ICD[1], **but we also identified two key issues of it, i.e., less effective and poor generalizability**.  To address these, we propose our Hallucination-Induced Optimization including three conponents, i.e.,CBTM, AMTH and ACI.\n\n**1.Less Effectiveness：**\n\n**(1)Sentence-Level vs. Token-Level Hallucination Induction**: The Supervised Fine-Tuning (SFT) strategy operates at the sentence level, which **limits its ability to precisely induce specific hallucinated tokens at token-level**, as discussed in Lines 203-207 of Section 4.3. This limitation impairs ICD's ability to effectively amplify the distinction between hallucinated tokens and correct tokens, thus reducing the effectiveness of Contrastive Decoding for hallucination mitigation. To address this issue, we propose a DPO-based hallucination mitigation method, **the CBTM, which precisely induces token-level hallucinations for the desired tokens**.\n\n**(2) Predefined vs. Self-Generated Hallucinations**: ICD [1] relies on hallucination samples generated by ChatGPT, rather than the model's own hallucinations as proposed in our Amplification of Multiple Targeted Hallucinations (AMTH) method (Section 4.2). Since these **externally-generated hallucinations may not accurately reflect the model's actual hallucination states, they can lead to less effective optimization or even result in overfitting**. To address this issue, **AMTH induces the model's own hallucinations, providing a more precise representation of the model's internal states**. As a result, our method more effectively targets and reduces the model's hallucinations. \n**To compare the effectiveness of our Hallucination-Induced Optimization (HIO) with ICD [1], we conduct experiments to assess their performance in mitigating open-ended and discriminative hallucinations**, as detailed in Tables 1 (CHAIR) and 2 (POPE). To implement ICD based on LLaVA, we first fine-tune an Evil-LLaVA-7B model using hallucinated answers generated by ChatGPT. We then apply this fine-tuned model to perform contrastive decoding against the original LLaVA-7B model. As shown in Tables 1 and 2, our method achieves significant improvements over ICD, demonstrating its superior capability in reducing hallucinations. **This enhancement is attributed to HIO's token-level induction, which effectively utilizes hallucinations generated by the model itself**.\n\nTable 1: Comparison with ICD on CHAIR\n|Row|Method|Length|CHAIRS↓|CHAIRI↓|Recall↑| \n|-|-|-|-|-|-|\n|1|ICD|106.3|50.8|15.0|78.5| \n|2|Ours|110.3|**41.4**|**10.5**|77.4| \n\nTable 2: Comparison with ICD on POPE\n|  Dataset   | Setting  | Decoding | Accuracy↑ | Precision | Recall | F1 Score↑ |\n|-|-|-|-|-|-|-| \n| MSCOCO |Random|ICD|89.5|88.7|90.6|89.6|\n|||Ours|**90.2**|**93.2**|86.8|**89.9**|\n||Popular|ICD|86.1|83.1|90.6|86.7|\n|||Ours|**88.1**|**88.9**|86.8|**87.8**|\n||Adversarial|ICD|79.7|74.3|90.6|81.7|\n|||Ours|**84.3**|**84.2**|84.3|**84.3** |\n\n**2.Poor Generability：**\nThe model tends to produce numerours different hallucinations for each visual semantic scene. However, **ICD only considers only one of them annotated by ChatGPT, ignoring other possible hallucinations**. As illustrated in the lower part of Figure 2, where 'People,' 'TV,' and 'Clock' are all hallucinations. For ICD, it can only induce one hallucinated token, such as 'Dog', while ignoring the other potentional hallucinations, such as 'Clock'. Finally, ICD overfis on the annotated hallucinations given by ChatGPT, failing to address most potential hallucinations, and further reducing the model's ability to generalize to unseen hallucinations.\nIn contrast, our proposed Amplification of Multiple Targeted Hallucinations (AMTH) generates multiple potential hallucinations using a beam search mechanism based on their predictive logits, as described in Section 4.2. Since these potential hallucinations can be any word from the 32,000-word dictionary of the LLM, **AMTH demonstrates improved generalizability to unseen objects, thereby more effectively mitigating hallucinations**.\n**To demonstrate the generalization capabilities of our method, we collected two OOD test set as Unseen-N and Unseen-P that the model had not encountered during training for evaluation**.\nUnseen-N contains 426 samples gathered from the web, while Unseen-P includes 69 samples collected from MSCOCO, A-OKVQA, and GQA. This resulted in a total amounts of 495 samples with 10 unseen object classes. \nFrom Tab.3, we observe that our method significantly outperforms ICD[1] across Accuracy, Precision and F1-Score on unseen catagories by a large margin, thereby **proving more powerful generalization capability of our proposed HIO method in comparision to ICD**.\n\nTable 3: Comparison with ICD on unseen datasets.\n|Dataset|Decoding|Accuracy↑|Precision|Recall|F1 Score↑|\n|-|-|-|-|-|-| \n|unseen-N|ICD|88.6|84.5|95.6|89.7|\n||Ours|**92.9**|**91.9**|94.75|**93.3**|\n|unseen-P|ICD|79.7|63.1|100.0|77.4|\n||Ours|**85.5**|**75.0**|87.5|**80.7**|\n\n#### W2: This paper is not complete.\n**W2-A2:** Due to space constraints, the conclusion section was initially included in the supplementary materials. We have incorporated it into the main manuscript in our revised submission. Moreover, **we have enriched the ablation study to analyze the generalization capability of our proposed components to unseen categories, as detailed in Tab.4**. Finally, **we have integrated the ablation study into the experimental results section, rather than presenting it separately**. \n\nTable 4: Ablation study on the generalization of each component on unseen datasets\n|Dataset|CBTM|AMTH|ACI|Accuracy↑|Precision|Recall|F1 Score↑ |\n|-|-|-|-|-|-|-|-| \n|unseen-P||||81.1|64.8|100.0|78.6| \n||✓|||82.6|66.6| **100.0**|80.0|\n||✓|✓||84.0|72.4| 87.5|79.2|\n||✓|✓|✓|**85.5**|**75.0**|87.5|**80.7**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yVXLIb5XSl", "reviewer": "Reviewer_pE7J", "summary": "This manuscript presents a novel perspective on implementing contrastive decoding for hallucination mitigation in large visual language models. In contrast to methods such as image perturbation, it achieves a quasi-min-max optimization by enhancing model hallucination followed by contrastive decoding for multiple hallucination targets. The proposed framework and its improved components, i.e., CBTM, AMTH, and ACI, have solid theoretical and empirical derivations. The overall writing of the paper is commendable, and it has achieved state-of-the-art performance.", "questions": "1. In analogy to adversarial training, the utilization of hallucinatory information during the training dynamics is unclear. The subsection 'Acquisition of Multiple Candidate Hallucinations' seems relevant to this aspect, yet the discussion could be further enriched. \n2. In comparison to the proposed method, is it feasible to directly calibrate the hallucinatory and targeted logits and achieve both efficiency and effectiveness? \n3. Please carefully revise the writing details and enrich the experimental descriptions. I believe this will significantly enhance the clarity and soundness of the paper.", "limitations": "The authors have noted the additional overhead introduced by the proposed method during model training and inference. The potential social impact is positive, yet the discussion on this is lacking. The reviewer thinks that the current limitations in relevant datasets may also constrain the method's development. How to seek the annotation of hallucination tokens during dynamic optimization merits consideration.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors provide a fresh perspective on the overall uncertainty in visual input as well as the exploitation of multiple hallucinatory targets, demonstrating notable gains. They observe and deduce the limitations of the existing optimization frameworks and offer effective solutions.", "weaknesses": "1. The description of experimental details is weak. For instance, the experimental setups for Figure 1, Table 2, and Table 5 are not clearly stated, yet the results vary significantly.\n2. The manuscript fails to provide the results of the three models purportedly studied.\n3. The training details, time and hardware expenditure, and the selection of the alpha parameter, among many other experimental details for the Evil Model, have not been provided.\n4. While the overall writing is satisfactory, many details betray a rather hasty preparation of the article, such as the repeated occurrences of acronym definitions, entirely duplicated result descriptions, missing spaces before parentheses in several instances, inconsistent terminologies used for metrics, etc.\n5. There appears to be an error in Figure 2: 'benches' should be the correct category?\n6. Inappropriate caption for Table 4.\n7. Inappropriate bolding for Tables 1 and 3.\n8. A Conclusion section is missing from the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### W1: The description of experimental details is weak.\n**W1-A1:** Based on your suggestions, **we have revised the descriptions of the experiments in Tab.1, Tab.2, and Tab.5 as follows**:\nTable 1 presents the experimental results on the POPE dataset across random, popular, and adversarial settings. Our method consistently outperforms state-of-the-art decoding methods across three settings, demonstrating its effectiveness in mitigating object hallucinations in diverse scenarios.\nTables 2 and 5: We assess our model's performance in open-ended caption generation by evaluating 500 images from COCO val2017 and val2014 in Tab.2 and Tab.5. We observed that our method significantly reduces object hallucinations in generated captions with lower CHAIRS and CHAIRI scores, while enhancing caption detail, as indicated by higher Recall scores. It demonstrates that our method can achieve an effective balance between accuracy and detail in open-ended caption generation by widening the gap between hallucinated and correct tokens.\n\n#### W2: Lack results of the three models.\n**W2-A2:** We have **included the results for the LLava, MiniGPT-4, and InstructBLIP models on the MSCOCO, A-OKVQA, and GQA datasets in Tab.1**. \n\nTable 1:Results of the three models on MSCOCO, A-OKVQA, and GQA\n|Dataset| Setting  | Decoding | Accuracy↑ | Precision | Recall | F1 Score↑ |\n|-|-|-|-|-|-|-|\n|MSCOCO|LLaVA1.5|Regular|83.2| 92.1|72.8|81.3| \n||| VCD |87.7|91.4|83.2|87.1| \n|||Ours |**90.2* |  **93.23** |**86.8**|**89.9**| \n||miniGPT4|Regular|67.1|69.1|66.5|67.7| \n|||VCD|69.6|72.7|66.7|69.6| \n|||Ours|**77.9**|**74.1**|**85.8**|**79.5**| \n||InstructBLIP|Regular|80.7|81.6|79.1|80.4| \n||| VCD|84.5|88.5|**79.3**|83.6| \n|||Ours |**87.3**| **96.1**|77.7|**85.9**| \n|A-OKVQA|LLaVA1.5|Regular|83.4|87.2|78.3|82.5| \n|||VCD|86.1|85.1|**87.5**|86.3| \n|||Ours|**90.6**|**94.9**|85.7|**90.1**| \n|| miniGPT4|Regular|64.7|65.2|65.7|65.5| \n|||VCD|66.6|66.4|68.2|67.3| \n|||Ours|**74.7**|**69.4**|**88.1**|**77.6**| \n||InstructBLIP| Regular|80.9|77.9|86.1|81.8| \n|||VCD|84.1|82.2|**87.0**|84.5| \n|||Ours|**88.5**|**90.2**|86.4|**88.3**| \n|GQA|LLaVA1.5|Regular|83.7|87.1|79.1|82.9| \n|||VCD|86.6|84.8|**89.2**|86.9| \n|||Ours| **89.0**|**93.5**|83.9|**88.4**| \n||miniGPT4|Regular |65.1|65.3|66.7|66.0| \n|||VCD|67.0|68.3|69.0|68.6| \n|||Ours|**73.8**|**70.0**|**83.2**|**76.0**| \n||InstructBLIP|Regular|79.6|77.1|84.2| 80.5| \n|||VCD|83.6|81.8|**86.6**|84.1| \n|||Ours|**87.2** |**89.0** |84.9| **86.9**| \n\n#### W3: Lack detailed implementation.\n**W3-A3:**  **We provide an detailed description of our implementation as follows**:\nImplementation Description: We evaluate our HIO by incorporating it into three LVLMs: LLaVA 1.5, InstructBLIP, and MiniGPT-4. For decoding, we use Llama-7B and Vicuna-7B as the linguistic decoder for LLaVA and InstructBLIP/MiniGPT-4. To ensure a fair and rigorous comparison, we adhere to the configurations and guidelines from the original works and codebases of the compared models. The training is conducted on 4x RTX 3090 GPUs for LLaVA 1.5, 8x V100 GPUs for MiniGPT-4, and 4x A6000 GPUs for InstructBLIP. Each training session lasts approximately 2-4 hours. Hyperparameters including alpha and beta are set to 1.0 and 0.1, in accordance with the VCD model's specifications.\n\n\n#### W4: Details betray a rather hasty preparation of the article.\n**W4-A4:** We have revised all your concern based on your suggestions. \n\n#### W5: Error in Figure 2.\n**W5-A5:** We have fixed this issue in our revised manscript.\n\n#### W6: Inappropriate caption for Table 4.\n**W6-A6:** Based on your suggestion, we have revised the caption for Table 4. \n**Table.4: Ablation study with different components of our model on CHAIR-COCO.**\n\n#### W7: Inappropriate bolding for Tables 1 and 3.\n**W7-A7:** We have reviewed the formatting and make sure that the bolding is applied consistently and appropriately. \n\n#### W8: Conclusion section is missing.\n**W8-A8:** Due to the length of the initial submission, the conclusion was placed in the supplementary materials. In the revised manuscript, we have included the conclusion in the main text.\n\n#### Q1: Utilization of hallucinatory information is unclear.\n**Q1-A9:** The following discussion has been incorporated into our revised manuscript:\nWe **utilize hallucinatory information—multiple potential hallucinated tokens—to sharpen the contrast between hallucinated and correct tokens, thereby enhancing the effectiveness of contrastive decoding**. The rationale is that while the **single-target DPO mechanism addresses a specific hallucinated token, it may inadvertently trigger other potential hallucinations**. For example, as shown in the lower half of Fig. 2, single-target DPO can correct the hallucination of \"Dogs\" but may introduce \"Tracks\" as a new hallucination with the second-highest confidence. To strengthen the contrastive decoding process, we generate multiple potential hallucinated tokens and ensure that the model considers them simultaneously. We then apply the HIO mechanism for adversarial training between the correct token and the generated hallucinations, creating an \"evil\" model that favors these hallucinated tokens. **This approach significantly improves the effectiveness of contrastive decoding, as the contrastive logits of the correct token surpass those of all hallucinated tokens, a fact theoretically validated in Sec. 5**.\n\n#### Q2: Efficiency and effectiveness.\n**Q2-A10:** Unfortunately, **achieving a trade-off between efficiency and effectiveness remains challenging**. As noted in [1], using pretrained models with lower capacities (e.g., BLIP2-7B) can significantly reduce training costs but results in decreased effectiveness in mitigating hallucinations.\n[1] Contrastive Decoding: Open-ended Text Generation as Optimization\n\n#### Q3: Revise the writing details and experimental descriptions.\n**Q3-A11:** We have revised writing details and enriched experimental descriptions in Tab.1, Tab.2 and Tab.5 of Sec.6.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qiGZT5fkeI", "reviewer": "Reviewer_6ngE", "summary": "This paper addresses object hallucination in large visual language models (LVLMs), a common phenomenon that such models generate texts not consistent with images. A viable approach for this issue is contrastive decoding. By comparing logits derived from images and distorted images, visual contrastive decoding reduces statistical bias and uni-modal priors thus alleviates hallucination. This paper points out that distorting images induces uncertainty which leads to unexpected results. To overcome this issue, this paper proposes to train an evil LVLM first by direct preference optimization, as illustrated in Figure 2, and contrast outputs from this model with LVLM responses. Theoretical foundations have been given and experimental results show that such strategy alleviates object hallucinations on multiple benchmarks, including POPE and CHAIR, which are commonly used metrics in this field.", "questions": "- For Table 1 where POPE is evaluated, I notice that previous approach VCD has three evaluation setups, including MSCOCO, GQA and A-OKVQA. Only MSCOCO is given. Authors are suggested to include the other two as well for more comprehensive evaluations.\n- Typos. In line 135, An overview of the proposed HIO method is shown in Fig. 3 (3 should be 2).", "limitations": "Authors have indicated limitations in Appendix D Discussion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method is simple and easy to follow.\n- Contrastive decoding is to contrast outputs from a model with an amateur model. It is an interesting perspective to train an amateur model first using DPO approach.\n- Writing and presentations are clear.\n- The performance gains against previous methods look good.", "weaknesses": "- **Generalization issue**. In Figure 2, if I understand correctly, this paper induces an LVLM to generate hallucinated outputs (which is referred to as Hallucination Induction Optimization). Both hallucinated and corrected responses are involved in this Figure. If thinking more over objects (e.g., people, tv, dog, clock in Figure 2), a potential issue for this is, trained model in such manner can only be applied to these objects. In proposed hallucination-induced optimization, an LVLM is taught to hallucinate on some objects, these objects can work for later contrastive decoding. But I doubt that such capability can generalize to other objects which are not seen during training, making this a close-vocabulary approach, just like post-hoc correction methods in this field.\n- Given that proposed hallucination-induced optimization resembles RLHF methods [A] in technical pipeline. Authors are suggested to include some results comparing these methods. Whether an amateur model can help an RLHF model seems an interesting question and may improve this paper.\n\n[A] Aligning Large Multimodal Models with Factually Augmented RLHF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 6ngE\nWe thank all the reviewers for the insightful comments and the recognition of our work \n\n**\"Novel Idea\"** (Reviewers #1, #3, #4)  \n**\"Promising Results\"**(Reviewers #2, $3, #4)  \n**\"Interesting\"**(Reviewers #1, #4)  \n**\"Easy to understand\"**(Reviewers #1, #4). Now, we carefully answer your question as follows. \n\n#### W1: Generalization issue. \n**W1-A1:**\nOur proposed Hallucination-Induced Optimization (HIO) strategy **significantly enhances the generalizability of hallucination mitigation**. Specifically, HIO induces multiple potential hallucinations (as claimed in Sec.4.2) for the current token using a beam search mechanism based on their predictive logits. Given that these potential hallucinations can be any word within a 32,000-word dictionary in the LLM, HIO theoretically has the capacity to generalize to unseen objects, effectively mitigating hallucinations.\nFuthermoe, to validate the generalizability of our method, we collected samples containing out-of-distribution object categories and evaluated the model's performance on these unseen classes. The test sets, Unseen-N and Unseen-P, consist of 426 images crawled from the Internet and 69 images sourced from MSCOCO, A-OKVQA, and GQA, with no overlap with the model's training set. These test sets include a total of 10 unseen categories. The evaluation results, shown in Table 1, demonstrate that our method outperforms the baseline model (i.e., Regular), highlighting its superior generalizability to open-vocabulary objects.\n\nTable 1: Generalization experiments of our method on two unseen datasets\n|  Dataset  | Decoding | Accuracy↑ | Precision | Recall | F1 Score↑ |\n|  ---- | ---- | ----  | ----  | ---- | ----  | \n| unseen-N   |  Regular baseline | 88.88 | 84.88 | **95.63** | 83.93 |\n| |   Ours  | **92.97** | **91.94** | 94.75 | **93.33** |\n| unseen-P  |  Regular baseline | 81.15 | 64.86 | **100.00** | 78.68 |\n|   |   Ours   | **85.51** | **75.01** | 87.51 |  **80.76** |\n\n#### W2: Authors are suggested to include some results comparing these methods. Whether an amateur model can help an RLHF model seems an interesting question and may improve this paper.\n[A] Aligning Large Multimodal Models with Factually Augmented RLHF.\n\n**W2-A2:**\nFollowing your suggestion, we have included experimental comparisons with method [A] under the POPE benchmark. As shown in Table 2, **our method consistently outperforms LLaVA-RLHF7B across three settings in POPE**, demonstrating the effectiveness of our approach in mitigating hallucinations. Furthermore, we believe **our method can help RLHF models generate multiple hallucinated answers with minimal effort**, thereby augmenting method [A] with high-quality instruction-tuning data for fine-grained RLHF alignment. We plan to explore this in future work.\n\nTable 2:Comparative experiments between our method and the RLHF method on POPE\n|  Model  | Random |   |  |Popular  |   | | Adversarial  |   | | Overall  |   |\n|  ----   | ----   | ----   | ----   | ----  | ----  | ----  | ----  | ----   |----   |----   |----   |\n|  - | Acc↑  |  F1↑ | Yes (%)  | Acc↑ |  F1↑ |  Yes (%)  | Acc↑ |  F1↑ |  Yes (%) |  F1↑ |  Yes (%) | \n| Shikra  | 86.9  | 86.2  | 43.3 |  84.0  | 83.2  | 45.2 |  83.1 |  82.5  | 46.5 |  84.0 |  45.0 | \n| mPLUG-Owl7B |  54.0  | 68.4 |  95.6 |  50.9 |  66.9 |  98.6  | 50.7 |  66.8 |  98.7 |  67.2  | 97.6|\n| LLaVA-SFT+7B | 86.1 |  85.5 |  44.5 |  82.9 |  82.4 |  47.2 |  80.2 |  80.1 |  49.6 |  82.7 |  47.1 | \n| LLaVA-RLHF7B  | 84.8 | 83.3 |  39.6 |  83.3  | 81.8  | 41.8  | 80.7  | 79.5  | 44.0  | 81.5 |  41.8 | \n| LLaVA+Ours7B  | **90.2** | **89.9** |  58.2 |  **88.1**  | **86.8**  | 48.7  | **84.3**  | **84.3**  | 51.5  | **87.5** |  52.8 | \n\n\n#### Q1:Authors are suggested to include the other two as well for more comprehensive evaluations.\n**Q1-A3:** In response to your suggestion, **we conducted additional experiments on the MSCOCO, GQA, and A-OKVQA datasets for a more comprehensive evaluation**. The results, presented in Table 3, show that our method consistently outperforms VCD across all three setups, demonstrating its effectiveness in mitigating hallucinations in LVLMs.\n\nTable 3: Results of our method on MSCOCO, A-OKVQA, and GQA\n|  Dataset   | Setting  | Decoding | Accuracy↑ | Precision | Recall | F1 Score↑ |\n|  ----  | ----  | ----  | ----  | ----  | ----  | ----  | \n| MSCOCO  |Random    |  Regular | 83.29 | 92.13 | 72.80 | 81.33 |\n|  | |   VCD     | 87.73 | 91.42 | 83.28 | 87.16 |\n|   |  |   Ours     | **90.21** | **93.23** | **86.85** | **89.94** |\n|  |Popular      |  Regular | 81.88 | 88.93 | 72.80 | 80.06 |\n|  |    |   VCD     | 85.38 | 86.92 | 83.28 | 85.06 |\n|   |    |   Ours     | **88.13** | **88.96** | **86.83** | **87.84** |\n|   |Adversarial|  Regular | 78.96 | 83.06 | 72.75 | 77.57 |\n|  |   |   VCD     | 80.88 | 79.45 | 83.29 | 81.33 |\n| |   |   Ours     | **84.32** | **84.28** | **84.33** | **84.34** |\n| A-OKVQA  |Random |  Regular | 83.45 | 87.24 | 78.36 | 82.56 |\n|   |  |   VCD  | 86.15 | 85.18 | **87.53** | 86.34 |\n| |   |   Ours   | **90.61** | **94.97** | 85.73 | **90.19** |\n| |Popular    |  Regular | 79.90 | 80.85 | 78.36 | 79.59 |\n|||   VCD   | 81.85 | 78.60 | **87.5**3 | 82.82 |\n|||   Ours     | **86.93** | **87.84** | 85.73 | **86.77** |\n||Adversarial|  Regular | 74.04 | 72.08 | **78.49** | 75.15 |\n|||VCD | 74.97 | 70.01 | **87.36** | 77.73 |\n|||Ours | **80.83** | 78.08 | 85.73 |**82.71** |\n|GQA|Random    |  Regular | 83.73 | 87.16 | 79.12 | 82.95 |\n|||VCD| 86.65 | 84.85 | **89.24** | 86.99|\n|||Ours| **89.06** | **93.53** | 83.93 | **88.47**|\n||Popular |  Regular | 78.17| 77.64| 79.12| 78.37|\n|||VCD | 80.73| 76.26| **89.24**| 82.24|\n|||Ours| **84.76**| **85.35** | 83.93| **84.63**|\n||Adversarial|  Regular | 75.08| 73.19| 79.16 76.06|\n|||VCD| 76.09| 70.83 | **88.75**| 78.78|\n|||Ours| **82.11** | **80.96**| 83.93| **82.42**|\n\n#### Q2: Typos. In line 135.\n**Q2-A4:** Thank you for highlighting the issue. We have addressed it in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for the insightful comments and the recognition of our work.

**"Novel Idea"** (Reviewers #1, #3, #4)  
**"Promising Results"**(Reviewers #2, $3, #4)  
**"Interesting"**(Reviewers #1, #4)  
**"Easy to understand"**(Reviewers #1, #4). 


**Summary of Strengths:**

**R1-S1.** The high-level idea is **easy to understand**.   
**R1-S2.** The method itself is **interesting and novel**.  
**R2-S1.** The motivation of this paper is **reasonable**.   
**R2-S2.** The experimental results are **significant compared to previous works**.  
**R3-S1.** The work provides a **fresh perspective** to exploit hallucinatory mitigation with **notable gains**.  
**R4-S1.** The method is simple and **easy to follow**.    
**R4-S2.** It is an **interesting** perspective to train an amateur model first using DPO approach.   
**R4-S3.** Writing and presentations are **clear**.   
**R4-S4.** The performance gains against previous methods **look good**.   


We have checked the manuscript carefully and made **all necessary revisions** strictly following the kind suggestions. Next, we answer your concerns **point-by-point**.

Meta-review: This work deals with hallucination in large visual language models, which is a common and important problem. It provides a novel perspective on the overall uncertainty of the visual input and the exploitation of multiple hallucinatory targets. The authors have observed and mitigated the limitations of existing optimisation frameworks and have provided an effective solution. Due to the strengths of the work, three reviewers recommend acceptance. Meanwhile, one reviewer has doubts about the novelty of this work, and for this, the authors have provided detailed response showing the contributions of this work. AC thus recommends acceptance.
Decision: Accept (poster)