CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models
Authors: Junho Kim
Hyunjun Kim
Yeonju Kim
Yong Man Ro
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Vision-Language Models
Large Multi-modal Models
Hallucination
Primary area: machine_vision
TL;DR: We introduce CODE, a decoding method to reduce hallucinations in Large Multi-modal Models by using self-generated descriptions to improve response accuracy and cross-modal consistency.

Abstract: Large Multi-modal Models (LMMs) have recently demonstrated remarkable abilities in visual context understanding and coherent response generation. However, alongside these advancements, the issue of hallucinations has emerged as a significant challenge, producing erroneous responses that are unrelated to the visual contents. In this paper, we introduce a novel contrastive-based decoding method, COuntering DEscription Contrastive Decoding (CODE), which leverages self-generated descriptions as contrasting references during the decoding phase of LMMs to address hallucination issues. CODE utilizes the comprehensive descriptions from model itself as visual counterpart to correct and improve response alignment with actual visual content. By dynamically adjusting the information flow and distribution of next-token predictions in the LMM's vocabulary, CODE enhances the coherence and informativeness of generated responses. Extensive experiments demonstrate that our method significantly reduces hallucinations and improves cross-modal consistency across various benchmarks and cutting-edge LMMs. Our method provides a simple yet effective decoding strategy that can be integrated to existing LMM frameworks without additional training.

CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models
Junho Kim
Hyun Jun Kim
Yeon Ju Kim
Yong Man Rot
Integrated Vision and Language Lab, KAIST, South Korea farkimjh, kimhj709, yeonju7 kim, ymro) ac.
Abstract
Large Multi-modal Models (LMMs) have recently demonstrated remarkable abili- ties in visual context understanding and coherent response generation. However, alongside these advancements, the issue of hallucinations has emerged as a sig- nificant challenge, producing erroneous responses that unrelated to the visual contents. In this paper, we introduce a novel contrastive-baser decoding method, COuntering DEscription Contrastive Decoding (CODE), which leverages self- generated descriptions as contrasting references during the decoding phase of LMMs to address hallucination issues. CODE utilizes the comprehensive descrip- tions from model itself as visual counterpart to correct and improve response alignment with actual visual content. By dynamically adjusting the information flow and distribution of next-token predictions in the LMM's vocabulary, CODE enhances the coherence and informativeness of generated responses. Extensive experiments demonstrate that our method significantly reduces hallucinations and improves pross-modal consistency across various benchmarks and autting-edge LMMs. Our method provides simple yet effective decoding strategy that can be integrated to existing L.MM frameworks without additional training.
Introduction
With recent advancements of Large Language Models (LLMs) 18 54 11 46 Large Multi-modal
Models (LMMs), sometimes referred Large Vision-Language Models 24
31 have been
drawn great attention for their natural multi-modal interaction with users through back-and-fortt conversations. Leveraging their robust generation capabilities, various pioneering tasks PMe-LMM era such as image captioning
7 58 27), visual question answering [1, 21, object detection [17]. etc., have been integrated into single task rather than treated as sub-tasks and achieved significant
milestones [1 59 44) However, at the same time, the hallucination issue [49 67 has become one
of the emerging problems when adopting LMMs into real-world applications due to their potential spurious generation in critical areas.
Here, unlike hallucination studies LLMs (22 mainly focusing on factuality hallucination originated from the language knowledge, the hallucination problem in LMMs refers cross-modal inconsistency between the given visual contents and the generated responses for the user instructions. After the seminal works 13
42 68
giving eyes to LLMs to understand visual contents with visual instruction
tuning, numerous cutting-edge LMMS 61
36 actively have been proposed. Albeit the scaling
laws following more stronger versatile vision models [37 29 33 higher resolution 41 g deeper
alignment layers 5 45 g) larger model sizes, etc., LMMs still suffer from generating responses that
seem plausible but are factually incorrect for the given visual contents.
"Equal contribution. Corresponding author. Code is available at https:///iyy wwt.:/iwww..iiiiiCCCCCC
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Visual
CODE: COuntering DEscription Contrastive Decoding
Ongoing Resporse: the photo, the dog
What happering to dog's face this photo?
Vision
+
logito(V, (,
logita (1) y.z) agblogite
catching
canching
<image>
playtig
playing
Large
jumping
jumping
Multimodal Models M
In the image, there is a light brown dog in the foreground, standing grassy area. dog appears be in mid artion, possibly äfter catching frisbec, as has purple frisbee mouth...
Tokenber & Emb
contrast
Continuations
Instruction
catciing
Regular Decoding:
Eatching purple fribee lss mouth..
playing
Jump
Our Decoding:
Comprehensive Description:d [Self-een Task]: Detailed Description d= Mg(M)
(d.
purple Frisbee.
The overall decoding procedure of CODE. After LMMs generate detailed description for the visual content by themselves (see instruction in Appendix. A the model recursively outputs logits from each u and d. By contrasting between two log-likelihoods, CODE produces more contextual and correct responses that match the given visual content suppressing inconsistent words (catching->le -iit).
Figure I:
The origin of LMM hallucination is intertwined problem for their inherent training paradigm, which involves alignment projection matching during the pre-training, followed by fine-tuning with the limited instucctionfoolloiing data. Several approaches, aimed for mitigating hallucinatory effects, have addressed the inconsistent issues in the context of data-associated solution [39 (56).
scaling model architectures [65], or additional RL-based training [64 (52). Among them, reactive methods 23 14, 26 intervene the decoding phase of LMMs' inference and alleviate undesired
responses. Motivated by Li et al. [34] that have proposed contrastive decoding (CD) method between expert and amateur language models, recent CD-based approaches in LMMs have proposed several ways of contrasting model responses from visual inputs with their counterparts (e.g.. visual contamination 30. amage-biased models [69], or fine- the-grained visual information [IO]..
Our research question begins with "How effectively do contemporary LMMs capture visual evidences in their descriptive responses, and what information must be curbed to produce informative and consistent responses?". As illustrated in Fig [ (bottom-left), when asking LMMs generate comprehensive description for visual content, the output seemingly generates detailed description effectively, but closer examination often reveals missed fined-grained information hallucinatory instances in the responses. By recursively referring these incomplete descriptions generated by the models themselves, we aim restrict the incorrect information flow during the generation phase and enhance the alignment of the model responses grounded in true visual evidences.
In this paper, we introduce novel training-free contrastive decoding method, COuntering DEscription Contrastvie Decoding (CODE), designed to use self eeff-genrrated descriptions as a contrastive reference to mitigate hallucination issues in LMMs. The core idea of our proposed method is on harnessing the self-generated descriptions which possibly encompass both factual evidence and hallucinatory information from visual contents look-up reference for response correction. Specifically, within our contrastive framework as illustrated in Fig. [ the comprehensive descriptions from model itself alternatively propagate visual input tokens and contrast the discrepancy with the logits from actual visual contents to enhance next-token prediction. In addition, we introduce a dynamic restriction strategy that enables adaptive control of information flow during the auto-regressive decoding phase, laking into account both token-level predictions and their distribution within the vocabulary set.
By conducting extensive experiments and analyses on prevailing cutting-edy LMMs [40 51 16 41 62 S) we corroborate the effectiveness of our method in reducing hallucination and enhancing the coherence and informativeness in various benchmarks [35 42 (22 53 (57. Our decoding method can be seamlessly integrated into existing LMMs by simply substituting the image tokens with self- eeff-generated deseriptions in training-free manner.
Our contribution can be summarized into three-fold as follows:
We introduce COuntering DEscription Contrastive Decoding (CODE), a training-free decod- ing strategy that employs self-generated descriptions to minimize hallucinations in LMMs.

---

By contrasting logit information from descriptions with actual visual contents, CODE enhances visual consistency and coherence in the model responses.
Our approach incorporates dynamic restriction strategies within the contrastive decoding phase. It selectively regulates the information flow by adjusting token-level predictions based on their distribution in the vocabulary, thus ensuring more contextual responses.
We validate the effectiveness of our decoding method across various benchmarks using cutting-edge LMMs. The results demonstrate that CODE significantly reduces hallucination while enhancing the relevance and informativeness in the responses.
2 Related Work
Vision+I.LM: Large Multi-modal Models After the emergence of large-scaled L.L.Ms
4
54
that can interact with users with question-answer chat, various of vision+LLM studies ie.. L.MSs, have been proposed to integrate the robust linguistic capability into the visual understanding and reasoning in diverse vision-language task. As earlier works such LLaVA 2 Instruct-BLIP [3], and MiniGPT-4 168), which utilized visual instruction tuning, have bridged two modalities of vision and language through fine-tuning with learnable query, exemplified by Q-Former [31 projection layer-based alignments [40]. To enhance cross-modal consistency in vision-language representation, recent works have proposed several solutions to address the underlying weaknesses both modalities: (i) utilizing
visual inputs [1 91. (ii) deploying Mixture-of Mfxturr-of-Exppet (MoE) concepts
integrating versatile vision models 377 29 33), (ii) improving weak alignment interface [ 45
or (iv) adopting larger L.MSs to scale up the language model prior 622
Hallucination Issue, Harming Cross-modal Consistency Despite of the endeavor developments of LMMs, they cannot be free from moosl inconsistency between the visual contents and their generated responses, so-called hallucination 44). This not only leads to performance degradation but also provokes an over-reliance issue, resulting in incorrect model responses that are not grounded in true visual evidence, This critical concern regarding response trustworthiness and model reliability hinders the adoption of LMMs in real-world applications. To mitigate the hallucination problem, diverse works have been proposed employing additional training on curated datasets [52] 555 or
reinforcement learning under feedback systems 164| 666. Among them, by intervening during the response generation, decoding-basee approaches [34 12] are introduced to encourage models to represent more precise responses. We refer readers for more comprehensive survey papers [3]
addressing hallucination in LMMs. Our work is in line with CD-based approaches that utilize logit discrepancy from counterpart outputs enhance coherence. Unlike the previous works [30 23 [0] that focus on twisting visual information, we utilize self- -enerrend deseription as contrasting visual counterpart and correct hallucinatory responses based on the model understanding.
3 Proposed Method
Problem Setup and Preliminaries. Let Ma denote a vision-language LMM parameterized by that auto-regressively generates responses for the given visual contents u and input textual query Then the model maps the logit distribution to the next token prediction output yi E IR V at time step t in the vocabulary set V such that Y/2 ~ (V ju. x. V<t) x logit, (gee v, x, Vee), where y<c indicates all previously generated tokens. During the response generation, strategies to choose the next word using either deterministic search (e.g,, greedy, beam search) or stochastic sampling (e.g., top-k, Nucleus search 20
deploy several decoding
After the seminal works (CD) mechanism, which considering information disparities between expert and amateur models for more coherence and informativeness, various works have deployed this strategy into LMMs by twisting visual contents [30 or model information [69] for the contrastive approach. The next-token probability Per from CD can be generally formulated as follows:
in natural language processing have introduced Contrastive Decoding
Pea(yr Vee))= Softmax [(1 + logitt (yr u, Mee) alogitz 6, Mc)))
where 5 and indicates visual counterparts and sub-optimal amateur model, respectively- note that θ= can be regarded self-correction Intuitively, the objective of CD is amplifying model outputs
3

---

by reflecting information deviation between top candidate log- prg--abbbbliiies. Therefore, the selection of logit counterparts for referring is the key challenge for high-quality and consistent responses during the contrastive decoding frameworks.
3.1 Comprehensive Image Description as Visual Counterpart
As the visual counterpart, we deploy comprehensive image descriptions generated from the model as contrasting reference, which are inevitably less informative than the visual contents themselves. Our motivation is on the innate difference of information density between vision and language [9] While vision information exhibits relatively less redundancy for spatial signals- e.g., human can visually recognize objects with few masked patches on images, languages contain high-entropy information, resulting in spans that are more semantic and information-denss than visual signals. That is, it is difficult to infer blanked-our words- e.g., "I went to the store to buy some __.. " Building upon the property of each modality, we first delve into the self-generated model responses with specific instruction for the visual contents to elicit the encompassed visual evidences in the representation space and assess its potential subject role visual counterpart for contrastive decoding.
As illustrated in Fig. I we input query instruction into the model to generate a comprehensive visual description for the given visual content (please see the detailed instruc- tion zo for the self-generate description in Appendix. (A) Then, we exploit the generated description as recursive visual inputs, replacing the position of image tokens in the model input sequence (e.g., <image> token in LLaVA
Evaluationn Soure
LLaV A-Bach
series 42 40 41). Ideally, if the model sufficiently cov- ers whole visual evidences to answer any vision-related questions, the generated response should provide com- petent answers with solely utilizing the description-only embeddings as alternative of visual embeddings. How- ever, as shown in Fig 2 it is obvious that the results from description-only show sub-optimal performance to answer questions due to insufficient information in capturing vi- sual evidences. Consequently, we use the comprehensive description for the visual contents, which is generated by model itself- but partially incorrect or hallucinatory to capture visual evidences, as a contrasting visual counter- part enhance response coherence during the decoding phase, which also in line with the amateur model selection philosophy of contrastive decoding [34].
ELAVA-NAT IntvraVi-135
HLAVA-NAT Te
Description-only
Original Model
Figure 2: The comparison based on two benchmarks (MMVP [33] multiple choice / VAL.VA-Bende 44]: description- level). The plain and dotted bars indicate the results for the models that use self- generated descriptions as visual input re- placements and original model with ac- tual visual contents, respectively.
3.2 COuntering DEseription Contrastive Decoding
Based on our analysis in sec. 3.1 we can obtain pair of the visual content and its comprehensive
description (v, d). such that d corresponds to Mo(yle, v. 20). By contrasting the logit variation between the paired information into the model response generation, we can formulate the next-word prediction using our proposed method, COuntering DEscription Contrastive Decoding (CODE):
Peode(Bf Vee) Softmax [(1 + ar)logity [,,,,<e) aglogit, (v+ d, ar, Uct)]
(2)
that restrict the logit variations with fixed value as in [ we present a dynamic restriction a for the logit variations by comparing the information between visual contents and their comprchensive descriptions. Revisiting the role of it determines whether to promote or curb information from logit variation, thus directly influencing next-token generation- higher value results in more aggressive adjustment for the variations. However, when confronting that both u and yield similar logit score on the correct token, the variation gets closer to zero, thereby the next-token prediction than the correct token with a fixed a on a token-by-toke basis. Although this aligns with the initial intent of CD, a more robust selector is necessary to effectively restrict the logit information flow.
Here, unlike the previous approaches 34 30 2
be unexpectedly reversed if other tokens get rewarded
Accordingly, our method predicts next-token not only at the individual teken-level but also considering its distribution across the entire vocabulary set, enabling dynamic control of the information flow.

---

To measure the relative entropy between the token distributions from visual contents P and its comprehensive description p at time step i, we deploy Bounded Divergence (Dnd) 61, which is a type of statistical distance that ensures symmetric and bounded measure:
Dia(P
Σ 1=1
(p: + 9:) log2(p:: gel* + 1),
where Doa (Pl()) 2
0 and equals O, if and only p-q, and k denotes smoothing parameter. Here, the upper-bound of the divergence apparently exists, such that Dra S E Pt log, 2-1, due to the following condition |p:-qd < 1.
We define the dynamic restriction ar as Daa P.' Pt'), where enables a token-wise feedback control that adjusts the information weighting with respect to the closeness of the two distributions. The major role of the restriction term is maintaining a balance in the logit variation for the observed prediction disparities between the v and d distributions. That is, when the distributions are close enough (i.e, PV Ptt), the value of [| approaches zero, indicating minimal divergence. That is, o, approaches 1, allowing for higher amplification of logit variations in predicting the next-token outputs. This adjustment reflects the increased reliability of predictions when the two distributions from v and d are closely aligned. On the other hand, for the dissimilar distributions, ar decreases towards zero, compelling the model to restrict information flow from the variation. This reduction limits the potential for introducing erroncous or less probable predictions by focusing more on visual information, thereby maintaining coherence in the output when the model's understanding of the visual content significantly deviates from its textual description.
3.3 Adaptive Information Constraint
One major challenge in contrastive-based decoding the scenario where implausible tokens are rewarded, even when predictions are made with low confidence. This issue can also arise in our method, particularly when token distributions derived from textual descriptions provide more confi- dence than the visual content, ironically undermining the most predictive tokens. To address it, Li et al. [4]] have introduced an adaptive plausibility constraint, which filters out less plausible tokens by truncating them based the maximum token confidence from the expert model. While this approach simply penalizes false positive tokens in the candidate pool, it may also have unintended side effects cwwer-confieence tokens. Specifically, early threshold
by prematurely applying cutoff threshold settings can sometimes eliminate the possibility of identifying correct token predictions, which might otherwise be dismissed in pool considered to contain mostly false negatives.
Improving the previous constraint [34, 30,, we present adaptive information constraint (Vnead) de- signed to dynamically retain tokens that may be informative despite their lower confidence. By comparing prediction distributions between P. and Pd we filter out less relevant tokens from the candidate pool follows:
(y=r)))))) = {ye E V po(w | 1,
2 Br max pow v,
where 3r dynamically regulate the token candidate pool utilizing the divergence term in Equ. 3] defined 3, Daa (P This strategy can expand the token searching pool when the next- token prediction, derived from both visual content and comprehensive description, shows a similar distribution yet uncertainty in selecting the candidate token (i.e., false negatives). Finally, we only consider the next-token prediction within (B<e), and for the tokens satisfying Yt. & Vyead(yee). we their logits -oo to filter out from the candidate pool. Please see comprehensive Algorithm. in Appendix.
4 Experiments
Experimental Setup
To validate the efficacy of our method over various LMM families and sizes,
we implemented our method on contemporary LMMss LLaVA-13 115 (13B) 40], Emu2- Chat (14B) [1]] InternL M- 3-XMoMppserr2 (7B) [6], LLaVA-NeXTT (34B) 41. Yi-VL (34B) 62 and Intern 1.5 (26B) g We compared our method with five baseline decoding strategies. For the regular decoding strategies, we used greedy decoding, Nucleus sampling (0). and beam search decoding. Additionally, selected OPERA
23 and VCD [30] for contrastive decoding method, which designed to mitigate
5

---

40
Emu2-Chad
51
IXC2-VL 16
MMIlal Benca
POPE
MMIlal Deoch
POPE
MMilal--ees
POPE
15
78
13
85
2
QARGEE
ALL
113
46.1
132
QAAP
T1L
63.7
48.9
ROGA QARONA
$24
46.7
55.
GORON
1611
192
25
29
LLaVA-Bend
MMV2
LLT
Yi-VL.
62
Intern VL 1.5
g
MMflai-Bendh
POPB
POPE
MMitaa--ang Femnch
PPPE
17
683
12.3
29
80
512
27
78
QA9555
31.1
36.2
48.5
149
73
KOOA 0A90E2
R3.1
61.1
63.7
31
-LGaV--Decc
Greody
Bcam
Necleus
Opera
VCD
Ours
Figure 3: Overview of experimental results on baseline LMMs, 6 decoding method, and halluci- nation benchmarks in spider chart format.
hallucinations with contrastive frameworks. We used the default parameter settings for all methods, where top-p value 0.95 and temperature 1.0 for Nucleus sampling, the number of window size for searching is 5 (i.e., num-beams 5) for both beam search decoding and OPERA, and CD-a = 1, CD-3 0.1 for VCD, and & 0.3 for our method. Note that OPERA inference requires too much memory especially for LLa VA-NeTT (34B), so that we excluded OPERA results for this model.
Benchmarks and Evaluation Metries
The benchmarks for evaluating hallucinations in LMMs can be broadly categorized into discriminative and generative streams. The discriminative type assesses hallucinations by evaluating the predicted answer among given options (e.g. multiple choice or yes/no question), while generative benchmarks typically employ more advanced language models (e.g., GPT-aided evaluation) to rate the subject model descriptions. Within this taxonomy, we carefully select benchmarks to test baselines. Please see Appendix. for benchmark details.
As discriminative benchmarks, we utilize mainly three datasets for detailed evaluation. Specifically, POPE 35 is a commonly used benchmark for detecting object hallucination by converting object annotations sourced from MSCOCO 77) Under the three different subsets: random, popular, and adversarial, the metric for POPE measures binary classification performance for simple yes/no questions. MMVP (53) aims to evaluate the understanding of visual details for 9 different visual patterns using paired classification accuracy. Due to its evaluation design, which involves comparing two similar CLIP-blind image pairs, MMVP requires LMMs to capture subtle visual differences. RealworldQA (57] is the most recent dataset tailored to assess the capability of LMMs in basic real-world spatial understanding, using the accuracy metric within multiple-choice questions.
We use three benchmarks for generative benchmarks, extending the evaluation scope include open- ended captioning tasks beyond merely assessing classification within given answer options. Generally, ChatGPT [46] is used score the quality of the model- generated sentences. The metric for both LA VL-AA9Q QA90 42 and LLa VA- (In-the-Ve Wild) [42 is score ratio, where model responses [TT-score), where all scores are rated by GPT-4. It has three types of questions: conversation, detailed descrip- tion, and complex reasoning. MMHal-Bennch [52 evaluates the degree of hallucination for the various question types: object attribute, adversarial object, comparison, counting, spatial relation,
rated from GPT-4 47 are divided by GPT-4 answers such that Σ modell-scree]

---

Table The hallucination evaluation results for the discriminative benchmarks in MMVP colum cell indicates 9 different visual patterns (details in Appendix. C
35
53
Each emoji
POPE
MMVP
Model
AParam
Method
AccT
Prec
F11
Q
Ø
14
9
40
A
Avgg
Greedy
90.88
82.62
30.77
12.50
10.00
50,00
40.00
30.67
Beam
84.13
90.89
82.70
19.23
27.27
11.11
25.00
10.00
60.00
16.67
70.00
35.00
32.67
Nucleus
80.60
84.46
79.45
26.92
27.27
22.22
12.50
20.00
33.33
0.00
60.00
20.00
26.67
LLAVA-15
13B
Opem
"3.07
51301
43.31
-6.36
11:11
25.00
10.00
56.66
T6.67
"N0.00
"5..00
53.35
VCD
81.40
85.00
80.39
54.62
18.18
22.23
37,50
50.00
43.33
33.33
40.00
35.00
34.00
Ours
84.27
90.99
82.86
19.25
31.82
11:11
25.00
20.00
53.33
16.67
80.00
40.00
34,00
Greedy
51.18
85.35
30.77
11.11
12.50
20.00
63.33
70.00
20.00
Beam
87.07
93.99
85.96
38.46
27.27
11.11
25.00
20.00
66.67
16.67
60.00
25.00
36.00
EimuZ-Chot
148
Opera
Nuclens
87.07
86.17
54.06
92.22
35.95
85. 10
38.46
18.46
51.35
31,82
T1:11
16.67
25.60
0.00
221.00"
401.00
63.35
43.31
33.33
76.00
60:00
25.00
30.00
56.67
34.00
VCD
85.87
91 58
84.82
34.62
36.36
7.78
0.00
0.00
56.67
0.000
70.00
211001
34.67
Ours
87.47
92.77
86.64
12.31
40.91
7.78
0,00
30,00
63.33
50,00
70.00
30.00
42.00
Greedy
84_37
11.11
37.50
30.00
70.00
30.00
Ream
84.13
K3.12
84.37
34.62
27.27
11.11
37.50
50.00
6.33
33.33
70.00
35.00
40.00
IXC2-VL
7B
Open
Nucleus
83.47
79.58
82.355
77.41
3:5622
80.3 50
42..1
34,62
51.38
22.27
16.67
16.67
37,50
62,50
50.00
30.00
55.31
53.33
0.00
70.00
80,00
25.00
25.00
36.67
VCD
80.17
78.23
80.83
34.62
31.82
38,89
50.00
40.00
10.00
16.67
90.00
35.00
40.00
Ours
84.30
86 26
83 86
34,62
27.27
11.11
57,50
6,00
56.67
16.67
70.00
35.00
38,
Gready
86.50
83.88
87.01
44.46
40.91
16.67
37,50
30.00
60.00
80.00
35.00
Beam
84.13
90.89
82. 70
38.46
31.81
22.22
37,50
50.00
60.00
0.00
80.00
30.00
40.67
Nucleus
84.90
82.78
85.37
34.63
22.73
27.78
25.00
20.00
43.33
0.00
50.00
45.00
33.
LLAVANNeTT
Opera
VCD
85.20
83.00
85 68
42.31
22.73
22.22
$7,50
50.00
46.67
16.67
80.00
40.00
39 35
Ours
86.93
83.82
87.51
34.62
36.36
33.33
25.00
50,00
70.00
0.00
70.00
30.00
42.67
Greedy
19.83
77.50
23.08
16.67
$7.50
30.00
46.67
70.00
25.00
30.00
Beam
78.97
46
75.74
26.92
18.18
16.67
$7,50
34.00
45.33
16.67
70.00
15:00
29
Yi-VL
Opera
Nucleus
64.30
75.30
60:20
78.96
70.67
73.68
23.08
7.69
18.18
3.55
16.67
0.00
12.50
37,50
20.00
0.00
30,00
16.67
33,33
0.06
60.00
0.00
25.00
10.00
26.67
7.35
VCD
76.70
80.69
75.08
23.08
36.36
3.89
25,00
30.00
33.33
16.67
60.00
30.00
32.67
Ours
80.17
87.37
78.05
23.08
18.18
27.78
62.50
30,00
35.67
0.00
70.00
30.00
31.35
Greedy
85.83
82.83
42.31
36.36
27.78
25.00
34.00
80.00
33.33
80.00
45.00
40.00
Beam
86.80
87.45
86.68
38.46
45.45
22.22
37,50
50.00
$3.33
50,00
70.00
45.00
50.67
V1
Nucleus
81.23
79.45
81.76
50.00
31.82
27.78
12.50
60.00
70.00
33.33
60.00
25.00
44.00
Opera
86.30
84.44
86.06
42.31
27.27
16.67
25.00
30.00
76.67
50.00
70,00
50.00
45.33
VCD
81.70
80.08
82 18
30.77
36.36
11.11
12.50
50.00
66.67
50,00
50.00
55.00
42.00
Ours
86,93
83.70
87.53
42.31
50.00
44.44
12.50
30,00
83.33
50,00
70.00
40.00
51.33
environment, holistic description, and others. GPT-4 measures the severity of hallucination in a range of 0 to and the higher score denotes less hallucination.
4.1 Evaluation Results
Overview We summarize our comprehensive experimental results across six LMMs, six decoding methods, on six benchmarks in a spider chart format for visibility of the improvements in Fig. 3 As in the figure, CODE generally shows competent performance and consistent results among different measurements and benchmarks. We delve into each result in detail in the subsections below.
Results on Discriminative Benchmarks We conduct hallucination evaluation across 6 LMM baselines on two discriminative benchmarks. POPE (35) focuses on object-level hallucinations, where its prompts consist of asking about the existence of objects, such as "Is there <something> in the image?". Here, we exclude trivial splits and test with the most challenging subset: adversarial split, which requires models to identify highly-relevant objects to correctly answer yes/no. As shown in Table [ both the accuracy and Fl score from CODE lead to the best performance in 6 LLM baselines, among 6 decoding methods, which indicates that our decoding method can properly address object hallucination within simple binary question format.
As a more sophisticated visual assessment, we compare our method on MMVP benchmark 53 Beyond object hallucination, the benchmark divided into 9 systematic visual patterns within CLIPP blind pairs. These pairs consist of two images that clearly display visual differences, yet CL.IP 488 struggles to discriminate between them. The test subject models only receive scores when they correctly identify both images in the pair, requiring precise visual understanding. On the right part of Table. [] we can observe a remarkable gain in average, indicating that CODE effectively enhances visual consistency through the contrastive mechanism using self-generated descriptions.
Results on GPT-aided Benchmarks Although the discriminative datasets are intuitive benchmarks for evaluating the degree of object-level hallucination, they are limited in mcasuring deeper under-
7

---

Table 2: GPT-aided evaluation results among LMMs and decoding methods on generative bench- marks SLLaV--QA9 42] : score ratio for GPT answer / AMHHll-Bnnl [52]: score rated by GPT).
LLVA--Q90
MMiltal Bunlal
Model
Decading
Cone
Detail
Comp
Overall
Adv
Comp
Count
Rel
En
Other
Overall1
[[al]
Gready
79.5
77.1
90.3
82.4
1:92
2.25
1.75
3.50
2.17
2.39
52.08
Beam
81.8
77.7
90.7
83.5
2.92
1.67
2.92
1.83
1.30
2.92
2.33
2.58
2.33
53.13
Nucleus
70.4
78.1
89.0
79.3
2.08
0.75
1.83
2.33
1.90
3.42
1:67
2.25
2.03
60.42
LLNA-115
13B
Opera
"2.
735
"3.4
"00.7
306
0.75
2.58
"1333
3.06
2.08
1.42
2.25
55000"
VCD
72.1
76.4
89.5
79.3
2,83
1.58
2:33
2.42
1.67
2.67
2.92
2.28
54.00
Ours
1.75
1.58
Crealy
1030
76.4
1706
2.17
50.55
Beam
81.7
54.0
82.9
74.8
3.00
0.58
1.83
2.75
2.58
3.33
3.17
2.42
2.46
39.58
Emn2: Chax
14B
Opera
Nucleus
32.8
78.6
5455
57.9
668.2
87.3
68.0
74.9
2.75
2,50
1,06
0:67
2.83
1.75
2.83
2773
-208
1.90
2.67
1.25
7.92
3.00
1.58
3.17
1.35
2:52
"3375
37.500
VCD
80.7
56.8
87.6
75.5
0:75
2:08
2.92
2.92
2.42
Ours
77.3
2.25
Crealy
83.0
31.3
W03
84.9
3:58
3.08
2.17
4188
1.50
2.75
1:08
3.17
29.17
Beam
79.1
80.2
90.4
83.1
3.50
3:25
2.83
2.25
4:33
3.83
2.75
3.08
3.23
28.13
Nucleus
86.5
76.5
88.3
84.1
3:25
2:75
2.90
2.92
3.25
3.42
2.90
3.00
3.05
33.00
IXC2-VL
7B
Opera
78.3
79:5
"68.
"1.7
3367
3.25
2.92
"222"
4::25
4.17
3.00
3.00
3.35
VCD
79.5
92.0
84.5
2.42
2.50
5.33
3.92
2.75
Ours
85.3
92.9
3.67
3.92
Groody
8000
93.7
97.9
90.2
2.75
3.90
3.30
2:75
2.33
4.08
3.08
3.30
3.30
34.00
Beam
87.7
97.2
99.2
94.5
3.38
3.83
2.83
2.58
3.17
3.92
2.67
3.75
3.26
35.42
Nucleus
83.2
88.7
98.3
90.0
2.50
3:92
3.50
1,33
3.92
..42
2:33
2.25
3,08
40.63
LeXT
34B
Opera
VCD
86.4
93.8
96.6
3.75
3.08
2.17
3.92
3.58
39.58
Ours
89.2
98.3
95.3
2,92
4.08
2.90
Greedy
8022
76.4
88.9
81.9
2.67
0.00
3708
1.00
2.00
2:23
1.33
1.17
1.75
65.63
Beam
70.7
75.7
85.9
77.5
2.92
0:42
3.33
0.92
2:25
2.33
1.83
1.58
1,95
60.42
Yi-VL
34B
Opera
Nucieus
63.6
76.2
$1.2
20:1
16.0
82.7
79.9
36.3
2.33
2.50
1,25
1.42
2.58
1,83
0.83
1.08
2,50
100
1.383
2.42
1.52
1.33
L.08
1.50
1.58
1.84
61.50
62.50
VCD
0:33
1.58
2.42
1.84
Ours
73.9
89.4
82.1
2:75
0.58
1.00
2.50
1.00
Greedy
75.2
91.2
89.]
85.6
3.92
2.58
2.67
2,42
3.17
1.75
3:25
3.42
3.15
33.33
Beamn
83.7
91.3
91.6
89.3
3.83
2.67
3.83
2.50
4.00
3.75
2.75
3.58
3.36
31.25
InseraVL
268
Nucteus
81.0
75.1
94.6
95.1
91.5
91.1
88.7
86.4
2.92
3.25
2,83
2.75
3.00
2.92
3.50
2.75
4.17
3.33
3.83
3.33
2.62
3.25
3.33
3.92
3.14
37.50
52.25
VCD
83.9
89.5
88.3
2.17
2.67
3.00
Ours
83.1
103.3
91.9
92.2
4.25
7.93
2.93
4.17
3.83
2.97
3.67
3.52
30.31
standing of whether the given LMM responses encompass contextual or sequential hallucination. To
probe the expanded effectiveness of our method beyond simple multiple choice tests, we utilize two
generative benchmarks that can verify model responses at the sentence level. As in Table. 2 CODE
generally outperforms the overall score of LLaVA-QA90 22 showing up +13. 13%7% improvements
than other CD methods. Additionally, we compare our models in MMHal-Bench (52) specialized to evaluate hallucination effects sourced from more challenging image-question pairs- OpenIm- ages [50]. As in the result, our method generally not only improves overall average score with consistent results among 6 other baseline LMMs, but also effectively mitigates the hallucination ratio. Through combinatorial results from both discriminative and generative benchmarks, we corroborate the robustness and efficacy of our proposed method in addressing both object-level and contextual hallucinations, thereby ensuring more reliable LMM responses.
4.2
Analyses on CODE
Ablation Study Previous CD-based methods 34 30 [2] require heuristic choices to control the de-
Table 3: Ablation study on ae (DR) and 3,
(AIC) for (abbreviated) LV1.5 40 LV-N 41] IVL1.5 [ on two benchmarks 53 42 We
gree of amplification for logit variation a and pe- nalizing parameter 3 that filter out the implausible next-tokens in adaptive plausibility constraint. To tackle it, we proposed two regulation methods that can dynamically control information flow in CODE and token candidate pool, respectively: (i) dynamic restriction (DR), 0; in Eqn. [ and (ii) adaptive infor- mation constraint (AIC), 3 in Eqn. 4 To validate the effectiveness of such adaptive regulations built in CODE, we conducted ablation study on them.
report overall scores for the benchmarks. X indicates fixed hyper-parameter for a and 3
MMVP
LaVVA-Rench
LV1.5
LV-N
IVLI.5
LV1.5
LV-N IVL1.5
X
28.67
32.67
45.33
66.4
76.0
81.0
32.00
41.33
48.00
67.9
81.6
83.5
29.33
37.33
48.67
79.7
82.4
34.00
42.67
51.33
72.6
85.1
85.7
We implement baselines with same CODE framework using self--enerated descriptions as visual counterparts, but with default settings of a=1.0 and 1-0.1. As in Table. 3 either use of DR or AIC can enhance the benchmarks than the fixed a and B. Our CODE implementation that utilizes both DR and AIC to dynamically restrict information flow exhibits the best results among the basclines.
8

---

Question: What is the brand of the yogurt flavored with blueberry?
GT Answer: The brand of the blueberry-flvoore yogurt is Fage.
Original Tokets
The
brand
the
image
is
from
tbe
brand brand
Yoplait
CODE Tokess
The
yogurt
the
image
From
Bic
Fage
Visual contens
20.81
21.34
20.81
22.03
22.47
22.84
18-42
19.78
20.97
1861
15.34
loght,
20:81
21.13
20.8
27.03
22.47
22.84
18.42
19.78
20.97
18.63
15,02
Description only
16.98
20.14
18.69
25.55
21.00
23.00
1644
19.23
19.50
17.53
lngta
16.98
18.69
25.55
1641
19.50
CODE
22:01
21,85
22.17
26.34
23,40
22.72
19.78
20.26
22.06
19442
15.39
Tusi
22.01
23.30
22.17
28.34
23.40
22.75
19.78
20.26
22.06
19.42
16,60
Figure 4: An example of token-level case study for CODE. Each row indicates the logit score from
visual content logit,,, comprehensive description logit,, CODE applied loget respectively.
Beam search
Open decoding
VCD
CODE (Ours)
524
LLaVoo13
Brab-cha DOCALL
W-VL
(a) RealworldQA (57
(b) LLaVA-Bench (n-thh--Wldd) 42
Figure 5: Additional experiments on In-the-Wiin benchmarks. Note that, unlike other datasets,
OPERA 23 fails to generate consistent responses in rorl-world datasets using Yi- VL 622.
Computational Analysis We utilize comprehen- sive descriptions from models as additional infor- mation for contrasting with visual contents, thereby leading to computational loads similar to other de- coding methods. To analyze the computation, compare the token throughput (token/s) and decod- ing latency (ms/token) with other CD-based methods on 8 NVIDIA RTX A6000 GPUs as in Table. 4
Table 4: Computational analysis on decoding throughput and latency among CD-based meth- ods. We compare three different model sizes.
Throughput (loken/s]
Latency (ms/token))
VCD
OPERA
CODE
VCD
OPERA
CODE
7B
5.62
1.23
3.66
177.99
809.73
272.92
14B
53
4.04
1.04
2,82
247.6
960.14
354.09
34B
41
3.61
2.81
271.27
00m 355,81
Token-level Case Study As illustrated in Fig. 4 to verify whether the proposed CODE effectively mitigates object hallucination, we analyze the output logit values of LMM [4] at the token-level case study, with greedy search as the baseline. The first two rows in the table indicate the original greedy decoded tokens which are elected based on high logit,, from the visual content and CODE output tokens, respectively. As in the figure, we can observe that visual hallucination occurs at the "Yoplait" token highlighted in red. For relatively easy tokens at the beginning of sentence, logit code produces identical decisions maintaining consistency with logit,, which indicates the amplification of logit variation is effectively adjusted due to similar prediction distributions from visual contents and description-only information. However, at the hallucination-occuree deviated between the two information, resulting in a more confusing state to identify between GT token "Fage" and ballucinatory "Yoplait". In our framework, "Fage" is relatively more amplified from 15.02 to 16.66 than "Yoplait", which changes from 15.34 to 15.30. By simultancously considering both token-level and distributional prediction over the vocabulary, CODE changes the wrong next- token output to correct one, mitigating hallucination. For more case studies, please refer Appendix. [
time step, logit scores
Additional Experiments on In-the-Wild Contemporary apen-sourced LMMs are fine-tuned with various combinations of vision- language datasets 7 28 25), mostly composed of COCO-sources visual images (38] and their curated instruction. Although the existing hallucination benchmarks intentionally convert question queries to assess model robustness against inconsistency, the visual contents in benchmarks are limited to in-distribution COCO images. To validate our method in more challenging and real-world scenarios, we compared baselines on V.LaVA-Bench (In-the-wild) 42 and RealworldQA [57] as in Fig. and achieved competent performance (case studies in Appendix.
28

---

Discussion and Limitation
Albeit the computational analysis in Table. one of limitations, our contrastive decoding method requires additional computational resources than the use of vanilla decoding. However, considering an essential ongoing research topics and developments (60) 15] aimed at mitigating the negative effects of hallucination problems in both LLMs and LMMs, our work contributes important societal impacts towards more real-world applicability and robust Al system.
6
Conclusion
We present COuntering DEscription Contrastive Decoding (CODE), a novel and training-free de- coding method to mitigate hallucination in Large Multi-modal Models. By utilizing self-generated descriptions as corrective references during the decoding phase, CODE dynamically adjusts the information flow for next-token predictions, enhancing the coherence and informativeness of re- sponses while reducing the cross-modal inconsistency. Extensive experiments demonstrate that CODE effectively decreases hallucinations across various benchmarks and contemporary LMMs, significantly improving contextual relevance and response alignment with visual contents.
Acknowledgments
This work was partially supported by two funds: ITTP grant funded by the Korea government (MSIT) (RS-2022-11220984) and Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD230017TD).
References
[1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 6077-6086, 2018.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[3] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint
WwXiv22404.1893 2024.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality- enhanced projector for multimodal 11m. arXiv preprint arXiv:2312.0677 2023.
[6] Min Chen and Mateu Sbert. On the upper bound of the kullback-leibler divergence and cross entropy. arXiv preprint arXiv: / 1.08334, 2019.
[7] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv
preprint Xrxiv:15044 00325, 2015.
[8] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far commercial multimodal models with open-source suites, arXiv preprint arXiv:2404. 16821, 2024.
to gpt-4v? closing the gap to
[9] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312. 14238, 2023.
[10] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425 2024.
[11] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, lon Stoica, and Erie P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
10

---

[12]
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality large language models. In The Twelfth International Conference on Learning Representations, 2024.
[13]
Wenliang Dai, Jumnan Li. Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBIIPP Towards general-purpont vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, 2023.
[14] Ailin Deng. Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guides decoding. arXiv preprint arXiv:24009 1,5300. 2024. [15] Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chaain-f-vverffcattinn reduces hallucination in large language models. arXiv preprint arXiv:2309. 1495, 2023.
[16] Xiaoyi Dong. Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. anternlm-ccompse free-form text-image composition and comprehension in languae large model. arXiv preprint arXiv:2401. 2024.
Mastering
[17] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1444 1448, 2015.
[18] Google. Bard. https: http://///add..oooe..eeee com/ 2023.
[19] Kaiming He, Xınei Chen, Samnng Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009 2022.
[20] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. [21] Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 1439-14440 1449, 2021.
[22] Lei Huang. Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang. Qian- glong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint
arXiv:2311.0522 2023.
[23] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weim- ing Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and cnraspecttionalloaattin arXiv preprint arXiv:2311.17911
2023.
[24] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui,, Owais Khan Mohammed, Barun Patra, et al. Language not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36, 2024.
[25] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9700-6709, 2019.
[26] Junho Kim, Yeon Ju Kim, and Yong Man Ro. What if...?: Counterfactual inception to mitigate hallucination effects in large multimodal models. arXiv preprint arXiv:2403.1 13513. 2024.
[27] Yeonju Kim, Junho Kim, Byung-K Kwan Lee, Sebin Shin, and Yong Man Ro. Mitigating dataset bias in image captioning through clip confounder-fir free captioning network. In 2023 IEEE International Conference on Image Processing (ICIP), pages 1724.174. IEEE, 2023.
[28] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[29] Byung-Kwan Lee, Beomchan Park, Chae Won Kim, and Yong Man Ro. Moai: Mixture of all intelligence for large language and vision models. arXiv preprint arXiv:2403.075:0
2024. [30] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. arXiv preprint ar3iv:231 I. 16922, 2023.
[31] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning. PMLR, 2023.
[32] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-tnaining for unified vision-language understanding and generation. In International Conference on Machine Learning, pages
PMLR, 2022.
11

---

[33]
Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal Ilm with co-upcycled mixture-of ofeexpers. arXiv preprin! arXiv:2405.0594
2024.
[34] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke 7ettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 12286-122 12-1222222222 2.. Association for Computational Linguistics, July 2023.
[35] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292-305, Singapore, Dec. 2023.
Association for Computational Linguistics.
[36] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024.
[37] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401. 15947, 2024.
[38] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pages 740-755. Springer. 2014.
[39] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-moda. models via robust instruction tuning. In International Conference on Learning Representations, 2023.
[40] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint 130.0233, 2023.
[41]
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge. January 2024.
[42] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023.
[43] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402 00253, 2024.
[44] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In International
Conference on Learning Representations, 2023.
[45]
Brandon McKinzie, Zhe Gan, Jean-philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal I1m pre-training arXiv preprini arXiv:2403. 0060191,,, 2024.
[46] OpenAL. ChatGPT. pictps://ope.s
cow www.blog///cc/ccccccco hhaggg/chattt///// 2023.
[47] OpenAI. Gpt-4 technical report, 2023.
[48]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8766 8763. PMLR, 2021.
[49] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045, 2018.
[50] H Rom, Neil Alldrin, J Uijlings, I Krasin, et al. The open images dataset v4. Int J Comput Vis.,
128:1956-1981. 2020.
[51] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context Iearners. 2023.
[52] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang. Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF, 2024.
[53] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms, arXiv preprint arXiv:2401.06209, 2024.
12

---

[54] Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023.
[55] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for milms hallucination
evaluation. arXiv preprint arXiv:231 1. 1.07397, 2023.
[56] Lei Wang. Jiabang He. Shenshen Li, Ning Liu, and Ee-Peng Lim. Mitigating fine- ane-grained hallucination by fine-tuning large vision- vision-language models with caption rewrites, In International Conference on Multimedia Modeling, pages 32 45. Springer, 2024.
[57] xAI. Grok-1.5 vision preview., 2024.
[58]
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pages 2048-2031 PMLR,
2015.
[59] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In European Conference on Computer Vision, pages 521-539. Springer, 2022.
[60] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
[61] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu. Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2.34.. 14178, 2023,
[62] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang. Guanwei Zhang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprins arxiv:2403. 0403, 2024.
[63] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022.
[64] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:23 2200949, 2023.
[65] Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. Halle- switch: Controlling object hallucination in large vision language models. arXiv e-prints, pages arXiv-2310, 2023.
[66]
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivims through hallucination-aweer awaree direct preference optimization. arXiv preprint arXiv:231 1. 16839, 2023.
[67] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Conference on Learning Representations, 2023.
[68] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing aision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[69] Lanyun Zhu, Deyi Ji. Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprini arXiv:2402. 18476, 2024.
13

---

A Instruction for Comprehensive Description
In generating comprehensive description for the given visual content, aim to obtain detailed description as possible, ensuring that the response fully spans the visual representation space, even though may not be entirely feasible as discussed in 3.1 The specific prompt instruction used to describe the image contents in detail is described in Table.
Comprehensive Description Prompt:
Provide detailed description of the image, covering all visible elements and their interactions, so as to thoroughly answer potential questions about the image.
Table 5: simple instruction prompt for generating comprehensive description.
B Detailed Algorithm for CODE
We describe the complete details of CODE implementation for better understanding in Algorithm,
Algorithm COuntering DEscription Contrastive Decoding
Require: LVLM Me. Visual Content v, Text Query æ, Comprehensive Description Prompt Zo.
Target Token Length T
1: d + Mo(u, (0)
Step I: Generate comprehensive description d for the given visual content
2: Initialize 1 4-1
3: while
do
>> Step2: CODE decoding
4: Pv = Softmax [logito (Bt 2, ycc)]
>> Compute
5: Pa Softmax [logita (3t d, x. yce)
b Compute Pa
6:
Daa (pa) = >E (Po,i + pa,i) logg(1p.. Pa.i +
D. Bounded Divergence Dha Set Or for Dynamic Restriction Set Be for Adaptive Information Constraint (y=cr) {3l E V po (3 2x,)) 2 Br max Pe(w (,,,y<t))
7:
Set at 1-Dra (Pp..)
8:
Set Be + (Pe,PeellP....
9:
Set Veead
10:
Poode = (oftmax + at logito v, aglogit, d, (d,2,8<t) y<t)
(ee 1Bec) 0f be & Vread
Apply Adaptive Information Constraint > Select Token
12: Y1 argmax (Pcode)
Set
14: end while
C Benchmark Details
Discriminative Benchmarks
POPE (35 is widely used benchmark designed for evaluating object-level hallucination, which can be split into the three subset categories based on how to select object replacements: (i) random, randomly sampled objects (ii) popular, top-k frequent objects not existing in the image, and (iii) adversarial, top-k objects that have high co-occurrence. The number of images is 500 and each image has 6 questions along with the subsets, making total of 9000 images. In this work, we only consider adversarial split, which is the most challenging subset in POPE benchmark.
MMVP 53
includes 300 images with different visual patterns that CLIP model 48
struggles identify the visual differences (CLIP--airee) images): Orientation and Direction Presence of Specific Features (Q), State and Condition ( Quantity and Count (ft).
Positional and Relational Context ( Color and Appearance
Structural and Physical
Characteristics (*)) Text (A), Viewpoint and Perspective
(Q)) ). It follows multiple selection tests, but uses GPT-4 to map the model response the answer options.
RealworldQA [57 is recently introduced benchmarks for evaluating basic real-world understanding for multi-modal models. It consists of total 765 anonymized outdoor (mostly taken from vehicles) and indoor images with multiple selection questions.
14

---

Generative Benchmarks
LLaVA-QA90 & LLaVA-Bench (In-the-Wild) 42 consist of three subset response types for each image: (i) Conversation, which is conversation format between the user and assistant answering the vision-related questions for the given images. (ii Detailed description, which requires detailed description for the given image scene, and (iii) Complex reasoning, which involves in-depth reasoning questions for the image. The former benchmarks sourced from COCO images (total 30 images with 90 questions). while the latter benchmarks are gathered from web for challenging domain situations (total images with 60 questions).
MMHal-Bennh [52] is specially focused on penalizing hallucinations. It has total 96 image- question pairs composed with question categories for 12 objects: Object attribute (Attr), Adversarial object (Adv), Comparison (Comp), Counting (Count), Spatial relation (Rel), Enviromment (Env), Holistic description (Hol), and Others (Other). As like in the above LLaVA-Bent Bench benchmarks, MMHal- ALMHal-Bench also utilize GPT-4 to analyze and rate the model responses and score in range of 0 to 7.
D Additional Token-level Case Study
As additional token-level case studies. we explore how the logit information changes during our CODE decoding phase other examples. As a first example, LLLVA-NeXT [41] struggles to distinguish between Haleakala National Park and Diamond Head, both located in Hawaii, and predicts the former during inference with the vanilla decoding method. Using our CODE decoding method, as shown in Fig. 6 the information flow of "Haleakala" token is curbed, inducing token inversion to "Diamond", which matches the ground truth word. This occurs because the logit variation is dynamically adjusted based on both token-level and distributional information.
Question: What is the name of this famous sight in the photo?
GT Answer: The famous sight the photo is Diamond Head.
Original Tokens
The
famous
sight
in tbc
photo
Halenkala
CODE Tokens
The
famous
sight
in the
photo
18
Dianond
Visual content
23.42
20.89
24.78
26.45
30.55
28.16
27.05
18.70
logli,
23,42
20.89
24.78
26.45
30.55
28.16
27.05
17.45
Deseription-only
18.89
19.06
23.81
24.86
28.64
26.34
24.23
17.95
logit,
18.89
19.06
23.81
28.64
26.34
24.23
11.86
CODE
21.91
19.32
Ingifcooe
25.57
21.91
25.66
27.89
32.29
29.95
28.59
22.06
Figure 6: Additional token-level case study for analysis.
In another example illustrated in Fig. 7 we show how the adaptive information constraint prevent from rewarding implausible tokens, thus suppress hallucinatory prediction during contrastive decoding.
The original tokens from logit, predict the correct answer at the hallucinatory time step, highlighted in bold. In this case, the original prediction should be preserved, and token inversion should not occur. By our CODE decoding method, B, dynamically controls the adaptive information constraint, so that those hallucination tokens' (i,e., "four" and "dragon") ") logit values are cut off to oo and removed from candidate token pool.
E Further Discussion on Broader Impact
We proposed CODE, which can be seamlessly integrated into LMMs without additional training. There is still a lot of room for progress and mitigation of hallucination issues, as method cannot assure 100% removal of hallucinations. However, by providing more coherent and contextually accurate responses,
work can potentially be integrated into real-world applications, making user interactions with AI in customer service, education, and personal assistance more effective and satisfying in the near future.
Furthermore, enhanced accuracy and reliability using our method can reduce hallucinations in LMMs, improving the accuracy of Al-generated descriptions in critical fields such as autonomous
15

---

Question: How miny uncut fruits are the imuge?
GT Answer: There three uncut fruits the image.
A
0.27
no1
0.80
0.11
0.00
0,80
0.66
0.0g
0.01
0.18
Original
Thare
IBC
Image
Deseription Tokess
There
Visual laght
22.56
23.09
21.17
27.42
25.50
26.53
21.66
29.47
14.65
25.41
30,00
27.42
25.50
Descipiion-onl
2047
27.00
21.
23.05
28.20
20.75
25.14
26 52
21 15
legit,
27.0
24.28
28.00
26.55
21
CODE Tokens
Thrre
Friils
CDDE
26.18
25.96
23.39
20.56
30.73
21.25
25.47
31.63
26.34
28.65
lnde
24.0
25.90
-in
20.56
30.70
25.47
31.63
28.34
28.9
Figure 7: Additional token-level case study for analysis.
driving, robotics, healthcare, and augmented reality. This advancement not only enhances practical applications but also significantly benefits the research community working hallucination, area that is not yet fully explored. Our contributions can help pave the way for deeper understanding and new research directions to address these challenges, for more trustworthy Al systems.
F Additional Qualitative Results
Discriminative Capability Case Study: MMVP.
Image:
Question:
Is the following statement correct: In the image, there is one daisy that is under the shadow of taller daisy.
(a) Correct (b) Incorrect
Ground-truth:
(b)
MiternL.M-CCoppess (Opera):
The answer is (a).
MMterrLMM-CCoppsee (VCD):
The answer is (a).
InternL MiveraLM-XCone XCooposee22 (Ours):
The answer is (b).
Image:
Question:
In the image, is the snake's head oriented facing upwards or directed forwards? (a) Upwards (b) Forward
Ground-truth:
(a)
InternVi VL 1.5 (Opera):
The snake's head oriented (b) forward.
InternVL (VCD);
The smake's 's head is directed forwards, so the correct answer is (b) Forward.
Intern VL 1.5 (Ours):
The snake's head is oriented facing upwards, so the correct answer is (a) Upwards.
16

---

Image:
Question:
Is the person the picture already standing on weighing scale?
(a) Yes (b) No
Ground-truth:
(b)
InternVL 1.5 (Opera):
Yes
Intern 1.5 (VCD);
Yes. the person in the picture is already standing on the weighing scale.
Intern 1.5 (Ours):
The person the picture is not yet standing on weighing scale. The answer (b) No.
Image:
Question:
Is the needle pointing up or down?
(a) Up (b) Down
Ground-truth:
(a)
LLaVA-NeXT (VCD);
The needle of the syringe is pointing down.
LLaVA-NeXT (Ours):
The needle in the image pointing up.
Image:
Question:
Based on the image, is the following statement correct: We cannot see the window the school bus?
TOP
(a) Correct (b) Incorrect
Ground-truth:
(a)
LLAVA-NeXT (VCD):
(b) Incorrect. window school bus visible image, located just above the "School Bus" sign.
SCHOOL. BUS
LLaVA-NeXT (Ours):
The statement "We cannot the window the school bus" is image provided, there no visible window on the school bus. The view is focused on the back of the bus where the stop sign and the words "SCHOOL BUS" are visible, but the window not shown.
(a) Correct.
Table 6: More studies within MMVP, using IntemLM--XCompose)) Inter ViterVLi 1.5 and LLaVA- NeXT.
17

---

Discriminative Capability Case Study; RealworldQA.
Image:
Question:
How many circular objects in the image?
There are circular objects.
B. There is circular object.
C. There are circular objects. Please answer directly with only the letter of the correct option and nothing else.
Ground-truth:
A
InteraL.M-Comppen (Opera): The answer is C.
InternLM-XCooposee (VCD):
The answer is C.
InternL M-CMopopsr22222 (Ours):
A.
Image:
Question:
How many animals are this picture?
A. There are eight animals in this picture.
B. There are nine animals in this picture.
C. There is animal in this picture.
Please answer directly with only the letter of the correct option and nothing else.
Ground-truth:
B
InternVi VL 1.5 (Opera):
A
Intern VL 1.5 (VCD):
A
InternVE VL 1.5 (Ours):
B
18

---

Image:
Question:
Where the black material relative to tree?
A. The black material is at the top of the tree.
B. The black material is at the midpoint or halfway on the tree.
C. There is black material.
Please answer directly with only the letter of the correct option and nothing else.
Ground-truth:
B
LLaVA-NeXT (VCD):
A
LLaVA-NeXT (Ours):
B
Table 7: More case studies within RealworldQA, using IntemLL--Coomposee, InternVL 1.5 and LLVAA-NeXT.
19

---

Generative Capability Case Study: LLaVA-Bene (In-the- Wild),
Question: Descrihe this photo in detail.
Image:
(Opera):
The painting you' re referring to is famous "Mona Lisa" by Leonardo da Vinci. It is an oil painting on a poplar wood panel. The artwork depicts woman, believed to be Lisa Gherardini, the wife of Florentine merchant named Francesco del Giocondo. The woman is portrayed in three-quarter view, with left hand resting on her lap and her right hand holding a veil. She is wearing dress with high collar and pearl necklace. The background features landscape with mountains and body of water. The painting known for its enigmatic smile and the use of sfumato, a painting technique that creates a soft, hazy effect. The "Mona Lisa" considered a masterpiece of the Italian Renaissance and is housed in the Louvre Museum in Paris, France.
MiternLL--Coon Composer2 (VCD):
The image features painting of the renowned Mona Lisa, oil wood artwork. The central figure is a woman with long red hair headband, wcaring a blue dress adorned with gold accents. Her face is captured in close-up detail, showcasing her distinct smile and enigmatic gaze. The background a landscape painted in earth tones, adding depth to the composition. This painting widely recognized masterpiece, capturing the essence the Renaissance era with its realistic portrayal and intricate details.
InternLMr W-I.LM--Cooopeee (Ours):
The painting you'l re referring to is famous "Mona Lisa" by Leonardo da Vinci. It is an oil painting on a wooden panel, measuring approximately 77 x 53 centimeters. The painting is portrait of woman, who is the focal point of the artwork. She is depicted wearing dress and headscarf, with her hair styled a manner typical the Italian Renaissance period. The background of the painting features a landscape with mountains and a body of water, adding depth and context to the scene. The painting known for its realistic portrayal of the woman's features. her enigmatic smile, and the subtle interplay light and shadow. The "Mona Lisa" is considered a masterpicce of High Renaissance period and is housed in the Louvre Muscum in Paris, France.
20

---

Question:
Describe this photo in detail.
Image:
VL (Opera):
The image shows group of star fruit, also known carambola tamarind, arranged and presented. These kind of sweet, fizzi, and tart tasting, and popular fruit. There appears few different ways they re displayed, some whole or unpeel, some are half-cut open. There is clear visual focus and contrast, as well as natural and organic feel to the image.
VL. 1.5 (VCD):
This image depicts several purple-colored fruits, which appear to he durian. Durian is tropical fruit that is known for its strong, pungent aroma and creamy. sweet flesh. In the center of the image, one the durians cut in half, revealing flesh. which is a creamy white with a star-like pattern of seeds. The cut side of the durian shows the texture of the flesh and the arrangement of the seeds within.
fruit's inner
The fruit has distinctive appearance with a thick, spiky green outer shell protects flesh inside. The surface of fruit textured and somewhat rough, with the color transitioning from deep purple to green at the points where the spiky projections emerge. The durians resting provides rustic backdrop that contrasts with the vibrant colors of the fruit.
wooden surface, which
In the background, there is a wicker hasket containing more of these fruits, suggesting that they have been gathered for con- sumption sale. The basket is made of natural materials and complements the overall natural, organic feel of image. The lighting in the photo is soft and natural, which highlights the textures and colors of the durians and adds to the overall appeal of the image.
Overall, this image presents the durian fruit in way that show- cases both its visual and textural qualitics, as well as its role in natural, possibly agricultural, setting.
Intern VI. 1.5 (Ours):
The image shows several whole cut open mangosteens on wooden surface. Mangosteens are tropical fruits known for their distinctive appearance and sweet, slightly acidic flavor. The fruit has dark purple, almost black. outer skin that contrasts with the green, bulbous protrusions at the top, which are the remnants of the fruit's ' calyx. One of the mangosteens is cut in half, revealing the white, segmented flesh inside, which is the edible part of the fruit. The segments central seed pod. The cut-open fruit provides clear view of the fruit's structure and contrast between the outer skin and the inner flesh. The wooden surface and the natural lighting give the image rustic and organic feel, emphasizing the freshness of the fruit.
arranged in flower-like like pattern around
21

---

Question: Describe this photo in detail.
Image:
LLaVA-NeXT (VCD):
In the center of the image. black bowl with vibrant red and yellow trim commands attention. It cradles hearty serving of noodles, their creamy yellow hue contrasting with the bowl's dark color. A few pieces of meat, their rich brown tones adding to the variety of colors, are scattered throughout the bowl. Green peas peek out from under the noodles, adding touch of fresh- to the dish. To the left of the bowl, a small white cup filled with a green sauce waits patiently. Its color soothing contrast to the bold colors of the bowl and its contents. In the top right corner of the image, white plate holds a small pile of chopped vegetables. Their bright green color stands against the plate, drawing the eye to the top right corner the image.
these objects creates a visually pleasing scene. The howl of noodles is centrally placed, making it the focal point of the image. The green sauce in the white cup and the plate of chopped vegetables framing the bowl and creating balanced composition. The colors, objects, and their relative positions all contribute to a harmonious and appetizing scene.
The arrangement
positioned on either side,
LLa (Ours):
In the heart of image, black bowl with vibrant red and yellow border takes center stage. cradles a hearty serving of udon noodles, their light brown hue contrasting heautifully with the bowl's dark exterior. The noodles are not alone; they accompanied by generous portion of green onions, adding pop of color and hinting at the freshness the dish.
To the right of the bowl, a white cup filled with a green sauce waits patiently, ready to enhance the flavors of the udon noodles. The cup's 's pristine white color provides a stark contrast to the green sauce within, making it an eye-catching element the composition.
In the background, a blue and white plate holds serving of meat and green onions. The plate's colors echo those of the bowl, creating harmonious color palette across the image. Scattered around these main elements are various condiments and utensils, each playing their part in this culinary tableau. Their presence adds depth the image, suggesting meal ready to be enjoyed.
Overall, this image captures moment of anticipation, just before the enjoyment a delicious meal begins.
Table 8: More case studies within LLaVA-Bench (illhe-Wild), using InternL MiternLM-CCOne M-XCompossr2,, Intern 1.5 and LLLVA- We marked the expressions where hallucinations occurred in red.
22

---

G Failure Cases
As discussed in Appendix. B even CODE shows competent performance along various benchmarks and LMM baselines. The hallucination cannot be 100% eliminated. In this section, we attached some of failure cases shed some light for future work direction. As illustrated in Table. [8] the baseline models fail to correct hallucination even with our CODE method. Upon closer examination of the model responses, language priors such as evident that the hallucinatory responses tend to be biased towards "holding glass of beer". Our approach mainly uses self self-generated description as contrasting reference (i.e., close to the concept of self-correction, thus the strong assumption is that the amateur model (comprehensive description) should generate not too much deviated responses from true answers.
In the failure examples, we can infer that the reliance on self self-generated descriptions may not always suffice, especially when the descriptions themselves are biased or inaccurate. This indicates need for integrating more robust mechanisms to verify and correct these biases. Additionally, enhancing the model's understanding and processing of visual content could help mitigate such issues. Future work could explore the integration of external knowledge sources and more sophisticated bias detection techniques to further reduce hallucinations and improve the overall accuracy and reliability LMMs.
Image:
Image:
Question:
Question:
there any strawberry- flavorrd yogurt in the fridge?
What is the man doing in the image?
Generated Text:
Generated Text:
The image depicts group of people gathered in a room, with some sitting and others standing. There are least 12 people in the scene, engaged various activities. One man sitting at table, holding cell phone, while another man is sitting nearby, giving a thumbs-up gesture. There are several chairs in the room, with another towards the right side. A dining table is also visible in the seene, with cup placed on it. The people in the room appear to be enjoying their time together, possibly at social event gathering.
The image showcases the inside of refrigera- tor, filled with various food items. On the top shelf, there a carton of milk and container of yogurt. The middle shelf contains bag of carrots, package of strawberries, and blue plastic container. The bottom shelf holds carton of blueberries yogurt. Additionally, there are two bottles on the door of the refrigerator.
another carton of
the center and
InternLM-X MMMMMMMppsppr2222 (Ours):
Yes, there is container of strawberry- flavored yogurt in the fridge
LLa VA-1. (Ours):
The man in the image is sitting table. holding glass of beer, and giving thumbs-up sign.
Figure 8: Failure case of InterLM-XXC McCoMpoosemosr in LLaVA-Ba VA-Bench (i-the-Wild)) and LLaVA-1.5 in LLa VA-QA90.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They are included in this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include discussion section for potential limitation of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper does not involve theoretical result to prove.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation section is included, we will open the source public.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We will open to public in next step.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
ceiiioooPolcyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp.ccppulli//guiii wwwiiccgguie///eeeeeee CoodeSubeiiieeee ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we have baselines and benchmarks explanations in the manuscript. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
Our experiments do not need statistical results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the computational details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpurip///erriss... httprp/.......p.....ttccccccaaea
Answer:
[Yes]
Justification:
We reviewed CoE.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We add it in discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This is not relevant with work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our work do not include any license assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsiipteeeee www.rrewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
28

---

Answer:
[NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowd-soureing data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowd-soureing data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "0DnGoOwkDd", "reviewer": "Reviewer_Z6jD", "summary": "Large Multi-modal Models (LMMs) have made significant strides in understanding visual context and generating coherent responses. However, they face challenges such as hallucinations, where responses are incorrect and unrelated to visual inputs. To tackle this issue, this paper proposes COuntering DEscription Contrastive Decoding (CODE). CODE uses self-generated descriptions as reference points during decoding to mitigate hallucinations. By aligning responses with visual content through dynamic adjustments in token predictions, CODE enhances coherence and informativeness without requiring additional training. Experimental results demonstrate CODE's effectiveness in reducing hallucinations and improving cross-modal consistency across various benchmarks and state-of-the-art LMMs.", "questions": "see weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand.\n2. The use of description to enrich language information for contrastive decoding to address hallucination problems is interesting.\n3. Extensive experiments  are conducted to evaluate the proposed method.", "weaknesses": "1. From Table 1, it can be seen that CODE does not bring much performance improvement compared to greedy decoding in practice.\n\n2. Did the authors consider evaluating with the CHAIR metric on generative tasks?\n\n3. Can CODE be used in conjunction with VCD or OPERA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. From Table 1, it can be seen that CODE does not bring much performance improvement compared to greedy decoding in practice.**\n\n**A1.** We respectfully argue that the use of CODE shows consistent performance improvements across 6 different models with varying sizes. Especially, when considering the performance improvements on the more challenging generative benchmarks (Table 2) and in-the-wild benchmarks (Fig. 5), which extend beyond simple multiple-choice tests, our method shows improvements exceeding 10%, as described in lines 272-277. These gains are not marginal, and importantly note that the improvements are achieved without further training of the models.\n\n---\n\n**W2. Did the authors consider evaluating with the CHAIR metric on generative tasks?**\n\n**A2.** We thank for the reviewer’s valuable discussion point! As an initial design of our experiments, we have considered CHAIR benchmark for generative task. However, as Ben-Kish *et al.* [R1] pointed out, CHAIR is out-of-dated measurements that is limited to only 80 object annotations in the MS-COCO. As a result, we selected recently introduced GPT-aided measurements (MMHal-Bench and LLaVA-QA) for more detailed generative evaluation.\n\nHowever, we further conducted experiments to clarify the effectiveness of our method on CHAIR benchmark. We have randomly sampled 500 samples from COCO and reported two metric variations per-sentence ($C_{S}$) and per-instance ($C_{I}$) proportion with context length of 64 in the below table. As in the table, our method shows competent performance and consistent results than the baselines decoding methods (greedy, VCD, and OPERA) with varying size of LMMs (LLaVA-1.5, IXC2-VL, and InternVL 1.5). We will incorporate the full CHAIR results into the potential next version.\n\n| CHAIR |        |      | LLaVA-1.5 |          |        |          | IXC2-VL |          |        |      | InternVL |         |\n|:--------:|:------:|:----:|:---------:|:--------:|:------:|:--------:|:-------:|:--------:|:------:|:----:|:--------:|:-------:|\n| | greedy |  vcd |   opera   |   ours   | greedy |    vcd   |  opera  |   ours   | greedy |  vcd |   opera  |   ours  |\n|  $C_{S}$ |  26.4  | 28.6 |    25.6   | **24.8** |  26.8  |   23.8   |   24.8  | **22.0** |  18.2  | 19.2 | **17.4** |   17.6  |\n|  $C_{I}$ |  11.1  | 12.1 |    11.5   | **10.9** |   11.8  | **10.2** |   11.1  |   10.6   |  10.1  | 10.4 |   10.8   | **9.7** |\n\n---\n\n**W3. Can CODE be used in conjunction with VCD or OPERA?**\n\n**A3.** CD-based decoding methods essentially requires a contrasting counterpart to penalize the original information with the logit variance. Our method contrasts with the information from self-generated comprehensive description, while VCD and OPERA rely on distorted visual inputs and logit penalties on pre-mature layers for contrastive decoding, respectively. Due to the different contrastive designs of each framework, conjunction of each method is not a feasible option. Additionally, simply concatenating each decoding process may result in the over-penalization of the original information, so that we cannot assure the right models' responses. Also, considering the increased inference time for each decoding framework, we believe that directly combining CD-based methods is not practical.\n\n---\n[R1] Mitigating Open-Vocabulary Caption Hallucinations, *arxiv preprint*, 2312.03631", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V8rw14ZKfR", "reviewer": "Reviewer_FLJA", "summary": "The paper proposed a contrast decoding method named CODE for large multi-modal models. CODE, as mentioned by its name uses self-generated description as contrasting references during the decoding phase of LMMs to mitigate the hallucination issues. CODE works by dynamically considering the variations between the visual features and their corresponding pure language features (description) to improve response alignment with actual visual content and misalignment with the wrong part in descriptions. Based on contrastive decoding, the author proposed dynamic restriction, which regulates the information flow, and adaptive information constraint to filters out less plausible tokens constraint in the contrastive decoding phase. The proposed method is verified on 6 benchmarks.", "questions": "1. Have the authors tried other divergence measures such as KL divergence for the Bounded Divergence used in CODE?\n2. For the computational analysis in Table 4, have you considered the cost of generating the comprehensive descriptions?\n3. A question about the DC, I don't understand, when $\\mathcal{D}_{\\mathrm{bd}}\\left(P_t^v \\| P_t^d\\right)$, which means there is little difference between decoding based on visual features and decoding based on pure language features. It is intuitive to decode without regard to the variations, but why does the CODE highly consider the variation?", "limitations": "As mentioned in the first point of Weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe writing logic is relatively clear and easy to understand. The figures and tables are neat, beautiful and intuitive.\n2.\tThe proposed method is reasonable and has a good performance.\n3.\tThe paper conducts fundamental sufficient experiments on 6 VQA benchmarks. The ablation study shows that both DC and AIC can improve the performance of MMVP and LLaVA QA 90.", "weaknesses": "1. The method has to generate a comprehensive description for each image before doing the visual question answering, the inference time is long and not convenient.\n2. Only conduct ablation study of DC and AIC on MMVP and LLaVA QA 90, lack of ablation study on other benchmarks, such as, POPE.\n3. Lack of analysis: why the performance of proposed method is not optimal or even inferior to the underlying Greedy decoding on some problem types of MMVP. For example, Color and Appearance in MMVP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. The method has to generate $\\cdots$ convenient.** \n\n**A1.** Even if we have discussed in Discussion section and computational analysis in Table. 4, we acknowledge that our method requires additional computational resources to obtain (self-generated) textual descriptions from models themselves as visual counterparts for CD. However, we would like to highlight that, by contrasting with the models’ descriptive self-understanding, our method can mitigate the hallucinatory responses without further training, which is also one of important research venues for real-world applications.\n\n---\n\n**W2. Only conduct ablation study of DC and AIC $\\cdots$ such as, POPE.** \n\n**A2.** Thank you for the valuable feedback. We would like to clarify the effectiveness of DR and AIC designs by conducting further ablation studies. In Table. 3 of our manuscript, we have selected two benchmarks for the ablation study, one each from discriminative and generative categories (MMVP, LLaVA-QA). As the reviewer asked, we experimented additional ablation studies for the rest of the benchmarks (POPE and mmHalBench). Note that we randomly sampled 500 examples from POPE and reported accuracy. For mmHalBench, we reported overall scores. As in the below table, we can observe that the use of DR and AIC progressively enhance the performance in both benchmarks, which indicates that dynamically restricting information is an important design of the CODE implementation. We will incorporate these results in the potential next version.\n\n|              |              |       | POPE |        |       | mmHal |        |\n|:--------------:|:--------------:|:-------:|:------:|:--------:|:-------:|:-------:|:--------:|\n| DR | AIC | LV1.5 | LV-N | IVL1.5 | LV1.5 | LV-N | IVL1.5 |\n| X | X | 74.4  | 84.8 | 75.8   | 1.99  | 3.09  | 3.39   |\n| X | O | 78.8  | 85.0 | 82.0   | 2.08  | 3.33  | 3.49    |\n| O | X | 82.6  | 85.0 | 84.0   | 2.02  | 3.13  | 3.40    |\n| O | O | **86.8**  | **85.6** | **84.8**   | **2.49**  | **3.43**  | **3.52**   |\n\n---\n\n**W3. Lack of analysis: why the performance $\\cdots$ in MMVP.** \n\n**A3.** We appreciate the reviewer’s comment. First of all, there is a minor typo in the IXC2-VL result (in Table. 1, “Presence of Specific Features”), which should be 6.0 $\\rightarrow$ 60.0.\n\nWe have added statistical results to the table below for each category and conducted an analysis of 54 results (total 6 models and 9 question categories). Especially, as in the below summarized results, CODE boost performances for the specific visual categories where vanilla greedy decoding scored very low. This is attributed to the utilization of self-generated description that can effectively contextualize and integrate nuanced information to conduct complex categories like “State and Condition” and “Structural and Physical Characteristics”.\n\nEven if our method improve overall performance with consistency across 6 models, in visual categories such as \"Orientation and Direction\", our method slightly underperformed compared to the greedy decoding. This may because such categories (including other categories that the improvements seemingly marginal) require straightforward decisions that are enoughly handled by greedy decoding. For example, in orient. and quantity. categories, the answer is typically deterministic (*e.g.,* left, right, 1, 2, 3), which does not require a wider token searching pool compared to other types of questions. In these instances, the use of CODE does not seem to optimally benefit simpler tasks. We will definitely add these discussions and analyses to the potential final version.\n\n|        |  Total  | orient. | feature. |  state. | quantity. | position. | color. | structure. |  text. | viewpoint. |\n|:------:|:-------:|:-------:|:--------:|:-------:|:---------:|:---------:|:------:|:----------:|:------:|:----------:|\n| greedy |  34.11  |  33.34  |   30.15  |  13.89  |   27.08   |   28.33   |  57.78 |    13.89   |  70.00 |    32.50   |\n|  ours  |  37.95  |  32.70  |   34.09  |  22.59  |   27.08   |   36.67   |  60.38 |    22.22   |  71.67 |    34.17   |\n|  $\\Delta$ | +10.1\\% |  -1.9\\% |  +11.6\\% | +38.5\\% |   +0.0\\%  |  +22.73\\% | +4.3\\% |   +37.5\\%  | +2.4\\% |   +4.9\\%   |\n\n---\n\n**Q1. Have the authors tried $\\cdots$ used in CODE?**\n\n**A4.** Yes, our initial design was using KL-divergence to measure divergence. However, we would like to kindly remind that the most challenging point is that the vanilla KLD is not bounded (infinite upper bound), so that it is not practical to directly use of the divergence measurements. Therefore, Utilizing the Bounded Divergence (BD) that we used for CODE implementation can address such challenge by tightening the divergence range to 0~1, which is easily incorporated to DR and AIC as an adaptive hyper-parameter form.\n\n---\n\n**Q2. For the computational analysis $\\cdots$ comprehensive descriptions?**\n\n**A5.** Yes, the computational analyses in Table. 4 have considered all the computational costs of generating self-generated descriptions.\n\n---\n\n**Q3. A question about the DC, $\\cdots$ why does the CODE highly consider the variation?**\n\n**A6.** We would like to clarify the small misunderstanding. As the reviewer mentioned, if the two logit values from the visual input and its textual description are almost identical (thus minimal variation), the decoding process should indeed prioritize the original information (the reviewer’s understanding is correct). The one missing point is that our CODE process adds up the multiplication of the logit variation and the $\\alpha_{t}$. Thus, in ideal case, when the logit difference nearly gets to zero, the final addition to the original logit is canceled out, even if the $\\alpha_{t}$ is close to 1. Please note that this design can be universally applied without loss of generality across different logit difference scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XC1m56KhBv", "reviewer": "Reviewer_9CKq", "summary": "This paper introduce the CODE decoding method to contrast origin image with VLM-generated image description to reveal the missed or hallucinated content in naive decoding process. The method contains two innovations: 1) a Bounded Divergence guided selector to provide dynamic combining weight. 2) an adaptive information constrain based also on Bounded Divergence. The experiment results show promising and consistent performance improvement.", "questions": "See weaknesses.", "limitations": "the authors adequately addressed the limitations", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and addresses a crucial problem in MLLMs.\n2. The idea of contrast image and caption in VLM decoding process is novel. \n3. The proposed CODE method is simple and effective on multiple benchmarks.", "weaknesses": "1. The dynamic information flow control does not have that much technical novelty, as it is quite close to what IBD does.\n2. Like CODE, VCD also contrast image with modified inputs (i.e. corrupted images), but not discussed in detail in this paper. What makes contrasting image descriptions better than contrasting with corrupted images?\n3. What makes a good image description for CODE to use is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. The dynamic information flow control does not have that much technical novelty, as it is quite close to what IBD does.**\n\n**A1.** We thank the reviewer for the valuable feedback, and we would like to clarify the primary differences between our CODE and IBD [R1].\n\nThe dynamic information flow control in our method consists of two main regulating terms, $\\alpha_{t}$ and $\\beta_{t}$. The first term, $\\alpha_{t}$, is used to adjust the penalizing original logit values in contrastive decoding. In IBD, this term is obtained by constructing another image-biased model and calculating the JSD with the vanilla model. In contrast, our method uses only self-generated image descriptions from the model itself (thus reflecting its current understanding of the visual inputs in the textual representation) to calculate BD and obtain $\\alpha_{t}$. This approach eliminates the need to build a new model and allows for seamless application to other LMMs.\n\nThe second term, $\\beta_{t}$, is designed to dynamically control the threshold of $V_{head}$. In IBD, $\\beta$ is set to a constant value of 0.1 and is not dynamically regulated. Thus, $\\beta_{t}$ is our novel feature for CODE that can expand the token searching pool when the next token prediction considering distributional difference between visual contents and their comprehensive descriptions.\n\n---\n\n**W2. Like CODE, VCD also contrasts image with modified inputs (i.e. corrupted images), but not discussed in detail in this paper. What makes contrasting image descriptions better than contrasting with corrupted images?**\n\n**A2.** We respectfully argue that we have discussed the methodological differences of our framework with VCD in detail (line 43-44, line 102-104, line 160-162, and line 200-201), because VCD is one of the important baselines for experimental comparison. We would like to highlight again that, unlike the VCD that leverages the contaminated visual inputs with Gaussian noise, our method utilizes self-generated description from models themselves as contrasting counterparts. Our design is more reasonable because the comprehensive description reflects the current model's understanding of the visual inputs and integrates this understanding into the decoding process by penalizing original logit information, whereas VCD depends on the number of noise injection steps $T$ and could suffer from unexpected adversarial effects resulting from the noise. To improve clarity, we will add this discussion to the potential final version.\n\n---\n\n**W3. What makes a good image description for CODE to use is not discussed.**\n\n**A3.** Here, as we described in Table. 5 of Appendix. A, we carefully design the instruction for LMMs to obtain comprehensive descriptions for the given visual contents, and its curated prompt aims for the self-generated descriptions to span possible visual contents thoroughly, answering any potential questions about the given image. \nAdditionally, as we analyzed in Sec. 3.1 and Fig. 2, following the amateur model selection philosophy of contrastive decoding, the generated comprehensive descriptions reflect the given visual understanding from the models themselves and subsequentially contrasts with the original logit information to enhance response coherence during the decoding phase. \n\nAlthough we have demonstrated the effectiveness of self-generated descriptions in our framework across 6 LMMs and 6 benchmarks, as the reviewer pointed out, there could be more optimal descriptions that can further mitigate hallucinatory responses. This could be an interesting future research direction, and we will definitely include this discussion in the potential final version. \n\n---\n\n[R1] IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding, *arxiv preprint*, 2402.18476", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for the constructive feedback, which we will incorporate into the potential revised version. We also appreciate to all reviewers (*Z6jD*, *FLJA*, *9CKq*) for acknowledging the novelty of our paper, which is the use of self-generated comprehensive descriptions to mitigate hallucinatory responses from existing LMMs.

We have carefully reviewed several concerns that reviewers raised and responded to each question individually. Please feel free to discuss any remaining concerns that can improve our manuscripts during the discussion period.

Meta-review: This paper proposed a contrast decoding method called CODE for handling hallucinations of large multi-modal models. It is interesting to use descriptions for enriching language information to address hallucination problems. The idea of contrast image and caption in the VLM decoding process is new, and the proposed method is quite effective on multiple benchmarks. Due to these merits, most of the reviewers are positive about this paper. Though one of the reviewer still has concerns about the details of experiments, authors have addressed them during rebuttal. Thus, AC recommends acceptance.
Decision: Accept (poster)