DAMO: Decoding by Accumulating Activations Momentum for Mitigating Hallucinations in Vision-Language Models
Authors: Kaishen Wang
Hengrui Gu
Meijun Gao
Kaixiong Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-Language Models (VLMs)
Hallucinations
Decoding Method
Momentum Techniques
Primary area: foundation or frontier models, including LLMs
TL;DR: To address the hallucination problem in Vision-Language Models (VLMs), we propose a novel decoding method inspired by momentum techniques.

Abstract: Large Vision-Language Models (VLMs) exhibit significant potential in multimodal tasks but often struggle with hallucinations—responses that are plausible yet visually ungrounded. In this work, we investigate the layer-wise prediction tendencies of VLMs and conduct an in-depth analysis of their decoding mechanism. We observe that VLMs tend to ``overthink'' during the final stages of decoding, making significant prediction shifts in the last few layers often favoring incorrect results, which leads to a surge in hallucinative outputs. Leveraging this localized pattern, we propose a novel decoding strategy inspired by the momentum analogy used in gradient descent-based optimizers. Our method enforces decoding consistency across layers in an adaptive manner during forward passes—an under-explored approach in existing works. This strategy significantly improves the reliability and performance of VLMs in various multimodal tasks, while introducing only negligible efficiency overhead.

Published as a conference paper at ICLR 2025
DAMO: DECODING BY ACCUMULATING ACTIVA- TIONS MOMENTUM FOR MITIGATING HALLUCINA- TIONS IN VISION-LANGUAGE MODELS
Kaishen Wang Hengrui Gu²; Meijun Gao'; Kaixiong Zhou'
'University of Maryland, College Park 'North Carolina State University 'Michigan State University (ka meiishen73322 guhenry21 gma 1. com, gaome juemsu. edu, kzhou22 csu edu
ABSTRACT
Large Vision-Language Models (LVLMs) exhibit significant potential multi- modal tasks but often struggle with hallucinnttonn--respoone that are plausible yet visually ungrounded. In this work, we investigate the layer-wise prediction tendencies of LVLMs and conduct in-depth analysis of their decoding mech- anism. We observe that LVLMs tend to "overthink" during the final stages of decoding, making significant prediction shifts in the last few layers often favor- ing incorrect results, which leads to surge in hallucinative outputs. Leverag- ing this localized pattern, we propose a novel decoding strategy inspired by the momentum analogy used in gradient bescent-baaed optimizers. Our method en- forces decoding consistency across layers an adaptive manner during forward passes-an an under-explored approach in existing works. This strategy significantly improves the reliability and performance of LVLMs in various multimodal tasks, while introducing only negligible efficiency overhead. The code is available at https::/qithuut com wwt.om//unannttuAaa....
INTRODUCTION
The recent advancement in model architecture and training methodologies has led unpreedenttd development and wide adoption of Large Vision-L.anguage Models (LVLMs) (Devlin 2018 Chen
al., 2019 Liu et al. 2024c] Zhou al. 2023a Ye al. 2023b Peng al. 2025 Through bridging the gap between visual and textual modalities, they offer a viable and promising solution
for various multimodal tasks such visual question answering and image captioning (Liu et al.
2024d Ye et al. 2023a Zhu et al. 2023b Li et al. 2023b Lee et al.
2024). Despite their success, LVLMs continue struggle with hallucinations (Liu al., 2023a Yin al. [0023). a phenomenon
where they tend to generate syntactically plausible yet visually ungrounded responses (see Fig. [a). This intractable challenge significantly undermines users' trust their output, thereby hindering
their broader application in real-world scenarios (Chen et al. 2024b Hu et al. 2023).
Several studies have delved into the mechanisms behind hallucinations. attributing them to factors such over-reliance statistical pre-training biases (Agarwal et al. 2020: Agrawal et al. 2016), language priors, and attention deficiency (An et al. 2024). All these proposed conjectures suggest that the inherent neglect of modality information integrated at later stages can lead to inaccurate outputs. Additionally, indiscriminate associative reasoning on linguistic and visual data cause hallucinated responses gradually dominate the decoding distribution. Thus they leverage knowledge editing methods (De Cao et al.] [2021] Meng et al.] [022] Gu ct al., 22024a Khandelwal al.] 2024| Shi et al. 2024]) Gu et al.] 2024b) Zhong et al., [224]] Liu et al. [023c] Jiang et al.) to inject new answers or knowledge into foundation models and erase hallucination on mis-answered samples. The existing methods edit knowledge of
LVLMs by either fine-tuning specific memory parameters or maintaining an external memory of updated knowledge facts. Despite these efforts, we propose a critical doubt remains unresolved: Does LVLMs really don't 't know ground-truth answers for solving those hallucinated problems?
"These authors contributed equally. The order of authorship decided through dice rolling.

---

Published as a conference paper at ICLR 2025
Keynurd Protublliyy Acmas Layers
Seeple pigll
Suuple Tacre fyt poope
Somple There am Das
Semple D. Thee am 722 peips
How many people are this tmage?
14
25
2
(b)
(a) Hallucination example: the model is asked "How many people are in this image?" for image of two people skiing. And the analysis of probability distribution for key token. (b) Probability of predicting the token 'two' across layers. The blue curve shows Regular decoding, while the red shows the effect of DAMO. The shaded area marks the stages of Visual Information Extraction and Loss of Visual Information.
Figure 1:
This suspicion is originated from following observation that LVLMS generally can decode the desired outputs at early layers but later they turn focus on other unrelated contents.
To answer this question, we analyze the layer-wise prediction tendencies of LVLMs and summarize the patterns of hallucination progressing accordingly. Specifically. we leverage a vanilla MiniGPT- 4 ZZbu et al. (2023c) and manually collect an evaluation dataset on visual question answering task al.,
from COCO 4 Lin al. (2014). Motivated by the idea of early exit (Teerapittayaoon 2016 Elbayad et al. 2019) [Schuster et al. 2022), we decode the immature hidden states exited by LVLMs in advance and pay attention confidence variations of the ground-truth token across layers. A qualitative illustration of this phenomenon is shown in Fig. 1b The decoding probability of the ground-truth answer, 'two", steadily increases across the early layers and then dominates the decoding distribution, revealing that LVLMs internally owns the capability to extract the fine- ine-grained visual semantics. However, its decoding tendency drastically deviates at the later layers and token 'two" is not the first choice anymore at the last layer. Based this observation, we hypothesize that the hallucinations within LVLMS manifest as localized surges at the later layers, which in turn suppresses the pre-existing and visual related information in the decoding distribution, ultimately leading to visually ungrounded responses,
To address these concerns and improve the truthfulness of LVLMs, we propose novel decoding method, Decoding by Accumulating Activations MOmentum (DAMO). Specifically, DAMO accu- mulates a momentum by aggregating the activations layer-by-layer and uses this momentum, which contains the historical updating trends, to correct hidden states at the later layers of LVLMs. This approach amplifies visual semantics consistently extracted throughout the inference while reduc- ing hallucination biases that are intensively introduced at the later layers. This idea is built upon two foundations:
gradient descent, momentum has been widely proven to an effective accel- eration technique by accumulating persistent updates towards low-curvature directions, driving the model closer to the global optimum (Sutskever et al. 2013, Polyak| 1964). Ahn et al., 2024 have demonstrated that transformer can implement preconditioned gradient descent, and that the forward computation process of a L-layer transformer is conditionally equivalent to steps of gra- dient descent. Therefore, implementing "momentum" in the forward computation of transformers to counteract the localized and intense emergence of hallucinations is an intuitive and expectedly effective approach. Fig. lb shows that DAMO successfully reduces the hallucinations that occur during the inference of the vanilla MiniGPT-4.
Experiments on MME and POPE datasets across various LVLMS demonstrate that DAMO signifi- cantly mitigates hallucinations, resulting in more visually grounded and accurate predictions. When applied to mPLUG-OWl2 Ye et al. (2023b) without any additional training. DAMO achieved a re- markable 100-point improvement over regular decoding the MME dataset, outperforming all other baseline methods. Bulilding on this success we further extended DAMO to L.Mss, where is also delivered strong results on benchmarks such as TruthfulQA Lin et al. (202T). FACTOR Muhigay et al. (2023). and GSMSK [Cobbe et al. (202T). indicating that DAMO possesses notable transferability across different tasks and model types. The contributions are summarized as follows:
2

---

Published as a conference paper at ICLR 2025
We identify that hallucinations in Large Vision-Language Models (LVLMs) primarily occur during the inference process and are driven by localized surges in the later layers, which suppress visual information.
We introduce Decoding by Accumulating Activations Momentum (DAMO), novel ap- proach that reduces hallucinations by accumulating activation momentum, significantly improving visual grounding in LVLMS across multiple benchmarks.
2 RELATED WORK
Large Vision- Langua-L Models The evolution of Large Vision-Languagn Models (LVL.Ms) has progressed from BERT-based models (Devlin) 2018 Lu et al. 2019 Tan & Bansal 2019] Chen
al. 2019 to those integrated with Large Language Models (LLMs) (Luu et 2023b Zhu al.
2023c Zhou et al. 2023a Ye et al., (023b), which have significantly improved their capabilities.
Early models like ViLBERT Lu et al. (201).. LXMERT Tan & Bansal (2019), and UNITER Chen et al. (2019) effectively merged visual and textual features using BERT-style architectures. The introduction of LLMs enabled LVLMs such as CLIP Radford et al. (2021) and ALIGN [Jia et al.
(2021). which significantly enhanced adaptability and performance through end-to-end training. Recent works like LLaVA (Liu et al.] 2023b] (024b) and InstructBLIP Daa et al.] (2023) further refined these models using visual instruction tinettunigg demonstrating adaptability across diverse vision-language tasks and showcasing a growing trend toward task-specific approaches.
Hallucinations in LVLMs In Large Language Models (LLMs), hallucinations have been exten-
sively studied (Ji et al., 2023, Yao et al. 2023 Zhang et al. 2023b a Liu et al. 2024a Xu et al. )024). particularly in contexts where generated text diverges from the input factual reality. In LVLMs, hallucinations present additional challenges due to the alignment between visual and tex- tual data (Biten al.
2022 Li et al. 2023c; Wang et al., 2024), which increases the potential
for hallucinations. This issue is particularly prevalent in tasks such as image captioning and visual
question answering (Liu al. 2023a Yin et al. 2023 Zhou al. 2023b) [hu al. 2024a| Gunjal
et al. 2024). where maintaining coherence between modalities is critical.
Addressing Hallucinations in LVLMs Some researches (Gunjal et al. 2024 Wang et al. 2024 have attempted to mitigate hallucinations through fine-tune models. However, this approach can be resource-sensitvvy leading to increased computational costs, while also risking decline in perfor- mance on other tasks and limiting the model's 's overall versatility. Additionally, there are also some approaches Zeng et al., 202T: Kim et al. 2023 Zhou al. 2023b) trying to enhance alignment between visual and text modalities to address hallucinations.
At the same time, DoLA Chuang et al. (2023) has been proposed to address hallucination issues in LLMs through contrasting decoding. achieving promising results without the need for additional training. Subsequently, contrasting decoding was applied in LVLMs to address hallucination issues. VCD (Visual Contrastive Decoding) Leng et al. (2024) is training- method designed to mit- igate object hallucinations in LVLMs, contrasting output distributions from original and distorted visual inputs to reduce reliance on statistical bias and unimodal priors. HIO (Hallucination-Innuuedd Optimization) Chen et al. (2024a) addresses hallucinations in LVL.Ms by amplifying the contrast between hallucinatory and targeted tokens using a fine-tuned Contrary Bradley-Terry Model. IBD Image-BBased Decoding) Zhu et al. (2024b) mitigates hallucinations in LVLMS by contrasting pre- dictions from standard LVLM with those from an mmage-biased LVLM, amplifying image-related information and reducing over-reliance on text. The common issue with these methods is that they only correct the next token prediction through logits contrasting at the final layer, without attempting to identify and address hallucinations in the hidden states during the inference process. In addition, OPERA Huang et al. (2024) mitigates hallucination by reducing reliance on summary tokens during decoding and adjusting token selection based on previously generated tokens.
3 METHOD
We first introduce the decoding process in LVLMs to provide some preliminaries. Next. elabo- rate our motivation to transfer the idea of momentum into LVLM decoding by an intuitive example. Finally, propose novel, seamlessly integrated decoding method called DAMO, which aceu-
3

---

Published as a conference paper at ICLR 2025
mulates the raw activation updates into the momentum layer by layer correct the hidden states' updating direction, effectively smoothing the update process and mitigating hallucinations.
3.1 DECODING IN VISION-LANGUAGE MODELS
Given an input representation n. at time step 4, the standard decoding process of a N-layer LVLM is outlined as follows (Liu et al. 2024c 7hu al. 2023a Li al. 2023a):
1att f(())= + hy, j o=.,...,NN p(1++11/ = (6ftmax(((hh))))
N 1,
where f-) refers to the j-th transformer layer, and hig denotes the hidden states layer j at time step fh²) represents the activation output by layer j. which the sum of the activations pro- duced by both the MLP and the self-attention modules at the current layer. Φ(-) represents the language head that predicts the next-token probability over the entire vocabulary. Although o(-) is only trained to transform the final-layer activation (i.e., h into the next-token distribution, several studies (Teerapittayaoon et al., 2016 Elbayad et al., 2019 [Schuster et al., [022] [hou et al.) [025] Ouyang al. (2025) have demonstrated that applying the language head directly to intermediate ac- tivations exited from earlier layers can also produce meaningful distributions, reflecting prediction tendencies of LVLMs. We formally describe early exit as follows:
Pj P (x;e) = softmax(6(hk)) j= -
(2)
Here, b/ refers the intermediate activation exited by layer j at time step Section 3.2 utilize this technique to illustrate our motivation.
we
3.2 MOTIVATION
In Visual Question Answering (VQA), LVLMs take an image and a related question as input, pro- ducing an answer based on visual cues. We conducted preliminary experiment on scall-scale, self-constructed dataset to explore hallucinations in LVLMs. The dataset, derived from randomly selected images from COCO val2014 Lin et al. 2014)), included questions focused on quantifiable attributes such as color, number, and direction. We identified 100 instances exhibiting hallucinations and analyzed the probability distribution of key words (e.g., 'two" in Figure (b) across the model's layers during inference.
Our findings revealed that 75% of the samples shared a consistent issue: while LVLMs are adept extracting visual information from images, hallucinations frequently emerge in the later layers of the inference process. This analysis highlighted two key insights: LVLMs are already proficient in capturing detailed visual information, so further intensifying image-text fusion unnecessary. e Hallucinations primarily occur during later inference stages, so we only need to correct hallucina- tions at these stages of the inference process.
Motivated by this, we propose DAMO that refines activations by leveraging earlier layer informa- tion. helping to preserve visual context and reduce hallucinations, ultimately improving prediction accuracy.
3.3 DECODING BY ACCUMULATING ACTIVATION MOMENTUM
Momentum in Gradient-based Optimization Momentum, a popular technique in gradient-based optimization, has been proven to be fairly effective for providing better convergence rate (Polyak) 1964). The idea of this method is preserve the historical updating trends from previous steps in the momentum and use this value to correct the update direction of the current step. Specifically, the model parameters 0, at the current step t are obtained according to the following formula:
12= f + (1 (
(3)
0 = 0 - 770e
4

---

Published as a conference paper at ICLR 2025
Sturting Layer
Existence
Count
Position
Color
Celebrity
Landmark
Artwork
Vanilla Decoding
190,00
160,00
138.33
165.00
141.50
135.88
156.25
161.25
118.50
125.00
1491,71
0-th
185.00
115.00
123.33
153.33
78.57
90.29
139.75
107.00
105.75
132.50
1230.53
10-th
185.00
115.00
118.33
168.33
81.63
89.71
136.00
102.75
108.00
110.00
1214.76
16-th
195.00
145,00
138.33
165.00
143.54
136.76
157.75
166.00
118.50
140.00
1505,89
18-th
190.00
145.00
133.33
160.00
144.56
136.47
159.75
167.75
121.50
130.00
1488.
20-th
195,00
150.00
133.13
165.00
142.52
174.12
157.00
165.25
115.75
132.50
1490.47
22-th
190.00
143.33
165.00
144.56
156.25
167.5
118.25
132.50
1501,80
24-th
195.00
153.33
148.33
165.00
142.52
134.71
157.00
163.75
116.25
132.50
1508.39
Table 1: Using L.5 the foundation model, the performance of Activation Momentum with different refinement starting layer on the MME dataset. Each column refers to a hallucination category and the best results in each column are marked with bold.
1
Here, represents the learning rate, and V (O,,- denotes the gradient w.r.t. the training objective. The term U, accumulates past gradients, serving as the real updates at the current step, while 8 is the trade- off coefficient balancing historical and current infor- mation. This momentum- basudd approach is particu- larly effective in training deep neural networks, smooths the optimization trajectory, improving both stability and convergence speed (Sutskever et al. 2013).
(1/2
2/42
fr2()
N/12
fun()
[
W+1
W/11
Activation Momentum in Vision-Language Mod- els Inspired by traditional momentum, we introduce the Activation Momentum (AM) in LVLMs, expect- ing to transfer its beneficial properties guiding the change of hidden states toward consistent and smooth update direction while preventing abrupt localized deviations from historical updates. Specif- ically, given the hidden state ii at layer j at time step the j-th transformer layer f)(-) can produce the raw activation Vi++t = Sj (h²). In vanilla decoding process, the next-layer hidden state is ob- tained directly by 1 1/ + V To accumulate the previous activation status used to correct into the momentum term v/. The momentum
(
f
f0
Tw/
(a)
(b)
Figure 2: (a) The vanilla decoding scheme in
LVLMs. (b) DAMO's decoding scheme.
later layers, our proposed AM first integrates Vht+
variable formally defined as:
(4) where B, similar to its role in gradient descent, is the trade-off coefficient balancing historical and current information. And the term v²+1 represents the momentum after incorporating the raw acti- vation produced at the current step, which actually serves as the refined activation. Therefore, the revised updating process of hidden states is shown below:
Ey But + 31-3)V
1o hi + ut
(5)
where h: can partially reflect the update trends of the previous layers and will then as the input to the subsequent layer. The aforementioned decoding process, involving the accumulation of activations momentum, is illustrated in Fig. 2b
3.4 MECHANISM OF ADAPTIVE ACTIVATION REFINEMENT
AM represents naive implementation of momentum-basee decoding in LVLMs. However, unlike its use in optimization, we argue that while it essential to maintain the momentum value (Equ. 4 all the time in order to preserve more historical information, Equ. 5 does not need to be applied across all layers. This is because hallucinations primarily emerge in the later layers so continuously refining the raw activations may suppress valuable updates encoding rich visual semantics generated by certain early layers.
To validate our proposition, conduct preliminary experiment the MME dataset (Fu al. 2023), a comprehensive benchmark to measure the ability of LVLMs in various aspects. The results shown in Table. [] demonstrate that: Starting activation refinement too early (at layer 0 or 10)
5

---

Published as a conference paper at ICLR 2025
significantly impairs the visual reasoning ability of LVLMs. @ Varying the starting layer for refine- ment can enhance different model capabilities (e.g.. layer 16 excels in OCR tasks, while layer 24 improves positional perception). These observations suggest that activation refinement should begin in the later layers, and a fixed starting layer is insufficient to achieve comprehensive improvements across various key abilities of LVLMs.
Thus, we propose the mechanism of adaptive activation refinemens, which also leverages the idea of momentum and adaptively determines, during forward computation, whether the model has reached the hallucination surge and selects the appropriate refinement starting layer. This mecha- nism, together with AM, constitutes our complete approach, Decoding by Accumulating Activations MOmentum (DAMO). For simplicity of notations, we denote the early exit distribution Py (2++1 (x,)) at layer (from Equ. 2 as p), We focus on the update direction of the early exit distribution, main- taining momentum V Pd to accumulate the update trends of the previous layers:
P pi -i-1 -
(6)
where pi denotes the relative change of the early exit distribution caused by layer j. reflecting the current prediction tendencies of LVLMs. PPj is the momentum variable, preserving historical prediction trends of previous layers. Intuitively, if P diverges significantly from the momentum variable Pi., it indicates a notable deviation from the original trend, potentially signaling the onset of hallucination patterns. Based on this hypothesis, we use the cosine similarity between V pi and PPj as the criterion for triggering activation refinement. Building on this hypothesis, we use the cosine similarity between P and Pi as the criterion for triggering activation refinement. Specif- ically, if the similarity falls below predetermined threshold т, activation refinement is initiated and continues until the final layer.
3.5 COEFFICIENT ADJUSTMENT FOR HOISE-RESISTE MOMENTUM
After we have reached the hallucination surge, keeping the trade-off coefficient 3 (from Equ. 4 fixed leads to several issues: The hallucinated activations produced by the later layers will be incor- porated into the activation momentum uy acting as noise and gradually diluting the correct historical information contains. To address this, we set the initial value of 33 to B1 and change its value ac- cording the following rules:
B2 if Cosine Pi,, V Pi) <T
8
B1 Cosine (VP), P 2T
Here must ensure that 81 (3.. Transitioning from 31 to means that the onset of a hallucina- tion surge, accompanied by significant changes in prediction tendencies. Consequently, it becomes necessary to preserve more historical information (set a higher value for 3. i.c. B2), enhancing u/ the momentum term's 's resistance these hallucinatory elements.
EXPERIMENTS
In experiments, we first introduced the datasets, models, and baselines the Setup section, fol- lowed by a detailed presentation of the Results. We evaluated our DAMO method on the compre- hensive hallucination dataset MME and examined object hallucination using the -EEMM-annttted MSCOCO and A-OKVQA datasets from POPE. To assess robustness, tested the generalized dataset LLaVA-Bende Additionally, we evaluated the adaptive activation refinement for starting layer selection and adaptive coefficient adjustment. Finally, we applied our DAMO to LLMss to assess its transferability to mitigate hallucinations.
4.1 SETUP
Datasets We evaluate our proposed method using four datasets designed to assess hallucination is- sues in Large Vision-Language Models (LVLMss). MME Fu et al. (2023) offers comprehensive

---

Published as a conference paper at ICLR 2025
benchmark featuring 14 tasks categorized into perception and cognition. POPE (Polling-based OB- ject Probing Evaluation) Li et al. (2023c) is a scalable framework for detecting object hallucinations in LVLMs, utilizzig SEEM- -nnootated datasets from MSCOCO Lin et al.] (2014) and A-OKVQA Schwenk et al. (2022). I.LaVA VA-Bench,, a generalized dataset, comprising 24 diverse images and 60 questions across categories like simple QA, detailed descriptions, and complex reasoning, used to
further evaluate the model's generalization and robustness.
Models Numerous high-performing LVLMs have emerged recently. For our evaluation, we selected three models: LLaVA 1.5 Liu et al. (023b), INF-MLLMI [Zhou et al. (2023a), and mPLUG-OW12 Ye et al. (2023b). each demonstrating strong performance on established benchmarks. Notably, all these LVLMs are equipped with 7B Large Language Models (LLMs).
Baselines We compare our method with four baselines. Regular responses are generated using the original LVLMs. Visual Contrastive Decoding (VCD) et al. (2024) mitigates object hallucina- tions by contrasting output distributions from original and distorted visual inputs. DOLA CCuang al. (202) is transferred to LVLMs, with fixed mature layer index at 32 and multiple candidate premature layer indices ("O, 2, 4, 6, 8, 10, 12, 14") to fine-grain the model's internal decision- making process. For OPERA OPERAHHuang et al. (202)), due to the computational resource limitations, we set num_beams to 4, while keeping all other settings consistent with its official configuration. To ensure fairness, we set the temperature to 0 for all comparisons.
Hyperparameters Setting We have provided all the hyperparameters used in our experiments, in- cluding T, B1, B2 and a, which can be found in the Appendix. Additionally, we tested the model's sensitivity to these hyperparameters, with detailed experimental results also available in the Ap- pendix. The experimental results confirm that our hyperparameter selection is optimal.
4.2 RESULTS
Evaluation on Comprehensive Hallucination Dataset
b Implementation We adopted the MME benchmark to evaluate DAMO across 3 models and reported results in the perception category, which contains 10 tasks, following the settings used in other hallucination studies.
D. Q: Whether our DAMO outperforms other decoding methods on the comprehensive dataset? Yes, as shown in Table 2 present the experimental results of various decoding methods the MME dataset. Notably, DAMO almost outperformed other decoding strategies across all models, achiev- ing a total scores of 1515.89, 1520.74 and 1437.46 on LLaVA115, INF-MLLMI and mPLUG-Ow12, respectively. Specifically, on the mPLUG-OW points. Moreover, in the "Celebrity" task, it achieved the highest score of 164.41 44.41 among all meth- ods. Additionally, DOLA demonstrated no performance enhancement on the INF-MLLMI model, suggesting that directly transferring DOLA LVMs may not guarantee performance improvements.
model, DAMO surpassed the VCD method by 38.91
> Q: Does our DAMO also have relatively low memory consumption? Yes, our DAMO also exhibits relatively low memory consumption. As shown in the last column of Table 2 the memory consump- tion of DAMO is nearly identical to that of Regular. In contrast, VCD requires two forward passes to contrast output distributions from original and distorted visual inputs, necessitating the storage of logits from both passes. DOLA also needs to retain logits from intermediate premature layers, while OPERA utilizes beam search, resulting in significantly higher memory consumption to 50GB on the INF-MLLMI model. In summary. DAMO does not require storing any intermediate logits, as we continuously use momentum for modifications, which contributes to its efficiency in memory usage.
Evaluation on Object Hallucination Dataset
Implementation To evaluate the effectiveness of our method in addressing object hallucinations, we compared with various decoding strategies on the SEEM-anmottted MSCOCO and A-OKVQA datasets provided by POPE. We reported four key metrics: Accuracy, Precision, Recall and Fl Score, focusing primarily on Accuracy and Fl Score for brevity, with complete results available in the Appendix.
7

---

Published as a conference paper at ICLR 2025
Model
Desodingg
Exiseence
Fositinn
Color
Celebxity
Landmark
Artwork
OCR
Toeal
Menoxy
160.00
165.00 .00
141.5
135.88
156.25
161.2 25
125.00
1491.71
14.6CIR
VCD
188.33
140,00
133.53
153.25
166.00
125.00
1458.54
LLAVALS
DULA
190.00
159.33
142.33
105,00
133.24
157.25
160.50
119.50
125.000
1892.11
15:1GB
OFERA
195.00
158.33
148.33
175.00
142.52
131.18
157.00
161.50
117.00
132.50
1518.36
22 22,5GB
DAMO
195.00
150,00
143.33
165.00
144.56
135.00
157.00
166.00
120.00
140.00
1515.89
14.6BB
Rogular
195.00
150.00
151.67
160:00
150.00
140.20
157.75
155.50
122.50
110.00
1492.71
17 67.6
VCD
190.00
138.33
160.00
160,00
1:2.52
135.59
157.00
153.50
113.25
105.00
1455.19
18.6GB
INF-MLLMI
DOLA
195.00
150,00
151.07
100:00
150,00
1401.29
157.75
155.50
122.50
110.00
1492.71
18.2GB
OPERA
195.00
155.00
151.67
160.00
149 32
139.41
156.25
154.00
122.50
110.00
1491.15
48 SCBB
DAMO
195,00
150,00
158.33
146.76
156.25
154.75
128.50
117.50
1520.74
17.6GB
Regular
180,00
145,00
73,33
136.73
141.18
157.25
157.75
127.25
102.50
1337.66
I6.ICB
158.53
154.00
117.50
1398.55
16
WPLUUG-NEE
DOLA
150.00
160:30
70:00
1500-00
159.12
160.50
110.00
1427.11
16.2GB
OPERA
190.00
160.00
70.00
150.00
92
160.58
160.50
150.50
00
110.00
1428 80
24 DGB
DAMO
190.00
160.00
75,00
145,00
148.30
164.41
157.50
155.75
131.50
110.00
1437.46
16.1GB
Table 2: Experimental results of various decoding strategies on MME dataset across three models: LLaVAL.5, INF-MLLMI and mPLUG-OW12.
> Q: Does our proposed DAMO address object hallucinations effectively? Yes, DAMO achieves significant performance improvements in both accuracy and F1 score, as evidenced by results from MSCOCO and A- A-OKVQA datasets.
MSCOCO dataset Results from the MSCOCO dataset, as shown in Table 3 demonstrated the ef- fectiveness of DAMO. In the popular setting with mPLUG-OW12, DAMO improved accuracy from 80.73% to 82.77%, while F1 score increased from 82.52% 83. 33.80% In the challenging adver- sarial setting. DAMO further enhanced accuracy and Fl score on LLaVA1.5 compared to Regular decoding. with improvements of 0.73% and 0.36% respectively. In contrast, methods like VCD sometimes exhibited performance declines; for instance, in the popular setting, VCD's accuracy and Fl score dropped by 1.80% and 1.62% on LLaVA1.5 compared to Regular. The consistent superiority of DAMO reinforces the advantages of our approach.
A-OKVQA dataset Turning to the A-OKVQA dataset shown in Table 3 DAMO again outper- formed other decoding strategies. In the popular setting with MPLUG-OWLL DAMO achieved sig- nificant improvements over the Regular decoding, with accuracy and F1 score increases of 3.30% and 2.07%, respectively, while other methods showed only minimal enhancements, with DOLA achieving improvements of merely 0. 16% and 0.09%. The improvements on A-OKVQA were even more pronounced than those observed on MSCOCO, further underscoring the capability of our model.
Evaluation on Generalized Dataset
Implementation To extend our evaluation beyond binary tasks, we conducted experiments on the generalized LLaVA-Bench dataset on LLaVA1.5, employing GPT-API for performance assessment.
D Q: Can our proposed DAMO perform well on generalized dataset? Yes, our DAMO demonstrated strong performance the generalized V.L.VA-Bench dataset. As shown in Table 4 DAMO outperformed most decoding strategies across all categories. In the detailed description category, DAMO achieved score of 75.87, surpassing DOLA by 1.20 points. It also excelled in complex reasoning, with a score of 88.39, which is significantly higher than OPERA's 69.64. With an overall score of 78.13, our DAMO demonstrated enhanced robustness and generalizability across diverse tasks, suggesting its potential for solving other challenges as well.
Evaluation about Adaptive Activation Refinement
& Implementation To fairly compare the effectiveness of adaptive starting layer selection strategies for activations refinement, we conducted several ablation experiments with fixed starting layers at the 0-th, 10-th, 16-th, 20-th, 24-th, and 28-th layers. All other parameters were kept consistent across experiments to ensure a fair comparison.
Q: Is our Adaptive Activation Refinement superior to the fixed layer starting for momentum? Yes, the results presented in Figure obtained on the MME dataset with the LLaVA1.5 model, indicate that while fixed starting layers provide modest improvements over regular decoding, the adaptive layer selection consistently outperforms all fixed-layer approaches, demonstrating clear advantages by enabling more responsive and effective decoding process.
Evaluation about Adaptive Coefficient Adjustment
Implementation After determining the hallucination surge via adaptive activation refinement, compared the performance of three setups: using only 31, using only 83 (without switching), and employing the adaptive switching strategy between 31 and 32. The experiments were conducted on
8

---

Published as a conference paper at ICLR 2025
Dataset
MSCOCO
A-OKVQA
Setting
Model
Decoding
Accuracy
FI Score
Accuracy
F1 Score
Regular
89.63
89.74
87.30
88.49
VCD
87.53
87.81
85,00
86.49
LLVA1
DOLA
89.67
89.74
87.40
88.57
OPERA
89.87
89.95
87.27
88.50
DAMO
89.97
89.92
87.90
88.93
Regular
91.17
90.92
90.60
90.97
VCD
90.00
89.65
89.87
90.21
Random
INF-MLLMI
DOLA
91.17
90.92
90.60
90.97
OPERA
91.27
91.02
90.57
90.94
DAMO
91.33
91.00
91.33
91.59
Regular
86.27
86.88
81.57
83.89
VCD
84.40
84.79
82.53
84.16
mPLUG-OW12
DOLA
86.33
86.92
81.83
84.07
OPERA
86.23
86.84
81.53
83.86
DAMO
87.20
87.44
84.60
86.03
Regular
86.23
86.82
80,30
83.21
VCD
84.43
85.20
77.50
81.07
LLVVA1.5
DOLA
86.20
86.75
80.47
83.32
OPERA
86.30
86.88
80.47
83.38
DAMO
86.70
87.07
81.27
83.84
Regular
88.83
88.78
85.70
86.88
VCD
87.60
87.43
85.00
86.17
Popular
INF-MILMI
DOLA
88.83
88.78
85.70
86.88
OPERA
88.80
88.74 74
85.70
86.89
DAMO
89.30
89.11
86.67
87.62
Regular
80.73
82.52
75.97
79.98
VCD
81.00
81.12
75.70
79.21
mPL.G-OW12
DOLA
80.87
82.60
76.13
80.07
OPERA
80.70
82.48
75.93
79.94
DAMO
82.77
83.80
79.27
82.05
Regular
79.70
81.71
69.33
76.10
VCD
78.13
80.38
67.90
75.01
LLVVAL
DOLA
79.73
81.68
69.53
76.21
OPERA
79.77
81.77
69.20
76.09
DAMO
80.43
82.07
70.77
76.88
Regular
84.87
85.38
76.13
79.88
VCD
84.17
84.47
76.37
79.82
Adversarial
INF-MLLMI
DOLA
84.87
85.38
76.13
79.88
OPERA
84.87
85.36
76.10
79.85
DAMO
85.73
86.00
77.43
80.71
Regular
76.17
77.69
67.37
74.63
VCD
77.10
77.00
68.80
74.85
mPLUG-Ow12
DOLA
76.73
77.87
67.50
74.68
OPERA
76.87
78.01
67.30
74.58
DAMO
78.63
78.87
70.03
75.98
Table Experimental results of various decoding strategies on the SEEM- annotttdd MSCOCO and
A- A-OKVQA datasets from POPE using three models: LLa INF-MLLMI, and mPLUG-OWI2. The best values for each metric across all models and decoding strategies are highlighted in bold. the MME dataset using three models: LLaVA1.5, INF-MLLMI, and mPLUG-Ow with all other parameters kept constant across experiments.
Q: Does the adaptive momentum coefficient adjustment strategy outperform fixed coefficient se- tups? Yes, the results presented in Figure Figure3bjdemonee demonstrae that the adaptive 3 switching strategy sistently outperformed the fixed-coefficient setups across all models. For instance, the mPLUG- Owl2 model, using 31 resulted in total score increase of 48.66, while 8, provided larger increase of 90.49. In contrast, our adaptive approach achieved an impressive total score improvement of 99.8.
Transferability Evaluation

---

Published as a conference paper at ICLR 2025
Decoding
Conversation
Detail
Complex
Overall
Regular
58.82
70.33
87.79
75.22
VCD
60.88
59.67
83.46
71.12
DOLA
61.47
74.67
88.04
77.17
OPERA
66.76
51.00
69.64
64.17
DAMO
63.24
75.87
88.39
78.13
Table 4: Comparisons about various decoding methods on LLaVA using LLaVA-Bent dataset.
Compuriea about Adsptive Activation Refinemcnt
Evaluation on Adaptive Coefficient CootticienAAdujstteee Adjustment
Percepatin Scares
110
105 66
3-A
ELIVALS
INEMILMI
Sayt
(b)
Figure 3: (a) Comparison of different starting layens for DAMO (0-th, 10-th, 16-th, 20-th, 24-th, and 28-th) against regular decoding and our adaptive activation refinement on the MME dataset using the VAI. model. (b) Comparison results adaptive momentum coefficient adjustment across LL5 INF-MLLMI and mPLUG-ON Owll2 on MME dataset. 31 indicates the use of only 31, 82 denotes the use only 32, and Adaptive refers adaptive adjustment between 31 and B2-
Implementation To evaluate DAMO's transferability, we implemented it on the LLaMA2-7B model specifically targeting the effectiveness in addressing hallucinations in LLMs. The experi- ments focused on four datasets: TrthfulQA, FACTOR (News, Wiki), StrQA and GSMSK. We uti- lized Regular and DOLA decoding baselines.
P Q: Does our DAMO perform well when transferred to I.Ms? Yes, as shown in Table 5 DAMO improved performance on L.MMs. In the TruthfulQA dataset, it outperformed Regular decoding in MC1 with a score of 34.15, surpassing both Regular (33.66) and DOLA (33.29). DAMO also excelled in the FACTOR task, scoring 57.48 on Wiki, higher than DOLA's 56.51. These results
demonstrate DAMO's effectiveness when applied to LLMs, consistently improving performance across various language tasks.
TruthfulQA (MC)
FACTOR
CoT
Decoding
MC1
MC2
MC3
News
Wiki
StrQA
GSMSK
Regular
33.66
51.29
24.91
65.44
56.91
63.67
21.25
DOLA
33.29
60.84
29.79
61.58
56.51
64.59
21.83
DAMO
34.15
51.24
24.95
64.67
57.48
63.67
21.30
Table 5: Performance comparison of different decoding methods, including our transferred momen- tum decoding, applied to the LLaMA2-7B model across various language tasks.
CONCLUSION
In this paper, we introduced DAMO, momentum decoding method aimed mitigating hallucina- tions in Large Vision-Language Models (LVLMs) by accumulating visual information from earlier layers, where we found that correct information often appears the early stage. By refining acti- vations throughout the inference procedure, DAMO effectively preserves essential visual semantics, leading to more accurate and reliable predictions. Our method achieved excellent results on the MME, POPE, and LLa marks. Furthermore, when transferred to Large Language Models (LLaMA2). DAMO maintained strong performance across several datasets, including TruthfulQA, FACTOR (News, Wiki), StrQA, and GSM8K. These findings validate the robustness of our approach, underscoring its potential to enhance the reliability of models across wide range of tasks.
datasets, showcasing its effectivvness in visual-language bench-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is, in part, supported by NSF (#CNS2431516). We thank the anonymous reviewers for their constructive feedback on this work. The views, opinions, and/or findings contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
REFERENCES
Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqu: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9690-9698, 2020.
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. arXiv preprint arXiv: (606::/600007356 2016.
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to imple- ment preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qian Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision langunla models with assembly global and local attention. arXiv preprint arXiv:2406. 12718, 2024.
Ali Furkan Biten, Lluís Gómez, and Dimosthenis Karatzas. Let there be clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 1381-1390, 2022.
Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. Alleviating hallucina- tions in large vision-language models through hallucination-indum inducedd optimization. arXiv preprint 10X55:2405.15356 2024a.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with Ilms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 14093-14100 IEEE, 2024b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjin Liu. Uniter: Learning universal image-text representatioos.(001)) arXiv preprint arXiv:1909. 11740, 2019.
Lung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883 2023.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning, 2023.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint arXiv:2 104.001644 2021.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:
2018
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. transformer. arXiv preprint arxiv: 1910 10073, 2019.
11

---

Published as a conference paper at ICLR 2025
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang, and Xin Wang. Pokemqa: Programmable knowledge editing for multi-hop question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
2024a.
Hengrui Gu, Kaixiong Zhou, Yili Wang, Ruobing Wang, and Xin Wang. Pioneering reliable as- sessment in sext-to-image knowledge editing: Leveraging a fine-grained dataset and an innova- tive criterion. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp-
15303-15317, 2024b.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18135-18143, 2024.
Mingzhe Hu, Shaoyan Pan, Yuheng Li, and Xiaofeng Yang. Advancing medical imaging with language models: journey from n-grams to chatgpt. arXiv preprint arXiv:2304. 044900, 2023.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and eetrospection-allcaatie on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
In Proceedings of the IEEE/CVF Conference
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Lc, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4916- PMLR, 2021.
Zhimeng Jiang, Zirui Liu, Xiaotian Han, Qizhang Feng, Hongye Jin, Qiaoyu Tan, Kaixiong Zhou, Na Zou, and Xia Hu. Gradient rewiring for editable graph neural network training. In The Thirty- eighth Annual Conference on Neural Information Processing Systems.
Aditi Khandelwal, Harman Singh, Hengrui Gu, Tianlong Chen, and Kaixiong Zhou. Cross-lingual multi-hop knowledge editing. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 11995-12015, 2024.
Jae Myung Kim, A Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious correlations for cross- cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2585-2595, 2023.
Jusung Lee, Sungguk Cha, Younghyun I Lee, and Cheoljong Yang. Visual question answering instruc- tion: Unlocking multimodal large language model to domain-specifie visual multitasks. arXiv preprint arXiv:2402.0833 2024.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large laniunn models through visual contrastive de- coding. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 13882, 2024.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-ing -mnag pre- pre- veeitraining with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023b.
12

---

Published as a conference paper at ICLR 2025
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:230S.1035. 2023c.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human 2021.
falsehoods. arXiv preprint
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer, 2014.
Fang Liu, Yang Liu, Lin Shi. Houkun Huang, Ruifeng Wang, Zhen Yang, and Li Zhang. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint 240407440,, 2024a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL httpp://
vi.github. 1o/blccg/0224----- 01 30- llava-next/
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 36, 2024d.
Zirui Liu, Zhimeng Jiang, Shaochen Zhong, Kaixiong Zhou, Li Li, Rui Chen, Soo-Hyun Choi, and Xia Hu. Editable graph neural network for node classifications. arXiv preprint 15X5v22305.15529, 2023c.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and laiin-and-langguge tasks. Advances neural information processing systems, 32, 2019.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances Neural Information Processing Systems, 35:17359-17372, 2022.
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. arXiv preprint arXiv:2307.0690 0069000,8888...88 2023.
Yang Ouyang, Hengrui Gu, Shuhang Lin, Wenyue Hua, Jie Peng. Bhavya Kailkhura, Tianlong Chen, and Kaixiong Zhou. Leyer-level self-exposure and patch: Affirmative token mitigation for jailbreak attack defense. arXiv preprint arXiv:2501.0266 2025.
Jie Peng, Shuang Zhou, Longwei Yang, Yiran Song, Mohan Zhang, Kaixiong Zhou, Feng Xie, Mingquan Lin, Rui Zhang, and Tianlong Chen. Continually evolved multimodal foundation mod- els for cancer prognosis. arXiv preprint arXiv:2501.18170 2025.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr compu- tational mathematics and mathematical physics, 4(5):1-17, 1964.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456-11 17472, 2022.
13

---

Published as a conference paper at ICLR 2025
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong, Kaixiong Zhou, and Ninghao Liu. Retrieval-enhanced knowledge editing in language models for multi-hop question answering. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Manage- ment, pp. 2056-2066, 2024.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning. pp- 1139- 1111 147.. PMLR, 2013.
Hao Tan and Mohit Bansal. Lxmert: Learning modasity encoder representations from trans- formers. arXiv preprint 0rXiv:/908.074:00 2019.
Surat Teerapittayanon, Bradley McDanel, and Hsiang- Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference pattern recognition (ICPR), pp. 2464-2469. IEEE, 2016.
Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. Mitigating fine-grained hal- lucination by fine-tuning large vision-language models with caption rewrites. In International Conference on Multimedia Modeling, pp. 32-45. Springer, 2024.
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401. 210.2244111177,, 2024.
Jia- Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, and Li Yuan. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01466, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 23044 /4/78, 2023a.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan. Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration, 2023b.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. arXiv preprint drXfv:2310.16045, 2023.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi dulti-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2 1..:::2/11.02276.6 2021.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 13534, 2023a.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.012.
2023b.
Shaochen Zhong, Duy Le. Zirui Liu, Zhimeng Jiang, Andrew Ye. Jiamu Zhang, Jiayi Yuan, Kaix- iong Zhou, Zhaozhuo Xu, Jing Ma, et al. Guns also deserve editing, and they need it more than once. In Proceedings of the 41st International Conference on Machine Learning, pp. 61727- 61746, 2024.
Huixue Zhou, Hengrui Gu, Xi Liu, Kaixiong Zhou, Mingfu Liang, Yongkang Xiao, Srinivas Govin- dan, Piyush Chawla, Jiyan Yang. Xiangfei Meng, et al. The efficiency vs. accuracy trade-off: Optimizing rag- enhaneedd I1m recommender systems using multi-head early exit. arXiv preprint arXiv:2501.02113. 2025.
Qiang Zhou, Zhibin Wang, Wei Chu, Yinghui Xu, Hao Li, and Yuan Qi. Infmllm: A unified frame- work for vinual-language tasks, 2023a.
14

---

Published as a conference paper at ICLR 2025
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination large vision-language models. arXiv preprint arXhv:2310.00554 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 2023a.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023c.
Jiawei Zhu, Yishu Liu, Huanjia Zhu, Hui Lin, Yuncheng Jiang, Zheng Zhang, and Bingzhi Chen. Combating visual question answering ballucinations via robust multi space co-debias learning. In ACM Multimedia 2024, 2024a.
Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. lbd: Alleviating hallucinations large vision-language models via image-biased decoding. arXiv preprint 24li::2002.187766 2024b.
15

---

Published as a conference paper at ICLR 2025
A HYPERPARAMETER SETTING
We provide our detailed hyperparameter setting in Table 6 is set to 0.7 for all experiments. And we present different hyperparameters including T. 31 and B2 for different tasks across different models. M represents MME dataset and P denotes POPE dataset.
LLaVA M
LLaVA P
INF M
INF P
mPLUG M
mPLUG P
B1
0.05
0.20
0.20
0.05
0.60
0,40
82
0.20
0,40
0.40
0.10
0.80
0.60
T
-0.30
-0.30
-0.60
-0.60
-0.30
-0.30
Table 6: Hyperparameters for different tasks across different models
B ABLATION STUDIES
B. EVALUATING THE SENSITIVITY OF DIFFERENT
We evaluate the sensitivity of different model and report F1 score. As shown in Table 7 by setting T to 0.3, 0, -0.3, and -0.6, we observe that the model's performance varies accordingly. This demonstrates the model's sensitivity т. Notably, when r is set to our default value of 0.3, the model achieves its best performance.
POPE (MSCOCO setting) dataset using LLaVA1.5
0.3
0
-0.3
-0.6
Random
88.83
89.56
89.92
89.62
Popular
86.60
86.43
87.07
86.66
Adversarial
81.95
81.56
82.07
81.78
Table 7: The sensitivity r on POPE (MSCOCO) dataset using LLaVA1.5 model.
B.2 EVALUATING THE SENSITIVITY OF DIFFERENT $1
We evaluate different 31 values to observe performance variations on the MME dataset using LLaVA1.5 model. For fair comparison, 33 is set to its default value of 0.2. In this experiment, we explore setting 31 to 0.01, 0.05, 0.1, 0.15, and 0.2, and analyze the corresponding results.
LLaVA1.5 is quite sensitive the value of 31 When 31 is set smaller values, the model shows improved performance compared to the baseline. However, as 3 increases to 0.20, noticeable performance drop is observed. This highlights that 0.05 is the optimal parameter setting.
As shown in Table 8
81
Existence Count
Position Color
Posters Celebrity Scene
Landmark Artwork OCR
Total
0.01
190.00
145.00
135,00
160.00
149.66
137.35
160.00
168.50
119.75
137.50
1502.76
0.05
195.00
150.00
143.33
165.00
144.56
135.00
157.00
166.00
120.00
140.00 1515.89
0.10
195.00
151.67
153.33
170.00
136.39
130.00
155.50
160.50
117.75
125.00
1495.14
0.15
190.00
153.33
143.33
180.00
125.51
122.94
151.00
150.75
113.00
132.50
1462.37
0.20
190.00
120.00
145.00
175.00
122.45
120.00
145.25
136.00
110.25
140.00 1403.95
Table 8: Comparison of different 31 on the MME datasct using LLaVAL.5 model.
B.3 EVALUATING THE SENSITIVITY OF DIFFERENT 32
Similar to the evaluation of 3, we fix 3, at 0.05 and evaluate the performance across different 32 values: 0.1, 0.2, 0.4, 0.6, and 0.8.
16

---

Published as a conference paper at ICLR 2025
B2
Existence Count
Position Color
Posters
Celebrity
Scene
Landmark Artwork
OCR
Total
0.10
195.00
156.67
143.33
170.00
138. 44
133,53
156.25
160.50
118.75
125.00 1497.46
0.20
195.00
150.00
143.33
165.00
144.56
135.00
157.00
166.00
120.00
140.00 1515,89
0.40
185.00
130.00
133.33
143.33
137.41
123.82
159.25
160.00
110.00
102.50
1384.66
0.60
188.33
103.33
93.33
120.00
123.81
107.94
132.75
124.00
88.25
105.00
1186.75
0.80
183.33
113.33
91.67
135.00
121.09
102.35
127.75
121.50
85.25
122.50 1203.77
Table 9: Comparison of different 32 on the MME dataset using LLaVA1.5 model.
As shown in Table 9 the model's performance is significantly affected by changes in 82_ When B is set too high, the model's performance rapidly declines, this interferes with the model's normal reasoning process. Our experiments also confirm that the default setting of 3g at 0.2 is optimal.
B.4 EVALUATING THE EFFECT OF OUR PROPOSED ADAPTIVE ACTIVATION REFINEMENT MECHANISM
In addition to testing the effectiveness of our adaptive activation refinement mechanism on LLaVA1.5, 1.5, we further evaluate on INF-MLLMI by setting the starting layer to the 16th, 20th, 24th, and 28th layers.
Existence Count
Position Color
Posters Celebrity Scene
Landmark Artwork OCR
Total
16-th
190.00
150.00
158.33
165.00 146.94
147.65
157.00
150.75
126.75
117.50 1509.92
20-th
190.00
150.00
158.33
165.00
145.92
147.65
156.25
150.00
128.50
125.00
1516.65
24-th
185.00
145.00
163.33
165.00
142.86
149.12
154.75
147.75
128.25
132.50
1513.56
28-th
190.00
140.00
158.33
165.00
143.88
148.82
154.75
147.00
127.25
132.50
1507.55
Adaptive
195.00
150.00
158.33
165.00
148.64
146.76
156.25
154.75
128.50
117.50
1520.74
Comparison of different momentum decoding starting layers (16-th, 20-th, 24-th, and 28-th) against regular decoding and our adaptive layer selection on the MME dataset using the INF- MLLMI model.
Table 10:
As shown in Table 10 our adaptive starting layer method outperforms all fixed starting layer settings, further demonstrating the effectiveness of our approach.
17


Reviews:
[{"review_id": "7fOH1HprnE", "reviewer": "Reviewer_au8F", "summary": "This paper observes that VLMs tend to make prediction shifts in the last few layers, which leads to a surge in hallucinative outputs. The authors propose a decoding strategy inspired by the momentum analogy used in gradient descent-based optimizers, which enforces decoding consistency adaptively across layers during forward passes. The proposed method outperforms existing approaches on several public datasets.", "questions": "My primary concern lies in the performance and  the quality of text generated at preceding layers. I will be happy to raise my score if my current questions and concerns can be addressed.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of hallucinations frequently emerge in the later layers seems interesting.\n2. The main technical pipeline is clear.", "weaknesses": "1. Writing and diagrams require improvement.\n1). Large Vision-Language Models (VLMs) mentioned in the abstract abbreviated as LVLMs would be better.\n2). A single-paragraph abstract would improve conciseness.\n2). Figure 1(c) requires a more detailed description.\n3). In Section 2, lines 151–152, the statement “However, these methods do not correct hallucinations during the inference process” could be reconsidered. In my view, methods like VCD, which contrast output distributions, are indeed part of the inference process.\n4). It would improve readability if the introduction were revised to emphasize the primary contributions of this work more clearly.\n\n2. Could you increase the number of instances shown in Figure 1(b) to provide a more comprehensive view?\n\n3. Section 3.2, line 198-200, \"VLMs are already proficient in capturing detailed visual information, so further intensifying image-text fusion is unnecessary.\" how to validate \"VLMs are already proficient in capturing detailed visual information\" from your results?\n\n4. Section 3.4, line 261-261, \"Varying the starting layer for refinement can enhance different model capabilities (e.g., layer 16 excels in OCR tasks, while layer 24 improves positional perception).\" (e.g., position scores do not consistently increase, nor do OCR scores consistently decrease). This makes it challenging to conclude that varying the starting layer directly enhances distinct capabilities. Could you provide additional evidence to support this claim?\n\n5. The performance improvement appears modest. For instance, on the MME benchmark, the score of 1515.89 is slightly lower than OPERA’s 1518.36 on the LLaVA baseline. Additionally, it would strengthen the evaluation if more hallucination benchmarks, such as CHAIR, were included.\n\n6. I am concerned about the quality of text generated at preceding layers. Further evaluation metrics for text quality, such as BLEU or other relevant scores, would provide a clearer understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jlZY8EQvrb", "reviewer": "Reviewer_5XME", "summary": "The paper proposes DAMO (Decoding by Accumulating Activations Momentum), a novel decoding strategy designed to reduce hallucinations in Vision-Language Models (VLMs) by maintaining consistency across layers during inference. DAMO introduces a momentum-based mechanism in transformer forward computation to smooth activation updates layer-by-layer, mitigating hallucinations by preserving visual grounding in predictions. DAMO shows good results on various benchmarks (MME, POPE) and models (LLaVA1.5, INF-MLLM1, mPLUG-Owl2) with minimal memory overhead. Its adaptability is demonstrated through successful application to LLM benchmarks such as TruthfulQA and FACTOR, highlighting DAMO’s transferability across various tasks.", "questions": "- The results on the MME dataset show only modest improvements in hallucination reduction, with much of the gains appearing in tasks that may be more knowledge-based (e.g., Celebrity, Posters, Landmark, Artwork) rather than directly related to hallucination robustness. Would you consider testing DAMO on additional hallucination-focused benchmarks? Alternatively, could you clarify how the MME dataset captures DAMO's effectiveness in mitigating hallucinations?\n- Given that the threshold τ\\tauτ is central to controlling the switch between coefficients \\beta_1​ and \\beta_2​, could you provide more information on how its value was determined? Additionally, could you discuss any observed sensitivity of DAMO’s performance to variations in \\tau?\n- Could you provide further insights on the actual layers where the adaptive coefficient adjustment activates? For instance, does the transition of \\beta from \\beta_1​ to \\beta_2​ tend to occur around specific layers, such as the after 24th, as observed in your analysis? Clarifying whether these transitions align with the predefined analysis layers would help in understanding DAMO’s consistency and effectiveness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The paper presents an interesting observation that VLMs are often able to generate accurate predictions in the initial layers, with hallucinations surfacing in later layers. This suggests that hallucinations could stem from disruptions in the decoding sequence rather than an absence of underlying knowledge, providing a novel diagnostic angle on VLM hallucinations.\n- The paper’s approach to using \"momentum\" within the forward computation of transformers is intuitive and impactful. By building consistency across activations, DAMO effectively mitigates hallucination emergence in later layers, reinforcing layer-wise stability and preserving visual grounding.\n- The authors extend their method beyond VLMs to yield strong performance in LLM benchmarks like TruthfulQA, FACTOR, StrQA and GSM8K, showcasing its versatility across various language tasks.", "weaknesses": "- While the Q&A format in the experiments section is engaging, it’s overused, which might dilute its effectiveness. Reserving this style for only the most critical insights could enhance focus and readability.\n- Although DAMO shows good results on the MME benchmark, the gains seem modest for hallucination evaluation. Emphasizing the \"celebrity\" score on mPLUG-Owl2 might not strongly correlate with reduced hallucinations. The MME benchmark may be more reflective of general VLM performance rather than specific hallucination robustness. To strengthen authors’ claims on hallucination mitigation, it would be beneficial to test its performance on other benchmarks that more directly target hallucination evaluation.\n- The paper introduces a threshold \\tau as a cosine similarity criterion for switching between coefficients \\beta_1​ and \\beta_2​, which is critical to DAMO’s performance. However, the manuscript lacks explanation on how \\tau was determined, its specific value, and its impact on model effectiveness. Additional details would improve understanding and reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9gWUTAelae", "reviewer": "Reviewer_XzDy", "summary": "This paper investigates the layer-wise prediction tendencies of VLMs and proposes a novel decoding strategy called DAMO to mitigate hallucinations. DAMO enforces decoding consistency across layers in an adaptive manner during forward passes, amplifying visual semantics consistently extracted throughout the inference while reducing hallucination biases. Experiments on various VLM benchmarks demonstrate that DAMO significantly mitigates hallucinations, resulting in more visually grounded and accurate predictions.", "questions": "1. How about some complex reasoning VQA performances in terms of Figure 1(b) (c). As indicated in DoLa [r1], deep layers facilitate the complex reasoning. Also, the reasoning steps is more complex as in [r1], please add more explanations.\n[r1] Dola:  Decodingbycontrastinglayers improves factualityinlarge languagemodels. arXivpreprint\n arXiv:2309.03883,2023.\n2. In 'Figure 1(c) Proportion of samples in a small dataset....',  Detailed descriptions in the Figure 1 caption are better illustrated.\n3. Better re-organize Figure 1 to save more space.\n\nI consider adjusting my score based on the clarification of DAMO and DoLa.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Novel insights consider the layer-wise outputs, especially regarding to language bias in LVLMs.\n\n2. Simple but effective solutions named DAMO based on the proposed findings.\n\n3. Extensive experiments and pleasant results are achieved.\n\n4. Source codes and partial experimental results are provided.", "weaknesses": "1. Although this method is proposed for LVLMs, this paper does not mention the DoLa [r1] in the main body. DoLa's claims are somewhat opposite to DAMO's. More analysis is needed to clarify the insights and differences.\n[r1] Dola:  Decodingbycontrastinglayers improves factualityinlarge languagemodels. arXivpreprint\n arXiv:2309.03883,2023.\n2. Applying DAMO to LVLMs and LLMs seems to be contradictory, especially considering the DoLa's findings.\n3. I check the appendix and think some critical experimental details of DAMO are missing, like temperature, max token, decoding strategies.\n4. GPT4-V experiments should be better conducted rather than GPT4, following VCD.\n5. typo: INF-MLLM1 $\\rightarrow$ INF-MLLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jhFAaLtNHi", "reviewer": "Reviewer_J6Q4", "summary": "This paper addresses the issue of hallucinations in Large Vision-Language Models (VLMs) by analyzing their layer-wise prediction tendencies and decoding mechanisms.  The authors propose a decoding strategy inspired by the momentum concept in gradient descent optimizers, which ensures adaptive decoding consistency across layers.", "questions": "Please refer to Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors conduct a detailed layer-wise analysis of VLMs' prediction tendencies, identifying that hallucinations often emerge in the final layers due to significant shifts in decoding. This insight may provide a deeper understanding of the underlying mechanisms causing hallucinations.\n\n2. The proposed Decoding by Accumulating Activations MOmentum (DAMO) method is somewhat effective. By accumulating momentum across layers, DAMO ensures consistent decoding and reduces the impact of late-stage hallucinations.", "weaknesses": "1. The paper draws inspiration from the momentum concept and contrast decoding, but it lacks a detailed analysis and comparison with recent related work, such as [1, 2]. Additionally, the paper does not adequately explain why the proposed method theoretically offers advantages over previous approaches.\n\n2. Poor presentation: The paper lacks comprehensive framework diagrams that clearly illustrate the specific content and workflow of the proposed method, making it difficult to understand the method's details.\n\n3. The paper does not provide a detailed analysis of how sensitive DAMO is to hyperparameters, such as the momentum coefficient. Understanding the optimal settings for different models and tasks could help in practical implementation.\n\n\n[1] Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization\n\n[2] Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)