AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
Authors: Chaeyoung Jung
Youngjoon Jang
Joon Son Chung
Venue: NeurIPS 2025 poster ()
Keywords: Audio-visual processing
Audio-visual LLM
Audio-visual LLM hallucination
Primary area: applications
TL;DR: We introduce Audio-Visual Contrastive Decoding (AVCD), a training-free framework for mitigating hallucinations in AV-LLMs by reformulating the existing contrastive decoding framework to support trimodal interactions.

Abstract: Hallucination remains a major challenge in multimodal large language models (MLLMs). To address this, various contrastive decoding (CD) methods have been proposed that contrasts original logits with hallucinated logits generated from perturbed inputs. While CD has shown promise in vision-language models (VLMs), it is not well-suited for AV-LLMs, where hallucinations often emerge from both unimodal and cross-modal combinations involving audio, video, and language. These intricate interactions call for a more adaptive and modality-aware decoding strategy. In this paper, we propose Audio-Visual Contrastive Decoding (AVCD)—a novel, training-free decoding framework designed to model trimodal interactions and suppress modality-induced hallucinations in AV-LLMs. Unlike previous CD methods in VLMs that corrupt a fixed modality, AVCD leverages attention distributions to dynamically identify less dominant modalities and applies attentive masking to generate perturbed output logits. To support CD in a trimodal setting, we also reformulate the original CD framework to jointly handle audio, visual, and textual inputs. Finally, to improve efficiency, we introduce entropy-guided adaptive decoding, which selectively skips unnecessary decoding steps based on the model’s confidence in its predictions. Extensive experiments demonstrate that AVCD consistently outperforms existing decoding methods. Especially, on the AVHBench dataset, it improves accuracy by 2% for VideoLLaMA2 and 7% for video-SALMONN, demonstrating strong robustness and generalizability. Our code is available at : https://github.com/kaistmm/AVCD.

AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
Chaeyoung Jung' Youngjoon Jang' Joon Son Chung Korea Advanced Institute of Science and Technology (KAIST)
Abstract
Hallucination remains a major challenge in multimodal large language models (MLLMs). To address this, various contrastive decoding (CD) methods have been proposed that contrasts original logits with hallucinated logits generated from per- turbed inputs. While CD has shown promise in vision-language models (VLMs), it is not well-suited for AV-LLMs, where ballucinations often emerge from both unimodal and cross cross-modal combinations involving audio, video, and language. These intricate interactions call for a more adaptive and modality-aware decoding strategy. In this paper. propose Audio-Visual Visual Contrastive Decoding (AVCD)-a novel, training-free decoding framework designed to model trimodal interactions and suppress modality-inducee hallucinations AV-LLMs. Unlike previous CD methods in VLMs that corrupt fixed modality. AVCD leverages attention dis- tributions dynamically identify less dominant modalities and applies attentive masking to generate perturbed output logits. To support CD in trimodal setting, we also reformulate the original CD framework to jointly handle audio, visual, and textual inputs. Finally, to improve efficiency, we introduce entropy entropy-guided adaptive decoding, which selectively skips unnecessary decoding steps based on the model's confidence in its predictions. Extensive experiments demonstrate that AVCD consistently outperforms existing decoding methods. Especially, on the AVHBench dataset, it improves accuracy by 2% for VideoL.LaMA2 and 7% for SAlco-SALMONN, demonstrating strong robustness and generalizability. Our code is available at http:////tttuboooooo/ss//m//////
Introduction
Large language models (LLMs) have achieved remarkable success, enabling Al systems to perform wide range text-based tasks such problem- solving. summarization, translation, and human interaction [1, 7, 41, 52, 57, 58, 72]. To extend these capabilities beyond language, recent advances have introduced multimodal large language models (MI.Mss), which integrate visual and auditory inputs alongside text. By incorporating multiple modalities, MILLMs enhance the model's '' ability to understand and solve complex tasks that require multimodal reasoning [11, 35, 43, 50, 64, 69].
However, despite these advancements, hallucination remains a persistent challenge. It refers to the generation of biased factually incorrect information that does not faithfully reflect the given input, posing a significant barrier to the reliable deployment of both LLMS and MLLMs [25, 37, 40, 53, 55, 63, 71]. In lisounlanguagg models (VLMs). hallucinations frequently occur when the model misinterprets visual inputs and generates responses that fail to align with the actual visual content. This issue largely attributed to an over-reliance on statistical biases [2, 17, 33] inherited from pretraining and the dominant influence of language priors [4, 20, 45, 61]. As result, models often produce responses based on familiar language patterns instead of accurately reflecting the visual input, problem that has been extensively addressed in previous studies [6, 18, 28, 56].
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Question Deseribe the given video and sudio detaiil.
VVoual risnazch
Cello and Drum
Speaking
Base
"Addo visal cissnorc
The video shows man playing the cello in front of a plano. The cello and plano are both
AVCD
wearing black shärr and hat, and he is smilligg The video playing drums in room with a man wearing plaid and jeams. He is sitting down and playing the drums with hands.
sounding the
The
The video shows playing the vello accompanied by shirt with a bird on and be semiling. The video also shows man playing the drums in with window. The sound of the drums load and clear, and accocspanied by the sound of the cello in the background.
The sound of the celle deep and rich, and drumss background.
also shows
wearing and
Figure 1: Hallucination mitigation with Audio Visual Contrastive Decoding (AVCD). Inaccurate visual and audio-visure information is highlighted in red and blue, respectively, and corrected during inference via AVCD, enabling the production of precise details such as 'a shirt with bird on it'.
To address these challenges, contrastive decoding (CD) has recently been applied to VLMs. These methods perturb fixed single modality and obtain the logits from the distorted information, which are then contrasted with those generated from the unaffected input. This balances modality reliance and effectively reduces hallucinations. As VLMs have two modalities (vision and language), one branch of research perturbs the language information, while another distorts the visual information for CD. For language perturbation, ICD [56] introduce distorted instructions to generate corrupted outputs. For vision perturbation, CODE [26] replaces visual input tokens with self- aeff-geneatee descriptions, while VCD [28] introduces Gaussian noise into image inputs. Similarly, SID [24] retains only the least informative visual tokens to generate distorted outputs. However, CD has not yet been explored in AV-LLMs, where hallucinations pose particularly difficult challenge due to the complexity of multimodal interactions [27]. Figure 1 shows that Base decoding can lead to hallucinations not only from single visual inputs but also from audio-visual combinations, highlighting the need for more sophisticated modeling of cross- modal interactions to mitigate such errors.
In this paper, we propose Audio- Contrastive Decoding (AVCD), training-free decoding framework tailored for AV-LLMs. The core innovation of AVCD lies in how it generalizes the idea of CD, which has been primarily explored in VLMs. Unlike prior methods that perturb a fixed modality, AVCD dynamically identifies the less dominant modalities, guided by the model's 's attention distribution across the three modalities. AVCD then applies attentive masking to selectively perturb the less dominant modalities and contrast the resulting biased logits with those derived from the original. This encourages the model to rely on more balanced multimodal evidence, thereby reducing hallucinations that may arise from over-dependence on any single modality or from misaligned cross-modal cues. Furthermore, we reformulate the original CD to jointly handle audio, visual, and textual modalities. This allows the model to consider the more intricate hallucination patterns that can emerge from combinations of multiple modalities, which prior methods overlook. Finally, to enhance inference efficiency, we also introduce an entropy-guided adaptive decoding mechanism. Specifically, when the model exhibits high confidence in its predictions, additional decoding steps skipped. This strategy significantly reduces the computational overhead associated with performing multiple forward passes, while preserving the benefits of contrastive reasoning. By combining dominance-aware attentive masking, trimodal-aware contrastive formulation, and efficient inference, AVCD effectively mitigates modality-induced hallucinations in AV-LLMs. As shown in Figure 1, AVCD enhances complex cross- modal interactions, resulting in reduced hallucinations.
We validate the effectiveness of AVCD through extensive experiments across range of MLLMs, including evaluations involving image [38], video [11, 35], and audio-visual inputs [11, 50]. The results consistently demonstrate that AVCD mitigates hallucinations during inference, regardless of the input modality. Specifically. we evaluate AVCD on the AVHBench dataset [51], a benchmark specifically designed to assess hallucinations in audio-visual settings. When applied to two widely used AV-LLMs, AVCD achieves a 2% relative improvement in accuracy on VideoLLaMA2 [1 and 7% on video-SALMOON [50]. These improvements underscore AVCD's robustness and its effectiveness in enhancing the reliability of AV-LLMS for trimodal understanding.
2 Related Work
Vision-language models. Recent advancements in MILMMs have significantly improved vision- language integration, enhancing both multimodal reasoning and interaction capabilities [8, 23, 30,
2

---

logit(*, xx, ,x,)x
Katropy-guided Adaptive Decoding
AVCD OFF
Prediction: "Three"
+
Attentive Masking
Modality Dominaace
Vidco
Moding
logit/k 1x xxxx
AVCD On
|||||||||||| 11111111111
Ag video
Aqastio
Agg.ongunge
Prodiction: "Thoo"
(4)
Audio Maskina
(1)
logit(s, I x
Question
Language Video Audio
Two
"Ilow many types of musical instrmmntt sound in the video!"
AV
logit xx
Ground Truth: "Twu"
input and a question, the model generates predicted logits along with stacked modality dominance score DM. computed by summing the attention values of the final query token across modalities from the attention map AQx (Eq. (3)). To improve efficiency, CD skipped when the model's 's prediction has high confidence (i.e., low entropy). Otherwise, once dominant modality is identified (e.g.. language > video > audio), AVCD applies all possible masking combinations across the less dominant modalities (Audio, Video, and Audio-Visual) for CD. An attentive masking strategy is used to perturb the less dominant modalities, and CD is performed using Eq. (10). This process promotes balanced multimodal reasoning by enhancing the influence of weaker modalities (e.g., audio and video) while maintaining efficient inference.
Figure 2: Overall AVCD pipeline. Given an
66, 70, 75]. LLaVA [39] extends the LLaMA [54] framework, foundational text-based LLM, by incorporating visual inputs to enable multimodal reasoning and cestruction-followinn Flamingo [5] introduces few-shot learning framework that bridges visual and textual information for contextual reasoning. Expanding upon this approach, InstructBL.IP [14] enhances BLIP-2 [30] through instruc- tion tuning. refining image-based based dialogue and multimodal understanding. Expanding beyond static images, GPe [43] extends GPT [7]-based models to engage with video content, enabling interactive video analysis. More recently, VideoLLaVA [35] builds on LLaVA, adding temporal reasoning to improve video-based dialogue and comprehension.
Audio-vise visual large language models. With the advancement of VLMs, AV-LLMs have emerged [10, 11, 12, 21, 42, 47, 65, 68, 69, 73], expanding multimodal capabilities to include audio perception. By jointly understanding visual and auditory inputs, AV-LLMs enable more context-aware reasoning, making them well-suited for complex real-world applications. VideoLLaMA [69] extends LLaMA by integrating both visual and audio inputs, enhancing iido-bbaedd reasoning. VideoLLaMM2 [11] refines this approach with improved temporal modeling and multi-frame integration, resulting in more accurate multimodal comprehension. Building on this, MAMO-S [50] unifies speech, audio, language, and video to enable context-aware multimodal interactions across multiple sensory inputs. Recently, Meerkat [12] improves audio-visual interaction by aligning signals at multiple levels using special modules before decoding. These improvements represent a major step forward in multimodal AI, uniting language, vision, and audio for a more comprehensive understanding.
Mitigating hallucinations in LLMs. A widely adopted approach is RLHF [46], which optimizes models using human preference data to enhance response reliability. Similarly, inference time interventions with human labels [31] incorporate real-time feedback to guide model predictions. Other strategies involve fine-tuning with hallucination-targee targettd datasets [19, 36] or post-hoc revi- sors [74] that refine outputs and correct errors. However, these approaches heavily depend on human supervision and further training, posing scalability challenges in real-world applications.
To address the challenges posed by supervised methods, recent research has explored non-training, non-human intervention techniques as more scalable alternatives [16, 22, 44]. In parallel, contrastive decoding (CD) has been proposed as a training-free approach that suppresses unreliable predictions by subtracting the logits of weaker model from those of stronger one [32]. This logit-level operation leads to more accurate and reliable outputs. Building on this, DoLA [13] improves model performance by conducting contrastive comparisons between the early, underdeveloped layers of the transformer and the later, more refined layers. This contrast helps the model identify and refine the outputs more effectively. Additionally, CD-based techniques have been extended to VLMs. For example, CODE [26] perturbs output quality by employing deff-descriptions and comparing them with the original VLM predictions, ensuring better consistency and accuracy. ICD [56] generate corrupted outputs by introducing distorted instructions for CD. Furthermore, VCD [28] tackles
3

---

hallucinations in VLMs by injecting noise into input images and contrasting the generated responses with their original outputs. SID [24] recently addresses lision-related hallucinations by preserving the least important visual tokens in the attention map and contrasting the resulting biased outputs. These approaches help mitigate hallucinations and enhance the reliability of VLMMs
However, despite their promise, their validation has been largely confined to VLMs, leaving the complex interactions among audio, video, and language AV-LLMs underexplored. Addressing the challenges of multimodal reasoning in this setting requires more sophisticated strategies. To address this, we reformulate the existing CD framework to generalize across a wide range of MLLMs.
3 Method
3.1 Preliminaries
Formulation for audio-visual large language models. For video data, a visual encoder such as CLIP [48] extracts frame-level features and converts them into sequence of M fixed-length visual tokens, denoted as x = (x1,X2, Xar.. Similarly, an audio encoder like BEATs [9] processes audio signals into sequence of N fixed-length audio tokens, represented as x (XM+1,
N. Text inputs are tokenized into L textual tokens using a tokenizer, form- ing x" X XM .... X. These modality-specifie token sequences are then concatenated to create unified representation x [x, where K = M + N + L is the total number of tokens. This unified representation used as the input to the LLM decoder. Given the input sequence x, the model generates a response autoregressively, predicting the next token based on previous generated tokens and modality-speciffc information:
Yt p(y, |x", xx ,x', y y<t) x exp (logit(y)) x"" xxx,,xxxx,,),))
(1)
Here, y, represents the token generated at timestep i, while y-r refers to the sequence of previously generated tokens. At each timestep, the decoder first computes hidden states, which are then mapped to the vocabulary dimension through a linear projection layer. This produces logit vector, where each element corresponds to a raw score for a token in the vocabulary. Finally, the softmax function transforms these raw scores into probability distribution over the possible next tokens.
Contrastive decoding for vision-language models. Existing multimodal CD methods aim to mitigate hallucinations, which arise when models excessively rely on language while neglecting visual inputs [18, 28, 56]. To counteract this, a common sampling strategy is employed:
logit = (1+ + (ooggi((y) x²) alogit(y/x "",
(2)
where a controls the contrastive effect, and x represents biased or corrupted visual information (e.g., through noise injection, augmentation, or masking) [28, 59, 60). Corrupting less dominant modality makes logits rely more on the unaffected modality. Subtracting these biased logits from the original reduces the dominant modality's influence, enabling more balanced multimodal reasoning.
To prevent the suppression of correct predictions, an adaptive plausibility constraint is applied the CD-enhanced outputs, ensuring that logits significantly deviating from the original distribution are excluded. This approach, originally proposed by [32], is also adopted in our method and remains consistent with its application in various CD-based studies [13, 24, 28, 56]. A detailed explanation of the adaptive plausibility constraint can be found in Supp. B.
3.2 Audio Visual Contrastive Decoding
In this section, we introduce Audio-Visual Contrastive Decoding (AVCD), a decoding strategy tailored for AV-LLMs. As illustrated in Figure 2, AVCD begins by generating an initial prediction from the full multimodal input while recognizing the dominant modality via attention distributions. An entropy-guides adaptive decoding mechanism then determines whether additional decoding is necessary. If the prediction is confident (i.e., low entropy). further decoding steps are skipped for efficiency. Otherwise, attentive masking is applied the less dominant modalities to generate perturbed outputs. Finally, we apply reformulated CD. specifically designed for trimodal settings, by contrasting the perturbed and original outputs. This enhances cross-modal understanding while maintaining efficient inference. A full description of the AVCD algorithm is provided in Supp. E.

---

Recognizing dominant modality via attention distributions. Transformer- based MLLMs process cross-modal information through an attention mechanism that integrates tokens from different modalities (audio, video, and language) via queries, keys, and values [11, 35, 43, 50]. The attention map captures the model's focus at each decoding step, providing a direct measure of modality influence. By analyzing the attention distribution of the final query token (AQx) ) in the attention map, inspired by [24, 49], we quantify modality dominance. Higher attention weights assigned to a specific modality indicate that the modality has stronger influence on the model's prediction.
Formally, the dominance score DM for given modality M (where M represents the set of indices corresponding to video, audio, or language tokens) is computed as:
Diu = ΣAOK AQUI SEM
1 J ED'u. DM
where D represents the dominance score of modality M at layer j E {1,2, J and AQx is the attention weight assigned by the last query token to i-th key token from modality M. The dominant modality is identified with the highest average DM across layers. This formulation adaptively measures modality importance at the attention level, enabling systematic identification.
Attentive masking via zeroing out. Once the dominant modality is identified by analyzing the attention distribution of the final query token (Aqx), we perform masking on combinations of the less dominant modalities. Specifically, we apply an attentive masking strategy using threshold defined by the top P% of the mean stacked AQx across transformer layers. Tokens in the non- dominant modalities that exceed this threshold are set to zero, intentionally suppressing informative signals in those modalities to produce logits biased toward the dominant modality for CD. Unlike existing methods that directly distort inputs [28, 59, 60], which may introduce undesired noise due to deviations from the trained input distribution, our attentive masking strategy preserves the structure of the learned input space. This design allows AVCD to avoid the noise from directly perturbing inputs, focusing instead on mitigating hallucinations that arise from cross- modal reasoning [24].
Figure demonstrates the adaptive behavior of the model under the attentive masking strategy for specific modalities. When the video modal- ity is masked, the model compensates by lever- aging audio cues, and in the absence of audio, it prioritizes visual semantics. When the text prompt is masked, the model shifts its reliance to visual elements in the video, which can lead to undesired output predictions, such as "3d art." These results demonstrate how our atten- tive masking strategy minimizes reliance on the masked modality, enabling the model to lever- age the remaining modalities for inference.
Video Masking
"Liskking firerackers are hwand!
Stacroo to deccibo sadio fosl to dorthe
Audio Mashing
"Lighting Enveraaion is truliionall Chircso oestoen"
Saacess decorbo Full dosiis
Question
"Cloect the whht you ooe Iana i1. singgle aemiloann.
Languric Madiing
"3d art."
Milde Irgand Inferration
Figure 3: Analysis of the attentive masking strat- egy. By masking specific modality, its influence is reduced, allowing the model to focus on the re- maining modalities when generating outputs.
reformulated contrastive decoding for trimodal integration. Existing CD methods operates on language and vision modalities. However, AV-LLMs generate the next token based on the joint distribution p(yr|x", x", x²), incorporating an additional audio modality. Assuming the language modality is dominant while the vision and audio modalities are less dominant, we can reformulate the CD method to account for the additional modality by modeling the probabilities in Eq. (2) as follows:
p(y:lx",x)
Σ
2
p x"" xx)..
(4)
a'E(a,aa)
ply.|x "
Σ
p(y xx" x0).
(5)
a' d'E(a.aa)
where {a, -a} denote the intact and corrupted audio states, respectively. Including the audio modality in the original distribution p(y+|x", x", xx) may interfere with accurately modeling the relationship between language and vision, represented by p(y+|x", xx,, x²). To mitigate this issue, we intentionally corrupt the audio information, which encourages the model to rely more on the language and vision modalities. By averaging the original and corrupted distributions, we simply reduce the impact of the
5

---

audio, making the model less sensitive to the audio modality. To convert the distribution into logit form, we apply the logarithm to Eq. (4) as follows:
logit(y, (x", x²) = log ply,p(y x²) - log (Folydxe ,²,,,,
xx))
(6)
However, directly adding probabilities inside the logarithm undesirable since the model outputs logits rather than probabilities. To address this, apply Taylor expansion for logarithm function:
A + B log(A) + log(B)
log(
with the error in the approximation being second order in the relative difference between and B. The detailed derivation of Eq. (7) is provided in Supp. A.
Based derivation, expect the difference between the original logits and those from corrupted signals to be mini- mal. To validate the effectiveness of our attentive masking strategy, we compute the exact approximation error (defined in Supp. A. between the logits from original and corrupted signals on 100 samples from the AVHBench dataset [51]. af- ter applying the adaptive plausibility constraint. As shown in Table 1, our method produces smaller errors than VCD across multiple masked modality combinations, indicating that in- troduces less distortion when masking modalities. This result supports the mathematical approximation that allows moving the addition operation outside the logarithm. Accordingly, we can approximate logit(y, [x", " x²) as:
Table I: Approximation error un- der the adaptive plausibility con- straint. On the AVHBench [51] dataset, AVCD produces smaller deviation from the original logits
compared VCD [28].
Masked Modality
Decoding
A1
Vi
AV
VCD
0.083
0.015
0.073
AVCD (Ours)
0.032
0.015
0.037
Tygit(y:/x's x') Ologp(ys x", x') log x" x))
Similarly, by applying the logarithm to Eq. (5) and substituting the results into Eq. (2), we derive the extended CD that leverages the visual modality:
logit x (1 + a") logp(y; xx, x²) a" log p(yp ", x,)')
x a1+ (logp(y./xx x" x') + p(yl"" , " ",x())
aa" (log x" x²) + logp(y.
x())
where a" controls the degree of contrastive influence. Same procedures can be applied the audio
modality. To simultaneously address hallucinations both video and audio, we sum the logits, applying CD to each modality follows:
logit = (2 a aogiity [x", x", x²) + (1 (1-a* a" + loggity y (y|x=", x", x!)
(10)
+ + a" - a )logit(y: |x", x x79 x!) - (a" + a )logit(y) "x x x(),
where a" and a control the contrastive strength in the video and audio domains, respectively.
Entropy-guided adaptive decoding. As described in Eq. (10), using AVCD can significantly increase computational cost. To mitigate this, we propose an seropy-ggiiedd adaptive decoding (EAD) that selectively applies AVCD based on the entropy of the original logit distribution. Entropy serves confidence measure, enabling the model to bypass unnecessary forward passes for high-confidence tokens. If the entropy low, indicating high confidence, standard decoding is applied. Otherwise, AVCD refines uncertain predictions. This adaptive mechanism enhances inference efficiency by reducing unnecessary computation while preserving the benefits of CD for ambiguous cases.
Experiments
This section presents a thorough evaluation of proposed AVCD method across multiple datasets and diverse set of MLLMs. For systematic analysis, the MILMSs are categorized into three groups based on their input modalities: AV-LLMs,, Video-LL.Ms, and image-LLMMs. Results for image-L..Ms and additional experiments are included in Supp. C and Supp. D. respectively.

---

Table Results on AV-LLMs. We evaluate two representative AV-LLMS across three datasets. For decoding. we compare the original model's 's decoding (Base). VCD [28] extended with audio via Eq. (2), and VCD*, which incorporates audio using our proposed formulation Eq. (10) along with adaptive dominant modality recognition. AVCD consistently outperforms all other decoding methods across the benchmarks, demonstrating the effectiveness of both our trimodal CD formulation.
Datasets
Model
Decoding
MUSIC-AVQA [29]
AVHBench [51]
AVHBencb-Cap
(8)
Acc. (%) 1
Score
Base
81. m1.30
70.52
2.84
LaM2 IIII
VCD [28]
77.6610m
2.86.00
VCD*
81,49
69.18
3.00
AVCD
3.03-00:
Bose
48.5010m
58.19
1.8
vidco-SALMONN [50]
VCD [28]
41.57.10x8
VCD*
49.00=a.11
60.66
2.44
AVCD
49.73
62.18
2.47
Question
"How many sounding acoustic guitar in the video?"
Violin Sound
Base "One"
VCD "One"
AVCD
"Zero"
Question
"Did snow ousside? It looks warm outside." "
Base
VCD
AVCD
don't think snowed."
"I know. can't tell."
don't know where you are. snowed where
Figure 4: Qualitative results on AV-LLM and video-LLM using VideoLLaMA2 [11]. AVCD effectively leverages all modalities by mitigating the issue of certain modalities being ignored.
4.1 Experimental Settings
Dataset and evaluation. We evaluate AV-LLMs MUSIC-AVQA [29], which tests synchronized audio-visual reasoning with question-answw answering (QA) pairs derived from the MUSIC dataset. We also use AVHBench [51], designed assess audio-visual hallucinations using adversarial QA pairs. We treat the entire dataset as the test set and use the initially released portion as the validation set, which is employed to evaluate inference speed reported in Figure 5. Since audio-video captioning in AVHBench follows a distinct evaluation protocol, we assess it separately. For video-LLMs, we use MSVD-QA [62]. which involves questions about objects, actions, and events in short video clips, using the first 1,000 test examples. We also use ActivityNet-QA [67], a more challenging benchmark requiring reasoning over long videos with complex temporal understanding.
Evaluation protocol. We evaluate model accuracy using a GPT-3.5-based framework. following the protocol of VideoLLaMA2 [11], except for AVHBench, which can be evaluated without GPT assistance. GPT-3.5 is used to perform binary verification of response correctness. To assess consistency, report the mean and standard deviation across multiple runs.
Baselines. For AV-LLMs, we evaluate the audio-visual variants of VideoLLaMA2 [11] and video- SALMONN [50], both of which integrate visual and auditory cues to support multimodal under- standing. Notably, video-SALMON
performs fusion at the feature level rather than the token level; therefore, we treat the fused audio-visual input a single modality when comparing it against language. For video-LLMs, we assess VidcoLLaMA2 in its vidco-only configuration and Vidco-LLaVA [35], an extension of LLaVA that incorporates temporal context across video frames.
Implementation details. In our experiments, the dominance-aware attentive masking method is applied to all transformer layers except the final layer. Based on the attention map, we mask the locations with the top 50% highest values (refer to Table A.3 in the Supp. D for more details). Based on our analysis that the modality dominance between video and audio is relatively balanced (see Figure A. in the Supp. D), we set a and a to equal. To determine their optimal values, we randomly select 100 samples from each dataset and vary the value from 0.5 to 3.0 in increments of 0.5. As result, we set to 2.5 for the AVHBench dataset, and to 0.5 for all other datasets. Furthermore, we set the entropy threshold to 0.6 for entropy -uidded adaptive decoding.
7

---

4.2 Experimental Results
Audio-visual LLMs. In Table 2, we compare three different types of decoding methods using two models, VideoLLaMA2 and Video-SALMONN. We use the term Base to refer to the original decoding method. We first extend VCD [28] by incorporating the audio modality, following the original CD formulation. This baseline extension, however, leads to performance degradation when applied directly to AV-LLMs. In contrast, VCD*, which integrates audio using our proposed AVCD formulation (Eq. (10)) with dominant modality estimation, improves performance over the VCD method in most cases. Finally, AVCD, which further introduces a dominance-avy strategy in place of the Gaussian noise used in VCD, consistently achieves the best results. This demonstrates the effectiveness of our approach in processing audio-visual inputs. Notably, on AVHBench, benchmark for audio-visual hallucinations, AVCD improves accuracy by around 2% for VideoLLaMA2 and 7% for MAleoo-SALMONN relative to the base method. In addition to simple QA tasks on MUSIC-AVQA and AVHBench, AVCD also excels in audio- captioning.
attentive masking
Video-ILMss To evaluate whether AVCD gen- eralizes beyond AV-LLMs, we apply it to video-LLM that processes only visual and tex- tual inputs, without audio. Specifically, we test AVCD on the VideoLLaMA2 model (Ta- ble. 3). The results show that AVCD consis- tently improves decoding performance across all settings, while VCD originally designed for image-LLMs- pessfrms worse than the base method. These findings indicate that AVCD not only mitigates hallucinations in AV-LLMs, but also serves as a generalizable decoding strategy for multi-modal models beyond its original scope. Additionally, AVCD surpasses VCD in image-LLLMs as well, despite VCD being tailored for such models (see Table A. in Supp. C for details).
Table 3: Results video-LLMs. AVCD sur-
passes both Base and VCD across all experiments.
Datsscts
Model
Decoding
MSVD-QA [62]
Activityis-QA [67]
Aec. (90)
Acc. (
Bore
7.
19:10:5
Videol (1)
VCD (28)
71. 9020.87
45 45.65:20044
AVCD
75,20.00.42
48.2210.004
Bote
70.20..0.00
47.48.10.02
Video-LLLaVA [S5]
VCD [28]
71.00:0.55
47.25:=0002
AVCD
72.16:10.24
480320014
Qualitative results. To qualitatively illustrate the effect of AVCD, Figure 4 presents QA pairs generated by AV-LLMs and video-LLMs using the VideoLLaMA2 model. In the AV-LLM example, the video and audio are misaligned, as the sound corresponds to violin instead of guitar. This requires complex visiul reasoning, where base decoding and VCD fail to correctly interpret the relationship between the modalities. However, AVCD successfully resolves this issue by properly understanding the modality interactions. In the case of the video-LLM, the video primarily depicts snowy scene, but the prompt contains misleading words such as "warm". Since language tends to dominate over visuals, both base decoding and VCD are misled, producing incorrect responses. In contrast, AVCD overcomes this bias by appropriately adjusting the influence of each modality, leading to the correct answer. Further qualitative examples are provided in Supp. F.
4.3 Discussions
Revisiting conventional CD. We extend the con- ventional approach (Eq. (2)) of performing single- instance CD to handle multiple instances from two less dominant modalities simultaneously in AV- LLMs. As shown in rows 2 to 4 of Table 4, con- ventional CD performs comparably or better than base decoding, confirming its effectiveness for single- instance CD. By simply adapting this approach for AV-LLMs, where language is dominant, the follow- ing formulation is derived.
Table 4: Ablation on CD with Eq. (2) and Eq. (11). AVCD effectively extends ex-
isting CD to trimodal configurations.
Masked Modality
Decoding
Acc
A
V
AV
Base
70.52
w/ Eq. 2
71.88
w/ Eq. 2
/
70.16
Eq. 2
72.07
w/ Eq. 11
70.94
AVCD
72.15
logit = (1 + 3a Sog)igii((x xxxxx) x²) ologit fx TV
xx,x²)
(11)
alogit(x(x"' x/) alogit(x(x""
xx)..
In the fifth row of Table 4, we report the performance of CD using Eq. (11), Although there is slight improvement compared to the base model, it is evident that the performance is lower than when using Eq. (2) with either audio or AV masking. This approach over-emphasizes audio-visual contrasts, as audio-induced hallucinations are partially resolved when CD is applied to the AV domain, and video-induced hallucinations are also partially mitigated. On the other hand, in the sixth row
8

---

of Table 4, AVCD effectively addresses imbalance by assigning positive coefficient to each domain's s output, achieving better performance than both Eq. (2)-based and Eq. (1 ba-ed contrastive decoding.
highlighting its effectiveness.
Efficient inference via entropy-guidet adaptive decod- ing, In Figure 5, we show the trade-off' between inference speed and accuracy for VideoLLaMA2 on the AVHBench validation set under varying EAD thresholds (T). The x- axis shows average inference time per generated token, and the y-axis indicates accuracy. Higher reduces addi- tional forward passes, speeding up inference but limiting
02.5
1011
Tolli
1-12
Accurncy (%)
03.0
12.5
78.0
performance gains. Note that FlashAttention [15] is not applied, as AVCD relies on full attention weights.
02.5
1.5
We observe that accuracy remains stable when the en- tropy threshold T is below 0.6, but gradually declines as increases to 0.8 and 1.0. A = 1.0, most additional decoding steps are skipped, resulting in faster inference and accuracy comparable to that of base decoding without FlashAttention. In contrast, setting T = 0.8 yields faster inference than VCD (2.25 vs. 2.5) while still improving accuracy over both the base method (78.05 788055% + 80.98%) and VCD. These results show that our EAD strategy successfully balances speed and performance. They also demonstrate that AVCD enables efficient inference, despite requiring multiple full forward passes.
Inference time (wiooken)
Figure 5: Comparison across entropy thresholds (7). 7 controls over the trade- off between inference speed and accu- racy. At T 0.8, it achieves faster in- ference than VCD while outperforming Base decoding in accuracy.
Evaluation of dominant modality recognition. To eval- uate the effectiveness of our adaptive dominant modality recognition, we compare our method against a fixed strat- egy that assumes a single modality is always dominant. We use VideoLLaMA2 for the experiments and evaluate on AVHBench for mudio-video-tex: inputs and ASSD--QA for video-text inputs. As shown in Table 5, treating language as the dominant modality matches the adaptive strategy, whereas fixing audio or vision leads to a clear drop in accuracy. This suggests that when all three modalities are present, the model identifies language as the dominant modality. This observation is supported by our analysis of the llyer-layer attention distribution (see Figure A. in our Supp. D), where 70% the attention is directed toward language modality.
Table 5: Ablation study on dominant modality recognition. We compare fixed dominant modality settings with our adaptive recognition strategy. AVCD consistently outperforms static configu- rations, demonstrating the effectiveness of dynamic modality selection.
Dominant Modality
Dataset
AVHBeach [51]
MSVD-QA [62]
Audio
68.67
Vision
67.79
70.70+:0.4
Langunge Adaptive
72.15
74.20:10.14
72.15
75.20:10.42
In contrast, for video-text input pairs, the attention is more evenly split between video and text (44% vs. 56%). indicating that no single modality clearly dominates. In such cases, adaptively selecting the dominant modality leads to better performance than relying on fixed choice, such as always prioritizing video or language. These results demonstrate that, when diverse modality pairs are provided, determining the dominant modality based on attention distribution offers flexible and effective alternative to fixed strategies, maintaining or even improving model performance.
Conclusion
In this work, propose Audio- Visual Contrastive Decoding (AVCD), training- maining-frree and generaliz- able inference-time decoding framework for AV-LLMs. AVCD extends CD to the trimodal setting by introducing three key components: (1) dominance- aware attentive masking to identify and perturb less dominant modalities based on attention distributions, (2) a trimodal contrastive formulation that captures hallucination patterns emerging from complex cross-modal interactions, and (3) an entropy guropy-guided adaptive decoding mechanism that improves inference efficiency. Together, these components enable AVCD to reduce modality-induced hallucinations effectively, while maintaining computational efficiency. Our approach is broadly applicable to MLLMs beyond AV-LLMs, offering a plug-and-ple and-play solution to hallucination mitigation during inference.

---

Acknowledgments
This work was supported by Institute of Information & communications Technology Planning Evaluation (IIP)) grant funded by the Korean government (MSIT, RS-2025-02263977, Develop- ment of Communication Platform supporting User Anonymization and Finger Spelling-Baas -Baeed Input Interface for Protecting the Privacy of Deaf Individuals).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv, 2023.
[2] Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proc. CVPR, 2020.
[3] Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. In Proc. EMNLP, 2016.
[4] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proc. CVPR, 2018.
[5] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Proc. NeurIPS, 2022.
[6] Ali Furkan Biten, Lluís Gómez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reducing object hallucination in image captioning. In Proc. WACV, 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proc. NeurIPS, 2020.
[8] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv, 2023.
[9] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers. In Proc. ICMI,,
[10] Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-aadioo-subiill-eex omni-modality foundation model and dataset. In Proc. NeurlPS, 2023.
[11] Zesen Cheng. Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporry modeling and audio understanding in video-Ilms. arXiv, 2024.
[12] Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. In Proc. ECCV, 2024.
[13] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In Proc. ICLR, 2024.
[14] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Proc. NeurIPS, 2023.
[15] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv, 2023.
[16] Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proc. ICML, 2024.
10

---

[17] Yash Goyal, Tejas Khot, Douglas Summers-Seee Dhruv Batra, and Devi Parikh. Making the vqa matter: Elevating the role of image understanding visual question answering. In Proc. CVPR, 2017.
[18] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proc. CVPR, 2024.
[19]
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations large vision language models. In Proc. AAAI, 2024.
[20] Vipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang, Yingwei Li, and Alan Loddon Yuille. Swapmix: Diagnosing and regularizing the over-reliance on visual context in visual question answering. 2022 ieee. In Proc. CVPR, 2022.
[21] Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proc. CVPR, 2024.
[22] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allccttoo In Proc. CVPR, 2024.
[23] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. In Proc. NeurIPS, 2023.
[24] Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective decoding: Alleviating hallucinations for large vision-language models. In Proc. ICLR, 2025.
[25] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1-38, 2023.
[26] Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. Code: Contrasting self-generated description to combat hallucination in large multi-modal models. In Proc. NeurlPS, 2024.
[27] Sicong Leng, Yun Xing, Zesen Cheng, Yang Zhou, Hang Zhang, Xin Li, Deli Zhao, Shijian Lu, Chunyan Miao, and Lidong Bing. The curse of multi modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio. arXiv, 2024.
[28] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large dision-language models through visual contrastive decoding. Proc. CVPR, 2024.
[29] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions dynamic audio-visual scenarios. Proc. CVPR, 2022.
[30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. Proc. ICMI., 2023.
[31] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. In Proc. NeurlPS, 2024.
[32] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proc. ACI, 2023.
[33] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object ballucination in large vision-language models. In Proc. EMNLP, 2023.
11

---

[34] Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, al. Omnibench: Towards the future of universal omni-language models. arXiv, 2024.
[35] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Yuan. Video-llava. Learning united visual representation by alignment before projection. In Proc. EMNLP, 2024.
[36] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. In In Proc. CoRR, 2023.
[37] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv, 2024.
[38] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proc. CVPR, 2024.
[39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. In Proc. NeurIPS, 2023.
[40] Yang Liu, Yuanshun Yao. Jean-Franconi Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy 1lms: a survey and guideline for evaluating large language models' alignment. arXiv, 2023.
[41] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang. Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radiology, page 100017, 2023.
[42] Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv, 2023.
[43] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proc. ACL, 2024.
[44] Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proc. ACL, 2023.
[45] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look at language bias. In Proc. CVPR, 2021.
[46] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proc. NeurIPS, 2022.
[47] Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: framework for aligning x-modal instruction-ave -warre representations to llms and emergent cross-modal reasoning, arXiv, 2023.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. ICML, 2021.
[49] Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung- Woo Ha, and Jinwoo Shin. Hierarchical context merging: Better long context understanding for pre-trained llms. In Proc. ICLR, 2024.
[50] Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced visuaal large language models. In Proc. ICLR, 2024.
12

---

[51] Kim Sung-Bin, Bin, Oh Hyun-Bin Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyum Oh. Avhbench: A cross cross-modal hallucination benchmark for audio-visual large language models. In Proc. ICIR, 2025.
[52] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv, 2022.
[53] SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. comprehensive survey of hallucination mitigation techniques in large language models. arXiv, 2024.
[54] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne I Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv, 2023.
[55] Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. Survey factuality in large language models: Knowledge, retrieval and domain-speciffiityy arXiv, 2023.
[56] Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision- vision-lanuuage models with instruction contrastive decoding. In Proc. ACL, 2024.
[57] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. In Proc. ICLR, 2022.
[58] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Proc. NeurIPS, 2022.
[59] Sangmin Woo, Jaehyuk Jang, Donguk Kim, Yubin Choi, and Changick Kim. Ritual: Random image transformations as a universal anti-hallucination lever in Ivims. arXiv, 2024.
[60] Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv, 2024.
[61] Yike Wu, Yu Zhao, Shiwan Zhao, Ying Zhang, Xiaojie Yuan, Guoqing Zhao, and Ning Jiang. Overcoming language priors in visual question answering via distinguishing superficially similar instances. In Proc. ACL, 2022.
[62] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proc. CVPR, 2016.
[63] Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv, 2024.
[64] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using va-vae and transformers. arXiv, 2021.
[65] Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios. In Proc. ECCV, 2024.
[66] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- fine-grained correctional human feedback. In Proc. CVPR, 2024.
[67] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qua A dataset for understanding complex web videos via question answering. In Proc. AAAI, 2019.
[68] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv, 2024.
13

---

[69] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction- tuned audio-vinn visual language model for video understanding. In Proc. EMNLP. 2023.
[70] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-ine init attention. In Proc. ICLR, 2024.
[71] Yue Zhang. Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's 's song in the ocean: survey on hallucination in large language models. arXiv, 2023.
[72] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv, 2023.
[73] Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv, 2023.
[74] Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In Proc. ICLR, 2024.
[75] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. In Proc. ICLR, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We highlight our contributions in both the abstract and the final two paragraphs the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We list our limitations in the supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We include the detailed proof in Supplementary Materials A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe the details of experiments and will release the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We will release the code open access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //o////ooaeeeeeeeeeee
/nips wwtppbccc//paeeccccccccccccc / ssisonolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We include experiments setting and the details in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report both the mean and standard deviation for the GPT-assisted evalua-
tion.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the computational resources in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the csGuidel ines?
NeurIPS Code of Ethics https: httpppss///////ss...ee// wwtppp/cccpppciccccceocccpppeepe http://///aaaappp..///////////aa
Answer: [Yes]
Justification:
We follow the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include that in the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose any foreseeable risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly open source assets following their license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't 't use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We don't include human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: In this paper, the core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tt///errss...e// /ttp///ips.ce/oooeeeeeeeeeeeeeee c for what should or should be described.
21

---

Supplementary Material: AVCD
This supplementary material complements the main paper by providing the following sections. To support code reproducibility, we include the source code along with a README file.
A Detailed Proof of AVCD
23
A.1 Proof of Eq. (7)
23
A.2 Detailed proof of Eq. (10)
23
Adaptive Plausibility Constraint
23
C Experiments on Image-LLMs
24
Further Discussions
24
D.I Modality dominance analysis
24
D.2 Impacts of a
25
D.3 Evaluation OmniBench dataset
26
D.4 Per component analysis
26
D.5 Impacts of masking ratios
26
D.6 Analysis on hallucinations
27
Algorithm of AVCD
27
F Further Qualitative Results
27
Computational Resource
27
H Limitations and Future Works
27
Social Impact
27

---

A Detailed Proof of AVCD
A.1 Proof of Eq. (7)
Since A and В represent probabilities as defined in Eq. (6), they are positive numbers and let M + ó and B M 6, where M is the mean and ó is small perturbation with |o| < M. Applying the Taylor expansion of the logarithm function, we have
8²
log( M + 6) = log M +
2M M
(12)
8²
log(M - 6) ~ log M
M 2M2
Now, taking the average of log A and log B, we get the right-hand side term of Eq. (7):
log( M + 8) + log(M - 6) log M
8²
(13)
2
2M
Given that the left-hand side of Eq. (7) is log M, the resulting error scales with the square of the difference between A and B.
A.2 Detailed proof of Eq. (10)
Considering the language modality dominant, CD can be extended to mitigate hallucinations in AV-LLMs by leveraging logit" and logit", dealing with the video and audio modalities in CD, respectively. Applying the logarithm function to Eq. (4) yields the following:
log (exp(x1(''' xx))
= log p
xx,x x²,x)) + 2 p(xx x x=",, ,x,x)
(14)
(logp(y:(x x", x²) log logply: xx
Similarly, applying the logarithm to Eq. (5) and substituting the results into Eq. (2), we derive:
logit" & (1+ (1+aa) a²) lx x') - a log p(x |x xx"",x"" x')
(1 + (log(()))lll x" + logp(x//x" x"" x'))
(15)
a" (2oglg
x", x²) 1x,
where a controls the degree of contrastive influence.
Following the same procedure for CD in the audio domain, we derive the corresponding logit as:
logit
(1 + (1og))) xx" x²) logp(x//x x", x')x
(16)
a" (logp(x;(x"" x xx)) + log p
where a regulates the strength of CD. Therefore, by summing Eq, (15) and Eq. (16) to account for both non-dominant modalities, we obtain Eq. (10) as the final logit expression.
B Adaptive Plausibility Constraint
CD penalizes model outputs that rely distorted inputs, thereby promoting more reliable generation process. However, a critical challenge arises when such penalization leads to incorrect outputs. In particular, overly strict penalties inadvertently suppress valid outputs that align with linguistic norms and commonsense reasoning, while promoting low cow-probability tokens that degrade overall quality.
To address this issue, we incorporate an adaptive plausibility constraint, inspired by [32], which dynamically truncates the candidate logits by retaining only those tokens whose probabilities under
23

---

Table A.1: Results on an image-LLM using the LLaVA-1.5 [38] model evaluated on the POPE [33] dataset. AVCD outperforms both the original model's decoding (Base) and VCD [28], demonstrating its strong generalization capability across AV-, video-, and image-II.Ms.
Random
Popular
Adversarial
Method
Acc.
FI Score
Acc.
FI Score 1
Acc.
Fl Score 1
MSCOCO
Base
82.93
80.87
81.13
79.27
81.10
77.60
VCD [28]
85.53
84.04
83.63
82.32
80.87
80.13
AVCD
86.03
84.87
84.23
83.24
81.27
80.70
AOKVOA
Base
84.03
83.22
80.20
80.00
74.23
75.33
VCD [28]
85,90
85,46
82,00
82.15
76.17
77.71
AVCD
85.97
84.46
83,90
82.57
81.63
76.20
GQA
Base
83.60
82.79
77.90
78.11
75.13
79.20
VCD [28]
85.97
85.54
79.27
80.01
76.53
77.97
AVCD
85.97
84.46
82.57
81.63
80.58
the original model exceed a predefined plausibility threshold after CD. The constraint is formally defined as follows:
Vnead (Y -yr E V Po(v x,y<t) 2 max (w x,y=,)),
(17)
where V/ represents the model's output vocabulary, and 3 E [0, 1] is hyperparameter that determines the level of truncation. higher 3 enforces more aggressive truncation, limiting the output space to high-conficn confidecne logits from the original distribution. For AVCD, we set 3 = 0.1, following the configuration used in VCD [28] and SID [24].
Given this constraint, we redefine the contrastive probability distribution as:
logit logit (Y: x,y = -iinf, y é (y
(18)
This formulation removes tokens that deviate significantly from the original output distribution, reducing the risk of generating implausible content. By integrating this constraint with CD, we refine the final token sampling process:
Yt (oftmax(logitAvci))
subject to ye Veea(Ver).
This mechanism narrows the candidate pool to retain only the most probable token and prevents the model from inadvertently favoring improbable tokens due to contrasting distorted inputs.
C Experiments on Image-LLMs
To demonstrate the broad applicability of AVCD, also evaluate it iMa. We the LLaVA- 1.5 [38] and compare the performance of VCD [28] and AVCD. As shown in Table A. the results show that AVCD consistently outperforms VCD most datasets, though VCD was originally designed for image-LLMM L..s. This indicates that AVCD is effective not only for audio-visual tasks but also for maage-based tasks.
D Further Discussions
D.1 Modality dominance analysis
We conduct detailed analysis of modality dominance in VideoLLaMA2 [11] using both its audio- visual and video-only variants. Following the methodology described the main paper, we compute attention weights based the final token and calculate the average dominance over 200 samples. For the AV-LLM, we use the AVHBench [51] dataset, and for the video-LLM, we use the MSVD-QA [62] dataset.
Figure A.1 (a) shows the modality dominance among language, video, and audio in the AV-ILM setting. Language accounts for 70% of the attention, while video and audio contribute 17% and

---

%6%
(a) AV-LLM
(b) Video-LLM
Figure A.1: Modality dominance analysis using VideoLLLaMA2 [11]. The analysis is conducted on the AVHBench [51] dataset for audio-visual inputs (AV-LLM) and the MSVD-QA [62] dataset for video-only inputs (Video-LLM).
Aoqurra (%)
VED
a value
Figure A.2: Ablation study on a values. We evaluate several CD methods originally designed for Mnage-LLMs, including VCD [28] and SID [24], along with proposed AVCD, which is specifically designed for AV-LLMs. We vary the a value from 0.5 to 3. VCD* and SID* denote extended versions of the original methods, adapted using our formulation in Eq. (10). This formulation consistently improves performance when applied to all methods, demonstrating its generalizability across decoding strategies. Moreover, VCD* consistently outperforms SID* across settings, highlighting its robustness beyond Image-language domains.
13%, respectively, revealing a strong bias toward the language tokens. Figure A.1 (b) presents the dominance between language and video in the video-LLM setting, showing more balanced distribution 56% vs. 44%.
In AV-LLM, language is the dominant modality for all 200 samples, which explains why fixing the dominant modality to language yields the same performance as selecting it adaptively, shown in Table 5 of the main paper. In contrast, video-LLM exhibits sample-speciftt variation in dominant modality. This highlights the benefit of adaptively selecting the dominant modality, which results in noticeable performance gains, as evidenced in Table 5.
D.2 Impacts of a
To evaluate the performance variation with respect changes in a, we measure the performance while gradually adjusting the value of a. As shown in Figure A. (a), the dominance between video and audio relatively balanced, which motivates us to set a a" a We also observe that even when is adjusted to reflect the slightly higher dominance of the video modality empirically, the performance remains largely unaffected. For instance, when using a" = 2 and a = 2.5, the accuracy the AVHBench validation set with VideoLLaMA2 remains at 1.95%, identical to the result obtained with the balanced setting a = a a 2.5.
As shown in Figure A.2, we compare the performance of VCD [28]. SID [24], their enhanced versions VCD* and SID (which incorporate Eq. (10)), and AVCD across different values. AVCD achieves the best performance when a 2.5, while the performance of the other models drops sharply as increases. Notably, the models that incorporate Eq. (10) exhibit a relatively smaller performance drop, indicating that our newly defined CD formulation contributes to improved model robustness. This also demonstrates that the attentive masking strategy employed in AVCD is resilient to changes in the value.
25

---

Table A.4: Evaluation on the OmniBench dataset. AVCD demonstrates robust generalization, con- sistently surpassing the base model. The following abbreviations indicate the evaluation categories: Action& & Activity, Story: Story Description, Plot: Plot Inference, Object: Object Identifica- tion & Description, Context: Contextual & Environmental, Identity: Identity & Relationship, Text:
Action:
Text & Symbols, Count: Count Quantity.
Decoding
Overall
Action
Story
Plot
Object
Context
Identity
Text
Count
Base
33.5
27.1
27.0
24.1
61.1
30.1
46.9
21.4
7.1
VCD
23.3
18.3
17.4
13.1
44.6
26.2
37.5
7.1
14.3
OPERA
30.8
27.9
23.0
20.7
52.1
36.9
34.4
14.3
7.1
VCD*
31.9
27.1
26.1
21.5
57.4
30.5
43.8
21.4
0.0
AVCD
34.5
28.3
28.7
24.5
60.7
30,5
50.0
21.4
7.1
D.3
Evaluation on OmniBench dataset
Following OmniBench [34], we adopt Video-SALMON, as the backbone model, for which the original paper reports an overall accuracy of 35.6%. Under the same setting, we reproduce the benchmark and obtain 33.5% accuracy with Base decoding, which serves as our baseline throughout experiments.
As summarized in Table A.4, AVCD emerges the only method that consistently surpasses the base model across most evaluation settings. In contrast, alternative approaches frequently fail to improve upon the baseline, often yielding comparable or even degraded results. This consistent superiority underscores AVCD's robust generalization capability and resilience dataset variability, demonstrating its effectiveness even on challenging benchmarks such as OmniBench.
D.4
Per component analysis
We conduct an ablation study on AVH- Bench validation set using Vid22 as the base model in Table A.2. Baseline refers contrastive decoding (CD) setup with randomly selected dominant modality and Eq. (11). Dominance- aware masking improves accuracy by approximately +5%. Trimodal CD with Eq. (10), our proposed extension of CD to trimodal alignment, further improves performance by +2.9%.
Table A.2: Per component analsysis on AVHBench. Results show that each component dominacce--------------- masking, trimodal contrastive decoding, and entropy- guided adaptive decoding- plays complementary role in improving either accuracy or efficiency.
Method
Accuracy (%)
Inference Speed (seciooken)
Raseline
74.15
4.4
Dominance-aware masking
79.02
Eq,
81.95
EAD (Oons)
81.95
311
Entropy-guided adaptive decoding (EAD) maintains accuracy while reducing inference latency by over 30%. These results validate the complementary roles of three AVCD components as each contributes meaningfully either accuracy efficiency.
D.5 Impacts of masking ratios
We investigate the impact of the masking ratio P in our attentive masking strategy, which determines the propor- tion of high-attention tokens be masked. As shown in Table A.3, we experiment with masking ratios of 25%, 50%. 75%, and 100% (i.e., full masking) and evaluate performance on the AVHBench validation set. Among these, 50% masking ratio yields the highest accuracy, When the ratio is too low (e.g., 25%). the contrast with the original model output is insufficient. limiting the effec- tiveness of contrastive decoding. Conversely, overly high masking ratios (75% or 100%) cause large deviations from the original logits, leading to greater error in the logarith- mic approximation (see Section A. and thus degraded performance.
Table A.3: Ablation study on masking ratio. high masking ratio significantly distorts the logit distribution, while low masking ratio retains similarity to original logits. A 50% masking ratio is found to be optimal.
Masking ratio P
Ace (%)
25
80.98
50
81.95
75
80.00
100
80.00
26

---

D.6 Analysis on hallucinations
We categorize hallucinations into three representative types defined in AVHBench. (1) A->: Vision- centric questions misled by irrelevant or misleading audio cues (e.g.. Q: Is the ship visible in the video? Video: No ship visible, Audio: Ship sounds present.). (2) V-+A: Audio-centric questions misled by visual content (e.g.. Q: Is the lion making sound? Video: Lion visible, Audio: No lion sound.). (3) AV Match- ing: Failures to correctly judge the consistency be- tween audio and visual modalities (e.g.. Q: Are the contexts of audio and visual content matching?).
Table A.5: Categorization of hallucinations addressed by AVCD. AVCD consistently re- duces hallucinations, particularly for V-A and AV Matching.
Category
Base
AVCD
Improvement
A-V
86.4%
86.4%
0.0%
V-+
81.3%
86.3%
+5.0%
AV Matching
64.4%
71.2%
+6.8%
We evaluate AVCD each category using VideoLLaMA2 as the base model on the AVHBench validation set. As shown Table A.5, while AVCD preserves the already high performance A- it achieves substantial improvements in V-+ and AV Matching. These results support our claim that AVCD more effectively balances the contributions of both modalities.
E
Algorithm of AVCD
Algorithm. shows the overall AVCD algorithm. We begin by computing the original logits and estimating the modality dominance Dar- If the entropy of the original logit distribution is sufficiently low, we directly use the original logits without applying further decoding steps. Otherwise, when language is identified as the dominant modality- as is often the case-wa compute the logits from audio-masked, video-masked, and audio-visual masked signals. We then apply our reformulated CD strategy described in Eq. (10). This process is repeated autoregressively until the end-of-sequence (EOS) token generated.
F Further Qualitative Results
We provide additional qualitative results for the AV-LLM in Figure A.3, the video-LLM in Figure A.4, and the image-LLM Figure A.5 and Figure A.6.
G Computational Resource
We run all experiments on a machine equipped with an AMD EPYC 7513 32-core CPU and single NVIDIA RTX A6000 GPU. To obtain reliable inference speed measurements, all background processes unrelated the experiment are disabled during runtime.
H Limitations and Future Works
While extensive experiments demonstrate that the proposed AVCD effectively mitigates hallucination in AV-LLMs at test time, introduces additional computational overhead due to increased forward passes as the number of modalities grows. To address this, we propose entropy-guie adaptive decoding strategy, which significantly improves inference speed. However, this approach may overlook certain types of hallucinations that oceur work, we aim to develop algorithms that address the potential issues caused by skipping, through a detailed analysis of cases where hallucinations occur despite low entropy.
when the model appears confident. In future
I
Social Impact
AV-LLMS are increasingly applied in real-world scenarios such as education, medical video analy- sis, assistive technologies, and interactive audio-visual systems. Our proposed AVCD framework contributes to this progress by enabling more accurate and robust multimodal understanding.
27

---

Algorithm Audio-Visual CD (AVCD)
Require: Multimodal inputs (x", x2, x²), Audio-visual Large Language Model LM, Contrastive
Weights a", a.. Entropy Threshold 7
Ensure: Decoded output sequence
1: Initialize empty output sequence <- 0
2: while EOS token $ y do
3: Compute original logits and dominance score:
logit, DM < LM(x", x²; x², yct)
4: Compute entropy:
Pt softmax(logit)
H Σ Pt log
5:
if H, then
6:
Append top token: De < argmax logit continue
7:
8:
end
9:
Apply modality masking based on DM Visual-masked logits:
logit" [M x x!, y<r)
11:
Audio-masked logits:
logit +- LM(x", x x', yer)
12:
Both-masked logits:
logit + L.Mx"",, x ", ,x,
13:
Compute AVCD-adjusted logits:
logit AVCD = (2 + a" + a logit
+ (1 a" + a logit
+ (1 + a"
logit
- (a" + logit
14: Append top token: O. + arg logit AVCD
15:
Append de to y
16: end while
return y
As test-time decoding method, AVCD requires additional training model modification, making a practical solution for enhancing existing models. This property also promotes energy efficiency by avoiding the need for computationally intensive retraining, which particularly beneficial when scaling AV-LLMs to real-world deployment scenarios. Overall, AVCD facilitates broader and more responsible use of multimodal models through improved inference quality and efficiency.
28

---

with
Prompt: Describe the given vidco and audio in detail.
Visual mismatch
Base
Under the soft glow of campfire, group of friends gathers around table, immersed in conversation and camaraderic. The warmth of the fire dances in their cyes as they share stories and laughter, punctuating the night air with the occasional crackle and pop the burning wood. As the evening deepens, the flickering flames cast an ever-changing light their faces, creating an atmosphere cozy comfort and shared joy. In the distance, the quiet hum of the wilderness echoes off the surrounding foliage, adding the serene ambiance of this nocturnal gathering.
x Fail describe the scene change and audio information
VCD
Under the calm of the night sky, small group gathers around a crackling campfire, engaged conversation and simple joy being together. The warmth the flames dances off their faces, as they share stories and laughter, the embers casting soft glow their smiling faces. As the night grows late, the fire's light becomes a beacon, drawing in the quiet stillness of the evening. symbolizing moment of camaraderie and connection under the vast, dark canopy above. X Fail describe the scene change and audio information
AVCD
Under the calm of a night sky, a small group gathers around crackling campfire, immersed conversation and the shared camaraderie of outdoor adventure. The warmth of the flames dances off their faces, accentuating the outlincs of their relaxed postures as they lounge their camping chairs. The rhythmic popping of the fire provides soothing backdrop to the soft murmur of their conversation.
As the night deepens, one of the group takes on the role of cook, reviving the fire with hearty breakfast. Amidst the sizzling and clinking of utensils, the aroma of cooking wafts through the air, mingling with the crisp, earthy scent of the campsite. The cook's skilled hands work methodically, flipping and stirring the food in black cast-iron skillet, ensuring that every morscl is cooked to perfection.
Figure A.3: Example of hallucination in the AV-LLM using VideoLLaMA2 [11]. The video
transitions from night to morning. However, except for AVCD, both the base decoding and VCD [28] fail to capture this scene change. Additionally, in the nighttime scene, people are seated in camping chairs around fire, but no table is present-an aspect incorrectly described by the base decoding.
29

---

Prompt: Is the athlete wearing pant?
Visual mismatch
Bala
Base
VCD athlete is wearing red pant.
AVCD
The athlete wearing pant.
No, the athlete not wearing pant. wearing shorts.
Prompt: Does the person in black wear a hat?
Base
VCD
AVCD
Yes, the person in black wearing in the video.
Yes, the person in black is wearing
No, the person in black does wear
Prompt: Is the person in white wearing glasses?
Soseeeooooa MALAN TUMEL
Base
Yes, the person white wearing glasses.
Yes, the person in white wearing glasses.
No, the person in white is not wearing glasses.
Figure A.4: Example of hallucination the video-LLM using VideoLLaMA2 [11]. AVCD effectively mitigates hallucinations caused by the original model's failure to incorporate video context.
30

---

Prompt: Describe this photo in detail.
*Visual mismatch
Base
The painting depicts beautiful woman in profile, wearing black veil and smiling for the camera. She appears relaxed and graceful, possibly part of a Renaissance-era portrait. Close to her face, there computer mouse that stands due the size difference between the two subjects. The painting itself is stunning piece art, capturing woman's expression and gaze that draw the viewer in.
VCD
The image depicts a close-up of a woman with veil covering her face. She wearing a red dress and smiling. The painting captures beautifully detailed portrait of the woman, emphasizing her features and expressions. The background of the painting features a mountainous landscape, which adds depth and sense of seale the scenc. The overall composition of the image creates captivating and intriguing visual experience for the viewer.
AVCD
The painting a close-up portrait of woman wearing a veil and Renaissance-style dress. She has smiling expression, and her eyes looking directly at the viewer. The woman's features, including her lips, nose, and chin, are distinct and well well-defined. In the background of the painting, there is mountainous landseape with rocky cliff visible, adding depth and context the scene. The overall composition of the painting captures the beauty and charm of the woman, while the background landscape creates captivating visual backdrop.
Figure A.5: Example of hallucination in the image-LLM using LLa [38].
31

---

Prompt: Deseribe this photo detail.
VViuaa mismatch
Base
The image features painting of Shiba Inu dog wearing tie and glasses, posing with smug expression as if engaged in theatrical stage make- believe. The dog stands as the central figure while wearing tiny hat and red gown, giving it unique appearance. Around the painting, there rocky landscape with several more rocks dispersed throughout the picture. A dining table can be seen the right side of the painting, possibly suggesting stage setting also present behind notion that this image evokes theatrical or performance feel.
this theatrical scene. chair tuble, further suppurting the
VCD
The image features a unique painting o dog, possibly poodle, wearing hat and a Renaissance- style gown. The dog sitting chair, positioned in way that it appears be looking the viewer. The painting gives off whimsical and artistic vibe. In background, there are several elements that add to the scene's ambiance. There are two chairs, one placed near the dog and the other in the right-hand corner of the image. There are also two people visible, one standing towards the right side of the scene and another near the top right comer. Additionally, there are two boats present the background, one on the left side and the other towards right.
AVCD
The image features a painting of person wearing Renaissance-style outfit, complete with gown, and hat. The person appears be dog dressed up in costume, giving the scene humorous and creative touch. In the background, there several hills and river, which adds overall artistic composition the painting. unique combination of cosummed dog and Renaissance-style outfit creates an interesting visual experience.
Figure A.6: Example of hallucination in the image-LLM using LLa [38]. AVCD generates more accurate image-based descriptions compared the base decoding and VCD [28].
32


Reviews:
[{"review_id": "6CS5T4r4BR", "reviewer": "Reviewer_yejw", "summary": "The paper tackles the problem of CD not being well-suited for AV-LLMs on hallucination by proposing AVCD to surpress modality-induced hallucinations. It leverages attention distributions to identify less dominant modalities and generated perturbed outputs and entropy guided adaptive decoding to selectively skip unnecessary decoding steps. It is training free and effective on multiple AV benchmarks.", "questions": "Have you tried using different masking probability P for different layers or even different models? Correct me if you have done so because intuitively a global P can't be optimal for different layers.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The method is training free.\n\n2. Clear ablations on dominant modalities ablation and masked modality.\n\n3. Proposed efficient inference by entropy guided adaptive decoding and clearly shows trade-off between accuracy and inference time.\n\n4. First CD framework that handles three modalities jointly (audio, video, language) instead of corrupting a single fixed channel.\n\nWeakness:\n\n1. I believe the masking ablation needs a row where you perform completely random modality-agnostic or modality-specific masking. Or you can compare to previous works that fabricate weak context such as noise, random corruption, adversarial inputs, etc. \n\n2. The attention mask potentially has weak foundation where the attention doesn't reflect the 'real strong modality' if it is noisy, or affected by certain prompt engineering. If the attention doesn't track true importance, then the entire framework will still hallucinate. It would be great if there is an empirical evidence that attention is mostly 'correct'.\n\n3. The speed is still 2x slower but it's shouldn't be a strong weakness.", "was_revised": true, "final_justification": "Thanks for the response. The authors have answered all my questions and I currently have no concern. I am not worried about the scoping of 'what is hallucination' here since general performance is still an indicator of levels hallucination. But it did seem that the scoping of this paper can be slightly more general by focusing on this method improves the general performance. Otherwise, the authors can include more analysis on hallucination, which fortunately is included in other reviewers' rebuttals.", "rebuttal": "We thank the reviewer for the valuable feedback. The comments helped us create a clearer and more understandable final version for readers. Based on the comments, we have:\n\n> **(1)** *conducted comparison experiments between random masking and attention-guided masking,*  \n> **(2)** *examined the reliability of analyses based on attention weights,*  \n> **(3)** *re-emphasized our efforts to improve inference speed,*  \n> **(4)** *supplemented the explanation that our masking strategy is based on global attention.*\n\nOur detailed responses are provided below.\n\n---\n\n**[W1] Validation of attention-guided masking against random masking or noise injection.**\n\nThank you for the insightful comments. In response, we conducted additional experiments to compare **Attention-Guided Masking** with **Random Masking** strategies.\n\n| Method | Accuracy (%) |\n| --- | :---: |\n| Base | 78.05 |\n| Random Masking | 71.71 |\n| **Attention-Guided Masking (Ours)** | **81.95** |\n\nWhen we apply random masking, the performance (71.71%) drops significantly compared to the base decoding (78.05%). In contrast, our proposed attention-guided masking selectively distorts the modality that truly affects the model output, resulting in improved performance (81.95%).\n\nAdditionally, as shown in **Table 1** of the main paper, AVCD consistently outperforms VCD, which perturbs the input by injecting noise, across all benchmarks. This suggests that simply corrupting the input with random noise is not as effective as selectively masking semantically meaningful tokens. Therefore, our method provides evidence that attention can serve as a useful signal for locating critical information, helping reduce hallucinations more reliably.\n\n---\n\n**[W2] How reliable are the attention weights in identifying key modalities?**\n\nIn **Table 4**, we validate the effectiveness of using attention weights to identify the dominant modality. Our adaptive strategy frequently selects “language” as the dominant modality, and this choice consistently yields better performance compared to manually assigning video or audio as dominant.\n\nMoreover, attention weights have been extensively studied in prior works **[1,2]** for identifying important tokens in LLMs, further supporting their reliability in modality attribution.\n\n> **[Reference]**  \n> **[1]** *Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models, ICLR 2025.*  \n> **[2]** *Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs, ICLR 2024.*\n\n---\n\n**[W3] A minor concern regarding the 2× slower inference speed.**\n\nThank you for recognizing that the speed gap is not a major weakness in the context of our overall contribution. We would like to emphasize that, despite handling three modalities in AV-LLMs, our method accelerates inference speed by incorporating an entropy-guided adaptive decoding strategy that effectively skips unnecessary steps. This mechanism helps reduce computational overhead and maintains a balance between richer multimodal reasoning and decoding efficiency.\n\n---\n\n**[Q1] Have you considered varying the masking probability (P) across different layers or models?**\n\nWe apologize for any confusion caused.\n\nTo clarify, we compute attention values across all layers and mask the top 50% of tokens based on the global distribution. As a result, the number of masked tokens varies per layer, effectively leading to a layer-wise adaptive masking scheme.\n\nTo validate this, we conducted a comparison against a setting where we mask the top 50% attention-value tokens separately within each layer. The results are shown below:\n\n| Masking method | Accuracy (%) |\n| --- | :---: |\n| w/o masking (Base) | 78.05 |\n| Fixed - top 50% per layer | 79.02 |\n| **Adaptive - top 50% (Ours)** | **81.95** |\n\nWhile fixed masking is already more effective than Base decoding, our adaptive strategy guided by global attention achieves significantly better performance. This supports the idea that masking based on global attention better identifies influential tokens across the entire model.\n\n---"}, {"review_id": "DErNyLgtU9", "reviewer": "Reviewer_seE2", "summary": "This paper introduces Audio-Visual Contrastive Decoding (AVCD), a novel, training-free decoding framework designed to mitigate hallucinations in audio-visual large language models (AV-LLMs). The authors suggest that existing contrastive decoding (CD) methods, primarily developed for Vision-Language Models (VLMs), are ill-suited for more complex trimodal interactions that cause hallucinations in AV-LLMs.\n\n*   Trimodal Contrastive Decoding Formulation: The paper extends the conventional CD framework to jointly handle audio, visual, and textual inputs. It provides a mathematical reformulation that allows for contrasting outputs based on perturbations across multiple modalities simultaneously.\n*   Dominance-Aware Attentive Masking: Instead of perturbing a fixed modality, AVCD dynamically identifies the less dominant modalities at each decoding step by analyzing the model's internal attention distributions. It then applies an \"attentive masking\" strategy to these less influential modalities to generate the perturbed logits for contrastive decoding. This approach avoids introducing noise from direct input distortion.\n*   Entropy-Guided Adaptive Decoding (EAD): To improve inference efficiency, AVCD calculates the entropy of the model's initial prediction; if the entropy is low (indicating high confidence), the more computationally expensive contrastive steps proposed are skipped. This balances the performance gains of CD with computational overhead.\n\n### Experimental Results\n\nThe method was evaluated on several MLLMs, including AV-LLMs (VideoLLaMA2, video-SALMONN), video-LLMs, and image-LLMs. Key findings include:\n*   On the AVHBench hallucination benchmark, AVCD improved accuracy by 6% for VideoLLaMA2 and 11% for video-SALMONN compared to the baseline decoding method.\n*   AVCD consistently outperformed both baseline decoding and an extended version of the VCD method across multiple datasets, including MUSIC-AVQA, MSVD-QA, and ActivityNet-QA.\n*   Ablation studies evaluate the the core components. The adaptive modality recognition strategy is compared to fixed-modality approaches, with some gains. The trade-off between accuracy and inference speed from the EAD mechanism is shown.\n*   The proposed method outperforms VCD on image-LLM benchmarks for which VCD was originally designed.", "questions": "1. Could you clarify the apparent tension in requiring a low KL divergence to validate the approximation in Equation 7, while also needing a sufficiently large difference between distributions to provide a meaningful contrastive signal, particularly for specific tokens.\n\n2.  On Computational Efficiency: The need to materialize the attention matrix precludes the use of optimizations like FlashAttention. Could you comment on the practical trade-offs in inference speed and how the \"Entropy-Guided\" mechanism mitigates this? Noting the need for the costly attention matrix materialization is only known once the forward pass for the original model has been done. \n\n3.  Equation 10 suggests a contrast against both audio and video modalities, while the text describes masking only adaptively identified less dominant modalities. Could you explicitly clarify how the definite formula is reconciled with the adaptive modality selection described?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n### Intersting Topic\nThe submission tackles the complex issue of hallucinations in trimodal audio-visual large language models (AV-LLMs), a challenging area that has received less attention than the more popular vision-language models (VLMs).\n\n### Extends existing work\nThe submission presents some mathematical extensions of previous CD frameworks.\n\n### Experiments\nThe proposed method is compared to a number previous comparable works and demonstrates some large and some more marginal gains.\n\n## Weaknesses\n### Justification of the Mathematical Approximation\nA core component of the proposed work is the mathematical approximation presented in Equation (7), which simplifies the combination of logit distributions from original and corrupted signals.\n\n$\\log(\\frac{A + B}{2}) ≈ \\frac{log(A) + log(B)}{2}$\n\nWhile this reformulation is helpful, a potential weakness arises from its justification. The authors validate this approximation by demonstrating that their \"attentive masking\" strategy yields a small Kullback-Leibler (KL) divergence between the logit distributions of the original and corrupted signals (Figure 4). This low divergence is presented as a strength, indicating that the method introduces less distortion than prior techniques and so the approximation is valid.\n\nHowever, this justification creates an apparent contradiction with the fundamental premise of contrastive decoding.\n*   **The core issue:** The method's mathematical validity rests on the assumption that the two distributions are **similar** (low KL divergence). Yet, the method's efficacy is presumed to come from the **contrast** between them. If the distributions are similar enough to validate the approximation, it is unclear whether there is sufficient difference between them to provide a meaningful contrastive signal that can effectively steer the model away from hallucinations. The two conditions seem mutually exclusive.\n\nThis concern seems particularly amplified when considering individual token predictions.\n*   **Local vs. Global Accuracy:** The KL divergence is a measure of global similarity across the entire vocabulary distribution. It is plausible that for a small but critical subset of tokens—for instance, a factually correct word versus a likely hallucination—the probabilities from the original and corrupted signals diverge significantly. In these high-impact cases, the Taylor approximation would be locally inaccurate, introducing error precisely where the contrastive mechanism is most needed. The main paper does not analyze or account for the error introduced by the approximation in these specific, high-divergence scenarios.\n\n### Computational Efficiency and Practical Implementation\n\nThe proposed method is reliant on materializing the full attention matrix to identify the \"dominant modality\" at each decoding step, when AVCD is triggered. This design choice has two primary negative consequences:\n\n1.  **Incompatibility with Modern Attention Mechanisms:** The requirement to inspect the numerical attention scores is fundamentally incompatible with state-of-the-art fused attention kernels like FlashAttention. These optimizations achieve significant improvements in speed and memory efficiency precisely by *avoiding* the materialization of the N x N attention matrix. Forcing a model to use a standard, non-optimized attention implementation to enable AVCD would negate these crucial performance gains, making the model inherently slower and more memory-intensive than a baseline that can leverage these common optimizations.\n\n2.  **An Efficiency Paradox in the Adaptive Strategy:** The authors introduce Entropy-Guided Adaptive Decoding (EAD) to reduce computational costs by selectively applying the expensive multi-pass AVCD only when the model's confidence is low. However, this creates an efficiency paradox. It would seem that the model incurs the high cost of materializing the attention matrix on **every single token**, even on high-confidence tokens where the subsequent AVCD step is ultimately skipped, unless the authors have done some inference management to only call the slow un-optimized kernel when AVCD is called.\n\nMoreover, it appears the inference speed with Base as shown in Figure 6 uses the same attention kernel as AVCD given $\\tau = 1.0$ and Base have the same inference speed and performance. Any realistic implementation of \"Base\" would use FlashAttention or another optimized kernel during inference.\n\n### Unifying the Mathematical Framework and Implementation\n\nOn reading Section 3.2, I find the method the authors outline confusing.\nThe core of the issue stems from a disconnect between the formula and the descriptive text:\n*   **The General Formula:** Equation (10) presents a complete trimodal framework where the final logit is adjusted by multiple separate contrastive terms: one for a corrupted audio signal and one for a corrupted video signal and one for both signals being corrupted. This mathematical representation implies that the model's output is shaped by contrastive pressures from both modalities simultaneously.\n\n*   **The Described Implementation:** The \"Attentive Masking Strategy\" (Section 3.3) details an adaptive process where, at each step, the model identifies and masks specific tokens in identified \"less dominant\" modalities. This description suggests that language tokens could be determined to be a less dominant modality in which case Equation (10) does not represent this possibility.", "was_revised": true, "final_justification": "I thank the reviewers for their comments. They have largely addressed my concerns well and additional concerns (for myself) have not been raised during the discussion period. I raise my score.", "rebuttal": "We sincerely thank the reviewer for the thorough and insightful feedback. The reviewer’s detailed critique greatly helped us deepen our explanation regarding KL divergence and improve the clarity of the paper. Based on the comments, we have:\n\n> **(1)** *provided an analysis of the relationship between low KL divergence and the strength of the contrastive signal,*  \n> **(2)** *explained how our implementation captures local token-level changes,*  \n> **(3)** *compared our approach to scenarios using FlashAttention,*  \n> **(4)** *revised the notation to a more general form reflecting the adaptive modality selection.*\n\nOur detailed responses are provided below.\n\n---\n\n**[W1.1 & Q1] Can low KL divergence still lead to sufficient contrastive effectiveness?**\n\nContrary to reviewer’s concern, low KL divergence does not mean the contrastive signal is ineffective or hallucinations are absent. To support this, we applied AVCD and VCD on the AVHBench test set using VideoLLaMA2. AVCD changed 13.1% of answers (8.5% improved, 4.6% degraded), while VCD changed only 7.4% (4.4% improved, 3.0% degraded), despite VCD showing higher KL divergence. This demonstrates that semantically coherent perturbations from attention-based masking (used in AVCD) provide a more effective contrastive signal than simply increasing KL. Therefore, the two conditions, low divergence and effective contrast, are not mutually exclusive.\n\n---\n\n**[W1.2 & Q1] Can KL divergence effectively capture local token-level changes that are critical to the final answer?**\n\nThank you for raising this important point. We clarify our use of KL divergence in the proposed framework.\n\nIn our implementation, KL is computed after applying the Adaptive Plausibility Constraint, as detailed in **Supp. Sec. B**. This constraint, commonly used in the CD literature, ensures that the final output is restricted to tokens with sufficient confidence in the original logit distribution. \n\nAs a result, the KL divergence is calculated only over the top-K logits that are most likely to influence output generation, addressing the concern that KL fails to capture local behavior.\n\nWe will revise the main paper to explicitly describe this procedure.\n\n---\n\n**[W2 & Q2] Is AVCD computationally efficient and practically applicable, for example in settings that leverage FlashAttention?**\n\nWe acknowledge the reviewer’s valid concern regarding efficiency. Indeed, our current AVCD implementation is incompatible with FlashAttention. To clarify the efficiency gap, we provide the following inference speed comparison (lower is faster):\n\n| Decoding          | threshold (τ) | Speed Scale (s/token) | Accuracy (%) |\n| ----------------- | :-----------: | :-------------------: | :----------: |\n| Base w/ Flashattn |       -       |        1 (1.30)        |     78.05    |\n| Base              |       -       |       1.3 (1.75)      |     78.05    |\n| VCD               |       -       |       1.9 (2.50)       |     62.44    |\n| AVCD              |      0.8      |       1.7 (2.25)      |     80.98    |\n| AVCD              |      0.6      |       2.4 (3.14)      |     81.95    |\n\nWhile FlashAttention is not currently applicable due to the need to access attention scores, our goal is fundamentally different: this paper is the first to design a contrastive decoding (CD) strategy tailored for AV-LLMs. Simply extending existing CD methods designed for unimodal or vision-language models often focused on a fixed modality, which leads to suboptimal or unsuitable behavior in AV settings. Our work identifies this issue and proposes a solution via modality-aware masking and new mathematically extended CD formula for three modalities.\n\nMoreover, instead of naively applying CD across all tokens with multiple forward passes, we introduce Entropy-Guided Adaptive Decoding (EAD) to selectively apply AVCD only when necessary, reducing the computational overhead.\n\nAlthough FlashAttention cannot be used at this stage, we are optimistic. Recent work **[1]** on optimizing CD for LLMs demonstrates that more efficient CD implementations are possible, and we believe future work can extend these techniques to AV-LLMs once modality-aware CD becomes more mature. Our contribution lays the foundation for this line of research, and we ask for your generous understanding of the current trade-off in efficiency in light of the broader methodological contribution.\n\n> **[Reference]**  \n> **[1]** *Fast Large Language Model Collaborative Decoding via Speculation, ICML 2025.*\n\n---\n\n**[W3 & Q3] On the generality of the framework.**\n\nThank you for pointing out this confusion. We apologize for the lack of clarity. In the final version, we will ensure that **Equation (10)** reflects the generalized case, allowing any modality to be adaptively masked based on its dominance score.\n\n---"}, {"review_id": "67vhFD6fmb", "reviewer": "Reviewer_BYoK", "summary": "The paper presents a significant advancement in hallucination mitigation for AV-LLMs with a well-designed, efficient, and empirically validated approach. However, its impact could be enhanced by addressing limitations, expanding evaluations, clarifying content, and exploring practical and ethical implications.", "questions": "Please provide a detailed introduction to the datasets used in the experiments. For example, what are the total numbers of samples in the three datasets?", "limitations": "1.\tIn Formula 1, how is the softmax function utilized? Please explain its working mechanism and potential impacts.\n2.\tThe term \"damaged and biased visual information\" (such as x¬v) mentioned in the formula refers to what? What extent is it considered, and does it reflect noise or bias in the data collection process?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Innovative Framework: Introduces Audio-Visual Contrastive Decoding (AVCD), a novel, training-free method to mitigate hallucinations in Audio-Visual Large Language Models (AV-LLMs), extending contrastive decoding to trimodal settings (audio, video, language).\nAdaptive and Efficient: Uses attention-based modality dominance detection and entropy-guided decoding to balance multimodal reasoning and improve inference speed.\nRobust Validation: Demonstrates consistent performance improvements (e.g., 6% and 11% accuracy gains on VideoLLAMA2 and video-SALMONN) across datasets like AVHBench, with statistical rigor.\nTheoretical Grounding: Provides mathematical support through KL divergence and Taylor expansion, with open-access code promised for reproducibility.\nComprehensive Context: Builds on extensive prior work in vision-language models and hallucination mitigation.\n\nLimited Scope: Validation is restricted to specific models and datasets, potentially limiting generalizability.\nTruncated Content: OCR errors and missing sections obscure critical details, such as derivations and full experimental setups.\nLack of Limitations Discussion: Does not explicitly address AVCD’s limitations or potential failure cases.\nDependence on Attention: Relies on attention mechanisms, which may be unreliable if biased.\nMissed Opportunities: Lacks discussion on non-CD method comparisons, ethical implications, and real-world applications.", "was_revised": true, "final_justification": "Thank you for your comprehensive response. They have addressed my main concerns, and I maintain my recommendation score of 4 points.", "rebuttal": "Thanks to the reviewer’s insightful comments, we were able to add detailed information that enhances reader understanding. Based on the comments, we have:\n\n> **(1)** *re-emphasized the broad validation scope,*  \n> **(2)** *referred to the sections describing derivations and experimental setups,*  \n> **(3)** *supplemented the discussion with failure cases,*  \n> **(4)** *added experiments to verify attention reliability,*  \n> **(5)** *included comparisons with non-CD methods and added explanations on ethical considerations and potential real-world applications,*  \n> **(6)** *provided more detailed descriptions of the datasets used,*  \n> **(7)** *explained the role of the softmax function in **Eq. (1)***,  \n> **(8)** *supplemented the explanation on modality distortion.*\n\nOur detailed responses are provided below.\n\n---\n\n**[W1] Limited validation scope with respect to models and datasets.**\n\nWe would like to clarify that AVCD is evaluated on four multimodal large language models (VideoLLaMA2, video-SALMONN, Video-LLaVA, and LLaVA-1.5) across five datasets (MUSIC-AVQA, AVHBench, MSVD-QA, ActivityNet-QA, and POPE). These evaluations cover a diverse range of tasks and modalities, demonstrating the robustness and generalizability of our approach.\n\n---\n\n**[W2] Insufficient details on derivations and experimental setups.**\n\nWe would like to clarify that our paper does not involve *OCR* in any part of the method or evaluation. Additionally, we provide detailed information about the datasets, evaluation protocols, baselines, and implementation details in **Sec. 4.1** of the main paper. The complete mathematical derivation is also included in **Supp. Sec. A**\n\n---\n\n**[W3] Absence of discussion on limitations and failure cases.**\n\nWe would like to note that we include a discussion of our method’s limitations in **Supp. Sec. H**. To further support readers' understanding, we will include additional failure cases in the final version of the paper.\n\n---\n\n**[W4] Concerns on attention reliability.**\n\nWe understand the reviewer’s concern regarding the reliability of attention. To address this, we conducted additional experiments on AVHBench dataset using VideoLLaMA2 model comparing attention-guided masking with random masking strategies.\n\n| Method | Accuracy (%) |\n| --- | :---: |\n| Base | 78.05 |\n| Random Masking | 71.71 |\n| **Attention-Guided Masking (Ours)** | **81.95** |\n\nWhen random masking is applied, performance drops significantly to 71.71%, compared to the Base decoding at 78.05%. In contrast, our proposed attention-guided masking selectively distorts the modality that truly affects the model output, resulting in improved performance of 81.95%. These results indicate that attention-based masking effectively identifies influential content and contributes to reducing hallucinations.\n\nMoreover, attention mechanisms have been extensively studied in prior works [1,2] for identifying important tokens in LLMs, further supporting their reliability.\n\n> **[Reference]**  \n> **[1]** Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models, ICLR 2025.  \n> **[2]** Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs, ICLR 2024.\n\n---\n\n**[W5] Limited discussion on non-CD methods, ethical considerations, and potential real-world applications.**\n\n**• Non-CD method comparisons**\n\nTo address comparisons with non-CD methods, we include experiments with the OPERA **[3]** method as a representative baseline using video-SALMONN model. \n\n| Decoding  | AVHBench | AVHBench_cap |\n| --- | :---: | :---: |\n| Base | 60.00 | 1.94 ± 0.05 |\n| OPERA | 56.59 | 1.73 ± 0.09 |\n| VCD | 59.51 | 1.65 ± 0.05 |\n| VCD* | 65.85 | 2.18 ± 0.07 |\n| **AVCD (Ours)** | **66.83** | **2.28 ± 0.02** |\n\nOur results show that OPERA underperforms in the AV-LLM setting, with lower scores on both AVHBench and AVHBench_cap. In contrast, our method achieves the best performance, demonstrating its effectiveness in handling complex audio-visual inputs and reducing hallucinations where OPERA struggles.\n\n**• Ethical implications**\n\nWe acknowledge that CD can produce more persuasive and coherent text, which raises ethical concerns. Specifically, it may be misused to generate fake news, misinformation, or manipulated content by making false information appear more credible. Responsible deployment and mitigation strategies should be considered in future work.\n\n**• Real-world applications**\n\nOur proposed method is model-agnostic and can be applied broadly in any environment using AV-LLMs to reduce hallucinations. Considering the trade-off between model performance and inference time, users can adapt the method flexibly according to their application needs, making it practical for real-world deployment.\n\n> **[Reference]**  \n> **[3]** *OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation, CVPR 2024.*\n\n---\n\n**[Q1] What are the total numbers of samples in the three datasets?**\n\nThank you for your question. Our evaluation covers diverse test sets across different model types. For AV-LLMs, we use AVQA (9,185 pairs), AVHBench (5,302 pairs), and AVHBench-Captioning (1,000 pairs). For Video-LLMs, we evaluate on MSVD-QA (1,000 pairs) and ActivityNet-QA (6,760 pairs). Lastly, for image-LLMs, we use the POPE dataset, which contains 9,000 samples across three categories: Random, Popular, and Adversarial (3,000 each). We will include detailed dataset descriptions in the revised paper.\n\n---\n\n**[L1] What is the purpose of applying softmax in Eq. (1)?**\n\nIn **Eq. (1)**, the softmax function is used in the standard way to convert the LLM’s output logits into a probability distribution over the vocabulary. This is an essential step for enabling word prediction, as it allows the model to assign interpretable likelihoods to each possible token.\n\n---\n\n**[L2] What does the term \"damaged and biased visual information\" refer to?**\n\nWe would like to clarify that the phrase “damaged and biased visual information” specifically refers to intentionally corrupted inputs introduced during inference, such as noise addition, data augmentation, or masking, as mentioned in **L144** of the main paper. It is important to note that this does not refer to any issues arising from data collection.\n\n---"}, {"review_id": "5PCmQE4mJ5", "reviewer": "Reviewer_Nvqo", "summary": "This paper introduces Audio-Visual Contrastive Decoding (AVCD), a training-free decoding framework designed to mitigate hallucinations in Audio-Visual Large Language Models (AV-LLMs). The core contributions of AVCD are threefold:\n- Dominance-aware Attentive Masking: Instead of perturbing a fixed modality, AVCD dynamically identifies less dominant modalities using attention distributions and applies attentive masking to generate perturbed logits for contrast.\n- Trimodal Contrastive Formulation: The paper reformulates the conventional contrastive decoding (CD) framework to explicitly handle trimodal inputs (audio, visual, textual).\n- Entropy-Guided Adaptive Decoding (EAD): To improve inference efficiency, AVCD selectively skips CD steps for high-confidence predictions based on the entropy of the model’s output distribution.   \n\nThe authors demonstrate through experiments on various MLLMs (AV-LLMs, video-LLMs, and image-LLMs in supplementary) that AVCD consistently outperforms existing decoding methods and base models in reducing hallucinations and improving accuracy.", "questions": "- The paper introduces the hallucination problem in AV-LLMs but does not clearly define what constitutes hallucination in the audio-visual context. How does it differ from or relate to hallucinations in general multimodal models?\n- Since the proposed method is based on attention mechanisms, the paper lacks an analysis of attention visualizations comparing the decoding process before and after applying AVCD.\n- Why do you justify that the attention distribution of the final query token (denoted as $A_{QK}$) is a valid and reliable measure for identifying modality dominance?\n- The paper omits an important related work OPERA, which analyzed hallucination from attention sink. It is recommended that the authors include a comparison with OPERA in the experiments.\n- Entropy-based decoding strategies are not novel, and have been used in prior works. What advantages does your method offer over existing entropy-guided approaches?\n- In line 223, you mention  $\\neg a$ denotes corrupted audio states, but seemingly do not tell how we do the corrupted operation, which is a key step in AVCD. Could you provide more information about this？\n- The paper states tokens in non-dominant modalities exceeding a top P% attention threshold are “masked out.” Could you clarify if this means their embeddings/features are zeroed out, or replaced with a generic [MASK] embedding?\n- Equation 10 implies up to four forward passes for a single token generation step in the worst-case scenario (original, ¬v, ¬a, ¬v¬a). While Figure 6 shows the trade-off with EAD, could you provide a more direct measure of the average inference time slowdown (e.g., as a percentage or factor increase) for AVCD compared to Base decoding on a challenging benchmark like AVHBench, perhaps at the optimal τ (e.g., τ=0.6)? How much of the computational graph can be shared across these multiple forward passes for different masked inputs, or are they largely independent computations through the LLM decoder?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths:**\n- The paper addresses a critical and timely problem: hallucination in AV-LLMs, which is more complex than in VLMs due to trimodal interactions.\n- The qualitative examples (Fig. 1, Fig. 5) help effectively illustrate AVCD’s ability to correct multimodal hallucinations.\n\n### **Weaknesses:**\n- The exact mechanism of “attentive masking” could be slightly more detailed in the main paper. For instance, are tokens completely zeroed out, or replaced with a special token? The paper states “mask out,” which implies suppression. The choice of P=50% for the threshold is justified by a reference to the supplement, which is acceptable.\n- AVCD involves multiple forward passes (up to 4 for the full trimodal contrast based on Eq. 10: original, ¬v, ¬a, ¬v¬a). While EAD mitigates this, the worst-case overhead is notable. Figure 6 shows inference speed vs. accuracy, which is good, but a more direct comparison of average inference time (e.g., relative slowdown factor) for AVCD vs. Base and VCD on a key benchmark like AVHBench could be beneficial.\n- It is unclear how hallucinations in AV-LLMs differ from those in VLMs. Also, it seems that AVCD could also be applied to general VLMs beyond the AV-LLM setting.\n- MINOR consideration: The idea of using attention scores to guide perturbations is not entirely new (e.g., SID uses attention to find the least informative tokens). However, AVCD’s approach of identifying less dominant modalities and then masking their most informative tokens for trimodal CD is a distinct application.", "was_revised": true, "final_justification": "I think this paper still has room for much improvement, including experiment and writing.", "rebuttal": "We appreciate reviewer's insightful comments. We were able to add crucial details that enhance the clarity of our work. Based on the comments, we have: \n\n> **(1)** *elaborated on the attentive masking mechanism,*  \n> **(2)** *provided details of the inference speed experiments,*  \n> **(3)** *clarified how AV-LLMs suffer from hallucination,*  \n> **(4)** *highlighted differences from the SID model,*  \n> **(5)** *demonstrated the reliability of attention via Attention-Guided Masking,*  \n> **(6)** *explained the motivation for using the final query token,*  \n> **(7)** *compared performance against a non-CD method (OPERA),*  \n> **(8)** *clarified how our entropy-based decoding differs from existing methods.*\n\nOur detailed responses are provided below.\n\n---\n\n**[W1 & Q6 & Q7] Could you clarify how \"masking out\" and \"corrupted states\" are implemented in your method?**\n\nWe apologize for the confusion. In our paper, terms such as “corrupted” or “masked out” consistently refer to the same operation: zeroing out the corresponding embeddings/features to reduce their influence during decoding.\n\nFor the corruption operation, we first compute attention values across all layers to derive a global attention distribution, which determines both the dominance score and the masking threshold. We then identify tokens that exceed this threshold and apply zero-out masking to those associated with less dominant modalities in the attention weights.\n\nWe agree that this clarification is important and will explicitly state this process in the final version of the paper.\n\n---\n\n**[W2 & Q8] Have you conducted a direct speed comparison on AVHBench? Additionally, can the multiple forward passes for masked inputs share parts of the computation graph, or are they executed independently?**\n  \nIn **Figure 6**, we measured inference time on the AVHBench dataset using the VideoLLaMA2 model. Specifically, we averaged the response time over 100 samples to compute the results. As suggested by the reviewer, we will include the inference speed scale in the final version, as shown in the table below.\n    \n|Decoding|threshold (τ)|Speed Scale (s/token)|Accuracy (%)\n| --- | :---: | :---: | :---: |\n| Base | - | 1 (1.75) | 78.05 |\n| VCD | - | 1.4 (2.50) | 62.44 |\n| AVCD | 0.8 | 1.3 (2.25) | 80.98 |\n| AVCD | 0.6 | 1.8 (3.14) | 81.95 |\n\nWhile base contrastive decoding typically requires up to four separate full forward passes in AV-LLMs, our Entropy-Guided Adaptive Decoding (EAD) effectively reduces unnecessary passes by leveraging entropy-based filtering. \n\nAlthough, like other existing CD methods, our implementation does not employ shared computation across passes, AVCD still achieves notable improvements. For example, with τ = 0.8, AVCD not only surpasses the Base method in accuracy but also runs faster than VCD, which generally requires two separate decoding passes.\n\n---\n\n**[W3 & Q1] Could you clarify how hallucination is defined in the audio-visual context, and how it relates to hallucinations in other multimodal models?**\n\nHallucination in AV-LLMs differs fundamentally from that in VLMs due to the presence of three modalities (audio, vision, and language), as opposed to two in VLMs. In AV-LLMs, hallucinations often arise when the model imagines non-existent audio from visual input or visual content from audio input, as noted in AVHBench **[1]**. This is further elaborated by CMM **[2]**, which attributes such hallucinations to over-reliance on unimodal priors and spurious inter-modality correlations.\n\nThis tri-modal structure introduces 7 types of modality interactions (3: unimodal, 3: bimodal, 1: trimodal), making hallucination more complex than in VLMs.\n\nOur proposed AVCD framework is designed specifically to handle this tri-modal challenge by applying modality-aware CD. Despite this challenge, AVCD shows strong generalizability to video-LLMs and image-LLMs, outperforming prior CD methods (see **Table 2**, **Supp. Sec. C**, and **Supp. Sec. F**).\n\nIn particular, **Table 2** shows that AVCD brings significant performance gains over the base model even on video-LLM settings, highlighting its broad applicability beyond the tri-modal setup.\n\n> **[Reference]**  \n> **[1]** *AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models, ICLR 2025.*  \n> **[2]** *The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio, arXiv, 2024.*\n\n---\n\n**[W4] Could you clarify how your method differs from SID?**\n\nWhile we acknowledge that attention-guided perturbations have been explored in prior work (e.g., SID), our contribution lies in extending this idea to the tri-modal setting of AV-LLMs. Specifically, AVCD adaptively identifies less dominant modalities and selectively masks their most informative tokens, introducing a novel application of CD tailored to multimodal interactions.\n\nTo our best knowledge, this is the first systematic exploration of CD in AV-LLMs, and we further demonstrate its generalizability across AV-LLMs, video-LLMs, and image-LLMs, underscoring its broad applicability beyond the original trimodal context.\n\n---\n\n**[Q2] Have you analyzed attention visualizations before and after applying AVCD?**\n\nWhile we cannot provide explicit attention maps, we conducted an indirect yet rigorous evaluation to verify whether the model's attention captures semantically meaningful auditory-visual information.\n\nSpecifically, we compared two masking strategies during decoding: a random zero-out masking strategy, which randomly masks 50% of tokens, and our attentive zero-out masking strategy, which masks the top 50% most attended tokens as identified by the attention scores. \n\n| Method | Accuracy (%) |\n| --- | :---: |\n| Base | 78.05 |\n| Random Masking | 71.71 |\n| **Attention-Guided Masking (Ours)** | **81.95** |\n\nAs shown in Table above, random masking significantly degrades decoding performance, while our proposed attentive masking improves performance by 3.9% over the base decoding.\n\nThese results suggest that the attention mechanism in AVCD is indeed focusing on informative and modality-relevant tokens. \n\n---\n\n**[Q3] Is it appropriate to use the last query token's attention distribution for identifying dominant modality?**\n\nWe use the attention distribution from the final query token because it is directly responsible for next-token prediction in autoregressive decoding. This design choice is supported by prior works **[3,4]**, where attention from the final token is commonly used to interpret token-level importance in LLMs.\n\nTo validate this decision, we compare our strategy against an alternative approach that averages attention across all query tokens. As shown in the table below, using final query token achieves higher accuracy, indicating that the final token provides a more reliable signal for analyzing modality dominance during decoding.\n\n| Method | Accuracy (%) |\n| --- | :---: |\n| average query token | 80.00 |\n| **final query token (Ours)** | **81.95** |\n\n> **[Reference]**  \n> **[3]** *Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models, ICLR 2025.*  \n> **[4]** *Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs, ICLR 2024.*\n\n---\n\n**[Q4] Could you provide a comparison with OPERA to demonstrate the effectiveness of AVCD?**\n\nwe attempted to apply OPERA **[5]** to VideoLLaMA2; however, due to its beam search with multiple attention rollbacks, we encountered out-of-memory (OOM) issues when handling its long input sequences (2K+ tokens). To circumvent this, we evaluated OPERA on a lighter AV-LLM—video-SALMONN, which processes only a few hundred tokens, to assess its feasibility in the AV domain.\n\nAs shown in the table below, both OPERA and VCD led to noticeable performance degradation on AVHBench and AVHBench-Captioning. In contrast, our trimodal extension of VCD (VCD*) improved over Base decoding, though it still lagged behind AVCD.\n\n| Decoding | AVHBench | AVHBench_cap |\n| --- | :---: | :---: |\n| Base | 60.00 | 1.94±0.05 |\n| OPERA | 56.59 | 1.73±0.09 |\n| VCD | 59.51 | 1.65±0.05 |\n| VCD* | 65.85 | 2.18±0.07 |\n| **AVCD (Ours)** | **66.83** | **2.28±0.02** |\n\nOverall, AVCD achieves the most significant performance gain over Base across both benchmarks.\n\n> **[Reference]**  \n> **[5]** *OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation, CVPR 2024.*\n\n---\n\n**[Q5] How does your entropy-based decoding differ from existing approaches?**\n\nWhile **[6]** utilizes entropy to set a hyperparameter (e.g., α) that controls the degree of contrasting in CD, simply adapting this approach requires up to four full forward passes—making it impractical for AV-LLMs.\n\nIn contrast, we treat entropy as a threshold signal to decide whether to apply CD at all, rather than how strongly to contrast. This design choice enables us to maintain reasonable inference speed even in the tri-modal setting. We believe this approach is more suitable for real-world deployment scenarios, where efficiency and scalability are critical.\n\n> **[Reference]**  \n> **[6]** *Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts, EMNLP 2024.*\n\n---"}, {"review_id": "4F33AnHdhn", "reviewer": "Reviewer_qKTD", "summary": "This paper proposes Audio-Visual Contrastive Decoding (AVCD), a novel, training-free decoding method for mitigating hallucinations in audio-visual large language models (AV-LLMs). AVCD extends traditional contrastive decoding (CD) to a trimodal setting (audio, video, and text) by introducing three key components: (1) dominance-aware attentive masking based on attention weights, (2) a new trimodal contrastive formulation, and (3) entropy-guided adaptive decoding to improve efficiency. The method is evaluated on AVHBench using VideoLLaMA2 and Video-SALMONN, where it shows improved performance over standard CD baselines. However, the scope of evaluation and the interpretation of results are limited.", "questions": "Questions and Suggestions for Authors\n- What kinds of hallucinations are being reduced? Is AVCD more effective against cross-modal inconsistencies, unimodal fabrication, or spurious alignments? A taxonomy and targeted evaluation would strengthen your claims.\n\n- Can you evaluate AVCD on harder benchmarks like OmniBench etc.? Current tasks may not fully stress-test hallucination mitigation in AV-LLMs. More complex reasoning tasks would better reveal the advantages and limitations of AVCD.\n\n- Please clarify the derivation and intuition of Equation (11). What assumptions or theoretical principles justify this formulation? The explanation around L318–322 is difficult to follow.\n\n\n- Provide per-component ablations. How much does each of the three components contribute to the overall performance? Are any of them redundant?", "limitations": "Yes, but not in detail.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Novel decoding framework: AVCD extends contrastive decoding to multimodal settings in a principled way, addressing an important gap in inference-time hallucination control.\n\n- Modality-aware masking: The use of attention-based dominance detection for perturbation is an intuitive and data-driven improvement over random or fixed-modality corruption.\n\n- Training-free: AVCD can be integrated with existing models without requiring re-training, making it practical and broadly applicable.\n\n- Efficiency-aware: The entropy-guided decoding strategy is a useful addition for real-world deployment.\n\nWeaknesses\n- - Disconnect from hallucination theory: While AVCD improves output accuracy, the paper does not clearly explain how the proposed method targets specific types of hallucinations (e.g., multimodal vs. modality-confused hallucination). This weakens the link between the design and the central challenge.\n\n- Limited evaluation scope: The AVCD method is only tested on AVHBench, with no experiments on more challenging or general benchmarks such as OmniBench or MM-Vet. Video-LLMs are not evaluated on complex reasoning tasks that would truly test hallucination resilience.\n\n- Unclear motivation and interpretation of some equations: Equation (11) is introduced without sufficient theoretical or intuitive justification. Lines 318–322 provide a vague interpretation that lacks grounding in either mathematical analysis or empirical results.\n\n- No ablations on individual components: The contribution of each AVCD module (dominance-aware masking, trimodal CD, entropy-guided decoding) is not independently assessed, making it difficult to attribute performance gains.", "was_revised": true, "final_justification": "The author provide detailed and professional experimental results in (1)(2)(4). \nAnd I would like to mention the performance increase on OmniBench (31.9%) to (34.5%) is marginal. But the results have a p-value less then 0.05, suggesting the benefits of proposed methods in (3).\nSo I change the score from 3 to 4.", "rebuttal": "We appreciate the reviewer’s insightful comments. The reviewer’s experimental suggestions enabled us to offer a more in-depth analysis of our method. Based on the comments, we have: \n\n> **(1)** *Categorized hallucination cases to demonstrate which types our model effectively mitigates*,  \n> **(2)** *Conducted broader performance validation using the OmniBench dataset*,  \n> **(3)** *Added clarification on the motivation behind **Eq. (11)***,  \n> **(4)** *Provided per-component ablation results.*\n\nOur detailed responses are provided below.\n\n---\n\n**[W1 & Q1] What types of hallucinations does AVCD primarily mitigate, such as cross-modal inconsistencies, unimodal fabrications, or spurious alignments?**\n\nThank you for your insightful suggestion regarding the taxonomy of hallucinations. In response, we provide a deeper analysis of the types of hallucinations mitigated by AVCD, using a subset of the AVHBench dataset. Following your guidance, we categorize hallucinations into three representative types defined in AVHBench:\n\n- **Case 1. Audio-driven video hallucination:** Vision-centric questions misled by irrelevant or misleading audio cues.\n    \n    *Example:* Q: *Is the ship visible in the video?* Video: *No ship visible*, Audio: *Ship sounds present.*\n    \n- **Case 2. Video-driven audio hallucination:** Audio-centric questions misled by visual content.\n    \n    *Example:* Q: *Is the lion making sound?* Video: *Lion visible*, Audio: *No lion sound.*\n    \n- **Case 3. Audio-visual alignment hallucination:** Failures to correctly judge the consistency between audio and visual modalities.\n    \n    *Example:* Q: *Are the contexts of audio and visual content matching?*\n    \n\nWe evaluate AVCD on each category using VideoLLaMA2 as the base model. The results are as follows:\n\n| Category | Base | AVCD  | Improvement |\n|---| :---: | :---: | :---: |\n| Case 1 | 86.4% |  86.4% | 0.0% |\n| Case 2 | 81.3% |  86.3% | +5.0% |\n| Case 3 | 64.4% |  71.2% | +6.8% |\n\nWhile AVCD maintains the already high performance in Case 1, it shows significant improvements in Cases 2 and 3. These results support our claim that AVCD balances modality contributions more effectively.\n\nWe will include this analysis and the above categorization in the revised version to better characterize the nature of hallucinations and the targeted efficacy of AVCD.\n\n---\n\n**[W2 & Q2] Could you evaluate AVCD on more challenging benchmarks like OmniBench to better assess its effectiveness in complex hallucination scenarios?**\n    \nIn response to the reviewer’s suggestion, we additionally evaluate various decoding methods (Base, VCD, OPERA, VCD*, and AVCD) on the OmniBench dataset, which is known to require more complex reasoning and better tests hallucination mitigation in AV-LLMs.\n\nFollowing the OmniBench **[1]**, we use video-SALMONN as the backbone model, where they report an overall accuracy of 35.6%. We reproduce this setting and obtain 33.5% accuracy with Base decoding, which we use as our baseline.\n\nAs shown in the table below, AVCD is the only method that consistently outperforms the Base model, whereas other methods often underperform or show comparable results. This highlights AVCD’s strong generalization ability, even on more challenging benchmarks.\n\n| Decoding | Overall | Action | Story | Plot | Object | Context | Identity | Text | Count |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Base | 33.5 | 27.1 | 27.0 | 24.1 | **61.1** | 30.1 | 46.9 | 21.4 | 7.1|\n| VCD | 23.3 | 18.3 | 17.4 | 13.1 | 44.6 | 26.2 | 37.5 | 7.1 | **14.3** |\n| OPERA | 30.8 | 27.9 | 23.0 | 20.7 | 52.1 | **36.9** | 34.4 | 14.3 | 7.1 |\n| VCD*  | 31.9 | 27.1 | 26.1 | 21.5 | 57.4 | 30.5 | 43.8 | 21.4 | 0.0 |\n| **AVCD (Ours)** | **34.5** | **28.3** | **28.7** | **24.5** | 60.7 | 30.5 | **50.0** | **21.4** | 7.1 |\n\n> **[Abbreviation in the table]**  \n> **Action**: Action & Activity,\n**Story**: Story Description,\n**Plot**: Plot Inference,\n**Object**: Object Identification & Description,\n**Context**: Contextual & Environmental,\n**Identity**: Identity & Relationship,\n**Text**: Text & Symbols,\n**Count**: Count & Quantity\n\n> **[Reference]**  \n> **[1]** *OmniBench: Towards The Future of Universal Omni-Language Models, arXiv 2025.*\n\n---\n\n**[W3 & Q3] What is the motivation behind Equation (11), and what role does it play in the proposed method?**\n\n**Eq. (11)** shows an intermediate step to verify whether hallucinations caused by masking audio, video, and both modalities could be treated as independent. The equation combines the three masked variants in a single formulation, assuming that their effects on hallucination could be simply additive.\n\nHowever, as shown in **Table 3**, the performance of **Eq. (11)** was lower than applying **Eq. (2)**. We interpret this result as evidence that hallucinations caused by different modalities are not independent, and that masking multiple modalities simultaneously can lead to redundant or overlapping corrections.\n\nThis finding became a key motivation for the design of AVCD, which explicitly considers interactions between modalities rather than naively aggregating all masked logits—thereby avoiding interference, as explained in **L204** of the main paper.\n\nAs a result, AVCD can treat hallucinations from different modalities as interacting factors, and improves decoding performance through more sophisticated formulation.\n\n---\n\n**[W4 & Q4] Could you provide per-component ablations to show how much each of the three components contributes to performance?**\n\nFollowing reviewer’s suggestion, we conducted an ablation study on AVHBench using VideoLLaMA2 as the base model. The results are summarized below:\n\n| Method | Accuracy (%) | Inference Speed (sec/token) |\n|----|:---:|:---:|\n| Baseline | 74.15 | 4.4 |\n| + Dominance-aware masking | 79.02 | 4.4 |\n| &nbsp;&nbsp;+ Eq. (10) | 81.95 | 4.4 |\n| &nbsp;&nbsp;&nbsp;&nbsp;**+ EAD (Ours)** | **81.95** | **3.1** |\n\nBaseline refers to a contrastive decoding (CD) setup with randomly selected dominant modality and **Eq. (11)**.\n\nDominance-aware masking improves accuracy by approximately +5%.\n\nTrimodal CD with **Eq. (10)**, our proposed extension of CD to trimodal alignment, further improves performance by +2.9%.\n\nEntropy-guided adaptive decoding (EAD) maintains accuracy while reducing inference latency by over 30%.\n\nThese results validate the complementary roles of all three AVCD components as each contributes meaningfully to either accuracy or efficiency.\n\n---"}]

Author rebuttal:

Meta-review: This paper proposes AVCD, a novel, training-free framework to mitigate hallucinations in audio-visual large language models by reformulating contrastive decoding for trimodal inputs. The reviewers agreed that the paper addresses an important and timely challenge in AV-LLMs, and highlighted several key strengths: the method is training-free and therefore broadly applicable, it introduces a principled extension of contrastive decoding to three modalities, and it balances accuracy and efficiency through entropy-guided adaptive decoding. Experimental validation across multiple datasets and models shows consistent gains, with particularly strong improvements on AVHBench, reinforcing both the novelty and practical relevance of the approach. The ablation studies and additional experiments added during rebuttal further strengthened the technical contribution.

Some concerns remain. One reviewer raised questions about the theoretical justification of the mathematical approximation used in the framework, noting that the explanation relies on empirical rather than rigorous analysis. Others mentioned the reliance on attention scores, efficiency trade-offs, and the need for deeper analysis of failure cases and hallucination categories. While these are valid points, the authors provided substantial clarifications, additional experiments, and broader evaluations that satisfied most reviewers. Overall, the paper makes a strong methodological contribution to hallucination mitigation in multimodal LLMs.
Decision: Accept (poster)