Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models

## Abstract

Abstract The goal of this work is to enhance balanced multimodal understanding in audio-visual large language models (AV-LLMs) by addressing modality bias without additional training.
In current AV-LLMs, audio and video features are typically processed jointly in the decoder.
While this strategy facilitates unified multimodal understanding, it may introduce modality bias, where the model tends to over-rely on one modality due to imbalanced training signals.
To mitigate this, we propose Fork-Merge Decoding (FMD), a simple yet effective inference-time strategy that requires no additional training or architectural modifications.
FMD first performs modality-specific reasoning by processing audio-only and video-only inputs through the early decoder layers ( fork ), and then merges the resulting hidden states for joint reasoning in the remaining layers ( merge ).
This separation allows each modality to be emphasized in the early stages while encouraging balanced contributions during integration.
We validate our method on three representative AV-LLMs—VideoLLaMA2, video-SALMONN, and Qwen2.5-Omni—using three benchmark datasets.
Experimental results show consistent gains in audio, video, and audio-visual reasoning tasks, highlighting the effectiveness of inference-time interventions for robust and efficient multimodal understanding.

## 1 Introduction

Recent advancements in large language models (LLMs) have demonstrated superior performance in various text-centric tasks such as problem solving, translation, and summarization (Achiam et al., 2023; Brown et al., 2020; Liu et al., 2023b; Thoppilan et al., 2022; Wei et al., 2022a; b; Zhao et al., 2023a).
Building on these successes in text processing, LLMs have evolved to handle additional modalities, including images (Alayrac et al., 2022; Chen et al., 2023a; Dai et al., 2023; Huang et al., 2023; Li et al., 2023a; Liu et al., 2023a; Yu et al., 2024; Zhang et al., 2024; Zhu et al., 2023b), videos (Lin et al., 2024; Maaz et al., 2024), and audio (Huang et al., 2024; Rubenstein et al., 2023; Tang et al., 2024), giving rise to multimodal LLMs (MLLMs). These models process diverse modalities through separate encoders and integrate their outputs within a decoder language model, achieving remarkable performance across a wide range of tasks.
Among these, audio-visual LLMs (AV-LLMs) are particularly notable for their ability to jointly integrate visual and auditory information, supporting more sophisticated reasoning and achieving closer alignment with human multimodal perception (Cheng et al., 2024; Sun et al., 2024; Zhang et al., 2023; Xu et al., 2025).

To effectively leverage pretrained LLM decoders in AV-LLMs, various fusion strategies have been proposed to integrate audio and visual information. One common approach (Chen et al., 2023b; Cheng et al., 2024; Chowdhury et al., 2024; Han et al., 2024; Lyu et al., 2023; Panagopoulou et al., 2023; Ye et al., 2024; Zhan et al., 2024; Zhang et al., 2023; Zhao et al., 2023b; Xu et al., 2025) is token-wise fusion, where audio and visual features are extracted by separate encoders and then concatenated along the sequence dimension before being fed into the decoder as a continuous input sequence. Several studies (Chowdhury et al., 2024; Ye et al., 2024) additionally introduce adapter modules that facilitate interaction between audio and visual features before they are passed into the LLM.
Another approach (Han et al., 2023; Su et al., 2023; Sun et al., 2024) is channel-wise fusion, in which modality-specific features are concatenated along the channel dimension to form a unified representation.
In most current AV-LLM architectures, the decoder receives both audio and visual inputs simultaneously, which raises a potential concern: if the model finds one modality easier to interpret—perhaps due to better alignment with its pretraining objectives—it may over-rely on that modality, leading to modality bias and modality-specific hallucinations (Leng et al., 2024a).

Figure: Figure 1: Attention weight analysis in VideoLLaMA2 on the AVHBench dataset. We analyze 100 samples and examine the attention weights from the last decoder layer, focusing on the final token of the question. Attention is disproportionately allocated to video inputs over audio, revealing a modality bias. Our proposed FMD method reduces this gap by encouraging more balanced contributions from both modalities.
Refer to caption: x1.png

To investigate this possibility, we begin by analyzing the attention weight distributions over audio-visual inputs using 100 samples from the AVHBench dataset (Sung-Bin et al., 2025).
Our analysis examines the final decoder layer of VideoLLaMA2 (Cheng et al., 2024), focusing on the attention weights of the last token. Since the token is critical for predicting the next token, we use it to quantify the relative attention allocated to each modality.
As shown in Figure [1](https://arxiv.org/html/2505.20873v2#S1.F1), the vanilla decoding setup, which reflects the default inference behavior of the model, exhibits a clear bias toward video inputs, with attention disproportionately concentrated on visual features over audio. This observation aligns with findings from recent studies (Guan et al., 2024; Leng et al., 2024a; Nishimura et al., 2024; Wang et al., 2024), which report that MLLM decoders often exhibit modality bias.
These studies highlight that such imbalances can lead to hallucinations or flawed reasoning, emphasizing the need to mitigate modality bias for more balanced multimodal understanding.

To address this issue, we propose Fork-Merge Decoding (FMD), which is a simple yet effective strategy that enhances multimodal understanding without altering the AV-LLM architecture or requiring additional training. The core idea of FMD is to divide the decoding phase into two stages: a *fork* phase and a *merge* phase, designed to improve both unimodal and multimodal understanding.
In the *fork* phase, the original multimodal input is split into two unimodal branches by zeroing out either the visual or auditory modality while retaining the text question. Each branch is processed independently through the initial layers of the pretrained AV-LLM, producing modality-specific hidden representations without requiring additional full forward passes. In the *merge* phase, these representations are combined and passed through the remaining decoder layers.
This separation enables the model to first attend to unimodal cues in isolation before integrating them for complementary multimodal understanding.
As shown in Figure [1](https://arxiv.org/html/2505.20873v2#S1.F1), FMD reduces the attention weight on video inputs by 14% and increase the weight on audio inputs by 7%. This adjustment balances the modality bias while still preserving the attributes of the pretrained model.
Building on this, since recent AV-LLMs commonly adopt either token-wise or channel-wise concatenation for audio-visual fusion, we propose a generalized decoding strategy that is compatible with both fusion methods. This unified approach improves performance across a variety of models, regardless of their fusion mechanism.

Furthermore, we evaluate the effectiveness of FMD by applying it to three recent AV-LLMs:
VideoLLaMA2 (Cheng et al., 2024) and Qwen2.5-Omni (Xu et al., 2025) (token-wise fusion) and video-SALMONN (Sun et al., 2024) (channel-wise fusion).
Applying FMD leads to consistent performance improvements in all baselines across three widely used audio-visual benchmarks: AVQA (Yang et al., 2022), MUSIC-AVQA (Li et al., 2022), and AVHBench (Sung-Bin et al., 2025). Notably, FMD enhances performance not only in tasks that emphasize a single modality but also in tasks that require balanced reasoning across both modalities.
These results show that FMD enhances multimodal understanding by fully utilizing information from each modality during inference.

## 2 Preliminaries

Input processing.
Most existing AV-LLMs (Chen et al., 2023b; Cheng et al., 2024; Chowdhury et al., 2024; Lyu et al., 2023; Panagopoulou et al., 2023; Sun et al., 2024; Ye et al., 2024; Zhan et al., 2024; Zhang et al., 2023; Zhao et al., 2023b) process audio and video separately before feeding them into an LLM decoder.
Visual inputs are encoded frame by frame into spatial embeddings, while audio signals are mapped to semantic representations capturing acoustic and prosodic cues.
Textual instructions or questions are tokenized and embedded by a tokenizer of language model.

For token-wise fusion models, video frames are transformed into $M$ embeddings ${\bm{x}}^{v}=\{{\bm{x}}_{1},\ldots,{\bm{x}}_{M}\}$ and audio into $N$ embeddings ${\bm{x}}^{a}=\{{\bm{x}}_{M+1},\ldots,{\bm{x}}_{M+N}\}$.
These are concatenated with $L$ text embeddings ${\bm{x}}^{l}=\{{\bm{x}}_{M+N+1},\ldots,{\bm{x}}_{M+N+L}\}$ to form the final sequence ${\bm{x}}={\bm{x}}^{v}\oplus{\bm{x}}^{a}\oplus{\bm{x}}^{l}$ of length $M+N+L$, omitting instruction tokens for simplicity.
For channel-wise fusion models, visual and audio features are projected to a fixed length $U$ embeddings and then concatenated along the channel dimension to form joint audio-visual embeddings, ${\bm{x}}_{i}^{av}=[{\bm{x}}_{i}^{v};{\bm{x}}_{i}^{a}]$.
The resulting set ${\bm{x}}_{i}^{av}=\{{\bm{x}}_{1}^{av},\ldots,{\bm{x}}_{U}^{av}\}$ is combined with $L$ text embeddings via token-wise concatenation,
${\bm{x}}=\{{\bm{x}}_{1}^{av},\ldots,{\bm{x}}_{U}^{av},{\bm{x}}_{U+1}^{l},\ldots,{\bm{x}}_{U+L}^{l}\}$, which is then fed into the decoder.

Decoding.
Both token-wise and channel-wise fusion-based AV-LLMs generate outputs from the input sequence ${\bm{x}}$ using an autoregressive decoding strategy, where each token is predicted by attending to previously generated tokens under a causal mask. At each decoding step $t$, the model generates the next token ${\bm{y}}_{t}$ conditioned on the input sequence ${\bm{x}}$, which includes video, audio, and text prompts, as well as the previously generated tokens ${\bm{y}}_{<t}$, with ${\bm{y}}_{t}\sim p({\bm{y}}_{t}|{\bm{x}},{\bm{y}}_{<t})\ \propto\exp\left(\text{logit}({\bm{y}}_{t}|{\bm{x}},{\bm{y}}_{<t})\right).$

Figure: Figure 2: Overview of the Fork-Merge Decoding pipeline. The AV-LLM takes video frames, an audio waveform, and a question prompt as input. In the fork phase, FMD masks one modality while preserving the question, enabling independent reasoning. After $L_{\text{fork}}$ decoder layers, the merge phase combines ${\bm{h}}_{\text{fork}}^{\neg v}$ and ${\bm{h}}_{\text{fork}}^{\neg a}$ with an attention-derived weight $\alpha$, and the merged representation is processed by the remaining layers to generate answers with balanced multimodal understanding.
Refer to caption: x2.png

## 3 Fork-Merge Decoding for Audio-Visual Understanding

This section introduces Fork-Merge Decoding (FMD) for token-wise (Section [3.1](https://arxiv.org/html/2505.20873v2#S3.SS1)) and channel-wise (Section [3.2](https://arxiv.org/html/2505.20873v2#S3.SS2)) fusion in AV-LLMs. An overview of the process is illustrated in Figure [2](https://arxiv.org/html/2505.20873v2#S2.F2).

### 3.1 Decoding with token-wise fusion in AV-LLMs

Input masking strategy.
In token-wise fusion models like VideoLLaMA2 (Cheng et al., 2024), the input sequence is composed of ${\bm{x}}={\bm{x}}^{v}\oplus{\bm{x}}^{a}\oplus{\bm{x}}^{l}$,
where ${\bm{x}}^{v}$, ${\bm{x}}^{a}$, and ${\bm{x}}^{l}$ denote visual, audio, and language embeddings, respectively.
To enable modality-specific processing for audio and video while preserving the textual question, we create two masked input variants as follows:

$$ ${\bm{x}}^{v\text{[masked]}}={{\bm{x}}}^{\neg v}\oplus{\bm{x}}^{a}\oplus{\bm{x}}^{l},\quad{\bm{x}}^{a\text{[masked]}}={\bm{x}}^{v}\oplus{{\bm{x}}}^{\neg a}\oplus{\bm{x}}^{l},$ (1) $$

where ${{\bm{x}}}^{\neg v}$ and ${{\bm{x}}}^{\neg a}$ denote the modality-masked embeddings for vision and audio, respectively. These are obtained by zeroing out the corresponding video frames or audio waveforms at the input level. This preserves original embedding shapes and positions while removing content information.

Fork processing.
To encourage independent understanding over each modality, each masked sequence is separately processed through the first $L_{\text{fork}}$ transformer layers of the decoder $\phi$:

$$ ${\bm{h}}^{\neg v}_{\text{fork}}=\phi_{\leq L_{\text{fork}}}({\bm{x}}^{v\text{[masked]}}),\quad{\bm{h}}^{\neg a}_{\text{fork}}=\phi_{\leq L_{\text{fork}}}({\bm{x}}^{a\text{[masked]}}),$ (2) $$

where ${\bm{h}}^{\neg v}_{\text{fork}}$ and ${\bm{h}}^{\neg a}_{\text{fork}}$ denote the intermediate hidden states obtained from the vision-masked and audio-masked inputs, respectively.
This design ensures that the model does not observe both audio and visual inputs simultaneously in the early stages, allowing it to focus on how each modality individually relates to the textual prompt. By reasoning over each modality in isolation, the model is less likely to become biased toward the more dominant or easier-to-interpret modality.

Merge processing.
After the $L_{\text{fork}}$ layers, the hidden states ${\bm{h}}^{\neg v}_{\text{fork}}$ and ${\bm{h}}^{\neg a}_{\text{fork}}$ are fused and passed through the remaining transformer layers. Each hidden state has a sequence length of $M+N+L$, corresponding to the visual, audio, and text tokens. The fusion is performed by summing the corresponding embeddings at each modality position, and is formally defined as follows:

$$ $\displaystyle{\bm{h}}_{\text{merge}}[\mathcal{V}]$ $\displaystyle=(1-\alpha)\cdot{\bm{h}}^{\neg v}_{\text{fork}}[\mathcal{V}]+\alpha\cdot{\bm{h}}^{\neg a}_{\text{fork}}[\mathcal{V}],$ (3) $\displaystyle{\bm{h}}_{\text{merge}}[\mathcal{A}]$ $\displaystyle=\alpha\cdot{\bm{h}}^{\neg v}_{\text{fork}}[\mathcal{A}]+(1-\alpha)\cdot{\bm{h}}^{\neg a}_{\text{fork}}[\mathcal{A}],$ $\displaystyle{\bm{h}}_{\text{merge}}[\mathcal{L}]$ $\displaystyle=\tfrac{1}{2}\left({\bm{h}}^{\neg v}_{\text{fork}}[\mathcal{L}]+{\bm{h}}^{\neg a}_{\text{fork}}[\mathcal{L}]\right),$ $$

where $\mathcal{V}=[1{:}M]$, $\mathcal{A}=[M{+}1{:}M{+}N]$, and $\mathcal{L}=[M{+}N{+}1{:}M{+}N{+}L]$ denote the index ranges corresponding to the visual, audio, and language embeddings, respectively.
The final merged representation ${\bm{h}}_{\text{merge}}$ is then constructed by concatenating the modality-specific segments along the token dimension as
${\bm{h}}_{\text{merge}}={\bm{h}}_{\text{merge}}[\mathcal{V}]\oplus{\bm{h}}_{\text{merge}}[\mathcal{A}]\oplus{\bm{h}}_{\text{merge}}[\mathcal{L}]$.
Here, $\alpha$ is a fusion weight for unmasked segments, reflecting their relative contribution when combining masked and unmasked segments. We refer to this approach as attention-guided fusion and describe in detail in the following section.

Attention-guided fusion.
To determine the fusion weight $\alpha$, we use the attention matrix $\mathbf{A}^{\text{final}}\in\mathbb{R}^{T\times T}$ from the final transformer layer. Specifically, we focus on the attention vector of the last token, $\mathbf{a}_{\text{last}}=\mathbf{A}^{\text{final}}_{T,:}\in\mathbb{R}^{T}$, which is critical in next-token prediction and follows the approach in prior studies (Huo et al., 2025; Song et al., 2024).
By leveraging the two masked branches ${\bm{h}}^{\neg v}_{\text{fork}}$ and ${\bm{h}}^{\neg a}_{\text{fork}}$, we compute the attention-based contributions of unmasked segments by summing the attention mass over the corresponding regions relative to the total mass, which are then used as $\alpha$:

$$ $\alpha=\frac{\sum_{i\in\mathcal{A}}\mathbf{a}_{\text{last}}^{v[\text{masked}]}[i]+\sum_{i\in\mathcal{V}}\mathbf{a}_{\text{last}}^{a[\text{masked}]}[i]}{\sum_{i\in\mathcal{V}\cup\mathcal{A}}(\mathbf{a}_{\text{last}}^{v[\text{masked}]}[i]+\mathbf{a}_{\text{last}}^{a[\text{masked}]}[i])},$ (4) $$

where $\mathbf{a}_{\text{last}}^{v\text{[masked]}}$ and $\mathbf{a}_{\text{last}}^{a\text{[masked]}}$ represent the attention weight distributions of the final token from ${\bm{h}}^{\neg v}_{\text{fork}}$ and ${\bm{h}}^{\neg a}_{\text{fork}}$, respectively.
The final attention weight $\alpha$ is then obtained as the fraction of unmasked attention over the total attention in Eq. [4](https://arxiv.org/html/2505.20873v2#S3.E4) and is used to interpolate between ${\bm{h}}^{\neg v}_{\text{fork}}$ and ${\bm{h}}^{\neg a}_{\text{fork}}$ in Eq. [3](https://arxiv.org/html/2505.20873v2#S3.E3).
In practice, instead of computing $\alpha$ for each data point, we estimate a representative value by sampling 100 random examples from the AVHBench (Sung-Bin et al., 2025) dataset. This approach is motivated by two considerations: (1) computing a separate $\alpha$ for each sample increases inference time, as it requires two additional full forward passes to obtain $\mathbf{a}_{\text{last}}^{v\text{[masked]}}$ and $\mathbf{a}_{\text{last}}^{a\text{[masked]}}$, (2) noisy sample-specific $\alpha$ values can act as outliers, causing performance drops, as shown by the results labeled Attention (Adaptive) in Table [3](https://arxiv.org/html/2505.20873v2#S4.T3). The resulting representative $\alpha$ is then applied consistently across all experiments, including datasets beyond AVHBench, to verify its generalizability.

This attention-guided fusion ensures that structurally aligned hidden states are preserved, while allowing the more informative modality to be emphasized. It enables a flexible and interpretable merging scheme without disrupting the architectural integrity of pretrained AV-LLMs.

Decoding.
The merged hidden state is then forwarded through the remaining transformer layers: ${\bm{h}}_{\text{final}}=\phi_{>L_{\text{fork}}}({\bm{h}}_{\text{merge}}),$
producing the final prediction logits.
By delaying modality fusion to deeper layers, where individual representations become semantically richer through the *fork* phase, our method enhances multimodal understanding while mitigating issues caused by modality imbalance.

### 3.2 Decoding with channel-wise fusion in AV-LLMs

Input masking strategy.
In channel-wise fusion models such as video-SALMONN (Sun et al., 2024), the input sequence is structured as ${\bm{x}}=\{{\bm{x}}_{1}^{av},\ldots,{\bm{x}}_{U}^{av},{\bm{x}}_{U+1}^{l},\ldots,{\bm{x}}_{U+L}^{l}\},$ where ${\bm{x}}^{av}$ denotes audio-visual embeddings and ${\bm{x}}^{l}$ corresponds to language (prompt) embeddings.
Here, $U$ and $L$ indicate the number of audio-visual and language sequence elements, respectively.
To allow for modality-specific processing, we construct two masked variants of the input:

$$ ${\bm{x}}^{v\text{[masked]}}={\bm{x}}^{a\neg v}\oplus{\bm{x}}^{l},\quad{\bm{x}}^{a\text{[masked]}}={\bm{x}}^{\neg av}\oplus{\bm{x}}^{l},$ (5) $$

where ${\bm{x}}^{a\neg v}$ and ${\bm{x}}^{\neg av}$ denote the video-masked and audio-masked embeddings, respectively, obtained by zeroing out the corresponding inputs before they are passed into the decoder layers.

Fork-merge decoding.
${\bm{h}}^{\neg v}_{\text{fork}}$ and ${\bm{h}}^{\neg a}_{\text{fork}}$ are obtained by passing the masked inputs ${\bm{x}}^{v\text{[masked]}}$ and ${\bm{x}}^{a\text{[masked]}}$ through the $L_{fork}$ layers (refer to Eq. [2](https://arxiv.org/html/2505.20873v2#S3.E2)).
To compute the merged hidden state ${\bm{h}}_{\text{merge}}$, we perform element-wise addition over the audio-visual embedding representations from both branches, based on the assumption that they capture complementary information. Since each branch processes modality-masked inputs (i.e., audio-masked for visual features and vice versa), their combination is expected to yield a more complete representation. Additionally, mean pooling is applied over the question prompt embedding positions to maintain consistency:

$$ ${\bm{h}}_{\text{merge}}[i]=\begin{cases}{\bm{h}}^{\neg v}_{\text{fork}}[i]+{\bm{h}}^{\neg a}_{\text{fork}}[i],&\text{if }i\leq U,\\ \frac{1}{2}\left({\bm{h}}^{\neg v}_{\text{fork}}[i]+{\bm{h}}^{\neg a}_{\text{fork}}[i]\right),&\text{if }i>U.\end{cases}$ (6) $$

We do not apply attention-guided fusion in the channel-wise setting, as the hidden states do not disentangle audio and visual embeddings.
Finally, decoding is then continued by forwarding the merged hidden state ${\bm{h}}_{\text{merge}}$ through the remaining decoder layers to produce the final output logits.
Notably, FMD provides a unified framework that seamlessly integrates input masking with modality-specific processing in separate decoder branches and merged decoding, making it broadly applicable to a wide range of AV-LLM architectures, regardless of their fusion strategies.

## 4 Experiments

### 4.1 Experimental setup

Baselines.
We evaluate our approach using three representative AV-LLMs: VideoLLaMA2 (Cheng et al., 2024), video-SALMONN (Sun et al., 2024), and Qwen2.5-Omni (Xu et al., 2025).

Datasets and evaluation protocol.
The AVQA (Yang et al., 2022) dataset contains 57,000 YouTube videos for evaluating real-world audio-visual understanding. MUSIC-AVQA (Li et al., 2022) offers 45,867 QA pairs from 9,288 music performance videos, focusing on fine-grained audio-visual reasoning such as identifying sound sources and temporally aligning auditory and visual cues. AVHBench (Sung-Bin et al., 2025) is the first benchmark specifically designed to assess audio-visual hallucinations in AV-LLMs. It comprises four subtasks: audio-driven video hallucination (A→V), video-driven audio hallucination (V→A), audio-visual matching (AV matching), and audio-visual captioning (AV captioning).
For the transformer layer analysis in Section [4.4](https://arxiv.org/html/2505.20873v2#S4.SS4), we select 200 samples from each task (A→V, V→A, and AV matching), with the remaining data used for evaluation.
For the AVHBench dataset, the three binary (yes/no) tasks except AV captioning, we report classification accuracy. For AVQA, MUSIC-AVQA and AV captioning, which involve open-ended responses, we follow the GPT-assisted evaluation protocol from the official VideoLLaMA2 implementation(^1^11[https://github.com/DAMO-NLP-SG/VideoLLaMA2/tree/audio_visual](https://github.com/DAMO-NLP-SG/VideoLLaMA2/tree/audio_visual)).

Implementation details.
For attention-guided fusion, we set the weighting parameter $\alpha$ as described in Section [3.1](https://arxiv.org/html/2505.20873v2#S3.SS1), using 100 randomly sampled examples from the AVHBench dataset: $\alpha=0.8$ for VideoLLaMA2 and $\alpha=0.9$ for Qwen2.5-Omni. The number of layers used for the *fork* phase $L_{fork}$ is chosen to be roughly one-seventh of the total decoding layers: the 4th layer for VideoLLaMA2 and Qwen2.5-Omni (28 layers) and the 6th layer for video-SALMONN (40 layers). The rationale for these *fork* layer selections is further analyzed in Section [4.4](https://arxiv.org/html/2505.20873v2#S4.SS4).

**Table 1: Comparison of audio-visual understanding performance. We evaluate FMD on VideoLLaMA2, video-SALMONN, and Qwen2.5-Omni. *Vanilla* denotes the original decoding strategy of each model. Experiments are conducted on AVHBench, AVQA, and MUSIC-AVQA datasets. FMD consistently improves performance across all benchmarks, especially in AV matching and captioning. For Qwen2.5-Omni, we leave the MUSIC-AVQA entry as N/A due to out-of-memory issues.**
| Model | Decoding | AVHBench | AVQA | MUSIC-AVQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| A$\rightarrow$V | V$\rightarrow$A | AV Matching | AV Captioning |  |  |  |  |
| VideoLLaMA2 | Vanilla | 80.02 | 77.03 | 57.75 | 2.84 | 60.23 | 81.30 |
| FMD | 80.45 | 77.52 | 59.01 | 2.95 | 61.46 | 81.50 |  |
| video-SALMONN | Vanilla | 68.69 | 62.39 | 49.46 | 1.83 | 28.20 | 44.48 |
| FMD | 70.51 | 65.41 | 54.77 | 2.01 | 40.46 | 50.60 |  |
| Qwen2.5-Omni | Vanilla | 80.77 | 71.20 | 77.45 | 3.25 | 86.49 | N/A |
| FMD | 81.30 | 71.77 | 78.22 | 3.36 | 86.61 | N/A |  |

### 4.2 Quantitative analysis

Comparison with vanilla decoding.
To verify the effectiveness of our proposed FMD method, we evaluate it with VideoLLaMA2, video-SALMONN, and Qwen2.5-Omni on three datasets: AVQA, MUSIC-AVQA, and AVHBench.
As shown in Table [1](https://arxiv.org/html/2505.20873v2#S4.T1), applying FMD consistently improves performance over vanilla decoding, which represents the original model inference, across all tasks. Notably, the gains are more pronounced in the AV captioning task (evaluated on a 5-point scale), which requires generating long and sophisticated answers, with relative improvements ranging from 3.4% to 9.8% across models. Among the models, video-SALMONN benefits the most, achieving a remarkable 12.26% increase on the AVQA dataset. This is particularly significant because video-SALMONN has not been trained on AVQA or MUSIC-AVQA, yet FMD still enhances its zero-shot inference. These results highlight the robustness and strong generalizability of our approach.

**Table 2: Comparison of decoding methods on AVHBench dataset using VideoLLaMA2. We compare vanilla decoding with DoLa, VCD, SID and two FMD variants (Gaussian noise injection and zero-out masking). Among them, FMD with zero-out masking achieves the highest overall accuracy, underscoring its effectiveness.**
| Decoding | Designed<br>for | AVHBench |  |  |
| --- | --- | --- | --- | --- |
| A$\rightarrow$V | V$\rightarrow$A | AV Matching |  |  |
| \rowcolorlightgray Vanilla | - | 80.02 | 77.03 | 57.75 |
| DoLa (Chuang et al., 2024) | LLM | 69.34 | 63.44 | 48.33 |
| VCD (Leng et al., 2024b) | VLM | 75.96 | 69.67 | 52.52 |
| SID (Huo et al., 2025) | VLM | 78.53 | 72.82 | 53.52 |
| FMD w/ noise | AV-LLM | 79.17 | 78.03 | 57.76 |
| FMD w/ zero-out | AV-LLM | 80.45 | 77.52 | 59.01 |

Comparison with other decoding methods.
We compare our FMD method with other test-time decoding strategies that operate at the logit level. The evaluation is conducted using the VideoLLaMA2 model on the AVHBench dataset. Specifically, the comparison includes the following methods: DoLa (Chuang et al., 2024), VCD (Leng et al., 2024b), SID (Huo et al., 2025), and FMD variants with Gaussian noise injection and zero-out masking.
DoLa contrasts intermediate-layer outputs with final predictions to factually correct outputs.
VCD reduces language bias by injecting Gaussian noise into the visual input and subtracting the resulting logits from the original ones.
SID adopts a similar approach with VCD but preserves the least informative visual tokens based on attention weights, contrasting their logits with those of the original outputs.
For DoLa, we follow the original setup and extract outputs from the same intermediate layer reported in the paper. For VCD and SID, which were originally developed for vision-language models (VLMs), we adapt their procedures to handle both audio and video modalities in AV-LLMs.

As shown in Table [2](https://arxiv.org/html/2505.20873v2#S4.T2), applying decoding strategies developed for LLMs or VLMs to AV-LLMs leads to degraded performance.
This highlights the need for decoding methods that are specifically tailored to the unique characteristics of AV-LLMs, which differ from those of unimodal or bimodal models. Additionally, injecting Gaussian noise into the inputs within FMD (denoted as FMD with noise) results in lower performance on A→V and AV matching tasks compared to vanilla decoding, although it does lead to improved performance on the V→A task.
We attribute this to the inability of Gaussian noise to equally isolate modality-specific information, as further discussed in Supp. [A.3](https://arxiv.org/html/2505.20873v2#A1.SS3).
In contrast, zero-out masking within FMD yields consistent performance improvements across all tasks, demonstrating that the proposed FMD design is suitable for audio-visual understanding.

Figure: (a) An example of audio-visual matching from AVHBench.
Refer to caption: x3.png

### 4.3 Qualitative analysis

Audio-visual matching from AVHBench (Figure [3a](https://arxiv.org/html/2505.20873v2#S4.F3.sf1)).
The video shows a man fishing in a stream, and the audio contains the sound of water flowing in a valley with occasional splashes from footsteps. The vanilla decoding mainly captures the visual content, describing only the presence of a man fishing in a stream. However, it overlooks the acoustic context, failing to reflect the sound of flowing water and splashes. By contrast, our proposed FMD generates a more comprehensive description that integrates both modalities, enriching the visual detail (e.g., specifying the fishing rod) and the audio detail (e.g., capturing the sounds of water flow and splashes). This highlights the strength of FMD in balancing and fusing multimodal cues, leading to richer and more faithful outputs.

Audio-visual question & answering from AVQA (Figure [3b](https://arxiv.org/html/2505.20873v2#S4.F3.sf2)).
The video shows a train, but in the early frames its appearance could be confused with a cable car. The accompanying audio contains the sound of a train gradually accelerating as it departs from the station.
The vanilla model, relying mainly on the ambiguous visual information, incorrectly answers (A) Cable car. In contrast, FMD correctly identifies the scene as a (C) Train, as it integrates both visual and auditory cues. This illustrates that FMD effectively leverages audio information to resolve visual ambiguity, resulting in more accurate and context-aware answers.
More qualitative results, highlighting the ability of FMD to capture both audio and visual information, can be provided in the Supp. [B](https://arxiv.org/html/2505.20873v2#A2).

### 4.4 Further analysis

Figure: Figure 4: Layer-wise hidden state similarity in VideoLLaMA2. $L_{\text{fork}}$ is chosen from the early stage.
Refer to caption: x5.png

Layer selection for merge point.
To determine the optimal layer for merging hidden states, we first measure the similarity across all hidden states of VideoLLaMA2, following the approach proposed in (Sun et al., 2025b). Their study on LLMs shows that layers can typically be organized into 4–5 clusters: 1–2 clusters in the early stage, a large cluster in the middle, and 1–2 clusters in the later stage.
In the early layers, modalities align in feature space, with intra-modal encoding strengthened and inter-modal interaction suppressed. In the middle layers, deeper modality fusion occurs, while in the later layers, it prepares task-specific outputs.
As shown in Figure [4](https://arxiv.org/html/2505.20873v2#S4.F4), we observe a similar clustering pattern in VideoLLaMA2. Based on this observation, we select the fork layer $L_{fork}$ from the early stage, where feature alignment and intra-modal encoding occur (Wei et al., 2024; Yu & Lee, 2025). This choice matches our decoding strategy, aiming to strengthen the independent intra-modal representations of vision and audio.

Analysis of attention and model performance across different merge layers.
To validate the suitability of the chosen fork layer $L_{fork}$ for subsequent merging, we further analyze the attention weight distribution across layers, as illustrated in Figure [5](https://arxiv.org/html/2505.20873v2#S4.F5). We find that deeper merge positions lead to reduced attention to visual tokens and increased attention to audio tokens.
However, forcing equal attention to video and audio can degrade performance because it deviates from the characteristics of the pretrained model.
To examine this, we analyze task performance across different *merge* layers in Figure [6](https://arxiv.org/html/2505.20873v2#S4.F6), using 200 AVHBench samples for each task. We also provide results with 500 and 1,000 samples in Supp. [A.2](https://arxiv.org/html/2505.20873v2#A1.SS2), which show similar trends. We observe that as the *merge* layer becomes deeper, the performance on A$\rightarrow$V and AV matching tasks decreases, while performance on the V$\rightarrow$A task improves. This implies that overly low visual dependence can hinder the interpretation of visual information.
Overall, performing the *fork* operation in the early layers achieves more balanced performance while largely preserving the original model characteristics. We propose a pipeline where forking occurs early and merging starts in the middle layers, promoting unimodal feature enhancement first, followed by cross-modal interaction and reasoning in later stages.

**Table 3: Ablation of fusion strategies in Eq. [3](https://arxiv.org/html/2505.20873v2#S3.E3) with VideoLLaMA2. Attention-guided fusion (Fixed) balances masked and unmasked inputs, improving performance.**
| Methods | AVHBench |  |  |
| --- | --- | --- | --- |
| A$\rightarrow$V | V$\rightarrow$A | AV Matching |  |
| \rowcolorlightgray Vanilla | 80.02 | 77.03 | 57.75 |
| Exclusion | 79.38 | 76.79 | 54.83 |
| Average | 75.11 | 55.31 | 57.16 |
| Attention (Adaptive) | 79.49 | 72.78 | 59.19 |
| Attntion (Fixed) | 80.45 | 77.52 | 59.01 |

Ablation on audio-visual fusion strategy.
To demonstrate the effectiveness of our attention-guided fusion, we compare it with three alternative audio-visual fusion strategies on the AVHBench dataset using the VideoLLaMA2 model in Table [3](https://arxiv.org/html/2505.20873v2#S4.T3).

In Eq. [3](https://arxiv.org/html/2505.20873v2#S3.E3), the aggregation of masked and unmasked features is controlled by the fusion weight $\alpha$.
Exclusion corresponds to setting $\alpha=1$, where the masked modality is entirely excluded. This leads to a performance drop compared to vanilla decoding. We attribute this to the causal nature of autoregressive models: fully ignoring one modality can disrupt the flow of information from previous tokens to the next prediction.
Average fusion, where $\alpha=0.5$, also results in degraded performance, likely because it gives equal weight to informative signals and noisy features.
Using an adaptive $\alpha$ for each input, referred to as Attention (Adaptive), also causes performance drops on A$\rightarrow$V and V$\rightarrow$A tasks compared to vanilla decoding.
Moreover, since obtaining the adaptive $\alpha$ requires an additional full forward pass as discussed in Section [3.1](https://arxiv.org/html/2505.20873v2#S3.SS1),
the inference speed becomes 2.89$\times$ slower than vanilla decoding.
This indicates that the model is not robust to continuously varying $\alpha$ values and, and given the added inference cost, the approach is also inefficient. By contrast, a fixed $\alpha$ computed from only 100 AVBench samples not only enables efficient inference without requiring full forward pass to calculate last-layer attention, but also generalizes well to other datasets such as AVQA and MUSIC-AVQA, as proven in Table [1](https://arxiv.org/html/2505.20873v2#S4.T1).
These results highlight the effectiveness of the proposed attention-guided fusion strategy.

Figure: Figure 5: Layer-wise attention weight comparison on VideoLLaMA2 using 600 samples from the AVHBench dataset. We analyze the attention weights from the final token in the last decoder layer, focusing on the distribution across video and audio segments. Deeper merging within the network results in reduced attention to visual tokens and heightened attention to audio tokens.
Refer to caption: x6.png

Figure: Figure 6: Layer-wise ablation results on VideoLLaMA2 using 200 samples from the AVHBench dataset for each task. To verify the suitablity of the selected fork layer $L_{fork}$, we evaluate performance across three tasks, each focused on a certain modality: A$\rightarrow$V for video-targeted understanding, V$\rightarrow$A for audio-targeted understanding, and AV matching for joint audio-visual understanding.
Refer to caption: x7.png

**Table 4: Decoding speed comparison. FMD achieves faster inference since the process after $L_{\text{fork}}$ matches the original model.**
| Decoding | Latency$\downarrow$ | Relative$\downarrow$ |
| --- | --- | --- |
| (sec/token) |  |  |
| \rowcolorlightgray Vanilla | 0.34 | 1 |
| SID | 0.71 | 2.09 |
| VCD | 0.69 | 2.03 |
| DoLa | 0.54 | 1.59 |
| FMD (Ours) | 0.43 | 1.26 |

Decoding speed comparison. To validate the efficiency of FMD, we compare its inference speed against three representative test-time decoding methods, SID (Huo et al., 2025), VCD (Leng et al., 2024b) and DoLA (Chuang et al., 2024). We apply each decoding method to VideoLLaMA2 and measure the time required to generate a single token, reporting both the absolute latency in seconds per token and the relative speed normalized to vanilla decoding in Table [4](https://arxiv.org/html/2505.20873v2#S4.T4). The results are obtained on 100 examples from the AVHBench dataset.

SID and VCD exhibit relatively slow decoding speed. Since they require two full forward passes, one on the original input and one on the modality-corrupted input to contrast the resulting logits, the computational cost nearly doubles. DoLA achieves faster inference than SID and VCD by leveraging intermediate layer outputs to refine the model’s predictions, thereby mitigating the inefficiency of repeated full forward passes. Our proposed FMD achieves the fastest inference speed among the three methods. This efficiency stems from the fact that only the fork layers require dual forward passes, making FMD not only computationally efficient but also more effective than prior decoding methods originally developed for LLMs or VLMs, as evidenced by its superior performance reported in Table [2](https://arxiv.org/html/2505.20873v2#S4.T2).
Additional experiments analyzing how inference speed varies with deeper $L_{fork}$ settings are provided in Supp. [A.4](https://arxiv.org/html/2505.20873v2#A1.SS4).

## 5 Related Works

Audio-visual large language models.
Building upon the success of LLMs, there has been a surge of interest in extending their capabilities to incorporate audio and visual modalities, with text serving as the central modality. ChatBridge (Zhao et al., 2023b) presents a text-centric modality bridging framework trained on limited paired data, whereas models such as PandaGPT (Su et al., 2023), ImageBind-LLM (Han et al., 2023), and OneLLM (Han et al., 2024) leverage unified encoders to accommodate various modalities. Other approaches (Chen et al., 2023b; Lu et al., 2024; Lyu et al., 2023; Panagopoulou et al., 2023; Zhan et al., 2024; Zhang et al., 2023) employ modality-specific encoders to better capture distinct feature spaces.
To enhance spatial-temporal modeling across modalities, CAT (Ye et al., 2024) introduces a clue aggregator for cross-modal reasoning, VideoLLaMA2 (Cheng et al., 2024) utilizes a spatial-temporal convolutional connector for video synchronization, and video-SALMONN (Sun et al., 2024) proposes a multi-resolution causal Q-Former for audio-visual fusion. Meerkat (Chowdhury et al., 2024) further refines multimodal interactions by aligning audio and visual signals at multiple levels through interaction modules prior to decoding.
Recently, Qwen2.5-Omni (Xu et al., 2025) is introduced as a model that perceives audio-visual inputs, generates text and natural speech, and achieves superior performance in audio-visual tasks.

Inference-time reasoning enhancement with LLMs.
Recent efforts have explored inference-time strategies to enhance the reasoning capabilities of LLMs without additional training. Chain-of-Thought (CoT) guides LLMs to produce intermediate reasoning steps, and has been extended to VLMs through structured textual representations (Himakunthala et al., 2023; Ni et al., 2024; Zhu et al., 2023a) or modular reasoning pipelines (Sun et al., 2025a; Xu et al., 2024; Yang et al., 2023). These approaches improve interpretability and robustness through a more explicit reasoning process.
In parallel, Contrastive Decoding (CD) improves inference-time decoding by comparing token-level logits between a weaker and a stronger model (Li et al., 2023b).
DoLA (Chuang et al., 2024) develops this idea by contrasting early and late layer outputs within a single model to refine predictions.
Recently, VCD (Leng et al., 2024b) extends CD to VLMs by injecting Gaussian noise into image inputs and contrasting the resulting biased predictions with the original outputs. Other CD-based approaches (Kim et al., 2024; Wang et al., 2024) enhance decoding robustness by utilizing self-descriptions or distorting instructions. SID (Huo et al., 2025) further advances this line of work by preserving the least informative visual tokens and contrasting their influence on predictions.

However, most of these inference-time reasoning methods have been developed for VLMs, while AV-LLMs remain relatively underexplored. This gap underscores the need for inference strategies specifically designed to address the unique challenges of audio-visual inputs.

## 6 Conclusion

We analyze modality bias in current AV-LLMs, where jointly processing audio and visual inputs can hinder balanced reasoning.
To address this, we propose Fork-Merge Decoding (FMD), a simple, training-free, efficient, and model-agnostic inference strategy that separates modality-specific understanding in the early decoder layers (the fork phase) and merges their representations in later layers (the merge phase).
FMD consistently improves performance on tasks requiring integrated multimodal understanding, as demonstrated across three audio-visual benchmarks using three representative AV-LLMs: VideoLLaMA2, video-SALMONN, and Qwen2.5-Omni.
Our approach is broadly applicable to AV-LLMs, offering a plug-and-play solution that enables deeper unimodal and multimodal understanding during inference.
We hope this work inspires further research addressing the unique challenges of AV-LLMs in complex, multi-sensory settings.

Limitations.
While our method demonstrates strong generalization across datasets without requiring ground-truth labels—relying only on 100 random samples to determine $\alpha$—we note that the optimal value of $\alpha$ can vary depending on model-specific characteristics (e.g., $\alpha=0.8$ for VideoLLaMA2 and $\alpha=0.9$ for Qwen2.5-Omni). This suggests that calibrating $\alpha$ is a lightweight yet necessary step when applying FMD to different architectures, and future work could explore automated strategies to further reduce this effort. In addition, although we identify fork layers that generally perform well across tasks and provide extensive analysis, the optimal choice of fork layer may still vary by task, as illustrated in Figure [6](https://arxiv.org/html/2505.20873v2#S4.F6). Investigating adaptive or task-specific fork layer selection strategies therefore represents an interesting direction for future research.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv*, 2023.
- Alayrac et al. (2022)
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.
Flamingo: a visual language model for few-shot learning.
In *Proc. NeurIPS*, 2022.
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
In *Proc. NeurIPS*, 2020.
- Chen et al. (2023a)
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.
Shikra: Unleashing multimodal llm’s referential dialogue magic.
*arXiv*, 2023a.
- Chen et al. (2023b)
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu.
Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.
In *Proc. NeurIPS*, 2023b.
- Cheng et al. (2024)
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al.
Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.
*arXiv*, 2024.
- Chowdhury et al. (2024)
Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha.
Meerkat: Audio-visual large language model for grounding in space and time.
In *Proc. ECCV*, 2024.
- Chuang et al. (2024)
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He.
Dola: Decoding by contrasting layers improves factuality in large language models.
In *Proc. ICLR*, 2024.
- Dai et al. (2023)
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
Instructblip: Towards general-purpose vision-language models with instruction tuning.
In *Proc. NeurIPS*, 2023.
- Guan et al. (2024)
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al.
Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models.
In *Proc. CVPR*, 2024.
- Han et al. (2023)
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al.
Imagebind-llm: Multi-modality instruction tuning.
*arXiv*, 2023.
- Han et al. (2024)
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue.
Onellm: One framework to align all modalities with language.
In *Proc. CVPR*, 2024.
- Himakunthala et al. (2023)
Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, and William Yang Wang.
Let’s think frame by frame with vip: A video infilling and prediction dataset for evaluating video chain-of-thought.
*arXiv*, 2023.
- Huang et al. (2024)
Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al.
Audiogpt: Understanding and generating speech, music, sound, and talking head.
In *Proc. AAAI*, 2024.
- Huang et al. (2023)
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al.
Language is not all you need: Aligning perception with language models.
In *Proc. NeurIPS*, 2023.
- Huo et al. (2025)
Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao.
Self-introspective decoding: Alleviating hallucinations for large vision-language models.
In *Proc. ICLR*, 2025.
- Kim et al. (2024)
Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro.
Code: Contrasting self-generated description to combat hallucination in large multi-modal models.
In *Proc. NeurIPS*, 2024.
- Leng et al. (2024a)
Sicong Leng, Yun Xing, Zesen Cheng, Yang Zhou, Hang Zhang, Xin Li, Deli Zhao, Shijian Lu, Chunyan Miao, and Lidong Bing.
The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio.
*arXiv*, 2024a.
- Leng et al. (2024b)
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing.
Mitigating object hallucinations in large vision-language models through visual contrastive decoding.
In *Proc. CVPR*, 2024b.
- Li et al. (2022)
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu.
Learning to answer questions in dynamic audio-visual scenarios.
In *Proc. CVPR*, 2022.
- Li et al. (2023a)
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
In *Proc. ICML*, 2023a.
- Li et al. (2023b)
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis.
Contrastive decoding: Open-ended text generation as optimization.
In *Proc. ACL*, 2023b.
- Lin et al. (2024)
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan.
Video-llava: Learning united visual representation by alignment before projection.
In *Proc. EMNLP*, 2024.
- Liu et al. (2023a)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
In *Proc. NeurIPS*, 2023a.
- Liu et al. (2023b)
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al.
Summary of chatgpt-related research and perspective towards the future of large language models.
*Meta-Radiology*, pp. 100017, 2023b.
- Lu et al. (2024)
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi.
Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action.
In *Proc. CVPR*, 2024.
- Lyu et al. (2023)
Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu.
Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration.
*arXiv*, 2023.
- Maaz et al. (2024)
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.
Video-chatgpt: Towards detailed video understanding via large vision and language models.
In *Proc. ACL*, 2024.
- Ni et al. (2024)
Minheng Ni, Yutao Fan, Lei Zhang, and Wangmeng Zuo.
Visual-o1: Understanding ambiguous instructions via multi-modal multi-turn chain-of-thoughts reasoning.
*arXiv preprint arXiv:2410.03321*, 2024.
- Nishimura et al. (2024)
Taichi Nishimura, Shota Nakada, and Masayoshi Kondo.
On the audio hallucinations in large audio-video language models.
*arXiv preprint*, 2024.
- Panagopoulou et al. (2023)
Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles.
X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning.
*arXiv*, 2023.
- Rubenstein et al. (2023)
Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al.
Audiopalm: A large language model that can speak and listen.
*arXiv preprint*, 2023.
- Song et al. (2024)
Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin.
Hierarchical context merging: Better long context understanding for pre-trained llms.
In *Proc. ICLR*, 2024.
- Su et al. (2023)
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.
Pandagpt: One model to instruction-follow them all.
*arXiv*, 2023.
- Sun et al. (2024)
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang.
video-salmonn: Speech-enhanced audio-visual large language models.
In *Proc. ICML*, 2024.
- Sun et al. (2025a)
Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Tianpeng Li, Fan Yang, Zenan Zhou, and Wentao Zhang.
Mm-verify: Enhancing multimodal reasoning with chain-of-thought verification.
*arXiv*, 2025a.
- Sun et al. (2025b)
Qi Sun, Marc Pickett, Aakash Kumar Nain, and Llion Jones.
Transformer layers as painters.
In *Proc. AAAI*, 2025b.
- Sung-Bin et al. (2025)
Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh.
Avhbench: A cross-modal hallucination benchmark for audio-visual large language models.
In *Proc. ICLR*, 2025.
- Tang et al. (2024)
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang.
Salmonn: Towards generic hearing abilities for large language models.
In *Proc. ICLR*, 2024.
- Thoppilan et al. (2022)
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.
Lamda: Language models for dialog applications.
*arXiv*, 2022.
- Wang et al. (2024)
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann.
Mitigating hallucinations in large vision-language models with instruction contrastive decoding.
In *Proc. ACL*, 2024.
- Wei et al. (2024)
Houjing Wei, Yuting Shi, and Naoya Inoue.
Phase diagram of vision large language models inference: A perspective from interaction across image and instruction.
*arXiv preprint*, 2024.
- Wei et al. (2022a)
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.
Finetuned language models are zero-shot learners.
In *Proc. ICLR*, 2022a.
- Wei et al. (2022b)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
In *Proc. NeurIPS*, 2022b.
- Xu et al. (2024)
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan.
Llava-o1: Let vision language models reason step-by-step.
*arXiv*, 2024.
- Xu et al. (2025)
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al.
Qwen2. 5-omni technical report.
*arXiv preprint*, 2025.
- Yang et al. (2022)
Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu.
Avqa: A dataset for audio-visual question answering on videos.
In *Proc. ACM MM*, 2022.
- Yang et al. (2023)
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.
Mm-react: Prompting chatgpt for multimodal reasoning and action.
*arXiv*, 2023.
- Ye et al. (2024)
Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao.
Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios.
In *Proc. ECCV*, 2024.
- Yu et al. (2024)
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al.
Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.
In *Proc. CVPR*, 2024.
- Yu & Lee (2025)
Zhuoran Yu and Yong Jae Lee.
How multimodal llms solve image tasks: A lens on visual grounding, task reasoning, and answer decoding.
In *Proc. COLM*, 2025.
- Zhan et al. (2024)
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al.
Anygpt: Unified multimodal llm with discrete sequence modeling.
*arXiv*, 2024.
- Zhang et al. (2023)
Hang Zhang, Xin Li, and Lidong Bing.
Video-llama: An instruction-tuned audio-visual language model for video understanding.
In *Proc. EMNLP*, 2023.
- Zhang et al. (2024)
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao.
Llama-adapter: Efficient fine-tuning of language models with zero-init attention.
In *Proc. ICLR*, 2024.
- Zhao et al. (2023a)
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.
A survey of large language models.
*arXiv*, 2023a.
- Zhao et al. (2023b)
Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu.
Chatbridge: Bridging modalities with large language model as a language catalyst.
*arXiv*, 2023b.
- Zhu et al. (2023a)
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny.
Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions.
*arXiv*, 2023a.
- Zhu et al. (2023b)
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
Minigpt-4: Enhancing vision-language understanding with advanced large language models.
*arXiv*, 2023b.

## Appendix A Further Analysis

### A.1 Analysis of attention weights beyond the merge point

Figure: Figure A.1: Attention weight analysis in VideoLLaMA2 on the AVHBench dataset. We analyze 100 samples and examine the attention weights in decoder layers after $L_{\text{fork}}$, focusing on the final token. FMD narrows the gap between audio and video attention weights, encouraging more balanced contributions from both modalities.
Refer to caption: x8.png

As shown in Figure [1](https://arxiv.org/html/2505.20873v2#S1.F1) of the main paper, the final layer of VideoLLaMA2 (Cheng et al., 2024) assigns higher attention weights to video inputs compared to audio inputs.
To examine attention trends across the entire model, we visualize the average attention weights after the merge point at $L_{\text{fork}}$ in Figure [A.1](https://arxiv.org/html/2505.20873v2#A1.F1).
Specifically, we focus on the last token in the sequence and aggregate attention across layers and heads over video and audio segments to estimate each modality’s contribution to the prediction.
The proposed FMD consistently produces a more balanced attention distribution between modalities.

### A.2 Layer-wise performance across dataset sizes

In the main paper (see Section [4.4](https://arxiv.org/html/2505.20873v2#S4.SS4) and Figure [6](https://arxiv.org/html/2505.20873v2#S4.F6)), we analyze task performance across $L_{\text{fork}}$ using 200 samples from the AVHBench dataset.
To examine whether this trend holds with more data, we repeat the analysis with 500 and 1,000 samples per task (see Figure [A.2a](https://arxiv.org/html/2505.20873v2#A1.F2.sf1) and Figure [A.2b](https://arxiv.org/html/2505.20873v2#A1.F2.sf2)).
We observe consistent trends across all sample sizes, demonstrating that $L_{\text{fork}}$ can be reliably validated even with a small number of samples.

### A.3 Comparison between gaussian noise addition and zero-out masking

We analyze two input perturbation strategies that aim to suppress modality-specific information by replacing the original input: Gaussian noise injection, as proposed in VCD (Leng et al., 2024b), and zero-out masking, as employed in our method. Specifically, we apply each perturbation method to the video and audio inputs, respectively, and compute the cosine similarity between the final-layer hidden states of the perturbed inputs and those of the original inputs.

As shown in Figure [A.3](https://arxiv.org/html/2505.20873v2#A1.F3), injecting Gaussian noise into the video input yields hidden states that remain highly similar to those of the original input, indicating that this approach fails to effectively isolate the visual signal. In contrast, when applied to audio inputs, Gaussian noise successfully disrupts the signal, leading to substantial differences in the hidden representations. This observation supports the performance improvement observed in the V$\rightarrow$A direction under the FMD with noise setting, as reported in Table [2](https://arxiv.org/html/2505.20873v2#S4.T2).

In comparison, zero-out masking completely suppresses both video and audio signals, resulting in hidden states that are clearly separated from those of the original input. This demonstrates that zero-out masking more effectively blocks modality-specific information from being encoded and consistently outperforms vanilla decoding across all tasks, as shown in Table [2](https://arxiv.org/html/2505.20873v2#S4.T2).

Figure: (a) Performance on 500 samples from the AVHBench dataset.
Refer to caption: x9.png

Figure: Figure A.3: Cosine similarity comparison on VideoLLaMA2 using 100 samples from the AVHBench dataset. We compute the cosine similarity between the final-layer hidden states of the intact input and those of audio- or video-perturbed inputs. The results indicate that video signals are not effectively isolated by additive Gaussian noise, whereas zero-masking reliably suppresses both modalities.
Refer to caption: x11.png

### A.4 Analysis of inference speed across L f ​ o ​ r ​ k L_{fork}

To better understand the relationship between inference speed and the fork layer $L_{fork}$, we evaluate FMD on VideoLLaMA2 using 100 randomly selected AVHBench samples across different fork layer settings.

As shown in Table [A.1](https://arxiv.org/html/2505.20873v2#A1.T1), deeper fork layers require more decoder layers to be forwarded, leading to longer inference time. At the same time, accuracy drops significantly because later forking disrupts the pretrained model attribute as illustrated in Figure [5](https://arxiv.org/html/2505.20873v2#S4.F5) and Figure [6](https://arxiv.org/html/2505.20873v2#S4.F6). Based on this, we set $L_{fork}=4$ in VideoLLaMA2 (one out of every seven layers), which yields consistent performance gains while increasing inference time by only 26%.

**Table A.1: Analysis between inference speed and fork layer $L_{fork}$ in VideoLLaMA2 on AVHBench. Deeper fork layers require more decoder computation, increasing latency while significantly reducing accuracy as illustrated in Figure [A.2](https://arxiv.org/html/2505.20873v2#A1.F2). Setting $L_{fork}=4$ (one-seventh of the 28 layers) achieves consistent performance gains with only 26% increase in inference time.**
| $L_{fork}$ | 0 (Vanilla) | 1 | 4 | 5 | 10 | 15 | 20 | 25 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Latency (sec/token) | 0.34 | 0.40 | 0.43 | 0.45 | 0.51 | 0.56 | 0.62 | 0.68 |
| Relative | 1 | 1.18 | 1.26 | 1.32 | 1.5 | 1.65 | 1.82 | 2 |

## Appendix B More Qualitative Results

In addition to the qualitative analysis presented in Section [4.3](https://arxiv.org/html/2505.20873v2#S4.SS3), we include further examples that illustrate both successful outcomes and failure cases, providing a more comprehensive understanding of the behavior of the model.

### B.1 Positive cases

In addition to Figure [3](https://arxiv.org/html/2505.20873v2#S4.F3), we further analyze various cases, including audio-driven video hallucinations, video-driven audio hallucinations, and audio-visual matching, as illustrated in Figure [A.4](https://arxiv.org/html/2505.20873v2#A5.F4), Figure [A.5](https://arxiv.org/html/2505.20873v2#A5.F5), and Figure [A.6](https://arxiv.org/html/2505.20873v2#A5.F6). We also examine complex audio-visual description scenarios in Figure [A.7](https://arxiv.org/html/2505.20873v2#A5.F7). In these cases, our proposed FMD effectively leverages both audio and visual modalities to accurately interpret the contexts. Notably, even when the audio and video are artificially constructed, FMD is able to detect inconsistencies and generate correct responses. Moreover, as shown in Figure [A.7](https://arxiv.org/html/2505.20873v2#A5.F7), FMD outperforms vanilla decoding by producing more precise and detailed descriptions of both modalities.

### B.2 Negative cases

Although our FMD significantly enhances multimodal understanding without requiring additional training, it occasionally produces inaccurate phrases alongside otherwise detailed and accurate descriptions. For example, in the top case of Figure [A.8](https://arxiv.org/html/2505.20873v2#A5.F8), FMD successfully captures both visual and audio content, whereas vanilla decoding describes only the visual scene.
However, the phrase “while she does it” is temporally incorrect, since the speaker talks before blow-drying.
In the bottom example, the mention of “cup” is inaccurate, as no cup is present. Although such cases reflect occasional misunderstandings, FMD generally produces richer and more informative responses by modeling cross-modal relationships (e.g., capturing “talking”). Moreover, at a broader scale, it tends to reduce hallucinations across datasets, thereby improving overall performance, as demonstrated in Table [1](https://arxiv.org/html/2505.20873v2#S4.T1). Further investigation into minimizing fine-grained hallucinations remains an important direction for future work.

## Appendix C Computational Resource

All experiments are conducted on a system equipped with an AMD EPYC 7513 32-Core CPU and a single NVIDIA RTX A6000 GPU. To ensure fair measurement of inference speed across all experiments, we terminate all non-experimental processes during inference time.

## Appendix D The Use of LLMs

We use LLMs to refine words and sentences for a formal academic writing style and to identify relevant related work.
For evaluation, we employ LLMs to assess long-form responses, such as AV captioning from AVHBench, following the protocol of VideoLLaMA2.

## Appendix E Social Impact

The rapid advancement of LLMs has significantly influenced various sectors, including technology, culture, and education, by making information more accessible and enabling efficient communication. In parallel, MLLMs have progressed through the integration of visual modalities into LLM decoders. Recently, AV-LLMs have emerged, extending these capabilities to both visual and auditory content.

By directly understanding and reasoning over audio-visual content, AV-LLMs offer practical benefits in everyday settings where information is naturally multimodal, such as videos, lectures, conversations, and real-world environments. This opens new possibilities for applications like multimodal search, video question answering, and assistive technologies that go beyond text-based interfaces. Much like how LLMs have made text-based knowledge more accessible, AV-LLMs can help users navigate and interact with rich multimedia content more effectively.

Despite these advantages, the inference-time behavior of AV-LLMs remains underexplored. Our proposed Fork-Merge Decoding (FMD) provides a training-free framework to analyze and isolate modality-specific reasoning, offering insights into how AV-LLMs process complex audio-visual information and guiding future improvements in model design and interpretability.

Figure: Figure A.4: Qualitative results for audio-driven video hallucination tasks using VideoLLaMA2. Compared to vanilla decoding, FMD generates more accurate responses by effectively leveraging both audio and visual modalities.
Refer to caption: x12.png

Figure: Figure A.5: Qualitative results for video-driven audio hallucination tasks using VideoLLaMA2. FMD produces more accurate responses in most cases, including instances where visual signals lead to confusion, as shown in the third example.
Refer to caption: x13.png

Figure: Figure A.6: Qualitative results for audio-visual matching tasks using VideoLLaMA2. FMD generates correct responses even when the synthesized audio and video inputs are semantically unrelated.
Refer to caption: x14.png

Figure: Figure A.7: Qualitative results for audio-visual description tasks using VideoLLaMA2. FMD effectively describes both audio and visual content, capturing fine-grained details (bold)—such as “A man is talking about the bus” in the top example.
Refer to caption: x15.png

Figure: Figure A.8: Failure case analysis on audio-visual description tasks with VideoLLaMA2 (Cheng et al., 2024). FMD may produce occasional errors (red), but captures both audio and visual content more effectively and with finer-grained details (bold) than vanilla decoding.
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: - The design of FMD avoids additional training or architectural modifications.
- The paper is easy to follow.
summary: This paper proposed Fork-Merge Decoding (FMD) inference-time strategy: first, audio-only and video-only inputs are processed separately through the early layers of the decoder (the fork phase), and then the hidden states are merged to perform joint reasoning through the remaining layers (the merge phase). This method has been validated on three representative AV-LLMs (VideoLLaMA2, video-SALMONN, and Qwen2.5-Omni) and three benchmark datasets (AVQA, MUSIC-AVQA, and AVHBench), achieving consistent performance improvements in audio, video, and audio-visual joint reasoning tasks.
weaknesses: - Lack of technical novelty: This method does not alter the fundamental nature of modality imbalance. During training, visual information has higher information density, and there are cases where the model can answer correctly by focusing solely on one modality—these factors lead to the model assigning different levels of attention to different modalities. Therefore, even though the model adopts a separate masking strategy, it still fails to address the aforementioned issues and thus cannot effectively resolve the modality imbalance problem.
- The modality masking method is relatively suitable for samples with consistent audio and video. For example, in the cases provided in Figure 2, the audio and video are completely consistent: either the model is required to output descriptions that include both audio and video content (case a), or the audio provides supplementary information for the video (e.g., the video fails to clarify the type of vehicle, but the audio contains the sound of a train, which helps the model make the correct selection, case b). However, this method cannot handle scenarios where audio and video are inconsistent. For instance, when asked about a sound-producing object that does not appear in the video, masking the audio and relying solely on the video is unnecessary.
- There is a feature space gap between audio features and video features. Therefore, directly fusing them in the merge phase is unreasonable, as it fails to account for this gap between the features.
- The method lacks performance analysis on unimodal understanding of audio and video. According to the modality masking strategy proposed by the authors, it seems impossible to achieve performance improvement in the understanding of video-only or audio-only content.
- This training-free approach cannot enhance the model’s inherent performance on fine-grained tasks, such as temporal grounding and spatial grounding. The authors do not conduct relevant evaluations in the paper.
- How does the method perform on unimodal benchmarks for video or audio? Examples include video-only benchmarks like Video-MME, Video-MMMU, MMVU, MVBench, MMBench-Video, LongVideoBench, EgoSchema, PerceptionTest, MLVU, LVBench, TempCompass, and Charades-STA, as well as other benchmarks for audio/music understanding 
- No performance evaluation on unimodal (video-only/audio-only) benchmarks: The article only tests FMD on audio-visual joint benchmarks (AVQA, MUSIC-AVQA, AVHBench) and completely ignores unimodal benchmarks—including video-only benchmarks (e.g., Video-MME, MVBench, MMBench-Video, LongVideoBench) and audio-only benchmarks.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. More base models and benchmarks. Suggestions: model - Video-SALMONN 2; benchmarks - Video-MME, WorldSense, AVUT
2. L235-236: "We do not apply attention-guided fusion in the channel-wise setting, as the hidden states do not disentangle audio and visual embeddings." This is confusing as audio and video features are concatenated along the channel axis. Do you mean audio and visual embeddings are not disentangled along the time axis?
3. I have concerns about 100 samples for calibration. Do they cover main domains? What is the influence of sample numbers on the performance?
soundness: 2
strengths: 1. **Novel yet simple method:** FMD is easy to implement, making it broadly applicable across different AV-LLMs.
2. **Comprehensive analysis:** detailed quantitative studies, including attention analysis and ablation experiments on fork layers are conducted
3. **Latency:** FMD sets the fork layer to be early layers. This leads to limited additional computation, and may be optimized by engineering (e.g., parallel processing masked audio and video inputs)
summary: This paper focuses on the issue of modality bias in current audio-visual large language models (AV-LLMs), where models tend to over-rely on the visual modality while overlooking the audio. To mitigate this, the authors propose Fork-Merge Decoding (FMD), an inference-time strategy that works for both token-wise fusion and channel-wise fusion AV-LLMs. The decoding process is divided into two stages:
  -  **fork**: the model performs modality-specific reasoning by masking out the other modality
  - **merge**: the resulting hidden states are combined for joint multimodal reasoning

The method is evaluated on three representative AV-LLMs, achieving consistent improvements over vanilla decoding strategies and other inference-time contrastive decoding strategies.
weaknesses: 1. **Dataset/task-specific $\alpha$:** The optimization of $\alpha$ for specific datasets limits the robustness of FMD. For real-world applications, it may be infeasible to retrieve representative samples in advance.
2. **Models and benchmarks for validation:** Although representative AV-LLMs have been evaluated on benchmarks, the robustness still needs to be assessed on more models, such as reasoning models, since the improvement on VideoLLaMA2 and Qwen2.5-Omni is slight

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. On the generalizability of α:
How sensitive is performance to the choice of α? Specifically, if we use a suboptimal α (e.g., from a different model or dataset), how much does performance degrade? Could you quantify this drop?

2. On the meaningfulness of masked features:
In the fork phase, one modality (e.g., video tokens) is zeroed out. Yet the resulting hidden states still carry meaningful unimodal signals. Why is this the case? Does the language prompt alone provide enough context for the model to generate coherent intermediate representations—even when one sensory modality is completely masked?

3. On adaptive Lfork selection:
Figure 6 shows that A→V tasks benefit from shallow Lfork, while V→A tasks prefer deeper Lfork. Have you considered a dynamic Lfork selection mechanism—for example, based on the modality bias inferred from the input prompt or early attention patterns? Would such an adaptive strategy further improve performance?

4. On text modality bias and counterfactual inputs:
The paper focuses on audio-visual imbalance, but text is also a modality. Could the model answer correctly even when both audio and video are masked, relying only on the textual prompt? Moreover, when audio and video are counterfactual or inconsistent, does the model tend to output stereotypical or common-sense answers (biased by pretraining) rather than reflecting the actual (possibly rare) content in the video? This would reveal whether text itself introduces another form of bias.
soundness: 2
strengths: 1. Important and practical problem: Modality bias is a core challenge in modern multimodal LLMs; this work tackles a real-world issue with clear applicability.

2. Simple and efficient method: FMD requires no additional training or architectural changes—only input masking and hidden-state fusion during inference—making it a plug-and-play solution that is easy to deploy.

3. Comprehensive experiments: The evaluation covers diverse AV-LLM architectures and multiple benchmark datasets, demonstrating consistent gains and robustness.
summary: This paper addresses the modality bias problem in current audio-visual large language models (AV-LLMs) by proposing Fork-Merge Decoding (FMD)—a training-free, architecture-agnostic inference-time strategy. FMD separates early-stage unimodal reasoning (fork) from later-stage multimodal integration (merge). The method consistently improves performance across three representative AV-LLMs (VideoLLaMA2, video-SALMONN, and Qwen2.5-Omni) and three standard benchmarks, especially on tasks requiring balanced cross-modal reasoning. The authors further validate FMD’s compatibility with both token-wise and channel-wise fusion architectures and support its effectiveness through attention analysis, ablation studies, and qualitative examples.
weaknesses: 1. Heuristic hyperparameter selection:
The fusion weight α is estimated as a fixed value from 100 samples on AVHBench (e.g., α = 0.8 for VideoLLaMA2). Although it generalizes well to other datasets, it lacks theoretical grounding or an adaptive mechanism. Manual calibration per model undermines true plug-and-play usability in real-world settings.
As shown in Figure 6, the optimal Lfork (fork layer depth) varies significantly across tasks (e.g., A→V prefers shallow layers, V→A prefers deeper ones), raising concerns about practical adaptability.

2. Inference speed overhead:
As shown in Table A.1, FMD increases latency by 1.26× compared to vanilla decoding. While modest, this overhead may limit deployment in latency-sensitive applications.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: I agree with the authors that the current choice of $\alpha$ and the fork layer is suboptimal. I would be interested to know if the authors have any thoughts on potential improvements or alternative approaches. This question will not affect my overall assessment of the paper.
soundness: 3
strengths: - FMD is convenient to deploy; as a training-free, plug-and-play method, it is adaptable to a wide range of applications.
- The comprehensive ablation studies presented in the paper demonstrate the authors' thoroughness in validating their design.
- The proposed method is simple and intuitive, and the manuscript is of high writing quality.
summary: This paper introduces Fork-Merge Decoding (FMD), a training-free inference strategy designed to mitigate modality bias in Audio-Visual Large Language Models (AV-LLMs). The core problem addressed is the tendency of AV-LLMs to over-rely on one modality (e.g., video) at the expense of another (e.g., audio), leading to imbalanced understanding and errors. The authors validate FMD on three representative AV-LLMs (VideoLLaMA2, video-SALMONN, Qwen2.5-Omni) across three challenging benchmarks (AVQA, MUSIC-AVQA, AVHBench). The results consistently show that FMD improves performance over vanilla decoding, particularly in tasks requiring balanced audio-visual reasoning. However, the performance improvement offered by this method is not particularly significant, and some of its design choices lack robust experimental validation.
weaknesses: Although the paper includes extensive ablation studies, I suggest that the following aspects require further investigation:
- I would like to clarify if the model employs Nucleus Sampling for decoding. If it does, given its stochastic nature, it is crucial to conduct multiple decoding runs and report the results with error bars. This analysis is essential because the performance improvement from FMD is currently marginal, which raises concerns about the stability and statistical significance of the method.
- For the model using CHANNEL-WISE FUSION, there is a lack of detailed experimental analysis regarding the audio and video weights during the merging stage. The majority of the analysis in the paper is based on VideoLLaMA-2, with insufficient corresponding analysis for the CHANNEL-WISE FUSION model.
- More powerful models should also be evaluated, such as the recently released video-SALMONN 2+, Qwen3-Omni.
- More benchmarks should be included, such as Video-MME, WorldSense, AVUT.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Have the authors tried soft masking?

2. How much additional computational cost does the merge decoding process introduce?

3. Any insights into how the model improves on hallucination induced by not having balanced modality attention? My experience is that audio-visual LLMs may make up things on a modality that they fail to attend to, so any insights?
soundness: 3
strengths: 1. The method is training free and trying to solve a critical problem in audio-visual LLMs

2. The proposed method achieved superior performance on a range of benchmarks over 3 different audio-visual LLMs in the experiments. 

3. The paper is very well-written.
summary: This paper proposes the fork-merge decoding method to improve audio-visual LLM performance in a training free way. The process first masks audio or visual inputs separately in earlier TFM layers, and then merge them in later layers so that the model is not overlooking any modality when answering questions.
weaknesses: I mainly have the following two concerns regarding the experimental setup, and require the authors to provide more information:

(1). Including more difficult benchmarks. I would suggest at least adding results on Video-MME [1] and/or AVUT [2] to demonstrate the consistency of improvements and the balance between audio and visual modalities. No need to do for all 3 models but at least 1 model is needed.

(2). More recent models, e.g. video-SALMONN 2+ [3] which achieved the best performance on Video-MME would be very helpful to show that the state-of-the-art models still benefit from this method.

[1] Fu et al. "Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis", https://arxiv.org/abs/2405.21075

[2] Yang et al. "Audio-centric Video Understanding Benchmark without Text Shortcut", https://arxiv.org/abs/2503.19951

[3] Tang et al. "video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models", https://arxiv.org/abs/2506.15220


Decision: Reject