Aligned Better, Listen Better for Audio-Visual Large Language Models
Authors: Yuxin Guo
Shuailei Ma
Shijie Ma
Xiaoyi Bao
Chen-Wei Xie
Kecheng Zheng
Tingyu Weng
Siyang Sun
Yun Zheng
Wei Zou
Venue: ICLR 2025 Poster (2025)
Keywords: Audio-Visual Learning
Multimodal Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce an audio-visual multi-scale adapter that can extract and merge spatial information from both modalities at multiple scales, thereby enhancing feature interaction and spatial alignment between modalities.

Abstract: Audio is essential for multimodal video understanding. On the one hand, video inherently contains audio, which supplies complementary information to vision. Besides, video large language models (Video-LLMs) can encounter many audio-centric settings. However, existing Video-LLMs and Audio-Visual Large Language Models (AV-LLMs) exhibit deficiencies in exploiting audio information, leading to weak understanding and hallucinations. To solve the issues, we delve into the model architecture and dataset. (1) From the architectural perspective, we propose a fine-grained AV-LLM, namely Dolphin. The concurrent alignment of audio and visual modalities in both temporal and spatial dimensions ensures a comprehensive and accurate understanding of videos. Specifically, we devise an audio-visual multi-scale adapter for multi-scale information aggregation, which achieves spatial alignment. For temporal alignment, we propose audio-visual interleaved merging. (2) From the dataset perspective, we curate an audio-visual caption \& instruction-tuning dataset, called AVU. It comprises 5.2 million diverse, open-ended data tuples (video, audio, question, answer) and introduces a novel data partitioning strategy. Extensive experiments show our model not only achieves remarkable performance in audio-visual understanding, but also mitigates potential hallucinations.

Published as a conference paper at ICLR 2025
ALIGNED BETTER, LISTEN BETTER FOR AUDIO-VISULA LARGE LANGUAGE MODELS
Yuxin Guo Guol:2.3, Stuol...,Shuuut Shuailei M Shijie Ma
Xiaoyi |2...2.3 Chenwei Xie Kecheng Zheng". Tingyu Weng". Siyang Sun Yun Zheng". Wei Zou' 1,2+
Schooo of Artificial Intelligence, University of Chinese Academy of Sciences ?AIIS, Institute of Automation, Chinese Academy of Sciences (CASIA)
TTonyyi Lab, Alibaba Group 4 *Ant Group Northeastern University (gueyuxin2021, wei. zou) @ia.a.iia..............
ABSTRACT
Audio is essential for multimodal video understanding. On the one hand, video inherently contains audio, which supplies complementary information to vision. Besides, video large language models (Video-LLMs) can encounter many audio- centric settings. However, existing Video-LLMs and Audio-Visual Large Language Models (AV-LLMs) exhibit deficiencies in exploiting audio information, leading weak understanding and ballucinations. To solve the issues, delve into the model architecture and dataset. (1) From the architectural perspective, we propose fine-grained AV-LLM, namely Dolphin. The concurrent alignment of audio and visual modalities in both temporal and spatial dimensions ensures a comprehensive and accurate understanding of videos. Specifically, we devise an audio-visual multi-scale adapter for multi-scale information aggregation, which achieves spatial alignment. For temporal alignment, we propose audio-visual interleaved merging. (2) From the dataset perspective, we curate an audio-visual caption & instruction- tuning dataset, called AVÚ. It comprises 5.2 million diverse, open-ended data tuples (video, audio, question, answer) and introduces a novel data partitioning strategy. Extensive experiments show our model not only achieves remarkable performance in audio-visual understanding, but also mitigates potential hallucinations.
INTRODUCTION
Humans perceive the dynamic world through their eyes and ears, with visual and auditory information complementing each other, both are indispensable. Similarly, the audio modality proves crucial for the comprehensive understanding capabilities of Multimodal Large Language Models (MLLMs) (Liu let al. [2023] Ma et al., [2024c; [aao et al.) 2024 Wu et al. 2025; Ma et al.) 2025a). On the one hand, audio modality can provide complementary information to the visual modality (Guo et al., 2024b), aiding MLLMs in more accurate comprehension. On the other hand, there many audio-centric tasks in audio-visual (AV) understanding, e.g., ... AV question answering. AV source localization, AV event localization, and AV segmentation (Tian et al. 2018 Yang al. 2022 Guo al.) 2023 Mo & Morgado 2022 Cuo et al. 20024)). However, most existing Video-LLMMs directly neglect audio, with only small portion incorporating both visual and audio modalities. A natural question arises: How proficient are these models in their audio-visual comprehension capabilities?
To answer the question, we design two progressive experiments, in Figure [ First, we evaluate the AV understanding abilities of current AV-LLMs, e.g., Video-LLaMA (Zhang et al. 2023a) and Video- LLaMA (Cheng al. 2024). We found they consistently neglect audio and the descriptions solely come from the visual content. Furthermore, when directly querying the audio content, the responses are always the speculations and associations derived from the visual input, rather than the audio itself. For example, when Video-LLaMA is presented with a cooking video, the response is "background music playing throughout the video." But actually the audio features a male commentator's 's narration. Besides, when replacing the background sound with white noise, the model's 's responses remain unchanged, indicating that the model does not extract information from the audio.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
(a) Audio-Viuul Understanding Capability
(b)) Vidao Understanding with
Different Modality
DOLPHIN
Audic Valea inpux
Gadio Captire sude of man speidin Frelec raging yerr sko poona itire you very yuic, and they cock und our you judt ther
oud
LIITER THE wOrLD WErrER
Visto
Quedtonn Wiritt doen this video and Its audio desc debail?
Quastton Please describbe the sudio
TE6
e03
The viss ono cooking boc lam The person sean uooig boo ae
The a the is roi inalists buciggrunn masic playing drouptoutte edda mase meaey that plays buc wtheut dhoaccigg
36.0
56.4
a in siee consant
Doble
uado mortranas eirting round Beay coming tro tyynng sound.
lagin shatg Stane Lafter cones
sud sockd
torme guiden parcerss then lipad
bwined aucto 4-50e data
TPK accompanien Englin
voice
ditaling Bie cooling prossi recidkiingg kng vau buiclb the food bowns
procees, descitiing
REECDIIICE the Lol and on booiing www.ggakkii eertesss of the capling accviset
talned purio input rext
Figure 1: (a) Audio visual capability of previous AV-LLMs and our Dolphin. We pose questions separately for audio-video and audio, discovering that LideoL.LaMA and LaMA exhibit significant hallucinations for audio understanding, while Dolphin produces accurate responses. (b) Audio could provide complementary information compared to video. Incorporating audio into training greatly enhances video understanding.
Based on the results, it naturally begs the question: Why do AV-LLMs tend to neglect the audio modality? We attribute the following reasons: (1) For alignment, most models lack fine-grained alignment and interactions between modalities, and simply concat the visual and audio tokens. (2) For datasets size, large-scale audio-visure works align vision-language and audio-language separately, resulting in less coordinated audio-video representations, (3) In current datasets, visual modality has relatively higher information density, where audio does not provide necessary content, consequently, AV-LLMs tend disregard audio.
cntructtion-folloiing datasets are scarce, and most
To solve these problems, we explore from two perspectives, i.e., model architecture and training dataset. From the model perspective, we propose novel fine-grained AV-LLM, namely Dolphin, which aligns audio and visual modalities both spatially and temporally and effectively harnesses both complementary modalities. For spatial alignment, we propose an audio-visual multi-scale adapter, which extracts multi-scale features and implements audio-vishn visuall interaction and merging at various scales. For temporal alignment, we propose audio-visual interleaved merging, where both audio and visual serve as context for each other through interleaved tokens. Finally, the fine-grained tokens aligned both spatially and temporally are projected into the input space of LLM to achieve remarkable joint audio-visual understanding.
From the dataset perspective, we propose a large-scale audio-visual understanding caption and cnstruction-oollwwiii dataset, called AVU, which consists of 5.2M AV captions and question-and- answer (Q&A) tuples. We extract video and audio meta- mnta-infoorattin and generate high-quality captions and Q&& pairs. The dataset is divided into several splits according AV consistency for different training objectives. Specifically, we incorporate datasets from several AV tasks, e.g., AVE (Tian et al. 2018), AVI (Mo & Morgado 2022), AVS (Zhou al. 2022) and AVVP (Tian et al., 2020), and convert them to fine-grained instruction-followt following data. Besides, some negative samples for rejection are devised avoid potential hallucinations. To comprehensively evaluate audio-visua understanding, we further propose benchmark, called AVU-Bench, for AV-LLMs. We highlight the importance of interaction from both modalities, as in Fig. (b).
Our contributions are summarized below:
We propose Dolphin, a fine-grained AV-LLM for audio-video multimodal and unimodal understanding. Dolphin could remarkably exploit audio information for understanding.
The core innovation lies in the architecture that enables audio-visual multi-scale spatial alignment as well as context-aided temporal alignment, which ensures fine-grained extraction of two complementary modalities and interaction between them.
We curate the first large-scale audio-visual caption and cdoruction-followiii dataset. It contains 5.2M samples with several splits and does not require rigid AV correspondence.
Extensive experiments show that Dolphin could not only achieve outstanding audio-vishe understanding performance, but also be competent in unimodal tasks, which validates that Dolphin effectively exploits the information of two complementary modalities.
2

---

Published as a conference paper at ICLR 2025
Praste Aariaa
Unbel
Choco Allly Madiane
Text Instruction
What does this video and is audio deseribal
Large Language Model Decoder
Ansiwer
Irpu
AL
batter Fipped stocked plate This asoompanied Englih-ppasking mor's voice detaling the coosing process, montooiing quickty the food brownd and inemuding to plls them up they come the
The deo shonecsaes the piccess making stack of puncresa Ricagina with dicee golde -hrown
pan
Audio Woruel Interioaved Merging
Figure 2: Overview of our Dolphin, which aligns on both spatial and temporal dimensions to fully exploit the natural consistency of videos and enhance the complementary roles of vision and hearing. Specifically, for spatial alignment, we introduced an visial multi-scale adapter using dual- feature pathway design, which extracts multi-scal features from both visual and auditory inputs and achieves fine- enn-grained alignment with the respective modality.
2 RELATED WORKS
Multi-Modal Large Language Models for Video Understanding. series of studies first de- compose videos into different representation dimensions and then integrate the inputs to enrich the prompts for MI.MMs. For example, Video-ChatGPP (Maaz et al. 2023) splits videos into spatial and temporal branches for pooling. VideoChat (Li et al. 2023b) decomposes videos into descriptions and video embeddings. LLaMA-VID (Li al. 2023c) represents each frame with context tokens and content tokens. Video-LaVIT (Jin et al. (022) performs video tokenization using keyframes and motion vectors. Other works incorporate images into unified video training to enrich the training data. Chat-UniVi (Jin et al.] (023) and Video-LLaVA (Lin et al.] (023)) employ strategies such as object-based adaptive claster-based tokens and aligning before projection unify image and video inputs. thereby achieving more powerful visual understanding.
Audio- Visual Large Language Models. Early works, like VideoChat (Li et al. 2023b), simply encode audio inputs using Whisper (Radford et al. (023b)) and directly overlay them to the textual input. Later works seek to align the output of audio and visual encoders before feeding them into LLMs. MACAW-LLM (Lyu et al. 2023) aligns the encoder outputs to the textual space through learnable alignment module. Audio Visual LLM (Shu al., 2023a) activates the embeddings of different modalities with different tags. Moreover, some works (Sun et al. 2023a Zhang et al.) (023a) explore temporal alignment between video and audio using a Q-Former (Li et al. (023a) structure, but most of them (Tang et al. 2024) neglect fine-grained spatial modeling. Meerkat (Chowdhury et al., 2024) explores fine-grained understanding but only focuses on images. To sum up, most existing AV-LLMs neither struggle to capture fine-graines local information nor handle temporal alignment, which motivates us to delve into the design and training of AV-LLMs.
3 MODEL ARCHITECTURE
Overview. Doiphin primarily focuses on effectively strengthening the fine-grained alignment and interaction between visual and auditory modalities. effectively exploits the complementary information of two modalities and prevents overlooking any of them. Specifically, Dolphin primarily comprises three components: to align audio and visual features across various spatial scales in fine-grained manner. (2) Audio- visual (AV) interleaved merging (Section (.2) for temporal alignment and extracting complementary information of two modalities. (3) Large Language Model (LLM) to handle the interacted audio-visual tokens and output responses according to the instructions.
(1) Audio-visual (AV) multi-scale adapter (Section 3.T). which aims
Notations. We split each video into T 8 visual frames and audio clips. Let H., W. denote the height and width of each frame, while each audio clip is transformed into a spectrogram of H, W.. N denotes the number of ViT blocks. In the AV multi-scale adapter, the visual modality
3

---

Published as a conference paper at ICLR 2025
contains duth-aat imput features, i.e., global feature V and multi-scale local feature v/, where i denotes the i-th block and & denotes the I-th frame. Besides, V, and of are features after modality interaction. Similar expressions of A,, at Ag aj are applicable to audio. The joint feature after modality interaction could be represented as arjj In AV interleaved merging, visual and audio tokens after contextual interaction could be written as Veemp and A Aemm,, respectively.
3.
SPATIAL: AUDIO- AUDIO-VISUAL MULTI-SCALE ADAPTER
The AV multi-scale adapter is designed to enhance the fine- ane-grained alignment of spatial features. Taking the spirit of ViT-Adapter (Chen et al. 2023b), we first independently extract each modality's s global and pyramid multi-scale features Lin et al. (017); |; Liu et al. (2025). Then, fine-grained alignment performed across different scales. In this section, we introduce the data flow with the visual modality an example, and the same principle applies to audio well.
Visual global and initial multi-scale features. For ViT-L (Dosovitskiy al., (021). we divide into N 4 blocks, each with 6 layers. To acquire multi-scale spatial features, we feed the images into the spatial module, ie., pyramid convolutional network (L.in et al. (017), to extract various multi-resolltiinn features, Le., 1/8, 1/16 and 1/32 of H, X W forming the initial D-dimensional multi-scale features i as an imput to the adapter, as shown below:
o
)
aj R
X
Inter-modality feature interaction. To achieve fine ineegrained alignment across scales, audio features are injected into the multi-scale visual features vi, where i denotes the i-th block, we incorporate audio global feature A into of through cross-attention, and obtain audio-guided multi-scale visual feature and as follows:
av; = C-ross v/ AI,
Intra-modality feature fusion. As shown in Figure 2 in the fusion process, the audio-guided multi-scale visual features av), which contains visual spatial priors and audio information, are injected into original global features Vū through cross-attention and added to V as follows:
D = Vi + 3 CrossAttn(V); avi, av;).
Here, 3 is learnable vector initialized as O, ensuring the initial outputs the same VIT's global features V Subsequently, the global features D are transmitted to the next block through the i-th standard VIT block and the fine fine-grained features of are passed to the next block as v/ through cross-attention and FFN layers:
VI+I ViT-Block' (D')
01 i+1 = 0i + FFN (6), 0: = :i CrossAttn (ui, Vi+1 v(+1).
(5)
Final outputs. The AV multi-scale adapter obtains V, BBXTXL and AN BBXTxL,XD where B, D denote batch size, number of frames and feature dimension, and Lv, are the number of tokens of two modalities. We perform average pooling across the temporal dimension T and obtain the final spatial tokens. In this way, we gradually incorporate audio informa- tion into the visual features and enhance the interaction between audio and visual modalities. The guidance provided by audio to the multi-scale visual features facilitates fine-grained alignment.
3.2 TEMPORAL: AUDIO- VISOO INTERLEAVED MERGING
In this stage, we merge the final audio and visual features to implement alignment and interaction at the temporal axis. Specifically, by concatenating visual and audio tokens in the same frame, we could obtain pairs of audio-visual interleaved tokens, each referred to as an AV group, as shown in Figure 2 Within each group, we perform bi-directional contextual attention visual and audio tokens, which produce visual-contexen contxxtulnne audio tokens and audio-contexttualize visual tokens.
yeemp CrossAttn (A,, Ve, V.), A femp = CrossAAtnnVV,,44
(6)
Then each AV token group mapped into the input space of L.LM through a joint audio-visua projector. Here, we condense each frame of the video into two tokens. In this way, achieve integration and merging of audio and visual information, which enhances the aadio-visual information exploitation of AV-LLMs.
4

---

Published as a conference paper at ICLR 2025
Drislied rec recapiion reteieflmmaion
InterVLL
Video ClOCke
Yitro Fats
GPT
Hit OPCOARRO
Mibile dV cara EaTT
Lea CENARRIYY
DieerA
vidca
Coption Integrating
playaa guoka
Keywordd
A/S-
KXU Specis
AXU Soguueee
40U-Ta
Liamn
wib
B Owee2--aclo
Audio
Addo Rela
AFL P1J00A
AVI Dalenet
Figure 3: The integration pipeline of the audio-visur understanding dataset (AVU-dataset).
Table 1:
The detailed statistics for our AVU (Audio- Understanding Dataset).
Subsees
AVU-Pretrain
AVU-MaltiOA
SVU-Specific
AVU-Negetive
AVU-Tasks
Instruction
Type
Description
Detailed
Conversation
Mulsitum
Visual-Spcific Info
Audio-Spocifie Info
Negative Samples
Temporal and Spatial
Total
Reasoning
Audio- Video
Statisties
LIIM
500K (subect AVU-Pretrain)
554k
186k
283k
2.13M
Instrsction Statisties
LIIM
109M
370k
5.24M
3.3 TRAINING STRATEGY
We employ Vicuna-v1. (Chiang et al. 2023) as the LLM. The video encoder is ViT-L/14 from CLIP (Radford et al. 2021), and the audio encoder adopts ImageBind (Girdhar al. 2023b). (1) In the initial pre-training phase, we freeze the visual encoder, audio encoder, and L.MM and only update all the projectors as well as the AV multi-scale adapter to achieve alignment across visual, auditory, and LLM modal spaces. We use the AVU-Pretrain dataset. (2) For the instruction tuning phase, only the visual and audio encoders are frozen, while other modules are updated. We employ AVU-Multi Q&A, AVU-Specific, AVU-Negatives and AVU UUUTTasks subsets. See Section for more details.
4 AVU: AUDIO- VISUAL UNDERSTANDING DATASET
Motivation. The quality of data Ma et al. 2024b 2025b) plays vital role in performance.
Currently, there is a shortage of large-scale audio- visual instruction-follwwer and fine- une-grained captions, which hinders the model from focusing on modality-speciff information and potentially leads to audio hallucinations. To solve the issues, we propose an audio-visual understanding (AVU)- dataset, large-scale AV understanding and cnttuuction-oollwwiit dataset.
Overview. In this section, we introduce the construction the AVU-dataset (Figure 3 Specifically, we first re-caption (Section I.1) audio and video data and generate veta-infformatin (Section 4.2)
Then the dataset is split (Section 4.3) into several subsets according to the similarity between audio and visual mnttainforrmttion and we feed different prompt templates generate the corresponding mnstruction-folloviin dataset and training stages.
Dataset statistics. As shown in Table T, AVU-dataset contains 2.13M audio-video pairs, each has several Q&& A pairs, resulting in 5.24M Q&A pairs in total. AVU-dataset has four subsets, i.e., AVU-Pretrain (1. 11M samples and Q&A), AVU-Multi Q&A (500k samples and IM Q&A pairs), AVU-Specific (554K samples and 1.09M Q&A pairs for video and 1.11M for audio), AVU-Negative (186Ksamples and 370K Q&A pairs) and AVU-Tasks (283K samples and 559K Q&A pairs).
Datasets quality and verification. We design three types of filtering mechanisms, including CLIP-Score filtering, self-consi filtering, and Annotation filtering, to filter noisy samples and guarantee the dataset quality, and then human verification is implemented. Details are shown in the Appendix
4.1 DETAILED RE-CAPTION GENERATION
Source datasets. We collect widely used audio-visure datasets, including AudioSet-2M (Gem. meke et al., 2017). VGG-Sound (Chen et al. 2020a) and task-specific datasets, i.e., MUSIC (Li et al.] (222) for AVQA (Yang et al.] 2022).
(Arandjelovic Zisserman 2017).
VGG-SoundSource for AV source localization (Mo & Morgado! 2022). AVE dataset for AV event
5

---

Published as a conference paper at ICLR 2025
Video Experts Comparisons
(b) Audio Experts Compartsons
fed Dranmstoon
Cen dews
Popuol
Kaed-
Muse
Chatbon
Inwert
Queaty Pm dee
Spedicrs
CPF
Pendoot
Grmnnd Spr
(aap
One
Figure 4: Performance comparison of different task-specific experts.
IAdiio-idee information]:
This the detailed caption of the video: [.. and the meta-informatioon {--). This is detailed caption of the
audio: (...): and the metaa-nfformation (...). These ahove contain important information that needs read carefully."
AV U-S-epiii
INPUT:
[Audlo-video Information]
You have to finish the following two tasks, First, carefully compare the information typically included for audio versus video. Second, Idectify information that included audio not for video, generate a pair for each. Similarty, idenlify information that included for video but audio, and generate pair for each.
Each pair should simulate dialogue that focuses on various aspects the content without introducing information explicitly mentioned. The purpsse evercise is fine language model that generate accurate und relevant responses.
The should with "Q=" and contain instruction related audio and video, while second line should start with "A:" and provide response to the instruction.
Please emsure that your instructions are diverse and of high quality. Avoid making ussumptions or adding details that explicitly given.
Figure 5: Examples of prompt templates for generating AVU-dataset, others are in the appendix.
Sprcid
the vide.
Wharddttee tosk uienci an parclea prpportion?
Wher doed the bark?
with useall bog on wooken surticu The
The vinee shayetooes sass basit dnysen rosss blynng pas demon their
Ths sadoo baiing mome fomey
th banta fore
back and tog sespe ne revving
entweee the pe von thio
Whvl spanedd socrads to agg
Ergiin sudio com maiikly. dosall
Whsd clild poing the vices?
zamarts
is norking from the ait row sight
such maing boot spoidss maaking parickaes
The ahoucases basic
The video coss not now any tiee.
coaleeina of koo
Figure Some examples of AVU-dataset.
localization Tian al. 2018), Wdatssee for AV segmentation (Zhou al., 2022 and LLP for AV video parsing (Tian et al. 2020). Then we utilize expert models to re re-caption audio and video to obtain audio, video and audio-video captions.
Expert models. For MILM,, we employ fine-tuned version of Intern VL.33B (Chen et al. 2024). For the audio expert captioners, we choose performance of these models is illustrated in Figure 4
A (Chu et al. 2024). The
Prompt templates. We prompt expert models with hand-crafted templates to generate detailed captions, and set regularizations to mitigate hallucinations. Figure[5]dipplae dissplay subset: AVU- Specific prompt templates. See Figure and Figure i0 the appendix for more details.
4.2 META-INFORMATION GENERATION AND INTEGRATION
Meta-information generation. The function of meta- information is to maintain audio and video details when integrating unimodal audio and video captions into multimodal audio-video captions. Meta-information could utilized to guide subsequent AV caption integration and dataset splitting. Specifically,, design options of "event", 'object', 'scene", 'place", 'action' and 'emotion' etc.,

---

Published as a conference paper at ICLR 2025
and transform original annotations to 'keywords'. Then, employ GPT-4 (Achiam et al. 2023) to judge consistency based on keywords and eettainformation and filter noisy samples, which might be caused by experts' bias and hallucinations.
Meta-information integration. We choose LL meta-information. First, we employ GPT-4 to generate multiple examples of integrating keywords and inttrminfommttinn. After human adjustments, feed them to LLaMA3. The in-context learning abilities enable the generation of audio-visual captions.
(Dubey al. 2024 integrate
4.3 DATASET SPLITS
Splitting process. The dataset split according to the consistency between audio and video meta- information. Notably, we do not require strict audio-visual consistency, which is the main novelty of this work. For different types of data, we create various types of instruction-tuning datasets for corresponding training stages. The whole dataset is divided into three subsets based audio-visual consistency. Specifically, we feed audio and video meta-information to GPT-4 (Achiam et al. 2023) to obtain the consistency score. Figure 6 shows examples of subsets of the AVU-dataset.
AVU-Pretrain comprises samples with high AV consistency. The audio and video information nearly the same. These samples are suitable for the pre-training stage align AV modalities. We design fixed question templates (Figure ((a and randomly select one each time, and use the previously integrated AV captions as the answers. In this way, we obtain AVU-Pretrain subsets.
AVU-Multi Q&A also consists of high consistency samples. Different from templates (Figure g (b)) to transfer AV caption multi-turn Q&A and reasoning.
we design
AVU-Specific comprises samples with medium AV consistency. Both audio and video carry relatively additional information compared to each other. Questions are posed regarding this additional information to generate Q&A pairs. These Q&A pairs construct AVU-Specific subsets and could only answered by focusing on specific modality (Figure (C)...
AVU-Negatives consist of low-consistency samples, e.g., the sounding object is not present in the
frame. Taking the spirit of contrastive learning (Chen al., [0206] He et al.] (020). we create the
negative sample dataset, i.e., AVU-Negatives (Figure I0 (d)), whose answers are primarily used as rejection. This subset could teach LLMs the rejection option, mitigating potential hallucinations.
AVU-Tasks are curated directly from downstream AV tasks. Specifically, we transform the original annotations to the format of Q&A, including accurate details like time, spatial, and event. Notably, the AVU-Tasks are derived from the fine- grained annotations, and significantly contribute to the model's 's fine-grained alignment capabilities.
For pre-training, we mainly employ high-consistency samples enhance modality alignment, while for instruction-tuninn we make use of samples in which audio and visual do not exactly overlap and mix AVU-Multi Q&A, AVU-Specificc AVU-Negatives and AVU-Tasks for training. In this way, the issue of neglect of audio and hallucination is mitigated.
EXPERIMENTS
We introduce the experimental setup and comparisons among models. In the ablation studies, we explored and validated that fine- mme-grained alignment significantly aids LLMs in multimodal understand- ing. Temporal contextual alignment is an effective way to leverage the inherent consistency of videos and to exploit complementary audio-visual information. Additionally, we also conducted ablations on the proposed dataset, showcasing its assistance in audio-visual joint perception and its high quality.
5.1 EXPERIMENTAL SETUP
Implementation details. For each video, we extract frames of 224 x 224 resolution, and audio is sampled into frames, each turned into 128 x 204 spectrogram. Both pre-training and fine-tuning are conducted for one epoch, with batch sizes of 256 for pretraining and 128 for finetuning. Projectors for audio, video, and audio-video use two-layer MIPs with GELU (Hendrycks & Gimpel| 2016) activation. Training is performed on NVIDIA A 100 GPUs. More details are the appendix.
7

---

Published as a conference paper at ICLR 2025
Table Comparison with existing Video-LLMs, We conducted performance comparison with the existing Video-LLM and reported the scoring results of GPT on four zero-shot video- -A datasets.
MSVD-QA
VTTQA
ActivityNet QA
TGIF
Method
LLM
POPE
Acc Score
Acc
Score
Acc
Score
Acc Score
L.aMA Adapter
Zhang al.
20236
LLaMA 7B
54.9
3.1
43.8
2.7
34.2
2.7
Video- Chat
20236
Vicuna-7B
56.3
45.0
2.5
26.5
2.2
Video-ChatGPT
Maaz et 2023
Vicuna-7B
64.9
3.3
49.3
2.8
35.2
2.7
BT--Adaterr
Vicuna-7B
47.0
45.7
3.2
LLaMA VID
La el
21/-6
Vicuna 7B
69.7
3.7
57.7
3.2
47.4
3.3
Vicuna- 13B
58.9
47.5
Vidco-LLaVA
Lin et hl.
2025
Vicuna-7B
70.7
3,9
59.2
3.5
45.3
3,3
70.0
4.0
84.4
PandaGPT (Sun et 2024
Vicuna-78
46.7
23.7
78.5
VidcoLLaMA
Vicuna-7B
29.6
12.4
1.1
82.9
AV-LLM Sin el a. 202.3
Vicuna-77 7B
67.3
53.7
47.2
AVicuna
Tang al.
Vicuna 7B
59.7
53.0
Onel.LM HHin ai.
21ZA
LLaMA2
56.5
53.8
FAVOR
Sun al.
37245
Vicuna-7B
67.8
59.3
Lad (Cnene
2024
Mistral-Instruce
71.7
3.9
57.4
49.9
3,3
85.4
video SALMONK [Sun al
2024
Vicuna 7B
67.9
59.5
3.4
Dolphin 7B-LoRA (Ours)
Vicuna
61.3
3.4
70.9
Dolphin 7B (Ours)
Vicuna 7B
72.7
3.9
62.6
3.5
49.1
3.4
71.2
3.9
85.9
Dulphin- (Ours)
Vicuna
63.5
3.4
86.2
Table Comparison with Audio-LLMs. We conducted closed-ended and open-ended auditory tasks with LTU and LTU-AS, where ZS denotes zero-shot evaluation.
Audio
Andio Caption
Speech Recognition
Emotion Recognition
Gender Classif
Music Genrv Classif,
Audio
Speech Question
Audio Hallucination
Chosif
Question
Method
ESC-50 AndioCaps Librispeech JEMOCAP Vonceleb2 Noxeeteb2 (ACC)) (SPICE (WER_ J
GIZAN
Random (ACC1
(ACC1) (maro-F11) (MAED
(ACCT) (ACC1) (ACCT)
Rest specislized models trained supervisolly each datneet Not wawallllaalll umsenn Tabel tasks.
TASK SOTA
97.0
17.7
1.4
70.6
CLIP-lkke
model
AudioCTia
CLAP
Dionge Momaggeear K016
82.6
25.2
Audio Goog et al
2b2.bb
82.8
17.0
104.2
38.2
77.0
29.8
9666
69%
UIU-
Lonn or
20/30
10.9
0.5
12.9
69.8
90.1
7.9
23.5
65%
93%
LTU-AS
80,8**
15.0
4.9
65.2
90.8
7.3
50.3*
96/8
94%
50.1
VideoLLaMA Zhanz
2003a
62.6*
6.2
128.4
23.4*
43.5
22.2
56%
27%
VideoLLaMM
9.8**
video- SALMONN (Sen allll 0024
77.6"
16.6
3.9
65.5"
36.7"
95%
94%
58.2
Dolphin-LoRAA (Ours)
81.6**
17.2
128**
67.4
91.22*
7,25
33.6
9698
93%
58.6
Dolphin (Ours)
83,12>
17.8
8,3
69.2**
92.5*
7.0
37.8
96/8
948
63.2
Dataset. During training, we enhanced our model by mixing inputs from multiple modalities. Apart from using AVU, employ LLaVA (Liu al. 2023). 10% of Valley, and audio clips for pre- training. VA innstruutt Video-ChatGPT (Maaz et al. 2023), and Clotho (Drossos et al. (020) for instruction- based finetuning. We assessed our model's zero-shot capabilities on single-modality video and audio Q&A A benchmarks and compared it against existing audio-visual L.MMs using tailored downstream task benchmark.
5.2 COMPARISON WITH
Zero-Shot Video Understanding. To validate our model's efficacy, we compared its performance in video comprehension with existing video-LLMs. Specifically, we showcased Dolphin's comparison with various methods on MSR-VTT-QA MSVD-QA, and ActivityNet-QA benchmarks. As indicated Table our model demonstrated superior video understanding, proving that auditory information complemented visual comprehension in the training stage. Moreover, POPE (Li et al. 2023d results show our method could mitigate object hallucinations.
Closed and Open-Ended Audio Tasks. To test our model's audio understanding, we followed LTU and LTU-AS, evaluating closed-ended and open-ended (Ma et al. 2024a Zhu et al. 2024) audio tasks like classification and captioning in Table Despite zero-shot settings on many datasets, our model showed effective understanding, achieving or exceeding SOTA performance. We use the pre- pre-trained ImageBind (Girdhar et al. 2023a) audio encoder as our audio encoder and freeze it during training, whose structure is AST (also used in LTU)). It is less effective in speech recognition compared with
8

---

lighurll Published as a conference paper at ICLR 2025
Table Results on the proposed audio-visual understanding bench and AVSD.
Task
AVU
AVSL
AVE
AVVP
AVSD
Dataset
MUSIC
AVL,
AVS
AVE
LLP
AVSD
Metric
acc
acc
acc
Video L.LMMA (Zhang et al. Video LLaMA 2 Cinnnn ec 2024
202.3a
65.4
3.3
35.6
2.3
40.8
2.8
21.5
2.1
36.7
73.2
3.5
47.1
2.3
48.2
2.9
28.4
2.3
57.2
Miloo-SALMONN (Sun al.
74.7
3.5
48.3
2.4
51.5
3.0
30.9
2.4
51.6
Dolphin (OWrS)
78.2
3,9
51.8
3.0
52.1
3.2
31,4
2.8
59.1
Videe APCENDTION
Whi dio
Whartnee
The sudol canddaal piiss Thes in pl Becinn recedied with pline
wits strs ROOP for
reding wound The siee Dammar unmoriee reveriil ane pertans gridle
What's th manssyiig
what's intorsiing clout
sprcr ofliciite
WoN' gor YOU a
some thap elccti
bacipes ad sourd Firs sucuoor
payers aooucting with Map sper cheangg
lially
and manntarr Bedge
Per Iraslgase the ded crritig cande exctensee
Figure 7: Qualitative cases of Dolphin.
Whisper (used by LTU-AS). However, our model showed improved speech recognition, benefiting from our dataset mixing audio and speech. Additionally, the hadoo-hallucination (Kuan al. 2024) experiment shows that our model can effectively reduce audio hallucination.
Audio-Visual Understanding Bench. To assess our model's competency in audio-visual under- standing, we developed minibench tailored for Large Language Models (LLMs) that aligns with existing audio-visual tasks. We meticulously gathered test datasets from labeled audio- tasks, comprising MUSIC (AVQA), LLP (AVVP), AVE (AVE), AVS-Bench (AVS), Flickr-SoundNer and VGG-SoundSource (AVL), to ensure a fair and precise evaluation. Inspired by zero-shot question- answer evaluations and TTTTo-CharGPT, we transformed these tasks' ground truths into open-ended question-answer formats. We also evaluate our model on AVSD (Alamri et al. 2019) benchmark.
This method evaluates the precision of model's predictive output, awarding scores scale. Comparing the performance of existing methods, like Video-LLaMA (Zhang et al. (023a)). Video-LLaMA 2 [Cheng al., 2024), and SideoSSLLMONN (Sun in Table demonstrate a notable performance gap favoring our model on audio- vision tasks, which highlights our model's 's enhanced comprehension of audio and visual modalities.
1-5
2024). our results, presented
Qualitative evaluation. The qualitative cases of the proposed Dolphin are illustrated in Figure 7 Results show that Dolphin could remarkably comprehend both audio and visual modalities, together with enhanced video understanding.
5.3 ABLATION STUDIES AND ANALYSIS
In this section, through detailed ablation studies, we explored how to enhance an audio-visual LLM's integration of video and audio for better video understanding. We also validated the effectiveness of our proposed methods and datasets through model and dataset ablations.
Fine-grained spatial alignment effectively aids AV-LLMS in understanding multimodal semantic information. In prior analysis, we found that without fine-grained interaction between visual and auditory information, LLMs struggle to learn relevant information between videos and audios, the lack of prior knowledge regarding the two modalities leads the model to focus more the information- rich video content while neglecting audio. To investigate this issue, we conducted ablation studies fine- finennrained alignment module in Table 5a The results reveal that inter-

---

Published as a conference paper at ICLR 2025
modality interaction indeed enhances the LLM's attention both modalities. However, in comparison, when the two modalities undergo fine-grained alignment before projection, the model is not only able to simultancously focus on both modalities but also better extract the complementary information from visual and audio. Table 5a illustrates that fine-grained audio-visual alignment aids LLMs in cross-modal understanding. Inter-modal interaction boosts attention to both modalities, but fine- grained alignment before projection further allows simultaneous focus and better extraction of visual and auditory complementary information. Compared removing temporal alignment, understanding of videos significantly improves with the help of temporal alignment and interaction.
Contextual alignment effectively leverages the inherent consistency of videos. We analyzed our temporal context attention module in Table Sa and found that models without it or with only the contrast, employing temporal context attention,
chronological arrangement of tokens underperform. which aligns video and audio features over time, significantly boosts performance by tapping into their inherent temporal consistency, thus enhancing L.LM's 's understanding of both modalities together.
Table 5: Ablations on model architecture designs (a) and various dataset subsets (b).
Module
Variants
acc
AVU
score
acc
AVE
score
Variants
MSR-VTT-QA
AV Understand
POPE
Original
Dolphin (All)
78.2
3.9
52.1
3.2
score
score
Spatial
inter-modal
adapter
77.6
3.8
51.6
51.8
3,2
w/o AUU-Specific
w/o AVU-Pretrain
59.3
58.8
3.3
3.4
69.8
72.6
3.6
3.6
82.1
81.7
w/o AVU-Negatives
61.0
3.5
77.8
3.8
84.8
Temporal
w/o bi-dir context atto
w/o AV inter-merging
32.3
76.1
2.5
3.6
22.6
50.3
2.2
Full AVÚ
62.6
3,5
78.2
3,9
85,9
(a) Model architecture designs.
(b) AVU-datasel subsets.
Table 6: Detailed ablation on datasets and models.
Methods
MSR-VTT-QA
Audiocaps SPICE
AV Understand
acc
score
Viden-IJ.aMA, Video- Vidro-LLaMA AVU (Ours)
dataset
11.8
65.4
55.3 (425.7) 7)
(+1.)
(44.))
73.2 (+7.8)
3.6 (+0.3)
Dolphin + Video-LL.at L.LMAA dataset
42.8 (+13.2)
2.7 (40.9)
14. (+2.6)
69.9 (+4.5)
3.4 (+0.1)
Dolphin + AVU (Ours)
(+19
(+1.6)
(7.84)34)
78.2 (+6.3)
3.9 (+0.5)
Effectiveness of model structure and dataset quality. We conducted an ablation study on our model and dataset in Table [b) First, training without the audio dataset diminished the model's understanding of pure video content, indicating that our model effectively utilizes complementary information from audio. Moreover, comparing the performance without our dataset and training Video-LLaMA with our dataset (Table showed significant performance decline without the AVU dataset, whereas Video-LLaMA achieved better performance on our dataset. These result, along with the human validation in Appendix B jointly demonstrates the quality and effectiveness of our dataset.
6 CONCLUSION
In this paper, we primarily explored how existing AV-LLMs can effectively overcome insufficient focus on audio information. We innovatively propose Dolphin, an Audio-Visual Large Language Model for audio-video understanding, which features fine-grained interaction and alignment on both spatial and temporal levels. We design a multi-scale audio-visual adapter and a temporal context module to fully leverage the inherent consisteney of videos and realize the complementary function of visual and auditory information. Extensive experiments indicate the effectiveness of the model. Additionally, dataset for video understanding, containing 2. 13M pairs of AV samples and 5.24M Q&A pairs and providing diverse training objectives for audio-visual LLMs. This instruction-follot foolowiing dataset is suitable for large model learning and has been proven to effectively enhance the audio-visual understanding capabilities of existing models, and demonstrate the quality of our dataset. Finally, through experiments, we explore and conclude that fine-graine temporal and spatial alignment can effectively enhance visual and auditory perception abilities of audio-visual LLMs.
collected and labeled audio-visual caption and instruction fine-tuning
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, al. Audio visual scene-aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7558-7567, 2019.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp. 609-617, 2017.
Xiaoyi Bao, Siyang Sun, Shuailei Ma. Kecheng Zheng, Yuxin Guo, Guosheng Zhao, Yun Zheng, and Xingang Wang. Cores: Orchestrating the dance reasoning and segmentation. In European Conference Computer Vision, pp. 187-204. Springer, 2024.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-moda models with better captions. arXiv preprint arXiv:2311.12799 2023a.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In The Eleventh International Conference on Learning Representations, 2023b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 241998 2024.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et Videollama 2: Advancing spatial-tems nattial-emmporl modeling and audio understanding in video-lims. arXiv preprint arXiv:2406.07476, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. arXiv preprint arXiv:2407.0185., 2024.
Yunfei Chu, Jin Xu, Qian Yang. Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv preprint arXiv:24007 100.599999. 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEF, 2020.
11

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
Benjamin Elizalde, Soham Deshmukh, Mahmoud AI Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023a.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023b.
Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuchne, and James Glass. Contrastive audio-visual masked autoencoder. arXiv preprint
arXiv:2210.07839, 2022.
Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023a.
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint arXiv:2305. 200.0 2023b.
Yuxin Guo, Shijie Ma, Hu Su, Zhiqing Wang. Yuhao Zhao, Wei Zou, Siyang Sun, and Yun Zheng. Dual mean-teacher: An unbiased semi-supervised framework for audio visual source localization. Advances in Neural Information Processing Systems, 014446999466666 2023.
Yuxin Guo, Shijie Ma, Yuhao Zhao, Hu Su, and Wei Zou. Cross pseudo-labeling for semi-supervised audio-visual source localization. In ICASSP 2024-2024 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pp. 8356-8360. IEEE, 2024a.
Yuxin Guo, Siyang Sun, Shuailei Ma, Kecheng Zheng, Xiaoyi Bao, Shijie Ma, Wei Zou, and Yun Zheng. Crossmae: Cross-modality masked autoencoders for region-aware audio-visual pre-training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp-
26721-26731, 2024b.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. ICASSP 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03990 03905, 2023.
Jiaming Han, Kaixiong Gong, Yiyuan Zhang Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
26584 26595, 2024.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 00.11166.88800 08415, 2016.
12

---

Published as a conference paper at ICLR 2025
Jeff Huang, Charles Zhang, and Julian Dolby. Clap: Recording local executions to reproduce concurrency failures. Acm Sigplan Notices, 48(6):141-152, 2013.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311 10011...0066, 2023.
Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et Video-lavit: Unified rideo-langual pre-training with decoupled visual-motional tokenization. arXiv preprint arXiv:2402.0316/ 2024.
Chun- Yi Kuan, Wei-Ping Huang, and Hung-yi Lee. Understanding sounds, missing the questions: The challenge of object hallucination large audio-language models. arXiv preprint arXiv:2406.08402, 2024.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleaae:: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint wrXiv2240.007899 2024.
Guangyao Li, Yake Wei, Yapeng Tian. Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 17108-19188 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint urXiv:2305.06355 2023b.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311. 2023c.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340. Springer, 2025.
Yifan Li, Yifan Du, Kun Zhou, Jimpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXhv:230511033. 2023d.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint 1.Xii:2311.10/2: 2023.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023.
Jiangyuan Liu, Hongxuan Ma, Yuxin Guo, Yuhao Zhao, Chi Zhang, Wei Sui, and Wei Zou. Monocular depth estimation and segmentation for transparent object with iterative semantic and geometric fusion, arXiv preprint 2rxiv:2002.1 /460002./46166 /4616, 2025.
Ruyang Liu, Chen Li, Yixiao Ge, Thomas H Li, Ying Shan, and Ge Li. Bt-adapter: Video conversation is feasible without video instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13658-13667) 2024.
Ruipu Luo, Ziwang Zhao, Min Yang. Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint a0X.v:2306.0723 2023.
Chenyang Lyu, Minghao Wa, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, vidco, and text integration. arXiv preprint arXiv:2306.09099 2023.
13

---

in Neural Published as a conference paper at ICLR 2025
Shijie Ma, Fei Zhu, Zhun Zhong, Wenzhuo Liu, Xu-yao Zhang. and Cheng-lin Liu. Happy: A debiased learning framework for continual generalized category discovery. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang Information Processing Systems, volume 37, pp. 50850-50875, 2024a.
Shijie Ma, Fei Zhu, Zhun Zhong, Xu- Yao Zhang, and Cheng-Lin Liu. Active generalized category discovery. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16890-16900, 2024b.
Shijie Ma. Yuying Ge, Teng Wang, Yuxin Guo, Yixiao Ge, and Ying Shan. Genhancer: Imperfect generative models are secretly strong vision-centric enhancers. arXiv preprint arXiv:2503. 19480, 2025a.
Shijie Ma, Fei Zhu, Zhen Cheng, and Xu-Yao Zhang. Towards trustworthy dataset distillation. Pattern Recognition, 157:110875, 2025b.
Shuailei Ma, Kecheng Zheng, Ying Wei, Wei Wu, Fan Lu, Yifei Zhang, Chen-wei Xie, Jiapeng Zhu, and Yujun Shen. Learning visual generative priors without text. arXiv preprint arXiv:2412.07767, 2024c.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424. 2023.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mml: methods, analysis and insights from multimodal IIm pre-training. European Conference on Computer Vision, pp. 304-323. Springer, 2025.
Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localiza- tion. Advances in Neural Information Processing Systems, 35:37524-377 -3556,, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PMLR, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning. pp. 28492-28518 PMLR, 2023a.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 28492-28518. PMLR, 2023b.
Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual 1lm for video understanding. arXiv preprint arXhv:2312.067200 2023a.
Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual Ilm for video understanding. arXiv preprint arXiv:2312.06722 2023b.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305. 16355, 2023.
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Fine- -rained audio-visual joint representations for multimodal large language models. arXiv preprint 23 10.00880.3333 2023a.
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Fine- Fine-grained oudio-visual joint representations for multimodal large language models. arXiv preprint arXiv:2310.0550 2023b.
14

---

Published as a conference paper at ICLR 2025
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. sadoon: enhanced audio-visual large language models. arXiv preprint arXhv22406.15744 2024.
Yunlong Tang, Daiki Shimada, Jing Bi, and Chenliang Xu. Avicuna: Audio-visual Ilm with interleaver and bontext-boundary alignment for temporal referential dialogue. arXiv preprint arXiv:2403.16276. 2024.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference computer vision (ECCV), pp. 247-263, 2018.
Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 436-454, Springer, 2020.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, al. Cambrian-1: fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024.
Wei Wu, Kecheng Zheng. Shuailei Ma, Fan Lu, Yuxin Guo, Yifei Zhang, Wei Chen, Qingpei Guo, Yujun Shen, and Zheng-Jun Zha. Lotlip: Improving language-image pre-training for long text understanding. Advances in Neural Information Processing Systems, 37:64996-6500 2025.
Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu, Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pp. 3480-3491, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11975-11986, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.00885. 02855, 2023a.
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mml. 5: Methods, analysis & insights from 2024.
multimodal Ilm fine-tuning. arXiv preprint arXiv:2409.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. XrXiv preprint arXiv:2303. 2023b.
Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In European Conference on Computer Vision, pp. 386-403. Springer, 2022.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, al. Languagebind: Extending video-language pretraining to modality by language-based semantic alignment. arXiv preprint arXiv:2310.01822 2023.
Fei Zhu, Shijie Ma, Zhen Cheng, Yao Zhang, Zhaoxiang Zhang, and Cheng-Lin Liu. Open-world machine learning: A review and new outlooks. arXiv preprint arX1v:2403.01759, 2024.
15

---

Published as a conference paper at ICLR 2025
Supplementary Material
Differences from Existing Methods
16
Comparison With Existing AV-MILMMs
16
A.2
Comparison between Dolphin and LTU and LTU-AS.
17
Details of Dataset Curation and Verification
17
Further Implementation Details
18
Training Details.
18
The Proposed Spatial and Temporal Modules are Mutually-Promotee
18
D Further Experimental Results
18
D.I
Comparison With Various Audio and Visual Encoders.
18
D.2 Other Encoders and LLM ablations.
19
D.3
Impact of Using Image and Video Encoder
19
D.4
Impact of Freeze or Unfreeze
20
D.5
Impact of Different Connector
20
D.6
Speech Comprehension Capability
21
Dataset Details
21
The Motivation of the AVU--Daasee
21
The Contribution of the AVU--Daasel
22
Prompt Templates.
22
A DIFFERENCES FROM EXISTING METHODS
A. COMPARISON WITH EXISTING AV-MLLMS.
Here, we also explain the differences from the methodology perspective. MACAW-LLM: The training visual and audio signals come from different videos, which results in lack of fine-grained representation and alignment of modalities. The proposed dataset includes only images and vidcos, without any audio.
ImageBind-LLM (Han al. 2023); It includes six modalities but only utilizes image-text alignment for training, without specifically addressing the alignment and representation of audio-video pairs.
PandaGPT (Su et al. 2023): integrates a shared latent space derived from ImageBind, primarily facilitating zero-shot transfer across six modalities: text, image/video, audio, depth, thermal, and IMUs.
Video-LLaMA (Zhang et al. 2023a): It employs audio and visual Q-Former for respective modalities, but only trains the vision- vision-language branch and the A-L branch on video/image instruction data, without incorporating audio training.
AV-LLM (Shu et al. 2023a): It focuses solely on spatiotemporal modeling of the video modality, neglecting the fine-grained information from the audio.
16

---

Published as a conference paper at ICLR 2025
FAVOR (Sun et al. 2023a): It proposes causal Q-Former structure with causal attention module that aligns only temporally, lacking aeeeggriied audio-visual modeling.
Video-LLaMA2 (Cheng et al.
2024): It focuses solely the spatiotemporal representation of video, neglecting fine-grained audio representation and audio-visual interactions.
Meerkat (Chowdhury al. 2024): It is fine-grained audio-visual understanding model; however, it only models images and lacks video understanding capabilities.
AVicuna (Tang et al.,
2024): It focuses solely temporal modeling, neglecting spatial fine- fine-grained information. Additionally. it exhibits a sigmificant amount of hallucination.
OneLLM (Han et al., 2024): It achieves effective integration and instruction adherence across different modalities through progressive multimodal alignment and modality routing, primarily focusing on global alignment.
In summary, our work introduces innovations in both the framework and dataset pipeline, effectively addressing the challenges faced by AV-MLLMs. Furthermore, the proposed solutions have the potential to inspire research on existing AV-MLLM models and datasets, while also contributing to the broader MLLM community.
A.2 COMPARISON BETWEEN DOLPHIN AND LTU AND LTU-AS.
We summarize the following four differences: (1) Different model types. LTU (Gong et al. 2023b) and LTU-AS (Gong et al. 2023a) are audio/speech-specificc models, trained specifically with audio/speech- language models for audio or speech tasks. contrast, our Dolphin model is an AV-LLM that comprehends both audio and video, encompassing a broader range of modalities. (2) Different audio backbones. LTU-AS employs Whisper (Radford et al., 2023b), which is pre-trained on a large-scale speech-language dataset, resulting in stronger speech recognition performance. We use the ImageBind aud encoder (used by LTU), which has not been pre-trained on a speech dataset. Moreover, the primary tasks in Table such speech recognition, emotion recognition, and gender classification, are closely related to speech. Therefore, our performance reflects zero-shot results. (3) Compared to LTU-AS, our audio encoder's zero-shot recognition performance still achieves state-of-the-as results in the majority of tasks. This demonstrates our model's ability to pay attention to and comprehend audio effectively. (4) Compared LTU, our performance surpasses theirs by considerable margin, which can be attributed to our generated dataset that includes both audio and speech samples. This highlights the effectiveness of our dataset.
B DETAILS OF DATASET CURATION AND VERIFICATION
For the dataset filtering, we design three types of filtering mechanisms: CLIP-Score filtering, self- consistency filtering and annotation filtering. Subsequently, human verification is implemented to quantitatively verify the quality of the generated caption.
CLIP-score filtering. In this stage, the visual and audio experts first generate captions based on the input video and audio, and then we employ CLIP and CLAP to assess the similarity score for each caption, respectively. Captions with lower scores might have noise and hallucinations.
Self-consistency filtering. We further prompt the visual and audio experts to summarize the inttrmmioommononnn of the generated captions, and utilize GPT4 to assign the matching score given the initial caption and its meta-information. Captions with lower scores might have noise and hallucinations.
Annotation filtering. The original annotations of the datasets are transformed to 'keywords' Then, we employ GPT-4 to judge consistency based on keywords and meta-information and filter noisy samples, which might be caused by experts' bias and hallucinations.
We synthesized the aforementioned three factors to calculate a weighted confidence score for each sample and subsequently ranked them. The bottom 25% of samples, based on this ranking, will be eliminated. The weights for the sereening criteria are as follows: CLIP-Score filtering: Self-consistency filtering: 1, Annotation Filtering: 5.
17

---

Published as a conference paper at ICLR 2025
Human verification. After the former three filtering steps, 100 human annotators employed to give scores (1 to 5) to each the video-caption and audio-caption pairs, considering completeness and accuracy (related to hallucinations). We randomly sample 100 video-caption and 100 audio- caption pairs in the generated dataset. Besides, we also verify the caption after the integration from two modalities (Integration Effect). The results shown in Table
Table 7:
The mean scoring of human verification on completeness and accuracy of audio (A) and visual (V) modalities.
Captions
Completeness
V: Accuracy
Completeness
Accuracy
Integration Effect
Mean scoring
4.27
4,45
4.23
4,40
4.31
The verification results show that the generated dataset is precise and accurate.
C FURTHER IMPLEMENTATION DETAILS
C.1 TRAINING DETAILS.
Stage-1: I: Multi-modal text alignment pre-training. The model needs to read audio, video and corresponding textual instructions, which is related to the understanding of audio, video and audio- video contents. The ground truth annotations are the captions of AVU-dataset. Stage-2: Audio-visual dynamic instruction-tuning The model is required to respond accordingly various types of instructions. The instructions comprise complex visual, audio and audio-visual understanding tasks. Both the projector and LLM are updated. Dataset prompts, audio or/and video and questions are fed to the model to generate answers. The generated answers are then supervised by the ground truth captions
datasets to update both the projectors and LLMs. For both stages, the learning objective is autoregressive wext-word-predictio loss.
C.2 THE PROPOSED SPATIAL AND TEMPORAL MODULES ARE MUTUALLY-
The two modules are utilized to align fine-grained spatial and temporal information for audio- visual data. The AV multi-scale adapter separately extracts multi-scale features from visual and auditory modalities and interacts with the other modality for fine-grained alignment. This promotes fine-grained alignment between audio and video, avoiding the neglect of auditory information and effectively enhancing the model's 's performance in audio-visual dense prediction tasks. The primary innovation of the temporal interleaved merging lies simultaneously calculating bidirectional attention for both audio and video features, enabling the alignment of video and audio features in the temporal dimension. This effectively leverages the consistency of videos, improving the LLM's joint understanding of video and audio.
D FURTHER EXPERIMENTAL RESULTS
D.1 COMPARISON WITH VARIOUS AUDIO AND VISUAL ENCODERS.
We compared various variants of audio and visual encoders and reported the performance for video/audio/audiovviae understanding, shown in Table g
Table 8: Comparison with various audio and visual encoder variants.
Encoders
MSR-VTT-QA Audio Caption AV Understand
(a)
CLIP+CLAP
62.5
17.3
74.2
(b)
ImageRind-video--mmaee lww.--iddde-mmm--nii--------i
58.2
15.8
72.3
Image ViCA
59.8
17.6
76.5
ILii+ (Ours)
62.6
17.8
78.2
Then the following conclusions are drawm: (1) If both audio and visual aspects are aligned with language (as in CLIP (Radford et al., 2021), CLAP (Elizalde al. 2023)), the performance A/V
18

---

Published as a conference paper at ICLR 2025
Table Comparisons with various visual and LLM backbones.
Audio caption
MSR-VTT-QA
AV Understand
w/ SigLIP Zhai et al. 2023
16.5
63,5
77.9
w/ Whisper Radiord et al.
2023a
17.9
61.9
77.5
w/ Llama3--38 Baamma3-BBInnae In--trrucct (Dubey et al. 2024
18.6
64.0
80.3
Ours
17.8
62.6
78.2
understanding tasks tends to be good, but the performance AV tasks is not high. We attribute this the fact that the AV encoder, due to lack of alignment, is unable to effectively utilize the AV adapter. (2) Building upon this, we added stage 0.5 before pretraining and pre-trained for one epoch on AudioSet-2M (Gemmeke et al. 2017) using the audio-visual contrastive loss. We observed performance improvement (especially in audio), which validates that fine- fineggrineed audio- visual alignment effectively helps the model pay attention to the audio modality. (3) If both audio and video use ImageBind (Girdhar al. 2023b), which has undergone audio visual alignment, the A/V/AV understanding capabilities decline. We believe that in Machine Learning I Language Models, since the output format is language, using backhone that has not been aligned with language will affect the final understanding results. (4) If the audio modality is aligned with language (CLAP) and the visual modality aligned audio-visually (ImageBind-we Bind-vidoo), the results are inferior compared to the other mode of video is much higher than audio, and video-lane vidoo-laggaage alignment can better enhance the model's understanding and instruction followwng capabilities.
CLLII.. We attribute this to the fact that the semantic density
In summary, we believe that selecting visual encoder aligned with language and an audio encoder aligned with visual can effectively balance A/V/AV understanding capabilities. Therefore, we chose Image-ia as our backbone encoder.
D.2 OTHER ENCODERS AND LLM ABLATIONS.
The primary objective of our work is to explore effective learning schemes, demonstrating that fine-grained vidio-visual alignment can enhance the model's understanding of audio, and mitigate hallucinations. To ensure fair comparisons, we have selected the most widely used encoders with better AV alignment and LLMs.
Moreover, we also include the results with other encoders and LLMs, as shown in Table. 9
When choosing SigLIP, the visual abilities are enhanced while the audio-related abilities are degraded, because IPg has weaker alignment compared with CLIP. When choosing 1a-ma-88-Innstruue our method obtains better overall results. We attribute the superiority of Llama3 and more parameters (LLaMA3: 8B > Vicuna: 7B).
After replacing the aforementioned backbone, there an improvement some results, such as SigLIP's understanding of video and the performance of Alamaa-8B-Innsruuee This further demon- strates the effectiveness of our framework and dataset. It effectively helps the model focus on both audio and video modalities while also mitigating hallucinations. This also demonstrates the strong generalization capability of our model, allowing it to be applicable to a wider range of backbones.
D.3 IMPACT OF USING IMAGE AND VIDEO ENCODER
We explore the impact of whether our model uses video encoder.
Firstly, we observed that preceding video caption models predominantly utilize image en- coders (CLIP), c.g., LLaMA- LLaMA-VID(Li et al.) 2025) and ShareGPT4V (Chen et al. 2023a), Valley (Luo et al., 2023). AV-LLM (Shu et al. 2023b), AVi- cuna (Tng al. (024) and video-SAL SALMONN (Sun et al. 2024), the reason is that CLIP could align better with language and are pre-traine trained on larger numbers of data with stronger visual abilities.
(Li et al., 2024).
Additionally, using image encoder to process video allows for the selection of any frame rate, providing flexibility that aids in modeling temporal information. This is also a similar rationale behind the approach used in Videol VideoLLLaM.. LaMA2 (Cheng al. 2024).
19

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study of using video encoder and image encoder.
Audio Caption
MSR-VTT-QA
AV Understand
w/ LanguageBind video encoder
16.4
60.8
76.4
w/ ImageBind video encoder
15.8
58.2
72.3
w/ CLIP image encoder (ours)
17.8
62.6
78.2
Table 11: Ablation study of freezing or unfreezing encoder.
Audio Caption
MSR-VTT-QA
AV Understand
unfreeze av freeze (Ours)
15.3
58.7
75.9
17.8
62.6
78.2
Table 12: Ablation study of freezing or unfreezing encoder.
w/ 2 (Ours)
w/ MLP
w/ Q-Foormer
MSR-VTT-QA Understand
62.6
60.8
58.6
78.2
77.1
75.3
We also added the results using video encoders in LanguageBind (Zhu et al. 2023 and Image-
Bind (Girdhar al. 2023a), as shown in Table. 10
The results indicate that replacing the CLIP with other video encoders leads to decline in perfor- mance. This attributed to CLIP's superior semantic representation capabilities and its alignment with text. Besides, CLIP is pre-trained with large number of images and has stronger vision abilities.
D.4 IMPACT OF FREEZE OR UNFREEZE
Moreover, we explored whether freezing has an impact on model performance, and conducted experiments with unfrozen AV encoders, as shown Table. II
From the result, we can see that in the case of unfrozen encoders, the overall results degraded. The increased number of training modules made training more challenging and increased training consumption. The training time was approximately 2.8 times longer than before.
D.5 IMPACT OF DIFFERENT CONNECTOR
For audio, video, and audio-visual connectors, we follow LLaVA and use a 2x MLP with GELU activation. Regarding the selection of connectors, existing Multimodal Language Learning Models (MLLMs) predominantly opt for relatively simple structures. For instance. LLa (Liu et al. 2023)
and Video-LLaVA (Lin et al. 2023) utilize MLPs, Cambrian (Tong et 2024 employs Spatial Vision Aggregator, and MMI.5 (McKinzie et al. 2025 Zhang et al. 2024 selects an abstractor. An empirical study also indicates that the choice of simple connectors (such as C-Abstractor, average pooling, and attention pooling) has marginal impact on the results. Consequently, we followed LLaVA and opted for the simplest configuration of MLP 2x with GELU activation.
We found that Sideo-SALMONN (Sun et al., 2024). and VideoLLaMM (Zhang et al.) 2023a use Q-Former as a connector. Therefore, we chose Q-Former and a one-layer MI.P conduct the connector ablation experiments.
From the Table. 12 we can observe that the performance of our model declined slightly after adding the Q-Former, which might be due to the complexity of training the Q-Former, making it difficult to train well. AAMo-SN (Sun et al.) 2024), and VideoLLaMA (Zhang et al. 2023a) mainly explore speech and audio with variable resolution and length, while our main concern is the fine- grained audio-visual alignment and mitigation of hallucination. Therefore, we decided not to adopt the Q-Former.
20

---

Published as a conference paper at ICLR 2025
saption
caption
Describe the overall audio and its significant aioments.
Explain the audio and key alements.
The audio is of woman speaking, She is describing the process applying makeup to the face. She starts by saying I'm going proceeds
The audio contains speaking foreign language. saying language
apoty your makaup" appllying makeup. She talks about how to use a foundation bruch to apply the foundation, how to use a concedler to cover any blemishes, and how to use powder sot the foundation. She also talks about how (he lashes. She ends by seying l'and that's you're donelt
you
("Woo The man responds the woman repeats the question. The man then continues speaking in the same language. There is knock on the door again and woman's voice heard seying I'Open the doar." The raspands again the language.
sleps
blush add color mescara lengthen and thicken
UUU-Specific
AVU-Spectfic
What happens after the woman hears knock on the door?
What is the wornn in the video descriiing?
Aller the woman heers knock on the door, aska 1"Who responds
The woman the video describing the process of applying makeup the face. She explain the invalved. including using foundation concealer cover blemishes, using powder fourdation, using a blush add color the cheeks, and using mascara longthen and thickan the lashes. She conciudie seying "ann that's you're donel"
IP! repeats question. in the same guugge There snother knock on the door. and the woman saya '"Open the doar!" The man responds again in the same language.
language, continues speaking
apply
Figure 8: Some cases dataset samples containing speech information. On the left is the audio caption generated by Qwen2-Audin Notably, these samples of audio carrying speech information are split into the AVU-Specific subset in our dataset generation pipeline.
D.6 SPEECH COMPREHENSION CAPABILITY
We have conducted an exploration into the speech capabilities of Dolphin. Firstly, we discovered that Dolphin has a certain level of speech recognition ability. From the perspective of performance, the speech capabilities are not bad. In Table 3, Dolphin outperforms other audio-centric models by a large margin. We attribute the result the fact that the generated AVU dataset contains significant amount of speech data.
Qwen2-Audio generate captions, it will perform real-time transcription for speech. We specifically calculated that samples containing speech transcription, the results account for 39.6% of all audio captions. These speech-related data help enhance speech abilities. As shown in Figure. g They contain a lot of speech information in the captions and are generally categorized into the AVU-Specific subset.
Specifically, when
Similar findings that speech-centric training datasets could help enhance the speech recognition of audio encoders have been concluded by some prior works. For example, in LTU (Gong al. 2023b), LTU-Speech and LTU-audio utilize freeze audio encoders (AST encoder using CAV-MAE (Gong 2022 objectives) trained with audio and speech data, respectively. Considering the performance, Speech outperforms LTU-Audio by a large margin on Speech Question datasets (93% >69%
LTU
E DATASET DETAILS
E. THE MOTIVATION OF THE AVU-DATASET
We list the purpose of the proposed AVU-dataset as follows: (1) There is a lack of relevant datasets. Currently, there are no large-scale audio-visua captioning and instruction tuning datasets available. Existing methods have only been trained on vision-language and audio-langen audio-language datasets, resulting deficiency of audio-visual alignment, which consequently leads to suboptimal performance in audio-visual tasks. (2) We analyze that one of the primary reasons existing models overlook the audio modality is that, in most cases, audio does not provide more information than video. Therefore, one of significant innovations lies in our dataset, which specifically selects samples where audio conveys more information than video. We have transformed this audio-specific information into question-and- answer pairs, effectively addressing the issue of AV-LLM's neglect of audio. (3) Existing audio-visual
21

---

Published as a conference paper at ICLR 2025
datasets exhibit diverse annotation formats (bounding boxes, timestamps, masks). Consequently, have standardized the input and output formats for audiovisual tasks such as AVE, AVL, AVQA, and AVVP, facilitating the training of AV-LLM. Additionally, have provided a dataset with fine-grained temporal and spatial granularity (AVU-tasks).
E.2
THE CONTRIBUTION OF THE AVU-DATASET
The contribution of AVU-dataset is summarized as follows: (1) The AVU dataset addresses the current. lack of large-scale, high-quality audiovisual instruction tuning datasets within the community, (2) The AVU dataset features a rich variety subsets, effectively addressing the issue AV-ILM neglecting the audio modality and significantly reducing the occurrence of hallucinations. (3) The introduction of meta-information in the AVU dataset, which is categorized based on AV consistency, allows for the extraction of modality-specine information (AVU-specific) and the generation of negative samples (AVU-negative). This approach can be widely applied the process of creating other datasets.
E.3 PROMPT TEMPLATES.
The prompt templates are shown in Figure 9 Figure I0 and Figure 11
22

---

Published as a conference paper at ICLR 2025
IAudio-video information]:
This the detalled caption the video: {...): and the meta- intormmfoomm (...). This is the detailed caption of the
audio: {...]: and the meta-information: (...). These above contain important information that needs be read carefully.*
INPUT:
[Audio-video information]
Your task is generate detailed caption an audio-video segment based on provided meta- inferaiiformatioonn detailed descriptions, Ensure caption comprehensive and accurate, using only provided information without adding any fictional content.
EXAMPLE:
video starts woman standing in kitchen, holding a knife and potato. She begins slicing the potato thinly, showcasing her skill precision. The camera captures her novements as she continues slicing, demonstrating the food preparation process. Soft ambient sounds the kitchen, including faint simmering from nearby stove and occasional utensil clinks.
IL.AVU-MuliQA
INPUT:
(Auiio-ideeo information]
Your task is to generate ten pairs of instructions and responses based on the provided meta rta-innormaaion and detailed captions of the audio and video content. Each pair should simulate dialogue that focuses on various aspects of the content without introducing information not ex expliityy mentioned. The purpose of this exercise is fine- tune language model so that it can generate accurate relovant responses.
The QA pair should start with "Q:" and contain an Instruction related to the audio and video, while the second line should start with "A:" and provide response the instruction.
Please ensure that, your instructions ons are diverse and of high quality. Avoid making assumptions or aaddnng details that are not expicitly given.
EXAMPLE:
Q: Describe how the video integrates visual steps and audio tips to guide viewers in making pancakes. A: The video combincs close visuals of pancake preparation with descriptive audio commentary, emphasizing quick cooking and stacking instructions.
Q: Provide summary of video's appmach to combining visual demonstrations audio expl anations. video seamlossly intogrates detailed visual demonstrations pancake preparation with insightful audio explanations, offering viewers comprehensive guide achieving perfect pancakes.
Figure 9; Prompt templates for generation of AVU-dataset subsets.
23

---

Published as a conference paper at ICLR 2025
[Audio-video information]:
This is the detailed caption of video. {...): and the meta-information: (...). This is the detalled caption of the
audio: (...) and the meta-information: (...). These above contain inportantiffomaatinn information that needs to be read carefully.*
INPUT:
[Audio-video information]
You have to finish the following two tasks. First, carefully compare the information typically included for audio versus video. Second, Identify information that is included for audio but not for video, and generate QA pair for each. Similarty, identify information that is included for video but not for audio, and generate QA pair for each.
Each pair should simulate dialogue that focuses on various aspects of the content without introducing information not explicitly mentioned. The purpose of this exercise is to fine- tume language model so that it can generate accurate and relevant responses.
QA pair should start with Q:" contain instruction related to audio and video, while secund line should start with "A:" and provide response the instruction.
Please ensure that your instructions diverse and high quality. Avoid making assumptions or adding details that explicitly given.
EXAMPLE:
Q: W kitchen tools or utensils are prominently featured in the video during pancake preparation? A: The video showcases basic kitchen tools such as mixing bowl, whisk, spatula, and frying pan, demonstrating their use in making pancakes.
Discuss overall tone and style the English audio commentary. How it emhance the tutorial? The commentary's friendly and instructive tone makes pancake making feee appruachable and enjoyable, encouraging viewers to try the recipe themselves.
Neeaaivee
INPUT:
[Audio-video information]
The information provided for audio and video does not fully match. Find out what the audio and video information does not match. Create negative question-answer pairs where the question asks for information not included in the provided data, Provide negative responses or rejections indicating the information is unavailable. Don't design questions that can be answered with yes or no.
The QA pair should wiah Q:" and contain an instruction related to the audio and video, w hile the second line should start with "A:" and provide response to the instruction.
EXAMPLE:
Q: Can you describe the sound of the frog the audio?
A: The color of the frog is not mentioned in the audio. The background audio is funny music.
Figure 10: Prompt templates for generation of AVU-dataset subsets.
24

---

Published as a conference paper at ICLR 2025
[Video caption information]:
This the detailed caption of the video: (...). These above contain important information that needs read carefully."
Meta-Information
INPUT:
[Video caption information]
You are an Al assistant for video information integration. will provide you with caption about video, and please extract some information from including "suhject"(the character the picture). and the location), "action" (the movements the main character the viden), and "emation" (the emotion of the subject the video). If the information not available or nol explicilly mentioned, please outpul the words "None". Avoid making assumptions or adding details that are not explicilly given.
"scene"(the
the vidco). "lecation" (the location of the video. Not geographical
EXAMPLE:
Original video caption: The video shows large crowd people gathered stadium, with some individuals holding flags cheering. The flags are positioned in the foreground, while crowd in the background. The people the crowd are standing and cheering, with individunls holding up their hands. The stadium is filled with people, and the atmosphere is lively and energetie.
Meta-information:
Subject: Large crowd of people, individulls with flags
Scene: Stadium
Location: Stadium with large crowd
Action: Individunls holding flags and checring, crowd standing and checring, some individuals bolding up their hands
Emotion: Lively and energetic almosphere
[Audio caption information]:
This the detailed caption of the audio: [...]. These above contain important information that needs to be read carefully..
Mcta-Inforrmttin
INPUT:
[Video caption information]
You Al assistant audio information intogration. will provide you with caption about audio, and please extract some meta information from it, including "subject"(the main character in the audío). "scene"(the scene and the events in the audio), "location" (the exact location of the audio. Not geographical location). "action" (the movements of the main character in the audio), and "emotion" (the emotion of the subject or the audio). the making assumptions or adding details that explicitly given.
available
explicitly mentioned, please output the words "None", Avoid
EXAMPLE:
speaking clear confident She deseribing dealership and the various cars that are available. She mentions the different fentures of each car and the benefits of purchasing from the dcalership. The music is playing in the background.
Original vidco caption:
Meta-information:
Subject:
Scene: dealership description
Location: None
Action: Speaking in a clear and confident tone, describing cars and their features
Emotion: Confident
Background: Music playing
Figure 11: Prompt templates for generation of meta- meta-information.
25


Reviews:
[{"review_id": "STHK26Shnv", "reviewer": "Reviewer_pdCN", "summary": "This paper introduces a new audio-visual LLM model called Dolphin and a new audio-visual dataset AVU. The authors discuss the existing problem with video LLMs, which is, how they often ignore the audio information present in the video and only attend to the visual information while understanding videos. The authors claim that the models do not learn any alignment between the audio and visual information in the video, which is the reason for this behavior of video LLMs. Hence the authors design the Dolphin model, which aligns the audio and visual information both spatially and temporally before feeding them to the LLM. Specifically, they use multi-scale vision transformers to extract visual features at different scales and apply cross-attention with audio features at each scale. These\nfeatures are again merged with the global visual representation using another cross-attention. Then temporal cross-attention is applied between these features bi-directionally to obtain visual-contextualized audio tokens and audio-contextualized visual tokens. This is fed to the LLM for the downstream task.\n\nSince most existing video datasets focus mainly on visual content, the authors have introduced a new audio-visual dataset by using existing unimodal datasets and leveraging LLMs to generate modality-specific question-answer pairs. They generate different types of questions and answers based on metadata correspondence of the audio and visual inputs by prompting LLMs. The experiments are designed to test the new model architecture on existing video QA datasets and other unimodal tasks such as captioning and classification.", "questions": "1.  What are the effects of using pre-trained models to create a pipeline for various captioning and QA creation steps? What if any of the models hallucinated? Was there some kind of quality check done?\n\n2. I am intrigued by some of the examples of the dataset that has absolute time information such as \"What time does the train whistle blow?\" and the model providing an answer. Do these models understand the concept of time and seconds?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem addressed by the authors is an important one. Most video-related datasets and models indeed ignore the information present in the audio almost completely. Hence this work is an important one to fill this research gap.\n\n2. The proposed model architecture achieves better results on existing video QA datasets and the ablation studies show the importance of spatial and temporal alignment layers introduced in the architecture.\n\n3. The dataset is large-scale and can be significant to the community to advance audio-visual understanding. \n\n4. The usefulness of the dataset is shown by comparing video llama trained with and without the AVU dataset.", "weaknesses": "1. The entire pipeline in the dataset generation is LLM-based. There are no discussions about the efficiency of the pipeline, hallucination effects, or error propagation in the dataset creation process.\n\n2. The authors claim in a lot of places in the paper that there is a significant reduction in hallucinations using their model and dataset. They design an AVU-negatives subset to train the model to say no to some questions.  However, the experiments are not designed to validate this claim in any manner. While Dolphin may outperform certain models, it is unclear whether the hallucination is reduced as there are no metrics or definitions to evaluate this. It is a tall claim without any experimental results to say that hallucinations are reduced. \n\n3. Minor comment: Clotho-V2 which was is used as a dataset for training is not referenced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3eIpeo7To6", "reviewer": "Reviewer_JTeq", "summary": "The authors propose an audio-visual LLM Dolphin, which consists of a multi-scale adapter for spatial alignment and an interleaved merging module for temporal alignment. A large-scale audio-visual caption&instruction-tuning dataset AVU is also proposed, including 5.2M video-audio-qa tuples. Training on the proposed dataset, the proposed method achieves state-of-the-art performance on several audio-visual, audio, and video benchmarks compared with existing audio-visual LLMs.", "questions": "1. Considering the selected audio and visual encoders are far smaller than the LLM (ViT-L and AST), why not directly train these encoders to achieve better performance since the 7b/13b LLM is also involved in training in the instruction-tuning stage? \n\n2. Why select ViT-L, AST, and Vicuna as encoders and decoders when tons of more powerful alternatives are available (such as SigLIP, InternViT for image, Beats, Whisper encoder for audio, and Qwen, llama3, mistral for LLM)? Is there any ablation?\n\n3. Why not use some video encoders to perform visual encoding both for the Dolphin model and the data curation pipeline? Is there any ablation?  \n\n4. For the temporal integration, how does the proposed bi-directional cross-attention block 'enhance the audio-visual information exploitation of AV-LLM' as the author claims? What I see is just an attention block to perform cross-modal interaction for global features, yet how to model the temporal relationships, is positional encoding or RoPE being used? How to inject the so-called 'temporal integration information' into the dual-path framework? The descriptions are too vague and need to be improved.  \n\n5. What is the connector between the audio/visual encoder and LLM decoder? Q-former or linear projection? Is there any ablation?  \n\n6. How does the model tackle uni-modal tasks since the fine-grained alignment seems to be mandatory? For videos that missing the auditory part, will a modality mask perform on the input of the LLM decoder and the cross-modality integration module (both spatial and temporal)? For videos with semantic-irrelevant auditory parts, how does the model resist the potential negative information brought by the auditory modality? \n\n7. For the experiments, the authors only compare the proposed method with audio-visual LLMs, how much is the performance gap between the proposed AV-LLM with some uni-modal models?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The curation process of the AVU dataset looks sound and reasonable. The authors integrate several open-source and commercial LLMs into the data pipeline to generate high-quality audio-visual captions and divide the dataset into several parts based on audio-visual consistency. The community now is facing a shortage of a large-scale audio-visual instruction-tuning dataset. The proposed dataset, along with the data curation procedure, will help the following research in the related field.\n\n2. The results show the proposed method outperforms several previous audio-visual LLM on audio, video, and audio-visual benchmarks.  Apart from caption and question-answering, it also excels in some closed and open-ended audio tasks, which makes the framework more applicable.\n\n3. The ablations are comprehensive. Each component is well-ablated and clearly verified. The authors also conduct numerical analysis on the impact of the proposed dataset.", "weaknesses": "1. The method is trivial and questionable. The entire framework consists of three parts: audio and visual encoders with injected multi-scale uni-modal and multi-modal adapters, a cross-modal attention block to perform temporal integration, and a Vicuna as the decoder. The audio-visual adapters and the cross-modal attention have been proposed and utilized in many previous works[1-3], and the pipeline of training an audio-visual LLM is also not novel. The data pipeline for generating audio-visual captions is also been utilized by several previous methods[4-5]. Besides, the description of the model architecture is vague, many details are missing and the rationale of some model designs is unclear. Please see the question part below in detail.  \n\n2.  Speech is neglected in the model architecture designs. Since the audio feature is semantic and high-level, while the speech feature is low-level and dense, it is a common way to model the audio and speech separately via different encoders, such as [4, 6]. Besides, how does the proposed model outperform baseline methods on the speech recognition task as shown in Table 3 when no speech encoder or dense feature is involved? What does the model perform when compared with some speech-centric models?\n\n3. The application scenarios are limited. It seems that the proposed method is only suitable for audio-visual correspondence videos since the training dataset is constructed by at least medium-level AV consistency videos, while the low-level AV consistency data is used for negative samples, yet 1). how to decide whether an in-the-wild video is suitable for the model to infer? and 2). what is the purpose of aligning audio and visual encoders using high AV consistency videos? I believe the alignment stage is more likely to align the audio and visual encoder with the text decoder rather than align the audio encoder with the visual encoder. What will happen if videos with low AV consistency are introduced for training?\n\n4. Audio-visual capabilities are not fully probed. Some audio-visual tasks are not tested, such as audio-visual caption, audio-visual speech recognition, and audio-visual sound source detection as the previous method [6] does. I suggest the authors conduct experiments on these benchmarks and compare the proposed method with [6] to show the model's capability more comprehensively.\n\nReference:  \n\n[1] Lin, Yan-Bo, et al. \"Vision transformers are parameter-efficient audio-visual learners.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.  \n[2] Tian, Yapeng, Dingzeyu Li, and Chenliang Xu. \"Unified multisensory perception: Weakly-supervised audio-visual video parsing.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer International Publishing, 2020.  \n[3] Li, Guangyao, et al. \"Learning to answer questions in dynamic audio-visual scenarios.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.  \n[4] Chen, Sihan, et al. \"Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.\" Advances in Neural Information Processing Systems 36 (2023): 72842-72866.  \n[5] Wang, Yi, et al. \"Internvideo2: Scaling video foundation models for multimodal video understanding.\" arXiv preprint arXiv:2403.15377 (2024).  \n[6] Sun, Guangzhi, et al. \"video-SALMONN: Speech-enhanced audio-visual large language models.\" arXiv preprint arXiv:2406.15704 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h95h4QJ3jF", "reviewer": "Reviewer_gDzU", "summary": "This paper discusses the importance of audio-visual large language models (AV-LLMs) in multimodal video understanding, with a particular emphasis on the use of audio information. The paper proposes a fine-grained AV-LLM model called Dolphin, which ensures comprehensive and accurate video understanding by aligning audio and video in both spatial and temporal dimensions. To better define the task, this work proposed a related dataset(AVU) and benchmark(AVU-Bench), that contains 5.2 million diverse data pairs (video, audio, questions, answers), and a novel data partitioning strategy is introduced. Experimental results show that Dolphin performs well in audio-visual understanding and effectively reduce hallucinations.", "questions": "Please refer to the weakness. \nOverall, I think this article is quite comprehensive, but in this era of a large number of LLM works, I think this work needs to be supplemented with more comparisons to prove that this work is novel enough to be published in ICLR.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is soundness. The author put forward a fine grand alignment method, adding visual tokens audio, and special temporal Temporal tokens to achieve better alignment.\nThe.\n\nThis paper put forward a comprehensive dataset with a promising data processing pipeline and obtained large-scale data.\n\nThe paper gives a benchmark based on the task definition and its dataset and compares the baseline methods.\n\nExtensive experiments demonstrate that Dolphin significantly improves audio-visual comprehension and is effective in reducing errors related to audio neglect.", "weaknesses": "1. The experiment is comprehensive but the baseline is weak. The method mentioned VideoLLAMA2, but the experiment seems only to compare the result with VideoLlaMA1. Adding more comparisons against these baselines would be more persuasive.\n\n2. The author mentioned that AVU could reduce the hallucination; while the related analysis is not included in the experiments. \n\n3. The meaning of “fine-grained spatial modeling” lack of definition. Please provide a clear definition or explanation of \"fine-grained spatial modeling\" in the context of their work.\n\n4. Although the author compares video and audio captions separately, more experiments on other audio-visual datasets are expected.\nMany any-to-any models can have a visual-audio understanding ability. What is their performance on the given tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1RjI3wL8zH", "reviewer": "Reviewer_f89q", "summary": "This paper investigates the capabilities of audio-visual large language models (AV-LLMs) to enhance their reasoning and understanding capabilities. As existing AV-LLMs tend to neglect audio information, this paper addresses the issue from two perspectives: model architecture and dataset. For model architecture, the authors enhance both spatial and temporal alignment of AV-LLMs by proposing an audio-visual multi-scale adapter for aggregating multi-scale information and proposing audio-visual interleaved merging, respectively. For the dataset, this paper proposes a large-scale caption & instructional dataset using existing audio-visual data sources. Experimental results show that the proposed model achieves favorable performance in both audio-visual understanding and audio understanding tasks.", "questions": "- L32: Does the audio modality prove crucial for comprehensive understanding? Could you substantiate this claim?\n- Does reason (3) on L72 contradict the starting paragraph of the introduction, where the authors assert that audio is crucial for video understanding? Could the authors provide examples of when audio is crucial versus when it may be less informative than visual data?\n- In Table 2 and Table 3, did Dolphin use unimodal signal as an input, or use both of multimodal signal for unimodal task?\n- L67: It appears that the model trained with audio converted to text performs favorably. How would the model perform with video + audio (converted to text)? Could this combination outperform the Dolphin model? Could the authors conduct this experiment?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The motivation of this work, which identifies the weaknesses of AV-LLMs and aims to solve them from two different perspectives, is a sound approach to advancing research in this area.\n- The approach to enhancing spatial and temporal alignment in audio-visual LLMs is innovative.\n- Constructing an audio-visual caption and instructional dataset is beneficial for researchers, as there is a lack of such datasets.", "weaknesses": "**The main text requires further refinement. It contains typos, broken sentences, and inconsistent tenses. The reviewer has identified only some of these issues:**\n - L67: VideoLlama is mentioned twice.\n - L158: \"clap\" should be checked for correctness.\n - L231: \"detailed\" is misspelled as \"detrailed.\"\n - L414 contains a broken sentence.\n - The right figure in Figure 1 is not explained in the main text.\n - There is a typo in the right figure of Figure 2, \"his audio.\"\n - The text should use \"/citep\" for citations.\n\n**The reviewer is concerned about the reliability of the dataset. Since the paper proposes a large-scale dataset, it should include a more detailed explanation, such as dataset statistics. The reviewer points out some missing or problematic aspects that lessen the dataset's reliability:**\n- The prompt templates for constructing the meta information are not provided. These prompts are crucial as they differentiate dataset types and help manage noise in this automatically generated dataset.\n- In Figure 6, AVU-specific, although the questions differ, the answers are identical.\n- In Figure 9, the question asks about the sound of a frog, yet the answer discusses an unrelated aspect of color, highlighting the dataset's noisiness.\n- To address concerns about the dataset's reliability and its claim as a benchmark, human verification of the dataset is necessary. If the dataset is noisy, researchers might hesitate to use it for evaluating models.\n\n**The comparison experiments are not thoroughly conducted. Since the paper focuses on improving the audio-visual understanding of AV-LLMs, it should include comparisons with existing high-performing AV-LLMs. Here are several models that the paper should have considered:**\n- FAVOR: https://arxiv.org/pdf/2310.05863\n- video-Salmon: https://arxiv.org/pdf/2406.15704\n- PandaGPT:https://arxiv.org/abs/2305.16355\n- OneLLM: https://arxiv.org/pdf/2312.03700\n\n**The reliability of the model's design and training is questionable. The inconsistencies and errors in the paper amplify these concerns:**\n- The notations in Figure 2 and the main text differ, making it hard to understand the model's mechanism.\n- What does the superscript “i” stand for in all notations? And what is the difference from the superscript “1” in L178?\n- In Figure 1, how does the Dolphin model recognize the words a man says using the ImageBind audio encoder? Doesn't the ImageBind audio encoder take environmental sound as an input, not speech?\n- In L430, the authors mention that AST was used, but do not explain how they trained or integrated this model.\n- Table 6 not explained in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)