WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM
Authors: Changli Tang
Qinfan Xiao
Ke Mei
Tianyi Wang
Fengyun Rao
Chao Zhang
Venue: ICLR 2026 Oral (2026)
Keywords: audio-visual embeddings
multimodal LLMs
video retrieval
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper builds a versatile audio-visual embedding LLM, which can not only achieve any-to-any retrieval but also generate prompt-aware embeddings.

Abstract: While embeddings from multimodal large language models (LLMs) excel as general-purpose representations, their application to dynamic modalities like audio and video remains underexplored. We introduce WAVE (\textbf{u}nified \& \textbf{v}ersatile \textbf{a}udio-\textbf{v}isual \textbf{e}mbeddings), the first LLM-based embedding that creates a unified representation space for text, audio, and video modalities. WAVE employs a novel hierarchical feature fusion strategy and a joint multi-modal, multi-task training approach to enable two key capabilities: any-to-any cross-modal retrieval and the generation of prompt-aware embeddings tailored to user instructions. Experimentally, WAVE sets a new state-of-the-art on the MMEB-v2 video benchmark and achieves superior results in audio and video-to-audio retrieval. Its prompt-aware nature also yields remarkable performance in multimodal question answering, significantly outperforming existing embedding models. Ablation studies validate our joint training strategy, demonstrating improved performance across all modalities.  With a newly introduced benchmark for versatile audio-visual learning, WAVE opens up broad possibilities for cross-modal, any-to-any applications. Our code and checkpoints are released at \href{https://github.com/TCL606/WAVE}{https://github.com/TCL606/WAVE}.

Published as a conference paper at ICLR 2026
WAVE: LEARNING UNIFIED & VERSATILE AUDIO- VISUAL EMBEDDINGS WITH MULTIMODAL LLM
Changli Tang', Qinfan Xiao', Ke Mei?, Tianyi Wang', Fengyun Rao", Chao Zhang't Tsinghua University', WeChat Vision, Tencent Inc. tc1240mails tsinghua. edu. en, c22778tsinghua.co en
ABSTRACT
While embeddings from multimodal large language models (LLMs) excel as general-purposs representations, their application to dynamic modalities like au- dio and video remains underexplored. We introduce WAVE (unified & versatile audio-visual embeddings), the first LLM--based embedding that creates a unified representation space for text, audio, and video modalities. WAVE employs novel hierarchical feature fusion strategy and joint multi-modal, multi-task training approach to enable two key capabilities: any-to-any cross-modee retrieval and the generation of prompt-aware embeddings tailored to user instructions. Experimen- tally, WAVE sets a new state-of state-of-the-ate the-art on the MMEB-v2 video benchmark and achieves superior results in audio and video-to-audio retrieval. Its prompt-aware nature also yields remarkable performance in multimodal question answering, significantly outperforming existing embedding models. Ablation studies vali- date our joint training strategy, demonstrating improved performance across all modalities. With newly introduced benchmark for versatile audio-visual learn- ing, WAVE opens up broad possibilities for mooss- any-to-any applications. Our code and checkpoints are released
INTRODUCTION
Multimodal embeddings, which transform diverse data types such as text, images, video, and audio into shared representation space, are central cross-modal search, classification, and recom- mendation. The prevailing approach employs separate encoders per modality that are aligned in common space (Radford et al. [222T] Jia et al.) 2021 Zhai et al. [223]] Ma et al. [202]]
2022|Miech Miech al.
2020 Xu et al., 2021 Elizalde et al. 2023 Mei et al. 2024] Guzhov et al., 2022] Su et al.) 2024 success large language models (LLMs)
McKee et al. 2023 Chen al., 2024b). Recently,
has catalysed more integrated paradigm: using single multimodal LLM (MLLM) to produce embeddings for all modalities jointly. This shift is enabled by increasingly capable MLLMs that can process and reason
images (Liu al. 2024b Li al. 2023 Chen et al. 2023), audio TTng
2024 2023 Chu et al. 2024). and video (Li et al. 2025a Zhang et al. 2025c Liu al.) 2025b Zhu et al. 2025 Bai al. 2025 Li al. 2025b] Zhang et al. 2025a Xu
ett al.] 2024] Gong et al.
et al. 2025 Tang et al. 2025). Consequently, the field is rapidly moving toward using these models
produce potent, versatile multimodal embeddings (Tiang et al. 2025 Meng al. 2025 Yu al.;
2025a Zhang et al. 2024 Jiang et al. 2024 Gu et al. 2025 Lin ct al. 2025 Liu et al. (025a)..
A unified embedding paradigm built upon MLLMs fully leverages their strengths in semantic un- derstanding and representation. By processing all modalities within a single model, this approach naturally improves cross- cross-modal modal interoperability and semantic alignment, which benefits downstream tasks such as retrieval. Such a model can also ingest multiple modalities concurrently to form holis- tie representations-
example, a coherent embedding from paired audio and video streams. Fur- thermore, by inheriting the instruction-follwwiin capabilities of MILMs, the resulting embeddings can be prompt-aware, conditioning on user instructions to encode task-relevant semantics. Despite these advantages, most MLLM-based embedding efforts have concentrated vision, particularly static images, while underexploring audio and synchronised audio-visual streams. Consequently, the promise of truly universal audio-visual embedding space remains largely unrealised.
i Corresponding author

---

Published as a conference paper at ICLR 2026
To address these limitations, we introduce WAVE, unified & versatile audio vistall embedding MLLM. To the best our knowledge, WAVE is the first model to produce unified embeddings for text, audio, silent video, and synchronised audio-visual inputs. Built Qwen2.5-Omni (Xu et al. 2025), WAVE projects heterogeneous inputs into a shared semantic space, enabling seamless cross-modal interaction. Experiments confirm that WAVE produces powerful embeddings, achiev- ing state-of-the-art (SOTA) performance on the MMEB-v2 video track (Meng et al. (0255) and excelling tasks like any-to-any retrieval (e.g., text-to-video, video-to-audio). Moreover, can generate prompt-aware embeddings for downstream applications like multimodal question answer- ing (QA). Crucially, WAVE maintains or even surpasses the performance of the base Qwen2.5-0mni on multimodal understanding benchmarks, which is notable since most embedding models show significant decline in these capabilities compared to their foundational MILMs.
Our main contributions can be summarised as follows:
Versatile audio-visual embedding MLLM:
We introduce WAVE, the first audio-visual em- bedding MILM capable of producing unified, general-purpose representations for text, audio, silent video, and synchronised audio-visual inputs. By projecting heterogeneous modalities into a single semantic space, WAVE excels at challenging any-to-any retrieval and achieves SOTA performance on the MMEB-v2 video track.
Instruction-following for prompt-aware embeddings: Leveraging the instruction-follooinn ability of its MILMM backbone, WAVE generates prompt-aware multimodal embeddings. Unlike conventional models that yield task-agnostic representations, WAVE can condition embeddings on a user's 's task-specific prompt, which reflected in its strong results on embedding- mul- timodal QA.
Effective architecture: We propose a hierarchical feature-fusion strategy that aggregates repre- sentations from multiple MLLM layers, yielding stable gains on tasks such as multimodal re- trieval. In addition, dual-encoder design for audio captures complementary cues (e.g., speech and environmental sounds), further enhancing the expressiveness of the learned embeddings.
2 BACKGROUND
2.1 MULTIMODAL REPRESENTATION LEARNING
Multimodal representation learning seeks to construct a shared embedding space in which text, image, audio, and video can be compared and composed. major milestone is CLIP (Radford) et al. 202T). which uses contrastive learning with dual encoders to align images and text at scale. Building on this paradigm, ALIGN (Jia et al. 202I) shows that training on even larger, noisier corpora, exceeding a billion image-text pairs, yields strong gains on retrieval and classification. SigLIP (Zhai al. 2023) further simplifies and scales training by replacing the standard InfoNCE objective with a sigmoid loss, removing the need for in-batch negatives and improving efficiency.
This contrastive recipe naturally extends to video. X-CLIP (M
2022) adapts the dual-encoder design to vidco-text retrieval with explicit temporal modelling. To better exploit large but noisy web videos, Miech al. (2020) enhance contrastive learning with noise-contrassive estimation, enabling learning from loosely aligned narration. VideoCLIP (Xu et al., (021) strengthens discrimination by mining hard negatives via nearest-neighhour retrieval during training.
In audio-language learning, CLAP (Elizalde et al.) 2023) aligns audio and text in joint space, enabling zero-shot audio classification and cross-modal retrieval. To address the scarcity of high- quality paired data, Mei et al.] (2024) introduce the WavCaps corpus and build state-of-the-arr dio-lang retrieval models with HTSAT (Chen et al.) 2022a) and BERT (Devlin et al. 2019).
As audio and vision are naturally synchronised and complementary, learning unified audio-visual representations is an important next step. AudioCLIP (Guzhov et al., (022) generalises CLIP to a trimodal setting (audio, image, text), enabling richer cross-modal transfer. (Su et al. (2024) propose a unified framework for audio-visual representation and generation, and subsequent work explores emerging applications such video-to-musé retrieval (McKee et al. 2023 Chen al. 2024b).
2

---

Published as a conference paper at ICLR 2026
2.2 LLM-BASED EMBEDDING MODELS
Pretrained on vast corpora, LLMs exhibit strong semantic understanding and broad world knowl- edge, motivating their use as text-embedding generators. Two common strategies adapt decoder- only LLMs into embedding models: last-token pooling, which takes the hidden state of the end-of- sentence (EOS) token the sentence embedding, and mean pooling. which averages token-level hidden states. Using last-token pooling. Wang et al. (2024) first synthesize training data with propri- etary LLMs and then fine-tune target models with standard contrastive objective, yielding compet- itive text embeddings without complex pipelines. NV-Embed (I.ee et al. 2025a) removes the causal attention mask and introduces a latent attention layer to improve mean pooling, More recently, Gem- ini Embedding (Lec et al.) 2025b), Qwen3 Embedding (Zhang et al. 2025b), and Yichou-Embeddingg (Yu et al. 2025b) have set new marks on comprehensive benchmarks such as MTEB (Muennighofl al. 2022 through large-scale training.
In the multimodal setting, researchers extend MLLMs to carve out unified semantic space across modalities, aiming produce robust, general-purpose embeddings with single model. VeM2Vec (Jiang et al. 2025) an early effort that trains a visual LLM across diverse multimodal embed- ding tasks; VLM2Vec-V2 (Meng et al.) 2025) broadens coverage video and documents. /Zhang ett al. (2024) focus on multimodal retrieval, building an MI.LM-based universal retriever for im- ages and text, while E5-V (Jiang
2024) shows that training on text pairs alone can still im- prove image-text retrieval. MM-Embed (Lin et al. (2055) adopts an image-LLM bi-encoder with modality-awar hard nard-negative mining mitigate modality bias. Gu al. (2025) combine textual discriminative distillation with multimodal contrastive learning to construct
image embedding
LLM. LamRA (Liu et al.) 2025a) offers a general framework that equips visual LLMs with strong retrieval and re-ranking, and CAFe (Yu et al. 2025a) unifies visual representation learning and gen- cration via satrrastive-uttorgrrsss retrieval and image generation.
ane-ttnnng scheme, enabling a single model to excel at both
METHODS
3.1
MODEL ARCHITECTURE
Last Token
If: Only
Classification Retrieval
LLM Layer N
Token
L.LM Layer 2
Last Toker
Otherwise
Fusion Module
LLM Layer
Text-Only
Visual-Oni
Visual input token
Audio-Only
Speech input token
Audio-
Audio input token
Visual Encoder
Speech Encader
Audio Encoder
Text Embedding
Text input token
LLM jth layer output token
8tll
b
Deseribe the video. [BOS]
Multimodal embedding
Video
Speech Audio Event
Prompt (Text)
Figure Inputs can be text-only, vision-only, audio-only, audio-visual. For text-only cases, the final embeddings are obtained via last-token pooling over the LLM'' last hidden states. For multimodal inputs, the last output tokens from all LLM layers are concatenated and passed cnsture-fusion module to produce unified multimodal embedding. Note that text prompts are al- ways provided to instruct the LLM for multimodal inputs.

---

Published as a conference paper at ICLR 2026
The overall architecture of WAVE is shown in Fig. I The model can accept text, video frames, audio signals, or synchronised audio-visual data as input and generate multimodal embeddings for downstream tasks, such as classification, retrieval, and QA.
To handle this modality diversity, WAVE employs distinct encoders for non-text inputs. A pre- trained visual encoder extracts features from video frames, converting them into visual tokens for the LLM. For audio, we utilise dual-encoder architecture to comprehensively capture the input signal. A speech encoder and a separate audio encoder generate speech-related and audio event- related tokens, respectively. Text inputs are tokenised using the LLM's original embedding layer. Crucially, all non-text inputs are accompanied by a text prompt, which serves as an instruction to the LLM.
To structure the multimodal input tokens for the LLM, we employ specific interleaving strategies. For audio-only input, the speech-related and audio event-related tokens, which are equal in number due to identical encoder frequencies, are interleaved on one-to me-too-ne basis to form unified auditory token sequence. For synchronised audio-visual input, both the visual and auditory token sequences are partitioned into several segments corresponding to the number of sampled frames. These seg- ments are then interleaved to create the andio-visual token sequence. Finally, the text tokens of the prompt are appended to the end to form the input token sequence for the LLM.
There are four possible input configurations: text-only, visual-only, audio-only, and audio-visual. Among them, audio and video are both multimodal temporal signals. To enhance the LLM's ability to capture spatiotemporal structure, ding (TMRoPE) introduced in Qwen2.5-Omni (Xu et al.) 2025). Because the speech and audio en- coders are synchronised produce outputs at the same frequency, their tokens are naturally aligned in time. Tokens corresponding to the same frame, therefore, share the same TMRoPE, ensuring precise temporal alignment.
adopt the time-aligned multimodal rotary position Embed-
After TMRoPE is applied, the multimodal token sequence is fed into the LLM. Inspired by the use of the hidden state of the final "EOS" token by last-token pooling, we aggregate the last output tokens from multiple LLM layers to construct embeddings for non-text modalities. This design captures both low-level perceptual cues and high-level semantic abstractions. A lightweight fusion mod- ule ins-immlleeeted as two-layer multi-layer perceptron (MLP) with GELU activation (Hendrycks) & Gimpel, 2016) is then used to refine and compress the embeddings. For the text-only scenario, we retain the standard last-token pooling approach, which prior studies have shown to be highly effective.
3.2 TRAINING STRATEGY
Similar to previous work, we adopt contrastive learning as the primary training paradigm to align representations from different modalities into unified embedding space. The semantic similarity between any two embeddings is quantified using the cosine similarity metric.
Our training regimen is composed of two distinct but complementary tasks: multimodal retrieval and QA. The retrieval task requires the model to extract general multimodal embeddings with general prompt like "Describe the video", while the QA task requires the model to extract prompt- aware embeddings that well interpret the given question. During training, each sample provides a source-target pair of inputs, denoted as (s, which are processed by the model to produce their respective embeddings, and The training is performed on mini-batches of size N, and we build task-aware data sampler that ensures samples in the mini-batch belong to the same task.
Retrieval Task: For the retrieval task, and t belong different modality types, which can be any the supported formats: text-only, audio-only, visual-only, or audio-visual. This allows for arbitrary any-to-any cross-modal training. We employ bn-batch negative sampling strategy, where for a given positive pair, all other non-corresponding pairs within the mini-batch are treated as negative samples. To ensure robust alignment, we compute symmetric InfoNCE loss.
Specifically, for the th sample in the and its target embedding et, is the positive key, the loss La; is formulated as cross-entropy loss
when its source embedding es, serves as the query
4

---

Published as a conference paper at ICLR 2026
over the batch:
Lx; - - log exp(sim(es: et,)/T)
(I)
jj=1 exp(sim(e,, et, )/T)
where sim(:,) denotes the cosine similarity, is the temperature parameter, and the summation in the denominator is over all target embeddings ét, in the mini-batch. For the retrieval task, the source and target are interchangeable. Therefore, the symmetrical scenario is also considered, where et, serves as the query and ex; is the positive key. The symmetrical loss Lt, is:
C, - log exp(sim(er,, Cx;)/T)
Σj=1 exp exp(sim(er, Cx, )/T)
The final retrieval loss for the entire mini-batch is the average of these individual losses over all samples and both directions, ensuring a bidirectional alignment:
CRetrieval
Σ (Co, + Ct,). 2N
1
(3)
Question Answering Task For the QA task, the source 8; for the i sample in the mini-batch a multimodal signal accompanied by a textual prompt that poses a question. The corresponding 4; text-only input representing the correct answer. To train the model for this discriminative task, we augment each sample with a set of n incorrect or "distractor" answers, denoted as { ) The model then extracts embeddings for the correct answer, Ct2, and for each incorrect answer, {dr,, M==1- The objective is to maximise the probability of selecting the correct answer from the candidate pool. The QA loss for the i th sample, Lor,, thus formulated as cross-entropy loss:
(i.ssmm,(,,,,))). + Σ exp(sim (xp(sim((ss,ee../r)
Los, -og log
This objective function effectively trains the model to produce multimodal query embedding 6s, that is most similar to the embedding of the correct textual answer ex,, while being distant from the embeddings of incorrect answers. The total QA loss for the batch is the average of individual losses:
CoA =
Σ LOA
(5)
N
4 EXPERIMENTAL SETTINGS
4.1 MODEL. SPECIFICATIONS
WAVE is built on the 7 billion (B) parameter version of Qwen2.5 -mmii (Xu et al. 2025). Specif- ically, the LLM backbone, the visual encoder, and the speech encoder are all initialised from the pre-trained weights Qwen2.5-Omni, which allows WAVE to inherit the powerful multimodal per- ception and reasoning capabilities of the foundation model. For the dedicated audio encoder, we adopt BEATs encoder (Chen et al.] 2022b) and further append a trainable aligner to align its output with the LLM's input space. The aligner consists of two-layer MLP projector with GELU tivation function. To ensure efficient fine-tuning, we employ the low-rank adaptation (LoRA) (Hu al., 2022) technique on the LLM backbone. The LoRA modules are configured with rank of 128 and scaling factor of 2.0. A dropout rate of 0.05 is applied to the LoRA modules during training to mitigate overfitting. To generate the ultimate multimodal embeddings, two-layer MI. with GELU activation function serves the fusion module to fuse features from different layers of the LLM. The temperature r of the model set to 0.01.
As for the model input, videos, in general, are sampled 2 frames per second, with maximum of 128 frames sampled. For videos longer than 64 seconds, 128 frames are uniformly sampled to conserve memory resources. The maximum resolution for each frame is 176400 pixels. For audio input, the waveform signals are resampled to 16,000 Hz. Other preprocessing settings are identical to those of Qwen2.5-Omni.
5

---

Published as a conference paper at ICLR 2026
4.2 TRAINING SPECIFICATIONS
Before the large-scale contrastive learning, perform dedicated pre- pre-training phase for the BEATs aligner. This stage aligns the BEATs encoder with the backbone LLM so that the LLM can interpret BEATs features. Only the aligner's parameters are updated while all other components remain frozen. Given an audio clip and a simple text prompt (for example, 'Please describe the audio'), ) the model is trained to generate descriptive audio caption. Training uses audio from WavCaps (Mei) et al. 2024), AudioCaps (Kim et al. 2019), and Clotho (Drossos et al., 2020); clips longer than 180 seconds discarded avoid memory issues. We train for three epochs on 128 H20 GPUs.
Next, we proceed with the primary training stage as detailed in Section 3.2 Table 2 provides a
comprehensive overview of the multimodal tasks, data sources, the modalities of each sample pair (s. t), and the number of data samples used in our training. Notably, we re-annotate the 1 million (M) videos of the Panda-70M dataset (Chen et al. 2024a) using VL.-2-1L (Chen al., 2024c).
Besides, in some datasets, a video may correspond to multiple text captions. To enhance the diversity of text captions, we construct samples that share the same video but differ in text captions for these datasets. The final WAVE model is trained time is approximately 36 hours. We set the learning rate to 2 x 10-5, and configure a per-device batch size of 1, resulting in total batch size of 192. The data sampler is designed to ensure that samples in each training mini-batel are consistent in task types and data sources. The visual aligner and the LoRA module are trainable in this stage, while other modules will stay frozen. For ablation experiments, the training settings are similar except that we only use 128 H20 GPUs for training.
192 H20 GPUs for one epoch, and the total training
Table 1: An overview of training tasks and data. Four tasks are trained for our models: video-text retrieval, video-QA, video-autio retrieval and audio-text retrieval.
Task
Data Source
Modalities of (s,
# *Samples
Panda-70k (Chen et al.
2024a
(visual, text)
1.0 M
MSVD Chen & Dolan
2011
(visual, text)
24 K
DiDeMo (Anne Hendricks
2017
(visual, text)
8 K
ActivityNet Captions (Knshnna et al. 2017
(visual, text)
Video-Text Retrieval
MSR-VTT (Xu cl al.
2016
(audio-visual, text)
180 K
VATEX /Wang ct al. 2019
(audio-visual, text)
260 K
YouCook2 (Zhou et al. 2018
(audio-visual) text)
10 K
Shot2Story HHan et al. 2023
(audio-visure text)
530 K
Video-QA
-LaVA-Vidco-178 178k (Zhang et al.
2025c
(visual, text)
100 K
AudioSet (Gemmeke al. 2017
(audio, visual)
1.7 M
Video-Audio Retrieval
VGGSound (Chen et al.
2020
(audio, visual)
182 K
AudioCaps (Kim al. 2019
(audio, text)
49 K
Audio-Text Retrieval
AudioSet-SL. (Hershey al
2021
(audio, text)
108
Clotho Drossos et al.
2020
(audio, text)
19
Total
4.9 M
4.3 EVALUATION SPECIFICATIONS
To thoroughly assess the performance of WAVE, we have collected and organised comprehensive suite of evaluation tasks and benchmarks. This collection is designed to systematically measure the quality of the embeddings for each modality and to validate the model's cross-modal alignment capabilities.
Details of the evaluation data and metrics are shown in Table All evaluation tasks are formulated as "query-to-targe target""" retrieval. For video-centric tasks, we adopt the video subset from MMEB-v2 (Meng et al. 20255 as our foundation, and we also augment the evaluations on the recent benchmark LoVR (Cai et al. 2025). In the audio domain, our evaluation encompasses both retrieval and QA tasks as well. Beyond these text-centric scenarios, more challenging tasks such as video-to-audio retrieval and video-to-musis retrieval are evaluated, which test the model's ability map visual semantics to auditory concepts directly within its unified embedding space. More details of the inference procedure for each evaluation task are shown in Appendix B

---

Published as a conference paper at ICLR 2026
Table 2: Details of the evaluation benchmarks. We formulate all tasks as "query-to-targer" retrieval,
Data Source
Task
Subset
(Query, Target) Modalities Metries
Classfication
CLS
(visual, text)
Acc%
Video QA
QA
(visual, text)
Acc%
MMEB-v2-Video
Meng et al.
2025
Retrieval
RET
(text, visual/audio-visual)
R@I
Moment retrieval
MRET
(text, visual)
R@1
text-to-clip
R@1
LoVR Cai et al.
2025
Retrieval
(text, visual)
theme-to-clig
R@25
AudioCaps (Kim et al. Clotho Drossso
2019
Retricval
test
(text, audio)
R@1
2020
test
VGGSound (Chen et al. 2020
Retricval
lest
(visual, audio)
R@1
MusicCaps (Agostinelli
2023
test
MMAU (akkhii et al. 2025
test-mini test
Audio QA
(audio, text)
Acc%
MMAR Ma et al.
5
EXPERIMENTAL RESULTS
5.1
OVERALL RESULTS
The results of our model are shown in Table and Table WAVE demonstrates its capabilities to generate versatile multimodal embeddings, achieving strong performance across a range of video, audio, and audio-visual scenarios. In the video domain, WAVE comprehensively outperforms ex- isting open-source models across all sub-tasks on the video track of MMEB-v2, which systemat- ically evaluates various video understanding capabilities. Notably, the overall performance of our model even surpasses that of the industrial-grant model, Seed- 100---6-Emmmeeee 16-Embedding Furthermore, model also exhibits strong performance on LoVR, not only on caption- text-to-clip retrieval but also on concept-base based theme-to-clip retrieval, leading existing open-source multimodal embedding LLMs.
Table 3: Results of video embedding benchmarks. MMEB-v2 and LoVR.
Models are evaluated on the video track of
MMEB-y2- Video
LoVR
Model
Overall
CLS
QA
RET
MRET
text-to-clip
theme-to-clip
LamRA A 7B (Liu et al. 2025a
35.0
39.3
42.6
24.3
32.8
62.9
60.2
GME 7B (Zhang al. 2024
38.4
37.4
50.4
28.4
37.0
51.2
43.9
CAFe
Yuu el al. 2025a
42.4
35.8
58.7
34.4
39.5
Seed-1 Bood-1.6-Bg 16--mmeeddn
55.3
55.0
60.9
51.3
53.5
WAVE
59.9
57.8
72.5
54.7
50.8
62.9
66.0
Table 4: Results of audio and audio-visual embedding benchmarks. Different tasks are evaluated, including audio retrieval (A-RET), -isiull retrieval (AV-RET) and audio QA (A-QA).
A-RET
AV-RET
A-QA
Method
AudioCaps
Clotho
VGGSound
MusicCaps
MMAU
MMAR
Reference Model
Mei al. 42.2
2024
encoder-only retrieval model (ours)
Qwen2.5-Ommn 7B
Reference Value
21.5
10.3
8.6
71.5
56.7
WAVE
25.6
25.0
20.4
76.6
68.1
In the audio domain, on the widely used AudioCaps and Clotho datasets, WAVE achieves superior audio retrieval performance compared to previous models that rely separate-encoder architec- tures. Moreover, as unified multimodal embedding model, WAVE is also capable video-to-audio retrieval, more challenging task that directly bypasses the text modality. For fair comparison,

---

Published as a conference paper at ICLR 2026
train encoder-only retrieval model (columns 4 and 5 in Table 4 using the same video-to-aue audio retrieval data, where video embeddings are extracted by WAVE's visual encoder and audio embed- dings extracted by WAVE's speech and audio encoders. The results in Table show show that WAVE considerably outperforms the encoder-only retrieval model on audio-visual retrieval, not only on the in-domain VGGSound test but also the out-of-domain video-to-man tose-c MusicCaps data.
More evaluation results are shown in Appendix C
5.2
ANALYSIS OF PROMPT-AWARE EMBEDDINGS
Beyond retrieval, WAVE leverages its L.M backhone's reasoning to produce prompt-aware embed- dings conditioned on textual instructions (see Appendix D for a case study). The ability to follow instructions crucial for embedding MLLMs in QA tasks, which can be shown by the following question example from Video-MME (Fu et al.
2025): Which of the following features/items is not discussed in the video in relation to the tomb? A. Inkstone. B. Niche. C. Jade. D. Sacrificial table.
Models that cannot understand the input question may probably produce embedding that repre- sents the main content of the video, which will lead to incorrect predictions. We list the detailed QA results in Table To investigate the extent which text prompts contribute WAVE's ability to generate embeddings, we also instructed WAVE with a common prompt ("Please describe the video") when testing, instead of separate questions.
Table 5: Results of different models MMEB-v2 video QA data, including Video-MME (Fu et al. 2025 MVBench (Li et al. 2024). NEXT-QA (Xiao et al., [202]) EgoSchema (Mangalam et al.,
2023 and ActivityNerQA Yu et al. 2019) In the case of "w/ separate questions" each question is
used as different prompt.
MMEB-12-Video QA
Model
Average
Video-MME MVBench NEXT-QA EgoSchema ActivityNelQA
LamRA
42.6
34.1
37.2
43.7
44.8
53.2
GME
50.4
39.2
46.6
53.6
46.8
65.6
CAFe 7B
58.7
46.0
48.9
62.4
60.0
76.0
Seed-1.6 101--.6-EEm 6-EEmeedding
60.9
54.0
53.3
66.2
52.2
78.6
WAVE 7B, w/ common prompt
51.8
39.3
44.7
53.5
61.4
60.2
WAVE 7B, w/ separate questions
72.5
63.4
69.6
82.6
66.2
80.9
Compared with existing embedding MILMs, WAVE outperforms strong baselines when testing with separate questions, averaging about 12% higher than Sodd--.6-Embeddig However, using general prompts to extract embeddings Icads to drastic performance degradation across all QA datasets. This stark contrast not only highlights the strong instructiion-oollvvvii capability of WAVE, but also suggests the critical limitation of single, static representation for complex tasks like multimodal QA.
On audio-reasoning benchmarks, WAVE further surpasses its base, Qwen2.5-Omni model, Ta- ble [ shows. This is notable given that WAVE was trained only to generate question-conditionel embeddings for video QA. This cross-modal transfer underscores robust generalisation and supports the hypothesis that WAVE learns a unified, modality-agnostic embedding space.
5.3 BENEFIT OF JOINT MULTI-MODAL, MULTI-TASK TRAINING
core hypothesis behind our unified model that joint training across diverse modalities and tasks fosters more robust and powerful universal embedding space. We posit that learning from audio, video, and text data simultaneously enables positive knowledge transfer, where insights from one modality can enhance the understanding of another. To verify this, conducted ablation study comparing our fully-trained WAVE model against specialist models trained on modality-specine subsets of the data.
Specifically, using the data described in Table we train models under the following three task settings: training video-text retrieval and video-QA. QA, training audio-text retrieval, and training video- audio retrieval. Each model trained on only one fixed pair of modalities, without mixing data from other modalities. Then test the three separately trained models video, audio, and audio-visun
8

---

Published as a conference paper at ICLR 2026
benchmarks, respectively. The results shown in Table denoted "Separate". The final WAVE model jointly trained across all modalities is also reported for comparison, denoted as "Joint".
Table 6: Comparison of model performance under separate vs. joint training schemes. The model jointly trained on all modalities and tasks consistently outperforms specialist models trained on separate modality-tast pairs.
MMEB--2- Video
A-RET
AV-RET
Training
Overall
CT.S
QA
RET
MRET
AudioCaps
Clotho
VGGSound
MusicCaps
Separate 58.2
57.5
71.6
56.1
47.6
42.5
24.0
24.9
20.1
Joint
59.0
57.8
72.5
54.7
50.8
44.2
25.6
25.0
20.4
As shown in Table [ the model trained jointly across modalities outperforms separately trained specialist models on seven of eight tasks, indicating positive cross-modal knowledge transfer. Ex- posure to richer, more diverse signals encourages learning generalised, modality- seman- tic representations rather than modality-specifii features, underscoring the promise of model for general-purpose embedding extraction.
5.4 ANALYSIS OF FEATURE FUSION
When using an LLM for embedding extraction, a common choice last-token pooling, which takes the EOS token's is hidden state from the final layer as the sequence representation. However, Gou ct al. (2025) observe, different LLM layers specialise in distinct functions for video understanding, implying complementary information distributed across depth. Accordingly, aggregate signals from all layers to form the final embedding, preserving both low-level perceptual cues and high-level semantic reasoning. Concretely, collect the last-token states from every layer, concatenate them, and feed them to lightweight fusion module to produce the output embedding.
To efficiently assess beabeedding-extractinnn strategies while conserving compute, we conduct an ex- panded ablation primarily on pure-visual (no-audio) video retrieval, with an additional check in the audio-visual setting. Evaluations use the MMEB-v2 video-retrieval split. Beyond the two main strategies, 1) standard last-token pooling from the final LLM layer and 2) our all-layer last-token MLP fusion, we also test 3) the last-token output from the first layer, 4) the last-token from a middle layer (Layer 15), and 5) a learnable weighted sum (Peters et al. 2018) of last-token features across all layers. The L.M has twenty-eight layers in total. Results are shown in Table
Table 7: Results of embedding extraction methods on the MMEB-v2 video retrieval data, including MSR-VTT, VATEX, MSVD, DiDeMo, and YouCook2. Note that videos in MSR-VTT, VATEX, and YouCook2 are paired with audio. "V" and "A+V" refer to visual-only and audio-visual. respectively.
-iddee RET
Method
Modality
Average
MSR-VTI VTT
VATEX MSVD DiDeMo YouCook2
Last token pooling (first layer)
38.8
44.8
37.3
60.9
39.0
12.0
Last token pooling (middle layer)
45.0
48.9
41.4
67.5
47.3
17.8
Last token pooling (last layer)
49,6
52.1
46.2
69.7
53.0
27.2
All-layer last token weighted sum
48.3
49.4
45.6
69.4
50.6
26.3
All-layer last token MI.P fusion
50.5
53.6
47.5
68.7
55.4
27.3
Last token pooling (last layer)
54.7
58.2
56.3
69.3
54.8
34.9
A+V
All-layer last token MI.P fusion
56.1
58.5
58.4
69.3
57.4
36.8
As shown in Table 7 using only first-layer or middle-layer features yields marked drop versus the final-layer representation, consistent with a hierarchical abstraction in which the top layer carries the most semantically relevant information for retrieval. Early-layer cues are still useful, however: fusing last-token features from all layers with a small MLP consistently surpasses the strong last- layer baseline. By contrast, direct weighted cross-layer interactions for video tasks are complex and non- non-linear, and thus benefit from learned transformations. The pattern holds in the audio-visual setting, our all-layer MLP fusion again out-
across layers underperforms, suggesting that

---

Published as a conference paper at ICLR 2026
performs last-layer pooling, and Table further shows that audio substantially boosts video retrieval, reinforeing the value of a unified, general- multi modal embedding model.
CONCLUSION
We present WAVE, to our knowledge, the first unified, versatile audio-visual embedding MLLM that maps text, audio, silent video, and synchronised audio-visual inputs into single semantic space. A dual audio-encoder design combined with hierarchical all-layer feature fusion yields robust, com- prehensive multimodal representations. Joint multi-modal, multi-task training enables WAVE to achieve strong results (e.g., on the MMEB-v2 video track) and to generate prompt-aware embed- dings that translate into competitive multimodal QA performance. Ablations confirm the benefits of unification sihiiiinion-soowing positive cross- moos- transfer and the value of learned cross-layer fusion. WAVE establishes a new, powerful baseline for universal audio-visual representation learning and can serve springboard for cross-modal, any-to-any applications.
7
REPRODUCIBILITY STATEMENT
We provide detailed descriptions of the model architecture, training pipelines, training data and hyperparameters in Sections 3.1 3.2 and 4.2 All datasets, code, and model checkpoints will be released. These provide enough reproducibility for our work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text. arXiv preprint arXiv:2301. 1125, 2023.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing Moments in Video with Natural Language. In Proc. ICCV, Venice, 2017.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL 2-L technical report. arXiv preprint a3X9: 2025.
Qifeng Cai, Hao Liang, Hejun Dong, Meiyi Qiang, Ruichuan An, Zhaoyang Han, Zhengzhou Zhu, Bin Cui, and Wentao Zhang. LoVR: benchmark for long video retrieval multimodal contexts. arXiv preprint 1rXxv:2505.13228 2025.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proc. ACL-IILT, Portland, 2011.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. VGGSound: A large-scale audio-visual dataset. In Proc. ICASSP, Barcelona, 2020.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma. Taylor Berg- Kink-atirkttrkk, and Shlomo Dubnov. HTS- AT: A Hierarchical Token-semantic Audio Transformer for Sound Classification and Detection. In Proc. ICASSP, Singapore, 2022a.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058. 2022b.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang- teii Chao, Byung Fun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70M: Captioning 70M videos with multiple cross-modality teachers. In Proc. CVPR, Seattle, 2024a.
Zeyu Chen, Pengfei Zhang, Kai Ye, Wei Dong, Xin Feng. and Yana Zhang. Start from dideo-musis retrieval: An inter-intra modal loss for cross modal retrieval. arXiv preprint arXiv:2407.194/ 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- VL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint 2rXhv:2312.14253 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding Performance Boundaries of Open-source Multimodal Models with Model, Data, and Test-time Scaling, arXiv preprint arX1v:2412.05271, 2024c.
Yunfei Chu, Jin Xu, Qian Yang. Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-Audio Technical Report. arXiv preprint ar arXiv:2407.1075, 2024.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proc. NAACL, Minneapolis, 2019.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In Proc. ICASSP, Barcelona, 2020.
Benjamin Flizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP:: Learning audio concepts from natural language supervision. In Proc. ICASSP, Rhodes Island, 2023.
Chaoyou Fu. Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever comprehensive evalua- tion benchmark of multi-modal llms in video analysis. In Proc. CVPR, Nashville, 2025.
11

---

Published as a conference paper at ICLR 2026
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labelle dataset for audio events. In Proc. ICASSP, New Orleans, 2017.
Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass, Joint audio and speech understanding. In Proc. ASRU, Taipei, 2023.
Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass, Listen, think, and understand. In Proc. ICLR, Vienna, 2024.
Chenhui Gou, Ziyu Ma. Zicheng Duan, Haoyu He, Feng Chen, Akide Liu, Bohan Zhuang, Jianfei Cai, and Hamid Rezatofighi. An empirical study on how video-LLMs answer video questions. arXiv preprint M.Xiv:2508./3666 2025.
Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal em- bedding learning with multimodal LLMs. In Proc. ACM MM, Dublin, 2025.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. AudioCLIP: Extending CLIP to image, text and audio. In Proc. ICASSP, Singapore, 2022.
Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. Shot2Story20K: new benchmark for comprehensive understanding of multi-shot videos. arXiv preprint arXiv:2311.17043, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv: 08415, 2016.
Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit temporally-strong labels in audio event classification. In Proc. ICASSP, Toronto, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In Proc. ICLR, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. Proc. ICMI., 2021.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-V: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12588 2024.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VLM2Vec: Training Vision-language Models for Massive Multimodal Embedding Tasks. In Proc. ICLR, Singapore, 2025.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. AudioCaps: Generating Captions for Audios in The Wild. In Proc. NAACI-HLT, Minneapolis, 2019.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-Captioning Events in Videos. In Proc. ICCV, Venice, 2017.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. NV-Embed: Improved techniques for training LLMs as generalist embedding models. In Proc. ICLR, Singapore, 2025a.
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer. Madhuri Shanbhogue, Iftekhar Naim, Gus- tavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, al. Gemini embed- ding: Generalizable embeddings from gemini. arXiv preprint arXiv:2503.0789/ 2025b.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-One Onvei-onnisioo:: Easy visual task transfer. Transactions on Machine Learning Research, 2025a.
12

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- language-image pre-training with frozen image encoders and large language models. In Proc. ICML, Honolulu, 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. MVBench: A comprehensive multi-modal video understanding benchmark. In Proc. CVPR, Seattle, 2024.
Yixuan Li, Changli Tang, Jimin Zhuang, Yudong Yang. Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. Improving LLM video understanding with 16 frames per second. In Proc. ICML, Vancouver, 2025b.
Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-Embed: Universal multimodal retrieval with multimodal LLMs. In Proc. ICLR, 2025.
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual News: Benchmark and challenges in news image captioning. In Proc. EMNLP, 2021.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proc. CVPR, Seattle, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proc. NeurIPS, Vancouver, 2024b.
Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. LamRA: Large multimodal model as your advanced retrieval assistant. In Proc. CVPR, Nashville, 2025a.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang. Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. NVILA: Efficient frontier visual language models. In Proc. CVPR, Nashville, 2025b.
Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Zhang, and Rongrong Ji. X-CLIP: End-to-end multi-grained contrastive learning for video-text retrieval. In Proc. ACM MM, Lisbon, 2022.
Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang. Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al. MMAR: A challenging benchmark for deep reason- ing in speech, audio, music, and their mix. arXiv preprint arXiv:2505.13032, 2025.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A diagnostic bench- mark for very long-form video language understanding. Proc. NIPS, 2023.
Daniel McKee, Justin Salamon, Josef Sivic, and Bryan Russell. Language-guided music recommen- dation for video via prompt analogies. In Proc. CVPR, Vancouver, 2023.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumb- ley, Yuexian Zou, and Wenwu Wang. WavCaps: ChaiGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3339-323 -354, 2024.
Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang. Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, et al. VLM2Vec- V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. arXiv preprint arXiv:2507.04500 045500 2025.
Antoine Miech, Jean Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisser- man. End-to-end learning of visual representations from uncurated instructional videos. In Proc. CVPR, Seattle, 2020.
Niklas Muennighoff, Nouamane Tazi, Loie Magne, and Nils Reimers. MTEB: Massive text embed- ding benchmark. In Proc. EACL. Dubrovnik, 2022.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. NAACL-HLT, New Orleans, 2018.
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. ICML, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. ICML, Honolulu, 2023.
S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: A massive multi-task audio understanding and reasoning benchmark. In Proc. ICLR, Singapore, 2025.
Kun Su, Xiulong Liu, and Eli Shlizerman. From vision to audio and beyond: A unified model for audio-visual representation and generation. In Proc. ICML, Vienna, 2024.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In Proc. ICLR, Vienna, 2024.
Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang. Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang.
2: Captioning-enhan enhanced audio-visual large language models. arXiv preprint arXiv:2506.1522 2025.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. In Proc. ACL, Bangkok, 2024.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VATEX: A large-scale, high-quality multilingual dataset for undeo-and-language research. In Proc. ICCV, Seoul, 2019.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NEXT-QA: Next phase of question- answering to explaining temporal actions. In Proc. CVPR, 2021.
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In Proc. EMNLP, Punta Cana, 2021.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, al. Qwen2.5-0mni technical report. arXiv preprint arXiv:2503.202155 2025.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In Proc. CVPR, Las Vegas, 2016.
Hao Yu, Zhuokai Zhao, Shen Yan, Lukasz Korycki, Jianyu Wang, Baosheng He, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, and Hanchao Yu. CAFe: Unifying representation and generation with contrastive-autorn autorreesveu finetuning. arXiv preprint arXiv:2503. 19900, 2025a.
Peng Yu, En Xu, Bin Chen, Haibiao Chen, and Yinfei Xu. Q7hou-Emmeddiing technical report. arXiv preprint arXiv: 2632, 2025b.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet QA: A dataset for understanding complex web videos via question answering. In Proc. AAAI, Hawaii, 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- pre-tnnin In Proc. ICCV, Paris, 2023.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding. arXiv preprint 1xXi:25001131000 2025a.
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. GME: Improving universal multimodal retrieval by multi- modal llms. arxiv preprint arXiv:2412. 16855, 2024.
14

---

Published as a conference paper at ICLR 2026
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang. Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.051 057766 2025b.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. Transactions on Machine Learning Research, 2025c.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proc. AAAI, New Orleans, 2018.
Jinguo Zhu, Weiyun Wang. The Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. InternVL33: Exploring advanced training and test-time recipes for open-souree multimodal models. arXiv preprint arXiv:2504.10474 2025.
A THE USE OF LARGE LANGUAGE MODELS
We used Gemini-2.5-Pro to help check for grammar emors and polish the fluency of our sentences.
B INFERENCE PROCEDURE FOR EVALUATION TASKS
The evaluation tasks can also be divided into two categories: retrieval tasks and QA tasks. Tasks that use a single, unified general prompt to extract embeddings for all test samples are regarded as retrieval tasks. This includes the classification, retrieval, and moment retrieval subsets of MMEB-v2- Video. Conversely, tasks that use separate and specific questions as prompts to extract embeddings considered QA tasks.
For all retrieval tasks, we generate an embedding for each sample in the test set based on its input video/audio, using the fixed text prompt, "Please describe the video/audio." If the task involves trieving text, the corresponding ground-truth text captions for each sample are also used to generate text embeddings. The entire test set then forms the candidate pool for the retrieval task. The candi- date with the highest similarity score to the query embedding is selected as the model's prediction.
For QA tasks, all QA evaluations for the embedding LLMS are performed using embedding- methodology. To be specific, the inference procedure is as follows:
The model first generates a single embedding that is conditioned on both the source modality (video/audio) and the provided question text.
2. Separately, the model generates embedding for the text of each answer option.
The similarity between the question-conditionel video/audio embedding and each of the option text embeddings is then calculated.
4. The option with the highest similarity score is selected as the model's predicted answer.
C MORE EVALUATION RESULTS OF WAVE
In the retrieval task of MMEB-v2-Vicee Videeo,, only one direction, text-to-video retrieval, was evaluated. Similarly, for the audio retrieval and audio-visual retrieval tasks presented in Table [ only a sin- gle direction assessed, i.c. text-to-audio and video-to-audio retrieval, respectively. This focus was chosen because these directions more closely align with practical, real-world application sce- narios like multimodal search and recommendation. However, it is undeniable that performance in the reverse direction is also important. In Table 88 we provide supplementary results for video re- trieval (V-RET), audio retrieval (A-RET), and audio-visual retrieval (AV-RET) in the other direction. WAVE can also achieve competitive results in the other direction.
D CASE STUDY OF PROMPT-AWARE EMBEDDINGS
To provide a more intuitive and qualitative demonstration of WAVE's prompt-aware embedding ca- pability, conduct study single video. We select video from the MSR-VTT test set,
15

---

Published as a conference paper at ICLR 2026
Table 8: Results of video-to-text, audio-to-text and audio-to-video retrieval. Corresponding refer- ence models and their scores are also provided.
V-RET
A-RET
AV-RET
Method
MSR-VTT
VATEX
YouCook2
Clotho
VGGSound
Reference Model
GME 7B Zhang et al. 2024
Mei et al.
2024
encoder-only retrieval model (ours)
Reference Value
32.2
33.0
10.0
27.1
8.2
WAVE 7B
55.1
55.4
29.8
24.5
25.8
v1
0.4473
0.3711
0.3867
0.4121
Video Embeddings
Prompts of Video Embedding 1-4 (V1-4)
0.40
V1
Please describe the video.
V2
0.3516
0.3555
0.2969
0.3145
V2
What animal is the video?
V3
What sound
heard in the background?
0.35
V4
Who Is speaking?
V3
0.3301
0.2578
0.3438
0.3184
Text Embedding 1-4 (T1-4)
0.30
and woman walking dog beach
T2 dog
0.3203
0.2676
0.2988
0.3281
wind
T4 man
Text Embeddings
Figure 2: heatmap visualizing the cosine similarity between video embeddings (VI-V4) and text embeddings (T1-T4). All four video embeddings are generated from the same video but conditioned on different textual prompts. The text embeddings represent various concepts present in the video.
which shows man and a woman walking dog on beach, with wind blowing the background. We then generated four distinct embeddings (V1-V4) for this single video conditioned on different prompts, ranging from general description request to specific questions about visual animals, back- ground sounds, speakers. Concurrently, generated four text embeddings (T1-T4) representing a general description and the specific concepts of "dog", "wind", and "man".
The cosine similarities between these video and text embeddings visualised in the heatmap in Figure 2 The general prompt VI ("Please describe the video.") yields an embedding that has the highest similarity (0.4473) with the general text description TI. In addition, video embeddings conditioned on prompts about specific aspects of the video (V2, V3, V4) also show high similarity T1, indicating that these prompt-aware embeddings still retain the overall semantic context of the video. In addition, video embeddings generated from specific prompts are clearly biased towards the textual representation of that specific feature. For instance, the embedding V2, generated by the prompt "What animal is in the video?", has a slightly higher similarity than T1 and a significantly higher similarity with T2 ("dog") than with T3 ("wind"") or T4 ("man""). Similarly, the audio-focused V3 aligns best with "wind" (T3), and the speaker- speaker-focused V4 matches most closely with "man" (T4). This clearly demonstrates that WAVE can dynamically shift the semantic focus of its output embedding to produce representation that is precisely tailored to the user's query.
E ANALYSIS OF DUAL SPEECH & AUDIO ENCODERS
Speech and general audio events are both crucial elements within an audio signal. Our base model, Qwen2.5-Omni, possesses some capability to process general audio, but its encoder for audio pro- cessing is derived from Whisper (Radford et al. 2023), model optimised for automatic speech recognition. This speech encoder, which is primarily specialised for modelling speech, has an insuf- ficient understanding of non-speech audio events. To address this limitation, augment the existing speech encoder with dedicated audio encoder, BEATs (Chen et al. 2022b), which designed for comprehensive audio event understanding.
16

---

Published as a conference paper at ICLR 2026
To validate the effectiveness of this dual-encoder approach, compared its performance video retrieval, audio retrieval, and audio-visual retrieval, respectively, with that of using only the original speech encoder. For video retrieval, whose videos are paired with audio. The results are presented in Table 2 The dual-encoder con- figuration consistently outperforms the single speech encoder both the audio retrieval and audio- visual retrieval, and achieves comparable or better performance on video retrieval benchmarks. This indicates that the speech and audio encoders can complement each other further enhance the model's 's ability to interpret both speech and environmental sounds.
test the models MSR- VTT. VATEX, and YouCook2,
Table 9: Results of using dual speech and audio encoders and using speech encoder only. Video retrieval (V-RET), audio retrieval (A-RET), and audio-visua retrieval (AV-RET) are evaluated here.
V-RET
A-RET
AV-RET
Method
MSR -TT VATEX YouCook2 AudioCaps Clotho VGGSound MusicCaps
Single speech encader
58.6
56.6
34.3
39.6
22.4
23.3
18.3
Dual speech & audio encoders
58.5
58.4
36.8
42.5
24.0
24.9
20.1
F THE EFFECT OF IMAGE TRAINING
WAVE is primarily trained on video data and achieves strong performance video retrieval-related tasks. However, this success not due to a narrow specialization in video. fact, WAVE is not specialized only for video retrieval, and image data actually helps. In preliminary study, compared the results of training solely on video data against training on a mixture video data and nearly equal amount of image data from the MMEB-vl training This experiment utilized the same pure-visual video retrieval setup as described in Section 5.4 this comparison.
Table I0 presents the results of
Table 10: Comparison of results for training with and without image data. We evaluate text-to- image retrieval (Liu et al. 2021 on the VisualNews dataset and text-to-video retrieval on MMEB- v2-Video
Image RET
MMEB-r2-Video RET
Training Data
VisualNews
Average MSR- VIS VATEX MSVD DiDeMo YouCook2
Video Data
54.9
49.6
52.1
46.2
69.7
53.0
27.2
Video Data + Image Data (MMEB-v1)
74.8
50.2
52.4
46.2
69.0
55.8
27.8
the results show, including image data substantially improves image retrieval and provides slight gain in video retrieval. This shows that WAVE's strong performance is not due to avoiding image tasks or specializing exclusively video.
17


Reviews:
[{"review_id": "aWv00WS7sg", "reviewer": "Reviewer_zEZy", "summary": "The paper introduces Wave, a unified multimodal embedding model capable of embedding text, audio, video and audio-video inputs. Wave is validated on a variety of video, audio and text retrieval tasks. Furthermore, Wave is capable of robustly using instructions to improve multimodal embedding performence.", "questions": "Several question can be found in the above weaknesses section, in addition:\n- How were the “distractor“ answers used during QA training created?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Introducing audio to multimodal retrieval is relatively novel. The performance of the model is quite strong.\n- Using a concatenation of many layers last token hidden is a novel method for vector embedding in this setting.\n- The manuscript is clear and well-written.", "weaknesses": "- The approach is very similar to [1] except an omni-model is used instead of a VLM.\n\n- The paper is missing implementation details relating to model architecture, including how inputs are templated into LLM backbone, resolution and visual tokenization in the vision encoder.\n\n- The paper does not compare their proposed pooling strategy with mean pooling with bidirectional attention, which tends to outperform last-token pooling strategies [2].\n\n- The evaluations of audio-visual retrieval is limited. The reviewer can only find a single task that tests WAVE’s ability to retrieve audio-visual items (RET split of MMEB-v2-Video), and only in one direction. Furthermore, It is unclear if MMEB-v2-Video is designed to have audio used its retrieval task. This could result in the audio track trivializing the task or being useless.\n\n- It is unclear whether WAVE’s superior video retrieval performance comes from its different techniques or larger scale video retrieval training or its specialization into video retrieval (I.e. no image retrieval).\n\nMinor:\n- Potentially missing related work: the use of “distractor” answers for QA training seems very similar to [3].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJ8quF6mwR", "reviewer": "Reviewer_mAPG", "summary": "This paper introduces WAVE, a LLM-based embedding model specifically designed to create a unified representation space for text, audio, silent video, and synchronized audio-visual inputs. \n\nTo achieve this versatility, WAVE employs a hierarchical feature fusion strategy that aggregates representations from multiple LLM layers, alongside a dual-encoder architecture for audio inputs. \n\nThe model is optimized through a joint multi-modal, multi-task training approach to enable any-to-any cross-modal retrieval and the generation of prompt-aware embeddings that condition on user instructions for tasks like multimodal QA. \n\nExperimentally, WAVE outperforms other baselines on the MMEB-v2 video benchmark and yields descent results in audio and video-to-audio retrieval, with ablation studies confirming the performance benefits derived from both joint training and the learned cross-layer fusion technique.", "questions": "### 1. Analysis of Prompt-Aware Embeddings and Generalization\n\n*  Could the authors elaborate on the fundamental limitations preventing a single, static embedding from adequately capturing complex query semantics for QA? Does this performance gap imply that for high-level reasoning tasks, WAVE’s LLM backbone is using the prompt to select relevant features internally rather than deriving a truly universal, task-agnostic representation?\n\n*  For users who need a generalized embedding for downstream applications that lack explicit questions, what is the optimal and computationally lightest prompt recommended by the authors that can minimize performance loss while maintaining acceptable semantic coverage?\n\n### 2. Feature Fusion and Interpretability\n\n* Given that a direct weighted sum underperforms, suggesting that cross-layer interactions are complex and non-linear, can the authors provide deeper insights into what the two-layer MLP fusion module learns? Are there any visualization techniques or analysis that can illustrate the relative importance assigned to low-level (early-layer) perceptual cues versus high-level (late-layer) semantic reasoning during the fusion process?\n\n### 3. Dual-Encoder Necessity and Redundancy\n\n* Since BEATs is designed for comprehensive audio event understanding, did the authors explore replacing the existing speech encoder entirely with a second, possibly smaller, instance of the BEATs encoder, or fine-tuning a single, unified audio encoder? If the dual approach is mandated by specialized roles, how do the embeddings from the two encoders contribute uniquely to the final unified representation, beyond the observed performance boost in audio retrieval?\n\n### 4. Inference and Computational Cost\n\n*  While the training resources are impressive (192 H20 GPUs for 36 hours), could the authors provide a comparison of the average latency and total inference computation (FLOPs or time per sample) required to generate a WAVE embedding versus competing MLLM-based embedding models (e.g., LamRA or CAFe)? Specifically, how much overhead is introduced by processing and fusing features from all layers compared to standard last-token pooling from only the final layer?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The writing is easy to follow.\n\n*  The architecture features an effective hierarchical feature fusion strategy by aggregating representations across multiple LLM layers and a dual-encoder design for audio. Ablation studies confirm that the proposed joint multi-modal, multi-task training strategy enables positive cross-modal knowledge transfer and superior results compared to specialist models.\n\n*  WAVE achieves new state-of-the-art results on the MMEB-v2 video benchmark and shows superior performance in audio and video-to-audio retrieval compared to strong baselines.", "weaknesses": "1.  A major limitation is the model's reliance on prompt-aware embeddings for high performance in complex tasks like multimodal QA. Using a single general prompt to extract the embedding results in a drastic performance degradation across all QA benchmarks, highlighting the critical limitation of a single, static representation in handling complex query semantics. Although boosting performance, generating diverse embeddings can be very expensive in real-world scenarios. Consider the model that needs to generate different embeddings based on various user inputs.\n\n\n2.  Achieving optimal performance requires a complex, learned MLP fusion across the last-token outputs of all LLM layers. Ablation studies showed that simpler aggregation methods, such as a direct weighted sum across layers, underperform, suggesting the necessary cross-layer interactions are highly complex. This leaves readers wondering whether all the layers were necessary, or if there are other strategies to select meaningful layers that would be sufficient.\n\n3.   As an LLM-based model built on a 7B parameter backbone, Qwen2.5-Omni, the training demands are significant, requiring large-scale infrastructure (192 H20 GPUs for approximately 36 hours), which is hard for many labs to reproduce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QSBuMCBMLk", "reviewer": "Reviewer_gqQX", "summary": "This paper proposes a method to fine-tune a multimodal large language model (MLLM) to produce prompt-aware embeddings for audio and video. Starting from Qwen2.5-Omni, the authors first learn a BEATs-based adapter to ingest an additional audio input. They then apply contrastive training with LoRA using two objectives: bidirectional cross-modal retrieval and a QA objective that treats the source modality to text as retrieval. Text embeddings are taken from the last token of the last layer. For non-text modalities, the method conditions on a prompt, aggregates the last tokens from all layers, and passes them through a fusion module. Experiments show improved retrieval performance over other MLLMs, and the fine-tuning also improves the base MLLM’s text generation task performance.", "questions": "1. What is the architecture of the BEATs aligner, and is it similar to the fusion module?\n2. How do text prompts vary across training tasks and evaluation tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The task formulation and model design are clear and sensible. The experiments are comprehensive and the results strongly support the efficiency of the proposed method. The presentation is clear.", "weaknesses": "In the evaluation benchmarks, retrieval tasks appear to be reported in a single direction only, for example text-to-audio on Clotho. Reporting both directions would provide a fuller picture.\n\nThe use of the term “QA” is potentially confusing. During training, the QA objective is a source-modality-to-text contrastive setup, while in evaluation some QA tasks are executed via text generation with the MLLM. The same ambiguity appears for other evaluation benchmarks. It would help to specify, for each evaluation task, the exact inference procedure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)