OmniCVR: A Benchmark for Omni-Composed Video Retrieval with Vision, Audio, and Text
Authors: Junyang Ji
Shengjun Zhang
Da Li
Yuxiao Luo
Yan Wang
Di Xu
Biao Yang
Wei Yuan
Fan Yang
Zhihai He
Wenming Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Composed Video Retrieval; Multimodal Benchmark; Audio-Visual Queries
Primary area: datasets and benchmarks

Abstract: Composed video retrieval presents a complex challenge: retrieving a target video based on a source video and a textual modification instruction. This task demands fine-grained reasoning over multimodal transformations. However, existing benchmarks predominantly focus on vision–text alignment, largely overlooking the rich semantic signals embedded in audio—such as speech, music, and environmental sounds—which are often decisive for comprehensive video understanding. To bridge this gap, we introduce **OmniCVR**, a large-scale benchmark for omni-composed video retrieval that establishes vision, audio, and text as first-class modalities. OmniCVR is constructed via a scalable, automated pipeline integrating content-aware segmentation, omni-modal annotation, and a rigorous dual-validation protocol involving both large language models and human experts. The benchmark comprises vision-centric, audio-centric, and integrated queries, with the latter forming the majority to accurately reflect real-world multimodal complexity. Furthermore, we propose **AudioVLM2Vec**, an audio-aware extension of VLM2Vec. By incorporating explicit audio semantics, AudioVLM2Vec achieves state-of-the-art performance, highlighting fundamental limitations in the audio reasoning capabilities of current multimodal retrieval systems.

Published as a conference paper at ICLR 2026
OMNICVR: A BENCHMARK FOR OMNI-COMOSEED VIDEO RETRIEVAL WITH VISION, AUDIO, AND TEXT
Junyang ; Shengjun Zhang', Da Li Yuxiao Luo Yan Wang', Di Xu², Biao Yang²,
Wei Yuan Fan Yang"! Zhihai ! Wenming Yang'!
'Tsinghua University. "Kuaishou Technology, 'Southern University of Science and Technology. ' UUniversity of Chinese Academy of Sciences, *Peking University hezh@suse susttchh. edu. cn yang wenmi cenming@sze www.inngttnn.sssiiiie...... www.innguaauuuu...
Audio-Centric
Integrated
Vision-Centric
4/1-
Modification Text: the and interview on post events. Replace and cantered visuals that shift the convensation teward future zurolling Capocira, possbbiitics- gymmastics moving from recollection to forwand looking ciscussion.
Modification Text: Maintain the dise golf' seone and the player's action. Change the player's shirt color from yollow whitellght gray and slightly adjust the dogs' positions to center of video.
Modification Test: Mointain the graffti artist, the wall, and the hupp-llppa soundrack genre. Advance art from an carly, moatly while outline to nearly finished, detailed piece with and
new madio
stase
Replice the malio to depict fister-pazed sequence of throws.
Figure 1: Overview of the OmniCVR Benchmark.
ABSTRACT
Composed video retrieval presents complex challenge: retrieving a target video based on a source video and a textual modification instruction. This task de- mands fine-grained reasoning over multimodal transformations. However, existing benchmarks predominantly focus on vision-text alignment, largely overlooking the rich semantic signals embedded in audio-such speech, music, and environ- mental sounds-which are often decisive for comprehensive video understanding. To bridge this gap. introduce OmniCVR, large-scale benchmark for omni- composed video retrieval that establishes vision, audio, and text as first-class modalities. OmniCVR constructed via a scalable, automated pipeline integrating content-aware segmentation, omni-modal annotation, and rigorous dual-validation protocol involving both large language models and human experts. The bench- mark comprises vision-centri audio-centric, and integrated queries, with the latter forming the majority to accurately reflect real-world multimodal complexity. Fur- thermore, we propose VodiioVLM2ee. an audio-aware extension of VLM2 Vec. By incorporating explicit audio semantics, AudioVLM2 Vec achieves state-of- the-art performance, highlighting fundamental limitations in the audio reasoning capabilities of current multimodal retrieval systems. Our data will be available htttps://bugqin:
co/ httddaatssetete Jun- Yang/ OmnicVR
INTRODUCTION
Video has established itself as the dominant medium for global communication, education, and entertainment, precipitating an exponential growth in digital video content. This deluge has neces- sitated retrieval systems capable of searching vast repositories with both accuracy and efficiency.
Work done during internship at Kuaishou Technology.
Project leaders.
1 "orresponniig authors.

---

Published as a conference paper at ICLR 2026
While early content-based based retrieval depended on low-level visual features, the advent of large-scale
llnguaee models (Zhan et al. [024] Kelly et al. 2024] Peng et al. (024) has revolutionized the field, enabling robust text-to-viee viido retrieval (Radford et al. (021). Foundational benchmarks such as MSR-VTI (Xu et al. 2016), VATEX (Wang et al. 2019). and YouCook2 (Zhou et al. 2018)
have been instrumental in this evolution, pairing extensive video collections with natural language captions to advance vidco-language alignment.
Recently, the paradigm has shifted toward composed video retrieval (CoVR), which requires models to retrieve target video given source video and a specific textual modification instruction (Thawakar al. 2024, Hummel al., 2024] Thawakarr et al., (025). This formulation demands not only visual grounding but also fine-rained compositional reasoning for instance, "retrieve the same cooking scene but with a different ingredient." Such benchmarks have successfully pushed the boundary from simple retrieval toward complex reasoning tasks.
Despite these advancements, critical limitation remains: existing benchmarks overwhelmingly treat video purely visual-textual medium, neglecting the audio stream. Audio often carries semantic weight equal to or greater than vision; speech conveys intent, background music establishes mood, and environmental sounds define context. A scene depicting "a crowd cheering at a sports arena" is incompletely represented by visuals alone. By ignoring audio, current benchmarks fail to evaluate models in scenarios where auditory information is decisive. Furthermore, existing framework systematically addresses retrieval tasks requiring simultaneous modifications across both vision and audio.
To address this deficiency, we present OmniCVR: the first benchmark for omni-composed video retrieval, treating vision, audio, and text as unified, first-class modalities. OmniCVR introduces large-scale, compositional retrieval tasks spanning three distinct categories, as illustrated in Figure D vision-centric (modifying actions, objects, or scenes), audio-centric (altering music, sound effects, or speech while preserving visual similarity), and integrated (simultancously modifying both modalities). Unlike prior works, integrated queries dominate OmniCVR. reflecting the intricate multimodal nature of real-word world video. The benchmark is constructed via scalable automated pipeline that combines segmentation, omni-modal annotation, and a dull-validation mechanism (utilizing Gemini 2.5 Pro and human experts in an AND-gated protocol) to ensure both breadth and high-quality data.
In summary, our primary contributions are:
1. We introduce OmniCVR, the inaugural large-scale benchmark for omni-modal composed video retrieval, comprising 50K triplets derived from 160K clips and rigorously validated 5K SK-instance gold-standard test set.
2. We propose scalable data generation pipeline integrating content-aware video segmen- tation, omni-modal annotation, and dual validation, yielding high-quality, compositional instructions.
3. We evaluate seven baselines and propose Audio VLM2Vee, which achieves state-of-the-at results on OmniCVR, revealing significant gaps in existing methods regarding audio-centric and compositional reasoning.
2 RELATED WORK
2.1 VIDEO-TEXT RETRIEVAL BENCHMARKS
The cornerstone of modern video retrieval research is the availability of large-scale video-text datasets. MSR- VTT Xu et al. (2016) pioneered this space with 10,000 web video clips, followed by VATEX Wang et al. (2019), which expanded the scale to over 41,000 clips with bilingual captions. Domain-specific datasets such as YouCook2 /hou et al. (2018) (instructional cooking) and Charades Sigurdsson et al. (0016) (indoor activities) further diversified the field. Similarly, recent works like MultiVENT 2.0 Kriz et al. (2025) have scaled retrieval to massive multilingual and event-centric do- mains. While these benchmarks have been instrumental in advancing video- Ladgo-lan understanding, they predominantly focus visual content, largely neglecting the auditory modality. Consequently, they fail to incorporate Composed Video Retrieval scenarios involving audio modifications, such as keeping the visual scene but changing the background music.
2

---

Published as a conference paper at ICLR 2026
2.2 COMPOSED VIDEO RETRIEVAL BENCHMARKS
To transcend simple text-based retrieval, the task of Composed Video Retrieval was introduced (Ven- tura al. 2024 Gupta al. 2025 Yue al. 2025 CoVR tasks model with retrieving target video given a source video (or image) and a textual instruction detailing the desired modification. WebVid-CoVR Thawakar et (2024) established large-scale, synthetic dataset for this purpose. Subsequently.
CoVR Thawakar et al. (2025) utilized GPT-4o to generate more elaborate modification texts. the egocentric domain, EgoCVR Hummel et al. (2024) focused on subtle, temporal, and action-oriented modifications. As detailed in Figure 2 and Table recurrent limitation among these CoVR benchmarks their exclusive focus on visual modifications, leaving auditory dimension unexplored. OmniCVR is the first to introduce compositional queries involving audio changes, such as "change the background music to an upbeat pop track."
Table 1: Comparison of OmniCVR with existing video retrieval benchmarks. OmniCVR is the first explicitly incorporate detailed, searchable audio modality and support composed audio-visual queries.
Anmotation Method
Scale
Task
Audio Modality
MSRRVVTI Xu
2016
Web Videos
Cuwesourced
10K clips
Tex-tt- Video
VATEX Wiin o
Web Videos
Crowdhourced
41K clips
Text-to-Vidoo
Weh Tineakar tf
X124
WebVidtOM
Syntheric (L.MM
420K miplers
Visual Comp.
EgoCVR
Manual
queries
Temporal Comp.
Dense Warva-CavR Iiweakar
2025
WebVid
Syntheti (GPT-44)
samples
Fine- Fine-grained
OmaiCVV (Oues)
Diverse
Generatvee (Qwen-Omii)
160K+ clips
Ouni-Composed V
Input Video
Target Video
WebVid- CoVR
Turn the sea into lake.
EgoCVR
Place it on the table.
Dense-WebVide CoVR
cluster of white greenish grey buckground.
pink and white flowers smoying brorge with Mlrreed
OmniCVR
Maintain the ensiromment, the yellow and blue sled, and the continuous background motor sound. Change the camere perspective from shird-person view following the visual glowed hand holding the handlehar. Replace the original dialogue with new dialague consisting primartly of the word "No" und on initial exclamation.
first-parson point-of-siew
itself, adding
Figure 2: Comparison of OmniCVR with existing video retrieval benchmarks.
2.3 AUDIO-VISUAL LEARNING
Beyond video-text retrieval, there is growing interest in joint vision-audio learning. Tasks such as audio-visual source separation /Afouras et al. (2018) and audio-visual event localization Tian jet al. (2018) demonstrate the efficacy of multimodal fusion. Benchmarks like AV-SUPERB Tseng et al. (2024) evaluate multi-task representations across speech and sound, while recent long-vider benchmarks like MLVU et al. (2025) emphasize the challenges of integrating extended temporal and multimodal information. However, these efforts restricted primarily to classification or grounding; none provide compositional retrieval tasks that explicitly require models to adhere to natural language instructions spanning both audio and vision.
3

---

Published as a conference paper at ICLR 2026
3
THE OMNICVR BENCHMARK
3.1 OVERVIEW OF OMNICVR
We introduce the Omni Composed Video Retrieval Benchmark (OmniCVR), large-scale omni-modal framework designed to evaluate the compositional retrieval capabilities of multimodal foundation models. Unlike prior CoVR benchmarks focused solely on vision, OmniCVR systematically integrates video, audio, and text. Each instance is represented as a triplet: (source video, modification text, target video), where the text specifies the transformation required map the source to the target.
We curated over 50,000 original long-form videos, segmenting them into 160,000 coherent short clips. These clips are annotated and paired to produce 50,000 compositional triplets. From this corpus, a 5, 5,000-trrplet gold-standard test set was selected and manually validated. OmniCVR tasks fall into three categories:
Vision-Centric: Queries focusing modifying actions, objects, scenes.
Audio-Centric: Queries altering music, sound effects, or speech while maintaining high visual similarity.
Integrated: Queries requiring simultancous modifications across both visual and auditory modalities.
This design ensures OmniCVR evaluates not only perceptual grounding but also the capacity to follow complex multimodal modification instructions.
Table 2:
Core statistics of OmniCVR. Integrated queries dominate the benchmark, reflecting the focus on realistic cross-modal modifications.
Statisties
Number
Training Triplets
45k+
Unique Video Clips
160k+
Test Set (gold-standard)
5,000
Query Types (VisionnAudiooonntegrutee
22.82%
20.00% 57.18%
Avg. Query Length
52.6 words
Vocabulary Size
25k+
Avg. Video Length
11.8
Table 2 summarizes the benchmark statistics. Unlike existing datasets, OmniCVR emphasizes integrated queries, which constitute the majority of tasks. This distribution mirrors real-world complexity, where modifications rarely occur in isolation. Vision-centric queries form the second largest cohort, while audio-centric queries are rarer due to strict pairing constraints (high visual similarity, low audio similarity). It is worth noting that the average query length in OmniCVR (52.6 words) is higher than in visual-only CVR benchmarks. This is a deliberate design choice: Integrated queries (57.18% of the dataset) require specifying simultaneous transformations both visual and auditory domains to avoid ambiguity. Unlike prior works that overlook audio, OmniCVR necessitates denser descriptions to capture the full spectrum of multimodal evolution, reflecting the complexity of real-world video editing and retrieval scenarios.
To further demonstrate the semantic breadth of the benchmark, Figure visualizes the hierarchical dis- tribution of content across modalities. Regarding video content (Figure Baj. OmniCVR encompasses four primary domains: Entertainment Events, Instructional & Procedural, Daily Life, Nature & Travel, and Music & Performance. These are structured into granular Culinary Arts to Wilderness Wildlife-to ensure robust coverage of diverse visual scenes. Similarly, the auditory landscape is rigorously balanced, as shown in Figure 3b. We organize audio instances into a two-level taxonomy comprising three high-level classes (Speech, Music, and Sound) branched into 15 distinct subcategories (e.g., Scripted Dialogue, Instrumental Music, and Nature & Animal Sounds). This fine- fine-graine grained categorization highlights the acoustic richness and real- world complexity embedded in our benchmark.
peatteggrrrerrannniii from
4

---

Published as a conference paper at ICLR 2026
Batk Housda
Homes sooue
meng
Sewndd
Cun
OmeiCYR
Audise
Medi & Tee Se Solee
materr Thad
Dake LBb.
Lenove
Wilerwo naME
Nhw Renehwek
(a) Video Categories
(b) Audio Categories
Figure The data diversity of OmniCVR. (a) Illustrates the hierarchical taxonomy of video content, spanning four major domains and their subcategories. (b) Displays the distribution of audio types, categorized into Speech, Music, and Sound with fine-grained subdivisions.
3.2 DATA GENERATION PIPELINE
OmniCVR constructed via a three-stage automated pipeline, depicted in Figure ensuring both scale and high fidelity.
Stage 1: Video Curation and Segmentation. We begin by collecting long-form vidcos (typically ranging from several minutes to multiple hours) from a diverse set of public datasets, including HowTo100M (Miech al. 2019), MSR-VTT, VATEX, and others. These sources diverse domains such as instructional content, daily activities, and entertainment.
From these long-form videos, extract semantically coherent short clips (5-15 seconds, average 11.8 seconds) that serve as the final retrieval units. We employ PySceneDetect to define segments based inter-frame HSV differences minor camera motion. Post-segmentation, we filter clips using two metrics: Action Intensity (optical flow magnitude) and Scene Richness (visual feature variance). Only clips exceeding a combined threshold are retained, ensuring semantic density.
36), effectively capturing scene changes while ignoring
Stage Generative Omni-Modal Annotation. Clips are annotated using Qwen2. 5-0mn which jointly encodes video and audio. To enhance auditory detail, integrate automatic audio transcription. The annotation prompt requests structured descriptions of scenes, actions, objects, and audio events. Specifically for audio, capture the full acoustic spectrum, we enforce strict schema covering para-linguistic features, lexical content, environmental hierarchy, and temporal dynamics (see Appendix G for full details). These dimensions are validated Stage ensure retrieval is grounded in fine-grate lin-ggained audio semantics. Quality is enforced through two-stage verification process: automatic consistency checks with Gemin' ni 2.5 5 Pro and manual expert review.
Stage 3: Triplet Mining for Compositional Retrieval. We generate triplets via three strategies:
Vision-Centric:
Constructed either from (a) different segments of the same long video, or (b) distinct clips from the same video source, ensuring coverage of both coarse-graines and fine-grained visual differences. Audio is preserved to isolate visual reasoning.
Audio-Centric: Candidate pairs are first filtered by requiring high visual similarity (video CLIP cosine similarity 0.9). Among these, pairs with low auditory similarity (audio

---

Published as a conference paper at ICLR 2026
embedding cosine similarity 0.3, measured by the CLAP model) are selected. This guarantees that the visual scene remains constant while audio varies significantly.
Integrated: Pairs are chosen to differ in both modalities, with low similarity in CLIP embeddings (vision) and low similarity in CLAP embeddings (audio).
Modification texts are generated by prompting an LLM with the structured annotations of the source and target, explicitly encoding the relevant differences.
Step1: Video Curation and Segmentation
Clips
Y
Video Source Selection
Split Filter
Step2: Generative Omni-
How 10M
5-15s Action Intensity Scene Richness
YouCook2 Dataset
PySceneDetect
Qwen2. 5mmmi
VATEX MSRVTT
YouTube@M-ussiTexxCCC
Omni-Modal innotationn
Step3: Triplet Mining for Compositional Retrieval
Annotations
CLIP Score
Gemini
Expert
CLAP Score
Gemini2 5 Pro
Review
Input Video
Annotations
Target Video
Annotations
Qwen
Modification Text
Check
Two-stage verification
Figure 4: OmniCVR Benchmark Construction Pipeline.
3.3 EVALUATION AND TEST SET CURATION
We construct gold-standard test set of 5,000 instances using concurrent dual gate protocol. For every candidate triplet, both Gemini 2.5 Pro and human expert independently validate the paired videos and modification text. A triplet is admitted only if both approve. This AND -ated review guarantees semantic fidelity and consistency. The resulting test set preserves the natural query distribution (Integrated Vision Audio), creating realistic evaluation regime.
Table 3: Distribution of audio categories (speech, music, sound) in the source and target videos.
Video Type
Speech (%)
Music (%)
Sound (%)
Source video
56
23
21
Target video
55
23
22
Table Jillustatsss the modality distribution of the audio streams in the source and target videos. We decompose each video's soundtrack into three categories: speech, music, and sound.
EXPERIMENTAL SETUP
4.1 TASKS AND EVALUATION
OmniCVR is designed to rigorously assess compositional retrieval across multiple modalities, fo- cusing on Composed Video Retrieval (CVR). this task, a model given a source video and a natural language instruction and must retrieve the corresponding target video from candidate pool. Unlike traditional video retrieval, which emphasizes semantic similarity, CVR tests a model's 's ability reason transformations like object changes action alterations. Queries categorized

---

Published as a conference paper at ICLR 2026
into three groups: vision-only modifications (focused visual appearance and motion), audio-only modifications (based on acoustic cues like speech or background music), and joint Eision-audie modifications (requiring integrated multimodal reasoning). Performance is measured using standard retrieval metrics-Recall at K (R @ 1, R@5, @@ 10)-which reflect the accuracy of retrieving the correct target from the top candidates. This provides challenging evaluation that mirrors real-world multimodal retrieval tasks.
4.2 BASELINE MODELS
(Ma ct al. 2025 A unified model from Tevatron 2.0 that encodes text, image, audio, and video into a shared space, achieving state-of-the-art performance in modal video retrieval, particularly
Large Multimodal Embedding Models. (i) OmmiEmbed--v.--muileee
the MAGMaR Shared Task (Zhan et al. 2025). after fine-tuning on MultiVENT data (Kriz et al. 2025) with joint vision-audio-text supervision. (ii) VLM2Vee (Jiang et al. 2025): framework that converts large vision- lasoonlanguuae models (e.g., Qwen2-VL (Wang et al. 2024). LLaVA (Liu et al.
2023), Phi-3.5-V)
into universal embedding models through contrastive learning on the MMEB benchmark. VLM2Vec outperforms baselines like CL.IP (Radford et al. 2021) and BLIP (Li et al. 2022). showing 10- 20% improvement in retrieval tasks. (iii) Audio V Vec (Ours): Our extension of VLM22ec that integrates audio semantics by using Qwen2-Audin (Chu et al.) 2024) to generate fine-grained captions of video audio tracks. The captions are combined with user queries and fed into VLM2Vec, enhancing its representation power while focusing audio signals.
Lightweight and Task-Specific Models. (iv) CLIP (Radford et al. 2021): foundational vision- language model trained on large image-text pairs, adapted for video retrieval via frame-level temporal averaging. (v) CoVR (Thawakar et al. (2024): A video retrieval model optimized for Asion-centtic modifications, evaluated for its generalization beyond visual changes. (vi) BLIP (Li et al. 2022): A unified vision-language model adapted for retrieval tasks using its vision-language matching head. (vii) ImageBind (Girdhar al. 2023): A model learning a joint embedding across six modalities, including audio and video, ideal for audio-centric tasks.
LAST
LLM
Projection
Image Encoder
Audio
Description
Modification Text
Pweaz-Audia-7B-Bnutte
Figure 5: The framework Audio VLM2Vec.
Figure illustrates the design of our proposed Audio VLM2Vec model. The framework extends VLM2Vec by explicitly injecting audio semantics into the retrieval pipeline. Given a video, we first encode its visual content using pretrained image encoder followed by lightweight projection layer. In parallel, the audio track is processed by Bren--AAudoo7B-IIntteee natural language description of the acoustic scene. This audio-derived text is concatenated with the user's 's modification query and then fed into the LLM backbone, ensuring that both vision and audio cues are aligned
which generates a fine-grained
shared semantic space. By feeding the transcribed audio semanties alongside visual tokens into the LLM, we leverage the model's 's multi-head self-attention mechanism to jointly process both modalities. This allows the model to learn synergistic and causal relationships (e.g., aligning the text 'lips moving" with corresponding visual tokens) within a shared high-dimensional semantic space. The final multimodal embedding is obtained from the LLM and optimized for retrieval via contrastive learning. By translating audio signals into text and integrating them at the embedding stage, Audio VLM2 Vec effectively grounds compositional queries in both modalities, yielding substantial gains audio-centric and cross-modal retrieval tasks.
7

---

Published as a conference paper at ICLR 2026
4.3 EVALUATION STRATEGY
For each query, compute similarity scores between the query embedding and candidate video embeddings, ranking candidates accordingly. To mitigate potential variance introduced by candidate composition, shuffle candidate pools five times and report averaged metrics. For audio- centric tasks, we additionally control for modality imbalance by ensuring candidate pools always include visually similar but acoustically distinet distractors, and vice versa. This evaluation protocol ensures fairness across different model families while highlighting the challenges of multimodal compositional retrieval.
5
RESULTS AND DISCUSSION
5.1
MAIN RESULTS
Table 4 presents Recall @K on OmniCVR across overall queries, while iablatSissllttss the case of audio-centric queries. We summarize the main findings below. The symmetric vision-centric retrieval results on OmniCVR are reported in Table 10 Appendix
1. Audio-centric queries expose a critical failure mode existing baselines. While our AudioVLM21 adapts robustly to mndioddependent queries (77.2 R @ 1), strong baselines suffer catastrophic performance degradation. For instance, VLM2Vec drops from an overall R@1 1 of 38 48.44 to just 12.4 in the audio-centric setting. This disparity underscores the unique difficulty of audio compositionality compared to visual tasks, a challenge that prior models fail to address.
2. Large-scale multimodal models establish superior performance tier. Across all query types, large embedding models consistently and substantially outperform lightweight, task- specific retrievers, validating the efficacy of large-scale pre-training for compositional reasoning.
3. Audio Audio VLM2Vec achieves universal state-of-the-art performance. Our model ranks first across all categories and K values, achieving 66.98 overall R @@) and 77.2 on audio-centric queries. This consistency confirms that our architecture generalizes effectively across both unimodal and integrated multimodal retrieval tasks.
4. Explicit audio semantics are the decisive factor for performance gains. The impact of injecting audio descriptions is profound: Audio VLMio surpasses the VLM2Vec baseline by an impressive +64.8 absolute points (77.2 vs. 12.4) in the audio-centric setting and +28.5 points overall. These results demonstrate that audio-aware embeddings not merely beneficial but indispensable for handling compositional queries involving non-visual transformations.
Table 4: Overall performance comparison of baseline models on OmniCVR. We report Recall at K (R@ @ R@3, R@5, R @ 10). Best results within each group are highlighted in bold, and second-best are underlined.
Models
Backbone
R @@I
R@3
R@5
R@ 10
Lightweight and Task- Task-Specific Models
CLIP
CLIP BLIP2 (Li et al. 2023
27.54
50.46
56.70
62.62
CoVR BLIP IMAGEBIND
11.46
22.88
28.08
35.18
BLIP
6.3
11.84
14.12
17.00
CLIP
17.28
29.55
43.34
45.33
Large Multimodal Embedding Models
OmniEmbed-wee -va -0.1-multiven
Qwen2.5-Omni
31.90
51.50
57.04
64.00
VLM2Vec
Qwen2-VI.
38.44
55.48
60.44
66.60
VLMVVe (Ours)
Qwen2-Audio + Qwen2-VL
66.98
77.84
80.86
84.40
8

---

Published as a conference paper at ICLR 2026
Table 5: Audio-centric retrieval performance of Large Multimodal Embedding Models on OmniCVR. We report Recall at K (R @ R@3, @@5, R 10). Best results are highlighted in bold.
Models
Backbone
R@1
R@3
R@5
R@ 10
OmniEmbed-vO.. 10t.--00..1miiiiii 1-multivent
Qwen2.5-Omni
13.6
28.5
35.8
47.0
VLM2Vec
Qwen2-VL.
12.4
23.3
30.4
42.3
Mudio (Ours)
Qwen2-Audio + Qwen2-VL
77.2
87.3
90.7
94.2
5.2 DETAILED ANALYSIS AND ABLATION STUDIES
Breakdown of Audio-Centria Performance. To better understand the model's capabilities across different acoustic domains, we decompose the audio-centric performance by target audio category (Human Speech, Music, and Sound). As shown in Table 6 AudioVLM2Ves achieves dominant performance in Human Speech (+85.23% gain) and Music (+70.36% gain). This indicates that converting audio to text effectively captures both lexical content and para-linguistic features like genre and mood. The gain is slightly lower but still substantial for Sounds (+49. +449.56%). reflecting the inherent challenge describing unstructured acoustic events compared structured speech or music.
Table 6: Fine-graing breakdown of audio-centric retrieval (R 1) OmniCVR by target audio category. Best results are highlighted in bold.
Target Audio Category
VLM2Vec
AudioVI.M2Ve (Ours)
Absolute Gain
Human Speech Music
11.36
96.59
+85.23
16.07
86.43
+70.36
Sound
10.75
60.31
+49.56
Impact of Source Video. To determine if detailed modification texts render the source video redundant (effectively reducing the task to text-to-vided retrieval), we conducted a "Blind Retrieval" ablation. Here, we removed the visual frames of the source video, forcing the model to rely solely on the modification instruction and source audio. As shown in Table 7 performance drops catastrophically without the source video. For Vidio on audio-centric queries, R@ plummets by 49.1% (from 77.20% to 10%). This confirms that the modification text functions relative instruction rather than standalone description. The source video provides essential context (e-g., the visual scene "park") to filter distractors, validating that OmniCVR rigorously evaluates compositional reasoning.
Table 7: Ablation study the importance of source video for VLM2Vec on OmniCVR audio-centric retrieval.
Metric
VLM2Vec (With Source Video) Blind Text-Only (No Source Video)
Performance Drop (A)
R
77.20 %
10%
-49.10%
R@3
87.30
33.20%
-54, 50%10%
R@5
90.70
42.50%
-48, 40.20%%
R@ 10
94.20
57.80%
-36.40%
Efficiency Analysis. We benchmarked inference latency under the same hardware configuration as used in VLM2Vec (Jiang et al. 2025), evaluating performance on 10-second video inputs. While AudioVLM2Ver increases latency from 1.72s (VLM2Vec) 4.77s (approx. 1.77x increase in processing overhead due to audio transcription). this trade-off yields a 64.8% improvement in audio- centric retrieval accuracy. Furthermore, with a Real-Time Factor (RTF) of 0.5, the system remains faster than real-time playback, ensuring deployability.
Native Audio Tokens Audio-as-Text. To rigorously isolate the contribution of our Audio- as-Text fusion mechanism from backbone differences, we performed controlled ablation on OmniEmbed-vO..
I-multivent.. -iniEmmbee-....-mmlieee This model's backbone (Qwen2.5-Omni) natively accepts raw audio

---

Published as a conference paper at ICLR 2026
waveforms via dedicated audio tower. We compared the original model (audio tower enabled) against a modified version where the audio tower is disabled and replaced by our Qwen2-Audio- Instruct transcribed captions.
As shown in Table S simply replacing latent audio tokens with explicit textual descriptions keeping the backbone, projector, and all training data adentical-yielee dramatic improvement from 13.6 to 32.7 on R@1 (+19.1 absolute points, 2.4x relative gain).
Table 8: Controlled ablation on 1tt.Eembed-OO--mmiiiii backbone; native audio tokens vs. our Audio-as- fusion mechanism audio-centric retrieval. Best results are highlighted in bold.
Model Setting
Audio Mechanism
R@1
R@3
R@5
R@
10
OmniEmbed (Original)
Native Audio Tokens
13.6
28.5
35,8
47.0
OmniEmbed (Modified)
Audio-as Text (Ours)
32.7
48.0
58.9
69.1
We further break down the results of this ablation by target audio category. As shown in Table 9 our Audio-as- Text strategy delivers consistent and substantial improvements across all categories. including dear-doubling of R@ on Music (+13.93 points) and Sound (+13.37 points). These results demonstrates that explicitly converting audio into dense, semantically rich captions- than relying on latent audio tokens-provveee provides a far more effective and universal audio representation, successfully capturing non-lexical attributes such as musical genre, timbre, rhythm, and complex environmental events.
Table 9: Per-category breakdown of the controlled OmniEmbed ablation (R @ I). Replacing native audio tokens with our Audio-as-Text mechanism yields large gains across all acoustic domains, including non-speech categories. Best results are highlighted in bold.
Target Audio Category
OmniEmbed (Original)
OmniEmbed (Ours-modified)
Absolute Gain
Human Speech Music Sound
11.66
50.38
+39.02
16.07
30.00
+13.93
10.75
24.12
+13.37
CONCLUSION, LIMITATIONS AND FUTURE WORK
In this paper, we introduced OmniCVR, large-scale benchmark for omni-composed video reirieval that establishes vision, audio, and text as first-class modalities. By requiring models to retrieve target videos based source videos and natural-language instructions spamning vision-centrie, audio-centric, and integrated querics-Omnich OmmiCVR provides a rigorous testbed for multimodal com- positionality. Our systematic evaluation reveals that existing state-of-the oof-the-art art retrievers significantly underutilize acoustic information, struggling when retrieval hinges on non-visual transformations. To address this, proposed Audio VLM2Vec, which injects explicit audio semantics into the em- bedding pipeline. This approach achieves state-of-the-art results and exposes the failure of current "full-modality" systems to effectively attend speech and environmental sound cues.
Limitations. Despite its strong performance, a primary limitation is the inference latency caused by the intermediate audio-to-tee transcription. While effective for semantic bridging, this step incurs higher computational overhead than latent embedding methods. Future work will focus on optimizing this pipeline peeeiiee-poeentallly via lightweight adaptors distillation-to accelerate embedding generation for real-time applications.
Future Work. Moving forward, we aim to: (i) incorporate additional modalities (e.g., depth, 3D) for richer reasoning; (ii) develop diagnostic protocols to probe temporal grounding and cross-modal consistency; (iii) scale to cinematic long-form videos (ranging from minutes hours, e.g., full movies/TVV episodes) by utilizing dense cropping from complex long contexts to yield short clips with richer temporal and semantic density, thereby introducing harder distractors and larger retrieval pools; (iv) explore open-ended retrieval to study hallucinations; and (v) leverage OmniCVR as a training resource develop robust end-to-end omni embeddings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We sincerely thank the reviewers for their insightful comments and constructive feedback, which have greatly contributed improving this work. This work was supported by the Special Foundations for the Development of Strategic Emerging Industries of Shenzhen (No. KJZD2023 17Z222023023094000000 023094700001) and the National Natural Science Foundation China (No. 62331014). We acknowledge the compu- tational support of the Center for Computational Science and Engineering at Southern University of Science and Technology.
REFERENCES
Sami Abu-El Abu-El-Haije El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaynnrrasmmhn. Youtube-8m: large-scale video classification benchmark. arXiv preprint (xXv::660000665 2016.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visse visual speech separation. Interspeech 2018, 2018.
Yunfei Chu. Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp.
2023.
Animesh Gupta, Jay Parmar, Ishan Rajendrakumar Dave, and Mubarak Shah. From play to replay: Composed video retrieval for temporally fine-grained videos. arXiv preprint arXiv:2506.05227 2025.
Thomas Hummel, Shyamgopal Karthik, Mariana-Iuliana Georgescu, and Zeynep Akata. Egoevr; An egocentric benchmark for fine-grained composed video retrieval. pp. 1-17, 2024.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training rision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations, 2025.
Chris Kelly, Luhui Hu, Bang Yang. Yu Tian, Deshun Yang, Cindy Yang, Zaoshan Huang, Zihao Li, Jiayin Hu, and Yuexian Zou. Visiongpt: Vision-languse understanding agent using generalized multimodal framework. arXiv preprint arXiv:2403. 09002,, 2024.
Reno Kriz, Kate Sanders, David Etter, Kenton Murray, Cameron Carpenter, Hannah Recknor, Jimena Guallar-Blasco, Alexander Martin, Eugene Yang, and Benjamin Van Durme. Multivent 2.0: A massive multilingual benchmark for event-centric video retrieval. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24149-244 -4158,, 2025.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image imagg pre- training for unified lasion-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Jing Liu, Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, and Jinhui Tang. Valor: Fision-audion language omni meni-perception pretraining model and dataset. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.

---

Published as a conference paper at ICLR 2026
Xueguang Ma, Luyu Gao, Shengyao Zhuang, Jiaqi Samantha Zhan, Jamie Callan, and Jimmy Lin. Tevatron 2.0: Unified document retrieval toolkit across scale, language, and modality. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 4065,, 2025.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2630-2640, 2019.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models to the world. In The Twelfth International Conference Learning Representations, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-878 8763. PMLR, 2021.
Gunnar Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European conference on computer vision, pp. 510-526. Springer, 2016.
Omkar Thawakar, Muzammal Naseer, Rao Muhammad Anwer, Salman Khan, Michael Felsberg, Mubarak Shah, and Fahad Shahbaz Khan. Composed video retrieval via enriched context and discriminative embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26896-269002 2024.
Omkar Thawakar, Dmitry Demidov, Ritesh Thawkar, Rao Muhammad Anwer, Mubarak Shah, Fahad Shahbaz Khan, and Salman Khan. Beyond simple edits: Composed video retrieval with dense modifications. arXiv preprint arXiv:2508. 14039, 2025.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. Proceedings of the European conference on computer vision (ECCV), pp. 247-263, 2018.
Yuan Tseng, Layne Berry. Yi-Ting Chen, I-Hsiang Chiu, Hsuan-Har Lin, Max Liu, Puyuan Peng, Yi-Jen Shih, Hung- Yu Wang, Haibin Wu, et Av-superb: A multi-task evaluation benchmark for audio-visual representation models. In ICASSP 2024-2024 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 6890-6894. IEEE, 2024.
Lueas Ventura, Antoine Yang, Cordelia Schmid, and Gül Varol. Covr-2: Automatic data construction for composed video retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Peng Wang. Shuai Bai. Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing rision-languee model's perception of the world at any resolution. arXiv preprint 11ti::2409.11991 2024.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 4581-4591, 2019.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296. 2016.
WU Yue, Zhaobo Qi, Yiling Wu, Junshu Sun, Yaowei Wang, and Shuhui Wang. Learning fine- grained representations through textual token disentanglement composed video retrieval. The Thirteenth International Conference on Learning Representations, volume pp. 4, 2025.
Jigqi Samantha Zhan, Crystina Zhang, Shengyao Zhuang, Xueguang Ma, and Jimmy Lin. Magmar shared task system description: Video retrieval with omniembed. arXiv preprint Ttl:22066009409 2025.
12

---

Published as a conference paper at ICLR 2026
Yufei Zhan, Hongyin Zhao, Yousong Zhu, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon-g: Bridging vision-languag and vision cision-centric tasks via large multimodal models. arXiv preprint arXiv:2410.16163, 2024.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang. Yongping Xiong, Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13691-13701, 2025.
Luowei Zhou, Chen Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
A DATASETS USED IN STAGE 1 (VIDEO CURATION & SEGMENTATION)
In Stage 1, we curate and segment long-form videos (typically minutes to hours in duration) from six complementary datasets that cover diverse domains, modalities, and annotation styles. From each of these long-form videos, we extract multiple semantically coherent short clips (5-15 seconds) that constitute the final retrieval items in our benchmark (average length 1 11.8 seconds). Below we provide more detailed descriptions et al. 2016), VATEX (Wang et al., 2019), CouTuubemm-MusicTexxxee MusicTexxClipss (Abu-El-Haijy et al. 2016). YouCook2 (Zhou et al. 2018), and VALOR (Liu et al. 2024).
the six datasets: How 100M (Miech et al. 2019), MSR-VTT (Xu
How (Miech et al., 2019). A large-scale collection of narrated instructional videos from YouTube. It contains about 1.2M long videos and roughly 136M automatically transcribed narration clips, covering over 23k diverse tasks (cooking. DIY, fitness, crafts, etc.). Strengths include matched scale and domain coverage, making it ideal for weakly supervised pretraining of text-video embeddings, However, ASR transcripts are noisy and loosely aligned with visual content, and activity distribution is highly imbalanced. In Stage I, it serves as the primary source of long videos, where narration timestamps provide coarse cues for segmentation.
MSR-VTT (Xu et al. 2016). A widely used benchmark of veneral-domate short clips with human- written captions. It contains 10k clips and 200k captions (about per clip). Strengths are high-quality human annotations and balanced coverage across diverse scenarios. Limitations include its clip-level scope and lack of fine-grained temporal structure. Stage benchmark to evaluate the semantic quality of curated segments after segmentation.
primarily used as reference
VATEX (Wang et al. 2019). A multilingual video capioon dataset with both English and Chinese annotations. It consists of 41k videos paired with 825k captions, including 206k Einlish-Chhines parallel sentences. The dataset enables multilingual retrieval and cross-lingual transfer. Its limitations lie in its clip-level nature and absence of dense temporal supervision. In Stage 1, it complements other datasets by providing multilingual benchmarks to test cross-lingual robustness of curated segments.
werTTubeeSmm-MuiTTeetCCCCii (Abu-El-Haija et al. 2016). Derived from the large-scale YouTube- 8M dataset, this subset focuses on music and music- video content aligned with text or tags. YouTube- 8M contains millions of videos and billions of frames with noisy 4.8k entity categories. Strengths are its massive scale and coverage, especially in entertainment and music. Weaknesses are label noise and lack of dense natural-language captions. In Stage acts as supplementary pool and weakly labeled filtering resource for music/entertainment domains.
generated labels across
YouCook2 (Zhou et al. 2018). A domain-specific dataset cooking videos with step-level annotations. It contains 2k untrimmed YouTube videos (about 176 hours, across 89 recipes), each segmented into procedure steps with textual descriptions. procedure segmentation and dense video-text alignment. Limitations include its domain restriction to cooking and moderate video length. In Stage 1, used as a calibration set for validating segmentation quality and procedure-aware alignment.
gold-standard benchmark for
13

---

Published as a conference paper at ICLR 2026
VALOR (Liu et al. 2024). A tri-modal dataset (vision-audio- lannguugg designed for audiovisual captioning and retrieval. VALOR-IM provides about 1M audiovisual clips for pretraining, and VALOR-32K offers smaller high-quality evaluation benchmark with human-curated audiovisual captions. Strengths include explicit modeling of audio cues addition to visual and textual context, enabling tri-modal learning. I Limitations include smaller scale compared to HowTo To100M and potential annotation subjectivity. In Stage 1, VALOR supports audio-informed segmentation (c.g., boundary detection from speech or sound transitions) and tri-modal retrieval evaluation.
B BASELINE MODELS
To contextualize the performance of our approach, we benchmark against a diverse set of baselines, spanning both large multimodal embedding models and lightweight or task-specific retrievers. Below we provide detailed descriptions of each baseline model.
CLIP RRaford et al. 2021). A foundational vision-language model trained on large-scale im- age-tes text pairs. For video retrieval tasks, we follow standard practice by uniformly sampling 15 frames per video and averaging their frame-level embeddings to obtain the video representation. While CLIP provides a strong baseline for vision-ter text alignment, lacks explicit modeling of audio or video-specific temporal dynamics.
CoVR (Thawakar al. 2024). A model specifically designed for composed video retrieval, focusing on scenarios where the query involves modifications of existing video content. CoVR is optimized for vision-centrie transformations, and our setting, we uniformly sample frames per video and average their embeddings to form the video representation. We assess CoVR's ability to generalize beyond purely visual changes. lightweight design makes it efficient, though its limited multimodal scope is a constraint for audio-aware retrieval.
BLIP (Li et al. 2022). unified vision-language understanding and generation model. We adapt BLIP for retrieval by using its vision- language matching head to score candidate videos. Following our experimental protocol, we uniformly sample 15 frames per video and average their embeddings to construct the video representation. BLIP demonstrates strong cross-modal reasoning and captioning ability, making it a competitive retrieval baseline. However, like CLIP, does not natively incorporate audio cues.
ImageBind (Girdhar al., 2023). A multimodal embedding model that learns a joint represen- tation space across six modalities: images, text, audio, video, depth, and IMU signals. For video retrieval, we apply the same protocol as other lightweight baselines, uniformly sampling 15 frames video and averaging their frame-level embeddings. Its broad modality coverage makes it a natural fit for audio-video retrieval tasks. The ability to align heterogeneous modalities directly shared space provides a strong baseline for multimodal integration, though its representations may be less specialized than task-specific models.
OmniEmbed-vol 10.bbed-v.11-me 10mliiiitie (Ma et al. 2025). A unified multimodal embedding model built on the Tevatron 2.0 framework. It is trained to encode text, image, audio, and video into a shared representation space. OmniEmbed-vol 1achhedd--.1-muutee 1-multivie has achieved state-of-the-art performance in cross-modal video retrieval benchmarks, such as the MAGMaR Shared Task (Zhan et al., 2025), by fine-tuning the MultiVENT dataset (Kriz et al. (2025) with joint vision-audio-ter. supervision. Its strength lies in versatility across modalities, making it strong baseline for multimodal retrieval.
VLM2Vec (Jiang et al., 2025). A general framework that transforms existing large vision vision-language models (VLMs)-including Qwen2-VL (Wang et al. 2024). LLaVA (Liu et al.) 2023). and Phi- 3.5-V-into universal embedding models through contrastive learning on the Massive Multimodal Embedding Benchmark (MMEB). VLM2Vec consistently surpasses conventional baselines such as CLIP (Radford et al. 2021) and BLIP (Li et al. 2022), with improvements retrieval tasks. This approach highlights the effectiveness of adapting powerful pretrained VLMs into mabedding--ocuseee architectures.
14

---

Published as a conference paper at ICLR 2026
Audio (Ours). Our proposed extension of VeM2 that explicitly integrates audio semantics. We employ Qwen2-Audio (Chu et al., 2024 to generate fine- grained captions from the video's 's audio track, which then concatenated with the user's modification query and fed into VLM2Vec. This design leverages the strong representation capabilities of large VLMMs while ensuring sensitivity audio signals, thereby improving performance audio-centric retrieval scenarios.
C SUPPLEMENTARY EXPERIMENTS AND DISCUSSION
C.I OMNICVR DATA STATISTICS AND ANALYSIS
Figure 6 shows the distribution video lengths in the OmniCVR training dataset, where most clips fall between5 and 15 seconds.
Distribution Viceo Durations
14000
1780
Nurlee dFVobo0
19K0
0008
6000
400
2000
Figure 6: Video length distribution of the OmniCVR training dataset.
To better illustrate the breadth of real-world scenarios covered by OmniCVR, we provide a detailed breakdown of its content sources. OmniCVR aggregates videos from diverse publicly available datasets spanning multiple genres:
Instructional & Procedural Content: Videos depicting step-by-step tasks such as cooking recipes, DIY repairs, crafting tutorials, and fitness routines. These critical for evaluating a model's ability to understand fine-grained actions and temporal dependencies.
Daily Life, Nature & Travel: Unscripted, in-the-wild footage capturing everyday human activities, natural environments, and travel vlogs. This category provides rich visual and auditory context and serves as the primary source for environmental sound-related queries.
Clips featuring musical instruments, dance performances, and live concerts, These are specifically leveraged for "Audio-Centric" queries, enabling evaluation of model's capacity to distinguish between musical genres, tempos, instruments, and performance styles.
Music Performance:
A broad collection of web-sourced clips including sports highlights, news broadcasts, public ceremonies, and other event-driven content, ensuring coverage of dynamic and socially relevant scenarios.
General Entertainment & Events:
This multi-genre composition ensures that OmniCVR comprehensively reflects the heterogeneity of real-world audiovisual experiences, making it robust benchmark for cross-modal retrieval and understanding.
Complementing these visual genres, OmniCVR explicitly models the acoustic dimension with high granularity. To ensure comprehensive coverage of acoustic, semantic, and para-linguistic features, we categorize audio content into three primary domains, each enforcing a strict schema of fine-grained attributes:
Human Speech: This category addresses both the lexical and lina-sin dimensions of spoken audio. Our annotation pipeline explicitly captures:
15

---

Published as a conference paper at ICLR 2026
Lexical Content: Verbatim transcripts of the speech to ground accurate semantic understanding.
-Prra-liguuistc Features:
Identification of speaker characteristics and emotional tone (e.g., neutral, angry, fearful, surprised), distinguishing retrieval targets based how something is said, not just what is said.
Music: This category focuses temporal dynamics and stylistic attributes. The annotations provide detailed descriptions off:
Genre & Instrumentation: Identification of specific musical styles and the instruments
present.
Temporal Dynamics: Chronological descriptions of rhythm, pace, and intensity (e.g., distinguishing a "slow, steady beat" from "fast, erratic tempo") and the overall atmospheric mood.
Environmental Sound: To address the complexity of acoustic environments, we enforce
detailed hierarchy distinguishing between:
Nature Sounds: Elements such as wind, rain, water flow, and animal calls.
Mechanical & Urban Soundscapes: Sounds of engines, machinery, alarms, traffic patterns, and construction noise.
Foley & Action Sounds: Distinct, event-driven sounds such as footsteps, glass breaking, or doors closing.
By explicitly modeling these dimensions during the annotation and verification stages, OmniCVR ensures that retrieval queries are grounded in rich, ane-grained audio semantics rather than simplified labels.
C.2 VISTON-CENTRIC RETRIEVAL. RESULTS
Table reports vision-centric retrieval results on OmniCVR. Within lightweight baselines, IMAGE- BIND achieves the best performance, indicating that simple cross-modal alignment still provides competitive vision retrieval ability. Yet, large multimodal embedding models clearly dominate: VLM2Vec already surpasses OmniEmbed-vie -.0-mbbbd--...mmliieeenn and
further advances state-of-the-art performance by a large margin (e.g., +14.3% R@) over VLM2Vec). Interestingly, the consistent gains obtained by Audio WL2 under vision-only evaluation-Suggess that audio representations implicitly encode complementary visual characteristics, which can be leveraged during joint embedding learning. This finding highlights the underexplored synergy between audio and vision modalities: audio not only enriches multimodal fusion but also strengthens pure visual retrieval through cross-modal feature transfer.
Table 10: Vision-centric retrieval performance OmniCVR. We report Recall (R@1. R@3,
R@5, @@I 10). Best results within each category are highlighted in bold.
Models
Backbone
R@I
R@3
R@5
R@ 10
Lightweight and Task-Specific Models
CLIP
CLIP
39.53
51.18
56.27
63.10
CoVR
BLIP2
2.10
6.57
9.29
14.46
BLIP IMAGEBIND
BL.IP
6.13
8.85
10.60
13.41
CLIP
47.85
57.58
61.09
65.82
Large Multimodal Embedding Models
OmniEmbed-we -aaptteed-vO.--mmiiii 1-multivent
Qwen2.5-Omn
50.74
63.45
69.85
77.21
VLM2Vec
Qwen2-VL
55.04
66.78
70.99
75.99
VLMVvee (Ours)
Qwen2-Audio + Qwen2-VL
69.33
76.51
80.28
82.82
C.3 GENERALIZATION TO OUT-OF-DOMAIN TASKS
To further validate the generalization capability of the proposed audio-augmented representations, we evaluate AudioVLM2M4 in zero-shot manner the widely used MSR-VTT dataset (Xu et al.
16

---

Published as a conference paper at ICLR 2026
2016) for conventional text-to-video retrieval. As shown in Table IT. Vec consistently outperforms the strong VLM2Vec baseline across all metrics (+1.7-2.3 absolute points), even though MSR-VTT captions rarely mention sound events explicitly. Qualitative analysis reveals that our audio- to-text descriptions provide valuable implicit visual context (e.g.. "roaring engines and screeching tires" in racing clips, "crowd cheering" in sports scenes), thereby enhancing discrimination among visually similar videos. This result confirms that the multimodal representations learned on OmniCVR transfer effectively standard retrieval tasks.
Table 11:
Zero-shot text-to-video retrieval performance on MSR-VTT, Our AudioVLM22ec is trained only on OmniCVR and evaluated using its automatically generated audio-to-text descriptions as additional input. Best results are highlighted in bold.
Model
R@1
R@3
R@5
R@
VI.M2Vcc
36.10 (0)
53.00
60.70
70.10
AudioVI.M2 (Ours)
37.90
55,30
62.50
71.80
+1.80
+2.30
+1.80
+1.70
C.4 DISCUSSION ON THE PLAUSIRILITY OF COMPOSITIONAL. TRIPLETS
We clarify that our benchmark construction is strictly data-driven. We do not synthesize arbitrary instructions; instead, we first mine valid video pairs (Source, Target) from real-world distributions (e.g., HowToo10M YouTuube8M based on semantic similarity. The modification text is generated post-hoc to describe the actual physical and acoustic differences observed between the clips. Thus, all "transformations" in OmniCVR reflect natural variations found in large-scale video corpora, ensuring that matching targets physically exist within the dataset.
C.5 WHY DO "FULL-MODALITY" METHODS DIVERGE ON AUDIO-CENTRIC QUERIES?
Although ImageBind, OmniEmbed, and VeM all operate multi-modal settings, their behaviors differ markedly once retrieval is driven only by audio changes.
ImageBind. ImageBind is capable ingesting audio and text, yet in our audio-centric OmniCVR splits- splits-where srrech-guided queries constitute majority-it struggles to capture the lexical and semantic content of speech. Its audio pathway is optimized for generic cross-modal alignment rather than fine-grained speech understanding; as result, instructions that hinge who spoke, what was said, or subtle speech-state changes (e.g., tone, intent) are often mapped to embeddings dominated by background acoustics or coarse timbral cues. This mismatch can even mislead retrieval under audio-centric composition, causing the search to prioritize scenes with similar ambient sounds while ignoring the intended speech-drive driven modification.
OmniEmbed. OmniEmbed can encode audio, but token budget and fusion design weigh visual tokens much more heavily than audio tokens across the sequence. During joint pooling, the resultant embedding therefore dominated by vision features, and audio contributes weakly to the final representation. Under audio-centric queries-wher the visual stream is deliberately held constant- this imbalance suppresses precisely the information that differentiates the targets, yielding poor retrieval.
VLM2Vec. VLM2Vec does not include explicit audio branch. Consequently, it performs com- petitively in vision-centric and integrated (vision+audioo) settings-wherr the visual stream provides reliable discriminative cues and the composed instruction contains strong visual operators but fails when the only changing factor is audio. In audio-centric queries, the absence of an audio encoder leaves the model with no path to ground the modification.
Audio (ours). In contrast, our approach leverages QwenZ-Audio to first convert raw audio-especially speech-into fine-grained textual descriptions. These transcribed and summarized audio semantics are then injected into the VLM2 Vec pipeline, where they are aligned with vision- language embeddings. This design bypasses the limitations of weak audio tokenization and ensures that speech content is represented the same space as visual and textual cues. As reflected by the audio-centric results, Audio ML2 achieves 77.2 R @ and 94.2 R @@ 10, far beyond all other
17

---

Published as a conference paper at ICLR 2026
full- modalttyy methods, showing that preserving semantic detail from audio before fusion is decisive advantage.
In summary, the failure modes align with architectural choices: (i) speech-heavy audio-centric composition penalizes models that lack speech-aware audio representations, (ii) token and pooling imbalance dilute audio contributions, and (iii) omitting audio branch altogether leads to systematic failure whenever sound is the only supervisory signal. By contrast, our design demonstrates that converting audio rich textual descriptions before multimodal alignment allows audio to function as true first-class compositional signal in OmniCVR.
D ADDITIONAL QUALITATIVE EXAMPLES
Keep the boy and couch setting. Remove all sickness elements (lissues, recovery dialogue). Replace with playful physical movement the couch, ending in cheerful mutual "goodbye. Change the visual action and conversational tone from caring to playful.
Figure 7: Qualitative example of modifying emotional tone and action. The query instructs the model to remove sickness elements (tissues, recovery dialogue) and replace them with playful physical movement, shifting the scene from caring to cheerful while maintaining the characters and setting.
18

---

Published as a conference paper at ICLR 2026
Maintain the graffiti artist, the wall, and the Hip-Hop soundtrack genre. Advance the visual state of the wall art from an early, mostly white outline to nearly finished, detuiled piece with and black fill.
Figure 8: Qualitative example of temporal and state progression. The model is tasked to advance the visual state of the graffiti wall from an early outline to nearly finished, detailed piece, while preserving the specific artist and Hip-Hop audio backdrop.
Maintain golf player's 's shirt color from yellow white/light and slightly adjust the dogs' positions to the center video. Replace the andio to deplet faster-paced sequence throws.
player's Change
Figure 9: Qualitative example of fine-graine mne-ggriied integrated modification. The instruction requires simultancous changes in vision (changing shirt color from yellow to white/light gray) and audio (depicting faster-paced sequence of throws), testing the model's ability handle precise attribute.
19

---

Published as a conference paper at ICLR 2026
4fp1-1
Keep the person and interview context intact. Remove audio and visuals centered on past events. Replace them with new audio and shift conversation toward future possibiliries- envolling in Capoetra, other martial gymnastics-mone moving the interview from recollection to forward-looking discussion.
4ptb
Figure 10: Qualitative example of semantic conversation shift. The query directs transition from recollecting past events to discussing future possibilities (e.g., enrolling in Capoeira), requiring the model understand the semantic content of the speech retrieve the correct forward-looking segment.
We provide additional qualitative results to demonstrate the model's capability in handling diverse and complex compositional instructions. Figure 7 illustrates a vision-centric transformation of emotional tone, shifting scene from "caring recovery" "playful interaction." Figure captures temporal state progression, where the query directs the retrieval of completed graffiti piece based on an early outline. Figure Figure@Sshowcasse showcases fine-grained integrated query, requiring simultaneous reasoning over visual attributes (shirt color change) and auditory events (pace of throws). Finally, Figure I0 demonstrates a semantic conversation shift, where the model must distinguish between an interview segment about past recollections versus one focused on future aspirations.
E ETHICS STATEMENT
The OmniCVR benchmark proposed this work derived from existing public video datasets, including HowTo100M, MSR-VTT, VATEX, YouTubeSM, YouCook2, and VALOR. We strictly adhere to the respective licenses and terms of use for these source datasets. The vidcos originate from publicly available content and do not involve private information belonging individuals or organizations beyond what is already public.
For the human verification stage of pipeline (Stage 3), employed expert annotators to validate the quality of the modification instructions and video pairings. We ensured ethical working conditions and paid all annotators fair hourly wage that exceeds the local minimum wage standards. Furthermore, during the generative annotation process involving Large Multimodal Models (LMMs), we applied safety filters to prevent the inclusion of harmful, offensive, or biased content in the generated text descriptions.
F REPRODUCIBILITY STATEMENT
We are committed ensuring the reproducibility of our results and promoting further research in omni-modal retrieval.
Data Availability:
The OmniCVR benchmark, including the 160k+ curated clips, the 50k+ compositional triplets, and the gold soad-standard test set, will be made fully open-source to the public upon publication.
20

---

Published as a conference paper at ICLR 2026
Code and Models: Our proposed AudioVLM2Vec framework leverages soure pre- trained weights (Qwen2-Audio and Qwen2-VLJ and standard libraries. We will release the complete codebase, including data generation scripts, training code, and evaluation protocols.
Transparency: To facilitate the reproduction of our dataset construction pipeline, we have provided the exact prompts used for annotation, triplet mining, and verification in Appendix
G PROMPTS
Below are the prompts used for triplet mining.
Prompt for generating chronological video captions using Qwen2.5-Omni
You are precise observer. Write one paragraph that describes ONLY what directly visible and audible in the video, in strict chronological order with clear temporal markers.
For any notable action, break it into distinct stages and describe each in detail (e.g.. starting slowly. changing technique, altering body position, adding props). If stage missing or unclear, state "not shown" or "unclear." This is the highest priority. Do not just state an activity; describe precisely how unfolds and changes over time. Detail the sequence of movements. Example: Instead of "A person plays football," by slowly dribbling a white and black ball with their right foot across a green field. They then transition to kicking it against wall, and later, after lying their back, they attempt to juggle the ball with their feet in the air." Rules:
must write: "A person begins
No guesses or world knowledge.
If something is uncertain, say "unclear".
Include at least one sentence about AUDIO (speech, sound, or music). If silent, state "no audible speech; ambient silence/noise". If there were voices, specify who spoke, what was said in the video, and the emotion conveyed; if background music (BGM) is present, describe the genre and mood of the BGM; if there other sounds, indicate their nature.
Prefer concrete attributes (colors, materials, relative positions) over interpretations.
21

---

Published as a conference paper at ICLR 2026
Prompt for generating vision-only video modification instructions
You are an expert vision-only video understanding and creative language. Given two textual Snncriptions[SSurree Description] and [Target D gennerae concise, natural-language instruction that tells someone how to modify the visual content of the "Source Video" match the "Target Video". Your instruction must:
Focus only on visible differences (e.g., added/removed objects, changed actions, different person, altered background, lighting, color tone, camera view, scene layout).
Completely ignore sound, audio, dialogue, music, or any non-visual information.
Be phrased clear, direct command request.
Start from the visual context of the Source Video.
Now, generate vision-only instruction for the following descriptions:
[Source Description]: [sourceddesc)
[Target Description]: [target.desc] Generated Instruction:
Prompt for generating audio-only video modification instructions
You are an expert in audio-only video understanding and creative language. Given two textual descriptions- Description] and [Target Description] generate -eneerrteeeee concise, natural- language instruction that tells somcone how to modify the audio content the "Source Video" match the "Target Video".
Your instruction must:
Focus only on audible differences (e.g., added/removed speech, changed speaker, background sounds, sound effects, music type, volume, tone).
Completely ignore any visual information (objects, actions, people, backgrounds, colors, lighting, or camera view).
Be phrased clear, direct command
request.
Start from the audio context of the Source Video.
Now, generate an audio-only instruction for the following descriptions: [Source Description]: {source_dess) [Target Description]: (target.desc) Generated Instruction:
22

---

Published as a conference paper at ICLR 2026
Prompt for generating video-to-video modification instructions
You are an expert in creative language and video retrieval. Your task generate concise, natural language instruction that describes how to modify a "Source Video" to become "Target Video". You will be given textual descriptions: [Source Description] and [Target Description].
Your instruction should:
Identify the most salient difference(s) between the source and target (e.g., scene, action, object, person, setting).
Be phrased as clear command or user request.
Be natural and easy understand.
Start from the context of the Source Video.
Now, generate an instruction for the following descriptions: [Source Description]: fsource desc) [Target Description]: barget_dese)
Generated Instruction:
Prompt for generating detailed audio descriptions
You are expert audio analyst. Listen carefully to the given audio and provide comprehen- sive description.
Instructions:
First, identify the main category of the audio. Choose exactly one from:
human.speeche music, atvirronmental.sound event_sound, other.
Then, provide detailed explanation according the detected category:
Human speech: Transcribe the speech verbatim. Then identify the speaker's emotion (choose from: neutral, happy, sad, angry, fearful, surprised, disgusted).
Music: Describe genre, instruments, vocals (if any), rhythm, and the overall mood.
Environmental sound: Describe the environment, natural elements, and the acoustic atmosphere.
Event sound: Describe the specific event action represented by the sound, including temporal sequence if clear.
Other: Provide as precise description possible of what is heard.
Finally, summarize the entire audio in one concise sentence.
Rules:
Do not use world knowledge beyond the audio itself.
If something is uncertain unintelligible, state "unclear".
Prefer concrete acoustic attributes (e.g., pitch, tempo, loudness, timbre, clarity, background noise) over interpretations.
23


Reviews:
[{"review_id": "9tO6Lb0Qlx", "reviewer": "Reviewer_F9DJ", "summary": "This paper introduces OmniCVR, a new benchmark designed to evaluate omni-composed video retrieval, where the system retrieves a target video given a source video and a textual modification instruction that may involve changes in visual content, audio, or both.\nThe authors describe a large-scale data construction pipeline that combines LLM-assisted multimodal annotation (Qwen2.5-Omni, Gemini 2.5 Pro) with validation across multiple modalities. They also propose AudioVLM2Vec, an embedding model that integrates audio information via textual transcriptions to better handle audio–visual composition. The benchmark aims to capture fine-grained multimodal reasoning, extending beyond traditional text–video retrieval tasks..", "questions": "1. Many of the benchmark’s compositional instructions describe modifications that seem infeasible to find in existing datasets (e.g., changing lighting, replacing sound events, or altering temporal composition). Could the authors clarify why they chose to formulate this as a retrieval problem rather than as conditional video generation? Do they believe that realistic corpora can ever contain such compositional matches at scale?\n\n2. If the benchmark truly captures omni-compositional reasoning, models trained on OmniCVR should generalize to more conventional video retrieval datasets such as MSR-VTT, VATEX, or ActivityNet Captions. Have the authors tested such out-of-domain performance? If not, how can they claim that the benchmark teaches general multimodal retrieval rather than overfitting to its own synthetic compositions?\n\n3. Could the authors provide example showing that the retrieved videos actually reflect the compositional modifications specified in the text?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an underexplored and ambitious problem where compositional video retrieval jointly involves visual, auditory, and textual reasoning. By integrating audio-based instructions alongside visual composition, it broadens the conventional scope of video retrieval tasks. This is a valuable and forward-looking addition for multimodal benchmarks.\n\n- The benchmark creation process is well-documented and systematically structured. The use of multiple LLMs for multimodal captioning, filtering, and validation makes the dataset large, diverse, and reproducible. This methodological transparency is a clear strength.\n\n- The implementation details, model configurations, and evaluation metrics are clearly presented. The writing quality is high, and the paper provides good contextualization against prior work .", "weaknesses": "- The task assumes that, for a given source video and a detailed compositional instruction, another video exists in the corpus that satisfies all the described changes. Many examples involve complex spatial, temporal, or auditory transformations that are implausible to find in any real-world dataset. Such a setup feels conceptually closer to conditional video generation than retrieval. The paper does not convincingly justify why retrieval is a meaningful or achievable approach for these highly specific instructions.\n\n- If OmniCVR truly captures omni-compositional reasoning, models trained on it should exhibit strong transfer to conventional video retrieval datasets (e.g., MSR-VTT, VATEX, Dense-WebVid-CoVR). Yet the paper does not test this. The absence of out-of-domain evaluation makes it impossible to assess whether the learned representations generalize beyond the synthetic distribution or simply overfit to the compositional language space introduced by the benchmark. \n\n- While the benchmark emphasizes audio–visual–text reasoning, the proposed model (AudioVLM2Vec) primarily uses text transcriptions of audio, not raw acoustic embeddings. This simplifies audio reasoning into another text-conditioning signal, limiting the extent to which the benchmark truly evaluates multimodal understanding.\n\n- The paper does not present qualitative examples or human evaluations showing that retrieved videos actually match the intended compositional modifications. Without such validation, it is unclear whether the benchmark’s retrieval results are semantically meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X7FgOSMHI2", "reviewer": "Reviewer_2XpL", "summary": "The authors introduce OmniCVR, a composed video retrieval benchmark that incorporates audio as a key modality alongside vision and text. The authors argue that the queries included in the dataset are representative of real-world queries, and target audio in addition to vision. Additionally, the authors extend the VLM2Vec retrieval method to simultaneously model audio, which they name AudioVLM2Vec. This approach achieves SoTA performance in multimodal retrieval.", "questions": "- What key contributions does this dataset make compared to existing video retrieval benchmarks that center audio? What is the importance of composed video retrieval tasks?\n\n- What genres of video are included in the dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper accurately identifies that audio is an under-utilized modality in video retrieval. We do lack benchmarks that target this modality.\n\n- In addition to dialogue, the dataset also centers non-language audio like music and sound effects, which historically have been even less focused on than dialogue.\n\n- The dataset is large enough to serve as a training set in addition to validation.\n\n- AudioVLM2Vec achieves impressive performance compared to existing methods. Its development is sensible and is described clearly. The chosen comparison methods are representative of the current research space.\n\n- The paper is generally sound and well-written.\n\n- The limitations and future work section is detailed and thoughtful.", "weaknesses": "- The video-text retrieval benchmarks section in the related works segment is quite small. There have been many more video retrieval benchmarks, some of which center audio. Describing the differences between this dataset and others that center audio (VaTeX, MultiVENT 2.0, etc.) would be informative for those evaluating the novelty of the dataset.\n\n- It would be nice to describe the genres and types of content focused on in the dataset in more depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QlW3biFwk5", "reviewer": "Reviewer_Too4", "summary": "This paper introduces OmniCVR, a large-scale benchmark for composed video retrieval that treats visual, audio, and textual modalities as equally important. It addresses the oversight of audio in existing benchmarks by including audio-centric and integrated queries, where integrated queries dominate to reflect real-world multimodal scenarios. OmniCVR is constructed via a scalable pipeline involving video segmentation, omni-modal annotation, and dual validation (using LLMs and human experts). Evaluations show that current models struggle with audio-centric tasks, prompting the proposal of AudioVLM2Vec—an extension of VLM2Vec that explicitly incorporates audio semantics. This model achieves state-of-the-art performance, revealing critical gaps in handling audio-dependent queries and underscoring the necessity of audio-aware multimodal retrieval systems.", "questions": "1. The paper contains many figures and tables that are not explicitly cited or referenced in the text.\n\n2. Efficiency Benchmarking: Report the inference latency and FLOPs for AudioVLM2Vec compared to other baselines, highlighting the cost of the audio-to-text step.\n\n3. Inadequate Ablation Studies: The paper fails to provide sufficient ablation analysis to disentangle the contributions of different components in AudioVLM2Vec. Critical questions remain unanswered: How much performance gain comes from audio transcription versus the fusion mechanism? How does the model perform on non-speech audio domains?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Originality: This work first define and benchmark \"audio-inclusive composed video retrieval.\" This effectively eliminates the key limitation of prior CVR benchmarks: the neglect of audio as a first-class semantic modality. The proposed AudioVLM2Vec also offers a novel, pragmatic architecture for audio infusion.\n\nClarity: The paper is easy to follow and understand.\n\nSignificance: The paper proposes \"audio-inclusive composed video retrieval\", which enables direct future research on audio-visual-text reasoning and is an essential tool for developing genuinely holistic video understanding models.", "weaknesses": "1. While the paper's introduction of audio-centric queries is commendable, its treatment of \"audio\" remains oversimplified. A truly comprehensive audio-centric benchmark must consider a wider array of acoustic, semantic, and para-linguistic dimensions and provide specialized evaluations for each. The benchmark misses the opportunity to define and evaluate based on these essential audio dimensions:\n\n     (a)  Para-linguistic Features in Speech:  (i)Emotion & Tone, (ii) Speaker Characteristics \n\n     (b) The lexical content in Speech\n\n     (c) Hierarchy of Environmental Sounds: \"Environmental Sound\" is a vast category that should be broken down into:   (i) Nature Sounds (wind, rain, animals), (ii) Mechanical Sounds (engines, machinery, alarms), (iii) Urban Soundscapes (traffic, crowds, construction), (iv). Foley/Action Sounds (footsteps, glass breaking, doors closing)\n\n     (d) Temporal Dynamics: This involves rhythm, pace, and intensity over time. A query could be: \"Find the same scene but where the background music has a slow, steady beat instead of a fast, erratic one.\"\n\n2. Lack of Specialized Evaluation per fine-grained Dimension: The paper's aggregated \"Audio-Centric\" results (Table 4) are misleading. A model might excel at modifying speech content but fail completely at modifying musical timbre or emotional tone. This critical diagnostic information is lost.\n\n3. In the same way, Limited Exploration of Audio-Visual Interactions: The benchmark and proposed method treat audio and vision as largely separate streams that are fused late (e.g., by concatenating text descriptions). It does not explicitly model or evaluate the complex, synergistic relationships between audio and vision, such as causal links (a person's mouth movements and their speech). The query text may be that \"Take the source video of a man speaking and replace his audio with a different language while also modifying the video to show his lips moving out of sync.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3djdC42LCa", "reviewer": "Reviewer_H76e", "summary": "This paper introduces the OmniCVR benchmark for composed video retrieval, encompassing vision, audio, and text modalities. A key novelty of OmniCVR is its pioneering inclusion of the audio modality as a modification target, distinguishing it from previous text-video and composed video retrieval benchmarks. Furthermore, the paper proposes a corresponding AudioVLM2Vec framework, which effectively highlights significant limitations of existing models when evaluated on the newly-constructed OmniCVR benchmark.", "questions": "Please see the above weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. As a benchmark paper, this work contains the basic elements of a newly constructed benchmark. It covers essential aspects such as the limitations of previous benchmarks, the construction pipeline and its analysis, and the presentation of several baselines evaluated on the new benchmark.\n\n2. The identified problem, i.e., the absence of an audio stream in previous Composed Video Retrieval (CVR) benchmarks, is indeed a prevalent issue in existing literature. This establishes a clear and compelling motivation for the current work.\n\n3. The observed poor performance of several baselines when evaluated on the newly-introduced OmniCVR benchmark effectively highlights the limitations of existing models and, by extension, underscores the necessity and value of benchmarks that incorporate the audio modality.", "weaknesses": "1. My primary concern relates to the modification texts in the current OmniCVR benchmark. As shown in Table 2, the average length of queries is 52.6 words, which seems excessively long compared to previous literature, as illustrated in Figures 1 and 2. The modification texts appear to contain too many details, especially for integrated-type retrieval, which, in turn, makes the task easier to some extent.\n\n2. This observation is further supported by the performance of the newly proposed AudioVLM2Vec framework. By incorporating audio signals into LLMs, AudioVLM2Vec achieves substantial gains of +28.5% and +64.8% in overall and audio-centric retrieval, respectively. These results are obtained without training on the provided training set, correct? The extremely lengthy modification texts make this task much easier, resembling a text-video retrieval scenario where the source videos play a less significant role.\n\n3. I am also confused by the statements regarding long-form videos in Lines 212, 482, and 602. The average length of the source videos is only 11.8 seconds, which can hardly be categorized as long-form. Additionally, why is the real long-video benchmark MLVU introduced in Line 130? It appears that MLVU is not directly related to the composed video retrieval task.\n\n4. Are there any performance comparisons for vision-centric retrieval? The main differences appear to arise from the audio-centric retrieval results. How do previous works perform on the visual-centric category within the OmniCVR benchmark?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)