JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation
Authors: Jianghan Chao
jianzhang gao
Wenhui Tan
Yuchong Sun
Ruihua Song
Liyun Ru
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-modal Large Language Model Benchmark
Omni-Large Language Model
Video Understanding
Primary area: datasets and benchmarks

Abstract: Understanding videos inherently requires reasoning over both visual and auditory information. To properly evaluate Omni-Large Language Models (Omni-LLMs), which are capable of processing multi-modal information including vision and audio, an effective benchmark must comprehensively cover three key aspects: (1) multi-modal dependency (i.e., questions that cannot be answered using vision or audio alone), (2) diverse audio information types (e.g., speech, sound events), and (3) varying scene spans.
However, existing datasets fall short in one or more of these dimensions, limiting strict and comprehensive evaluation.
To address this gap, we introduce JointAVBench, a novel benchmark with strict audio-video correlation, spanning five cognitive dimensions, four audio information types (speech, sound events, music, vocal traits), and three scene spans (single-, cross-, and full-scene).
Given the high cost of manual annotation, we propose an automated pipeline that leverages state-of-the-art vision-LLMs, audio-LLMs, and general-purpose LLMs to synthesize questions and answers that strictly require joint audio-visual understanding. 
We evaluate leading vision-only, audio-only, and Omni-LLMs on our dataset. Results show that even the best-performing Omni-LLM achieves only 62.6\% average accuracy, outperforming uni-modal baselines but revealing substantial room for improvement, especially in cross-scene reasoning.

Published as a conference paper at ICLR 2026
JOINTAVBENCH: A BENCHMARK FOR JOINT AUDIO- VISUAL REASONING EVALUATION
Jianghan Chao', Jianzhang Gao', Wenhui Tan', Yuchong Sun', Ruihua Song' Liyun Ru² "Golling School of Artificial Intelligence, Renmin University of China
Baichuan Inc {chaojh, rsong)eruc.
ABSTRACT
Understanding videos inherently requires reasoning over both visual and auditory information. To properly evaluate Omni-Large Language Models (Omni-LLLMs), which are capable of processing multi-modal information, including vision and audio, an effective benchmark must comprehensively cover three key aspects: (I) multi-modal dependency (i.e., questions that cannot be answered using vision or audio alone), (2) diverse audio information types (e.g., speech, sound events), and (3) varying scene spans. However, existing datasets fall short in one or more of these dimensions, limiting strict and comprehensive evaluation. To address this gap, we introduce JointAV Bench,, novel benchmark with strict audio-vides correlation, spanning five cognitive dimensions, four audio information types (speech, sound events, music, vocal traits), and three scene spans (single-, cross-, and full-scene). Given the high cost of manual annotation, we propose an automated pipeline that leverages state-of-the-art vision-LLMMs, audio-LLMs, and general-purpose LLMs to synthesize questions and answers that strictly require joint audio-visual understanding. We evaluate leading vision-only, audio-only, and Omni-LLMs on our dataset. Results show that even the best-performing Omni-LLM achieves an average accuracy of only 62.6%, outperforming uni-modal baselines but revealing substantial room for improvement, especially in cross-scent reasoning. Project page: htt.p:://foittabbeeeettt github.
INTRODUCTION
Humans can understand videos and the real world by seamlessly perceiving and integrating both visual and auditory information across different scenes, where diverse audio signals (e.g. speech, sound, music, or even vocal traits) are used to complement the visual scene in analyses. As illustrated Figure [a). for a multi-scene video understanding task, such as determining the order across a visual object in scene 3 with the dialogue in scene and scene 23, requires complex joint audio-visual reasoning. This process involves recognizing visual and auditory cues, correlating them across distinct temporal and spatial contexts, and reasoning with the acquired relations. Toward the goal of artificial general intelligence, equipping multimodal large language models (MLLMs) with such joint audio-visud reasoning ability is paramount.
While newly developed Omni-LLMs (Team et al. 2023 Bai et al. 2025 Han et al. 2024 Wu et al.
2024a| (Su et al.] 2023) aim process both audio and visual inputs jointly, progress is hindered by the lack of a comprehensive benchmark dedicated to evaluating this crucial capability. Existing benchmarks exhibit several limitations: some lack strict audio-visual correlation controls (Hong
ct [al.|202)) Geng et al. 2024), others primarily focus on static images or simple videos (Li ct al. 2024c; Gong et al. (2024) and mostly cover only limited range of audio types (Yang et al. 20225) Furthermore, nearly all existing benchmarks neglect the complexities of multi-scene reasoning, which
a core component of human cognition.
To address this critical gap, we introduce JointAVBench,, the first comprehensive benchmark for evaluating Omni-LLMs joint audio-visual reasoning capabilities. Our benchmark features system- atic taxonomy covering five cognitive dimensions (e.g., temporal, plot, and long-form reasoning),
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Scene
Scene 3
Scene 23
boy with blue cap and striped shirt appeared...
Come don't act like that... Question: what order the following iterss mentioned the video?
Are serious? have baby...
(a) 'Come don't act like that like come get you'. The woman mention her baby with
(a)
surprised tone (c) The boy wearing a blue cap and striped shirt?
A. (a) (b) (c)
B. (c) (b) (a)
C. (a) (c) (b)
D. (b) (a) (c)
...The man black speaking...
... man brown jacket speaking with fearful tone...
Question: What's the emotion of the speaker that wears brown leather jacket? A. Confident
(b)
B. Angry
C. Calm
D. Fearful
Figure 1: Examples JointAVBench. (a) asks a cross-scene plot-related question that needs the visual information in Scene 3 and the speech information in Scene I and Scene 23 to reason the right answer. (b) asks a single-scene amotion-related question that needs the visual information of the speaker and his vocal traits to answer.
four audio signal types (vocal traits, music, speech, and sound event), and three distinct scene spans (single-, cross-, and full-scene). These features enable us to construct 15 challenging tasks with strict audio-visual correlations, providing unified and rigorous evaluation framework. For example, the task in Figure (b) tests Speaker Emotion Recognition (SER), i.e., single-scene, vocal traits involved, and emotion-related task.
To overcome the immense cost of manual annotation, we propose semi-automated pipeline to generate high-quality question-ansee annserr (QA) pairs. This three-stage process first generates detailed multimodal captions. then synthesizes questions that strictly require joint audio-visual reasoning, and finally performs a rigorous quality assurance step to ensure data fidelity. We then use human labor to filter out unqualified data. This approach enables us to construct high-quality multi-choice benchmark of 2,853 samples that are designed to probe complex reasoning abilities.
We conduct extensive experiments on JointAVBench to evaluate three types of MLLMs: Omni- LLMs, Video-LLMs, and Audio-LLMs. Our results demonstrate that current Omni-LLMs, such as Qwen-omni and Gemini-2.5 2.5 flash, significantly outperform their single-modal counterparts. However, our analysis also reveals that these models exhibit uneven capabilities across different audio types and suffer from substantial performance degradation with increasing scene complexity. Our comprehensive assessment highlights critical limitations current models' audio-visual reasoning capacities, posing the potential for future improvement.
In summary, our contributions can be summarized as follows:
We introduce JointAVBench, the first-ever comprehensive benchmark to evaluate joint audio-visual reasoning capability across five cognitive dimensions, four audio types, and three scene complexities.
We propose a novel three-stage semi-automated pipeline for generating high-quality QA pairs with striet audio-visual correlations while reducing annotation difficulties and costs.
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison between our benchmark and previous ones. Anno.: the construction method, where A for automatic pipeline, A+M for pipeline involving manual inspection, and M for manual pipeline. Modality: the modality involved. V for video, for image, and for audio. Aud. Type: number of different audio signal types included in the dataset or benchmark. AV Corr. Ratio: the ratio of true audio-visual correlated questions, discussed in detail Appendix C.T This metric is evaluated only on the three MCQ audio-visual tasks of AVUT (with two additional audio-visual open-ended tasks yet to be released).
Benchmari/DDtaset
Avg Duration
#QA
Anno Method
Modality
#Tasks
#Audio Types
AV Corn. Ratio
Video Benchmarks
EgoSchema (Mngaal
5,063
Vidco-MME (FU et 20250
1.017.9s
2,700
M
V
MV Bench Li (02415
16.0s
4,000
20
LVBench Wang 20244
4,101s
1,549
0
MMBench-Video Fang
2024
165.4s
1,998
26
Andiovisual Benchmarkk/Daiesets
Music-AVQA at. (002aa
OmniBench Li al 20746
1,142
M
I&A
3
100%
AV-Odyssey Loog ot aL. 2024
4,555
VI&A
26
3
100%
LongVALE
235s
A+M
V&A
3
76.2%
AVUT Yang el M125
67.8s
13,774
A+M
VIA
8
2
77.8%
WorldSense Hong clal.
2025
141.1s
3,172
VIA
26
62.9% 3
JointAVBench (ours)
97.2a
2,853
A+M
V&A
15
4
100%
We provide comprehensive evaluation of current MLLMs on JointAVBench, demonstrating their limitations and highlighting the importance of developing truly integrated audio-visual reasoning Omni-LLMs
RELATED WORKS
2.1
MULTIMODAL LARGE LANGUAGE MODELS
The rise of Large Language Models (LLMs) has spurred interest in extending their capabilities beyond text multimodal inputs (Bi 2024 Achiam al. 2023 Radford al., 2018). Early efforts, such as (Radford et al. 2021 Hurst et al. 2024 Li et al., 2022b: (023b), demonstrate effective fusion visual and textual modalities for cross- moss-modal understanding. Subsequent studies (Chu et al.)
2023 Radford et al. 2023) expand this paradigm to incorporate audio-text integration and achieve significant improvements. Later advances hardware and memory optimization enable video-text
modeling in MLLMS (Team et al. [023 Gao et al. [017 Geng et al.) [022] Huang et al., (024). spanning across various domains and achieving progress such as long video understanding (Wang
et al. 2024a| Yuan et al. 2025 Chen et al.] (024a) and movie understanding (He et al. 2024; Song
et al. 2024). Recent works (Chowdhury et al. | 2025 Shu et al., 2023 Cheng et al.) 20024] Tang et al.
2025b Fu et al.) 2024 2025b; Lu et al. [022 2024]] Su al. 2023 Wu al.] 2024a Han al.
2024) focus on achieving human-like audio-visual joint reasoning ability by interleaving audio, video,
and text. This requires datasets contain QAs with striet visual correlations. To facilitate the development of Omni-LLMs, we present DointAVBench to evaluate the models' audiovisual joint reasoning ability with questions that are fully audio-visual correlated.
2.2 AUDIO- VISUAL. BENCHMARKS
With the development of MLLMs, various benchmarks have been constructed for the evaluation of
MILMs' comprehensive abilities (Wu al. 2024b Li et al. 2024a 2023a Yue al., 2024 Zhang et al. | Sakshi et al.) 2024 Liu et al. [2024]]] et al. (2244)). Early datasets or benchmarks, such
as AVQA (Yang et al. 2022), ) Music-AVQA (Li et al. 2022a), and AVInstruct (Ye et al. 2024). only
focus certain types of audio signals and lack strict audio-visus, correlation. Subsequent works such as Omni-bench (Li et al. 2024cb and AV-Odyssey (Gong et al. 2024) consist primarily of only image and audio, lacking the evaluation of videos. The recent WorldSense (Hong al. 2025) has delved into the problem. However, it lacks strict audio-visual correlation and emphasizes the evaluation of visual tasks. These datasets cannot capture the complex and interleaved auditory and scene details
3

---

Published as a conference paper at ICLR 2026
Table 2: Task categories of our designed taxonomy. In audio signal type, we use SPE for speech, VOT for vocal traits, SEV for sound event, and MUS for music.
Scene Type
Cognitive Dimension
Andio Signal Type
Task Name
Task Code
SPE
Speech- based Timepoint I Localization
STI.
Temporal
SPE
Vision-Speech Sequence Recognition
VSSR
VOT
Speaker Spatial Localization
SPL
Single
Spatial
SEV
Sounding Object Grounding
SO0G
SEV
Sound Event Recognition
SOER
Speaker Emotion Recognition
Emotion
Musical Inference
Long-form
SPE
Cross-scent sces--scee A Asoociation
CSA
SPE, VOT
Multi-plot Ordering
MPO
Multiple
Plot
SPE
Plot Development Prediction
PDP
SEV, MUS
Audio Function Analysis
ATA
Temporal
SPE
Piot Temporal Grounding
PTG
Long-form
SPE, VOT, SEV,
Audio- Visual Detail Memory
AVDM
Full
Musical Emotion Shift Inference
Character Relationship
in video (such the details in Figure T In contrast, proposed lointAVBence focuses the evaluation of diverse audio signal types and multilevel scenes, aiming to conduct a comprehensive and systematic assessment of current MLLMs for joint audio-visual understanding.
JOINTAVBENT BENCH
We propose Bench, a benchmark for evaluating Omni-L.LMs' joint audio-visual reasoning ability. This section will first detail the benchmark's 's core requirements, then the carefully designed data generation pipeline, including (i) omni-caption generation, (ii) QA pair creation, and (iii) rigorous quality control, with statistics provided at the end of this section.
3.1 BENCHMARK REQUIREMENTS
The benchmark construction adheres to three fundamental requirements, ensuring comprehensive evaluation of joint audio- visual reasoning capabilities.
Strict Audio-Visss Visual Correlation. We design a hierarchical taxonomy comprising 15 tasks (detailed Table D. ensuring that each task requires the integration of both visual and audio information to generate answers.
High-quality Video Source. Movie scenes are a natural source of extensive and diverse multimodal data. For our benchmark, we leverage the Short-Films 20K (SF20K) (Ghermi et al. (0224) dataset, which comprises 1,072 professionally produced movies rich in narrative and balanced audiovisual features. We then remove unavailable grayscale videos, retaining 1, 1,046 films that can be used to construct our benchmark.
Multi-dimensional Task Taxonomy. To ensure comprehensive and fine- tne-grained evaluation, categorize our tasks along three key dimensions:
Cognitive Dimension: Derived from a systematic analysis of previous studies (Fu et al.. 2025al Hong et al. 2025), this dimension assesses cognitive abilitics essential for vidco understanding. We define types of cognitive dimensions: temporal, spatial, emotional, plot, and long-form.
This dimension enables a comprehensive evaluation of audio understanding capabilities across audio signal types. We divide audio into four types of signals: speech, vocal traits, sound event, and music.
Audio Types:
Scene Complexity:
This dimension evaluates model performance across videos with varying temporal characteristics, using different scene types to quantify temporal information. We define three types of scene complexity: single-scene, multi-scene and full-scene.
4

---

Published as a conference paper at ICLR 2026
Identification
Audio Caption Generation
Quality Control
Speech
Caption heflnee Aueen
SSpeech (SPE):
Question: What's the emotion the speaking man brown sweater?
Transcription
Where'd you
play poker like that?...
Vocal Traits Caption
Vocal Traits (VOT):
Angry B. Happy Disappointed D. Sad
speak with
Sound Event &
angry mood...
Caption
Sound Event (SEV):
Distractor
footstep
Generation
becomes audible...
Specific
Omni-modal Caption Generation
Music (MUS):
Verification
instrumental
General
electronic piece
Verification
(c)
Video
Captioner
(b)
Long Video
Audio Type Selection
Single-Scene
Video Description: The scene opens close-up shot hands engaged card game...The
SPE
Task-specile
QA
VOT
Caption
Multi-Scene
Pair
brown sweater speaks, gesturing slightly
SEV
Creation
hands.....
Full-Scene
(a)
Video Caption Generation
Pipeline for JointAVBench. Our construction pipeline is three-fold: (a) Omni-modal caption generation, (b) QA pair creation, and (c) Quality control.
Figure 2:
3.2
BENCHMARK CONSTRUCTION
Our dataset construction pipeline is illustrated in Figure 2 which adopts semi-automated process capable handling diverse modality characteristics. More details can be found in the appendix.
3.2.1 STAGE 1: OMNI-MODAL CAPTION GENERATION
Scene Identification. We first split the video into scenes with semantic consistency. Specifically, follow the procedure in Panda-70m (Chen et al. 2024b) to divide long videos into distinct scenes with PySceneDetec[ and and then merge scenes with high semantic similarity, ensuring in-scene consistency. These segmented scenes retain considerable length, enabling capture richer contextual information within each scene.
Video Caption Generation. After scene identification, directly generate visual descriptions for all segmented scenes, ensuring that static features (e.g. in-scene objects and characters) and dynamic features (e.g. transitions between shots and movements of characters) well captured.
Audio Caption Generation. To ensure the diversity of audio types, we follow the requirements to generate captions for each audio type shown in Figure Notably, observe that existing audio models have limitations in distinguishing between sound event and music, and therefore generate their captions simultaneously. Subsequently, refine the audio captions by addressing the hallucination in the caption and separating the sound event caption and music caption using different LLM judges.
3.2.2 STAGE 2: QA PAIR CREATION
To create QA pairs with strict audiovisual correlation, we design various question templates for tasks that LLM cannot easily understand (temporal, plot tasks that require complex audio-visual relation), while leaving other tasks to be curated by LLM to ensure question diversity (general tasks such as Character Relationship Inference). Additionally, when providing cross-modal descriptions, we strictly adhere to each task's modality and scene requirements by inputting only the required modality descriptions from the designated scenes. For example, when generating data for the task Speaker Spatial Localization, we provide video captions along with vocal traits descriptions from only one scene. This procedure can eliminate possible interference from extraneous modalities and scenes.
hhtnps/wwww.cccccdeeer.comm
5

---

Published as a conference paper at ICLR 2026
Scene Type
1000
800
Single
Multi
800
Full
Num of
Num of QAs
600
600
Pending Review
Discarded
26 20..0.4% 4%
400
$2.8%
200
200
Accepted
Vocal
Speech
Music
Sound
0-158
15-30s
30-60s
1-5min
5.1 5-1mmin
In- 15min
15+min
Traits
Event
Distribution of MCQ quality
(b) Distribution of audio types
(c) Distribution of scene durations
Figure 3: Statistics of JointAVBench.
3.2.3 STAGE 3: QUALITY CONTROL
We implement multi-stage quality control process to address issues identified in the collected 9.109 QA pairs, such mismatched question-answer pairs and redundant information. This process employs -popeeii-- verification strategy, where we guide models to use chain-of-thouge approach for step-by-step data filtering.
General Verification. We validate all QA pairs to ensure they meet fundamental standards. This includes Modality Check to confirm that each QA pair necessitates both audio and video informa- tion, and a Logic Check verify that answers are directly derivable from the question's 's context. For instance, question like "What is the emotion of the adull male speaker?" will be discarded if the audio contains only male speaker, the answer be inferred from single modality.
Specific Verification. This stage focuses on task-specific validations. We design the following three specific checks based QA's task: 1) Sequence Check to ensure the correct element order for sequence-based tasks; 2) Ambiguity Check to filter out overly generic QA pairs for complex reasoning tasks (e.g., "What makes the door closing sound?" with "door closing" the answer); 3) Audio Signal Type Check to confirm that the required auditory information cannot be deduced from visual information for sound event and music.
Distractor Generation. For each verified QA pair, we craft three plausible but incorrect distractors create challenging multiple-choice questions. These distractors incorporate diverse misdirections, such as replacing the sound source or confusing details.
3.3 HUMAN VERIFICATION
From the automated three-stage generation process, we obtain 3,974 MCQs. To ensure their quality and factual accuracy. we conducted a rigorous human verification process, and the results are illustrated in Figure 3a Specifically, team of human annotators rated QAs based on four key criteria: (i) answer correctness, (ii) information correctness, (iii) audio-visual dependency, and (iv) question difficulty. Based on these ratings, we categorize these data into three subsets: (1) Accepted: QAs that pass the answer correctness check and score highly on all other criteria, which are directly retained in the final dataset; (2) Pending Review: QAs that pass the answer correctness check but receive lower ratings on one or more additional criteria, which are subject to further selection according to their ratings; and (3) Discarded: QAs that fail the answer correctness check and are removed from the dataset. In total, we retained 2,853 QAs, achieving data retention rate of 1.88% which demonstrates that our automatic pipeline is highly effective at generating data of sufficient quality.

---

Published as a conference paper at ICLR 2026
Table Evaluation results three types of mainstream MLLMs. We evaluate the performance of Omni-LLMs, Video-LLMs and Audio-LLMs on JointAVBench to provide comprehensive analysis.
i For neatness, we use short names of models, where v-SALMONN represents video-SALMONE series.
Model
Size STL.
SPL.
SOOG
SOER SPER MPTI VSSR
CSA
MPO
PTG
AFA
PDP
AVDM
MEST
CRI
Avg
Omai-LLLMs
Gemini2.5-Pro
73,0
59,4
60.8
68.9
35.2
68.1
76.5
43.8
66.0
60.7
65.5
45.7
75.5
66.1
81,9
62.6
Qwen3-Ommi
30B
71.1
43.4
73.8
78.4
35.7
80.3
75.7
42.1
45.2
30.9
59.7
47.3
61.8
69.2
84.0
62.1
5-mm2.5-Ommi
71.3
59.8
72.3
30.6
63.4
77.6
40.4
20.8
47.3
47.3
Gemini2.5 S-ashsh
62.6
43.8
55.7
68.5
23.3
59.1
42.6
39.2
46.2
25.3
65.3
48.4
64.8
67.7
81.3
52.6
40.9
17.6
72.8
39.0
LaM2
7B
20.9
38,8
56.1
67.0
29.6
47.5
48.5
24.0
35.3
30.9
63.6
36.6
38.0
61.7
58.8
46.6
OneLLM
33.0
44.2
45.6
37.8
29.9
29.7
33.9
55.4
31.7
32.9
46.6
44.1
34,5
50.3
38.5
--SLMMNN--o
7B
32.2
30.0
35.1
43.5
14.0
44.7
32.0
25.6
20.0
36.2
55.4
30.1
35.5
66.9
58.8
37.3
SSALLMNN
7B
52.2
25.1
37.8
52.2
19.0
33.3
33.5
30.5
31.7
26.1
48.9
26.9
24.2
37.9
50.3
35.8
Video-LLMss
V1.-22
8B
28.7
37.9
59.8
71.1
23.6
64.1
52.2
42.5
44.2
27.5
63.6
41.9
50.0
68.4
68.3
51.3
LaM33
43.5
58.8
55.8
17.9
69.2
50.0
34.7
43.3
33.6
61.9
40.9
51.8
64.8
49,9
Qwea2.5-VL
7B
33.9
38.8
55.3
59.3
22.9
57.2
47.2
31.7
40.4
32.2
62.5
39.8
40.7
62.9
61.6
47.1
LLaVA Video
30,9
GPT-4o
30.4
34.8
55.7
69.7
11.6
53.6
24.8
40.5
13.5
14.1
51.7
47.3
50.9
56.4
70.9
43.3
Andio-LLMs
Kimi-Audie
56.5
48.6
32,9
53.3
34.3
26.2
65.3
38.7
Owenn-Audin
7B
54.1
24.3
39.5
54.3
34.6
40.0
34.3
33.0
32.7
27.6
55.0
29.8
32.9
46.6
58.1
40.0
3.4 BENCHIMARK STATISTICS
JointAVBench consists of 2,853 high-quality, manually verified MCQs spanning all scene levels and audio types, with an average duration of 97.2s (Table []. A detailed statistical analysis of the benchmark is presented in Figure 3 The number of QA pairs is balanced across diverse audio signal types (Figure (bb), showcasing our benchmark's s comprehensive coverage. Moreover, our dataset spans wide range of video durations (Figure 3c). with single-scene, multi-scene, and full-scene tasks mainly comprising videos of less than min, 1-10 min, and over 10 min, respectively.
EXPERIMENT
This section first demonstrates a comprehensive evaluation of mainstream MLLMs on our proposed benchmark, and then key factors that influence performance to provide valuable insights for future Omni-IL.Ms.
4.1 EXPERIMENT SETUP
Models. Our experiments are conducted on diverse set of mainstream MILMs. To comprehensively evaluate their joint audio-visual reasoning capability across different modalities. categorize them
into three groups: (i) Omni-modal LLMs: Qwen3-Omni (Xu et al. (2025b), Owen2.5-Omm
(Xu
al. 2025ah, VideoLLaMA2 (Cheng et al. 2024), Sadeo-SALMONN-. -2+ (Tang et al. 2025a video-
SALMONN-ol (Sun et al. 2025), video-SA SALMONN (Sun et al. 2024). OneLLM (Han et al. 2024).
AVicuna (Tang et al. 20256). Gemini2.5-PP P (Comanici
(023). and Gemini2.5-Flas
(omanici
et al. 2025); (ii) Video-I.LMs: Qwen2.5-VI. (Bai et al. 2025), LLa VA- (Zhang et | 2024b)
Video- aMM3 (Zhang et al. 2025). VL.2.5 (Chen al., 2024c). and GPT-4o (Hurst al.
2024); and (ii) Audio-LLMs: Kimi-Audio (Ding et al. (225)) and Qwen2-Audio (Chu et al. 2024).
Metrics and Experiment Settings. To achieve evaluation consistency, we follow previous works (Hong al. 2025 Fu et al. 2025a) and use accuracy as the evaluation metric. For fair evaluation, we adopt the following protocols for all experiments. For open-source models, their official codebase with default configurations, while for closed-source models, their official APIs while keeping the configuration by default. To maintain comparability, we select open-source models with comparable 7B parameter sizes and enforce unified sampling of 32 frames

---

Published as a conference paper at ICLR 2026
Vocal Traits
Speach
Music
50
Sound Event
Accuracy
40
30
20
10
Qwen2 5-Omi GeminiZ.5-Faatt
VideoLLaMA2
OnsLLM
Vidko-SALMONN
Kimi-Audio
QwenZ-Audio
Figure 4: Results JointAVBench across different audio types.
Single Scene
Multi Scene
Full Scene
Ansmeyy
50
30
Qwen2.5-0mii
Genini25-Flissh LadeoLLaMA2
OneLLM
IntemVL2.5
LaMe3
Kimi-Audio
Figure 5: Results JointAVBench across different scene types.
across all models. We also ensure that the text input all models is limited to the question text, without any additional contextual information.
4.2 RESULTS AND FINDINGS
Overall Performance. Our results, summarized in Table 3, reveal that current mainstream MLLMs perform sub-optimally on our benchmark, with the best performing model having an average accuracy of only 62.6%. This suggests significant gap in their ability to process omni-modal information. Importantly, Omni-LLMs consistently outperform Video-LLMS and Audio-LLMs, highlighting the critical role of native modality integration. For instance, Qwen2.5-0mni significantly improves upon Intern-VIL across most tasks.
Breakdown Findings. To gain deeper understanding, analyze model performance various task categories and have the following observations.
Models perform unevenly across different audio types, failing tasks requiring vocal traits and speech. We find a significant performance gap among different audio types (Figure Models excel tasks involving sound events and music, likely due to their stronger visual correspondence (where audio often matches visible objects or atmosphere). However, they struggle with more abstract audio, such as speech and vocal traits. This likely due a lack of training data focused on vocal traits, as most audio-visual datasets (I.i et al. 2022a Yang et al.) (022) overlook information like emotion and gender, leading to tasks like SPI., SPER, and MPO being the worst-performing overall.
2) Multi-scene tasks usually yield worse results compared to single-scene tasks, while full-scene tasks often achieve better results. Figure 5 shows the pronounced impact of scene complexity. Models perform well on single-scene tasks, particularly those requiring speech-based reasoning like STL and VSSR. This is likely because single scenes offer stable visual contexts and limited speech content, simplifying rross-modal correspondence. Conversely, multi-scene tasks requiring speech, such as MPO and PTG. yield worse performance, as they demand more complex processing of diverse scenes and cross-scene connections. Interestingly, while most models struggle with multi-scene tasks,
8

---

Published as a conference paper at ICLR 2026
Spatial
55
Qward 5-Omii
Long-for
Gemin S-Fissh
Owenz
GPT-4c Kimi-Astio
Gemiis2.55 5- wah
Accuracy
40-
MONN-SALMONN
Temporal
Hatm
35
GPT-4o
Emotonn
25
20
Plat
0-20
20-40
40-60
60+
Figure 6: Results on JointAVBench across 5 cog- nitive dimensions.
Figure 7: Results on JointAV with dif- ferent number of scenes.
Table 4: Evaluation results of open-source omni-LLMs with different modality utilization. i For neatness, we use short names models, where Qwen2.5 represents @0wen..5-mmni, ViL.LaM22 represents LaM22, and V-SALMONN represents video-SALMONN
Model
Modality N. No Arg
STL
SPL
SOOG
SOER
SPER
MPTI
VSSR
CSA
MPO
PTG
AFA
PDP
AVDM
MESI
CRI
A-V
56.2
71.3
35,3
59.8
72.3
30.6
63.4
77.6
51.2
40.4
20.8
69.9
47.3
47.3
69.9
70.3
Qwen2.5
8
49.3
38.3
33.0
57.8
64.6
18.6
60.1
50.4
43.8
40.4
24.2
68.8
37.6
48.2
70.7
68.5
23.9
64.4
66.5
34.2
57.0
41.3
45.2
A-V
46.6
20.9
38.8
56.1
67.0
29.6
47.5
48.5
24.0
35.3
63.6
36.6
38.0
VILLLaMA2
V
46.6
27.8
37.9
56.8
67.4
27.9
49.3
43.9
28.9
46.0
26.2
62.5
38.7
38.0
62.4
52.7
A
41.4
24.3
35.7
56.1
61.4
18.3
37.3
32.6
35.5
35.6
30.2
60.2
29.0
31.8
50.4
56.4
A-V
38.5
33.0
44.2
45.6
37.8
29.9
29.7
33.9
55.4
31.7
32.9
46.6
44.1
34,5
34.6
50.3
OneLLM
27.8
35.7
31.7
38.5
28.7
43.8
43.6
38.9
31,6
28.6
27.4
47.9
23.1
30.2
41.5
62.4
45.5
39.8
60.0
A-V
35.8
52.2
25.1
37.8
52.2
19.0
33.3
33.5
30.5
31.7
26.1
48.9
26.9
24.2
37.9
50.3
V-SALMONN
5
34.8
23.5
24.3
44.4
49.0
16.8
35.5
29.1
28.3
36.5
26.2
47.2
40.9
31.3
38.3
44.1
they perform better on full-scene tasks, which focus on global narratives rather than fine-grained details. This highlights that improving models' cross-scene reasoning capabilities is needed.
3) Omni-models perform worse on emotional and spatial tasks than single-modal models. As demonstrated in Figure 6 while Omni-LLMs generally perform best in 11 of our 15 tasks, they surprisingly fall behind angle-modality models on emotion based tasks. This suggests that some cases, single-modality models can better focus emotion cues without the distraction of additional modalities. Furthermore, Omni-LLMs perform poorly on spatial tasks like SOOG and SOER, even falling behind Video-LLLMs. This is likely because models primarily rely on spatial information from video and fail to effectively integrate complementary audio cues. This finding highlights a critical limitation in current models' ability perform true audio-visual spatial reasoning.
4) Increased scene number leads to models' performance degradation on multi-scene tasks. Our analysis reveals that increasing the number of scenes adversely affects multi-scene task performance (Figure Z) with accuracy dropping sharply by approximately 20% from 0-20 to over 60 scenes. This underscores the uneven performance of current MILMS across diverse scenes, highlighting a critical area for improvement.
5) Omni-modal Models Demonstrate Effective Modality Fusion. We quantify the effectiveness of joint reasoning by defining N, and N,, as the number of tasks where model's audio-visue (A+V) performance is better and worse than its single-modality baseline, respectively, in Table For all models, N, significantly outweighs N,, confirming that integrating audio and video fundamentally enhances overall reasoning capability. Furthermore, as a model's overall performance increases, its N, count rises while its N, count falls. This pattern indicates that more advanced models better adept at modality fusion. For instance, while early Omni-LLM, (Cheng et al.) 22024) Han et al. 2024 Sun al. 2024) show only marginal gains single-modality baselines, 1imen2.5-Omn's

---

Published as a conference paper at ICLR 2026
dual-m qual-modality performance significantly surpasses its single-modaliy results. This confirms that true joint reasoning a hallmark of mature omni-modal architectures.
CONCLUSION
In this paper, we propose JointAVBench, comprehensive benchmark for evaluating joint audio- visual reasoning, distinguished by hierarchical taxonomy and high-quality, automated generation pipeline. Each question in JointAV is meticulously designed to necessitate the integrated understanding of both visual and specific type of audio input. We further ensure benchmark quality through human verification. Our extensive experiments reveal that even the best-performing models achieve accuracy of only 62.6%, underscoring the substantial need for more powerful omni-modal models with enhanced audio-visual fusion capabilities.
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (No. 62276268) and Baichuan Inc. We also gratefully acknowledge the insightful comments and suggestions provided by the anonymous reviewers.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint 2XXi:24011.02554 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2024a.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wel Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross- coss-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13320-13311, 2024b.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et Expanding performance boundaries of open-souree multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 2024c.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 2024.
Sanjoy Chowdhury, Hanan Gani, Nishit Anand, Sayan Nag, Ruohan Gao, Mohamed Elhoseiny, Salman Khan, and Dinesh Manocha. Aurelia: Test-time reasoning distillation in audio-visual llms. arXiv preprint arXiv:2503.222 2025.
Yunfei Chu, Jin Xu, Xiaohuan Zhou Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou, Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023.
Yunfei Chu. Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo. Yichong Leng, Yuanjun Lv. Jinzheng Hc, Junyang Lin, et al. Qwen2-audia technical report. arXiv preprint arX1v:2407.10759 2024.
Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXi:22507.0626). 2025.
Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504. / 18425, 2025.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098-93 -9100889991,,,,11 2024.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal 1lm. arXiv preprint arXiv:2408. 20t.::2000..0.211... 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang. et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal ilms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24108-2411 2025a.

---

Published as a conference paper at ICLR 2026
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5; Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXhv:2501.0195/ 2025b.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp. 5267-5275, 2017.
Tiantian Geng, Feng Zheng, Xiaorong Hou, Ke Lu, Guo-Jun Qi, and Ling Shao. Spatial- pyramid graph reasoning for action recognition. IEEE Transactions Image Processing, 31: 5484-5497, 2022.
Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio- assion-audio-laguug--vveea benchmark towards time-aware omni-modal perception of long videos. arXiv preprint arXiv:2411.19777 2024.
Ridouane Ghermi, Xi Wang, Vicky Kalogeiton, and Ivan Laptev. Short film dataset (sfd): A benchmark for story-level video understanding. arXiv preprint arXiv:2406. 10221, 2024.
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng. Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, et al. Av-odyssey bench: Can your multimodal Ilms really understand audio-visual information? arXiv preprint 2XXiv:4422.02611 2024.
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26584-26593 26595, 2024.
Yichen He, Yuan Lin, Jianchao Wu, Hanchong Zhang, Yuchen Zhang, and Ruicheng Le. Storyteller: Improving long video description through global audio-visual character identification. arXiv preprint arXiv:241..07072 2024.
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal 1lms. arXiv preprint arXiv:2502.04326, 2025.
Bin Huang. Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower I1m grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14271-14280. 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card, arXiv preprint arXiv:2410.21276 2024.
J Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data. hiometrics, pp. 159-174, 1977.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension. arXiv preprint arX1v:2307.66125, 2023a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13308, 2024a.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108-19118, 2022a.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- -mage pre- training for unified langune understanding and generation. In International conference on machine learning, pp. 12888-1299 PMLR, 2022b.
12

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imag image pre-training with frozen image encoders and large language models. In International conference machine learning. pp. 19730-19742. PMLR, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 22195-22206 2024b.
Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, al. Omnibench: Towards the future of universal omni-language models. arXiv preprini arXiv:2409. 2224099.522., 15272, 2024c.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? arXiv preprint arXiv:2403.00477 00476, 2024.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916 2022.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-fon form video language understanding. Advances in Neural Information 2023.
Processing Systems, 36:46212
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp. 28492-28518 PMLR, 2023.
Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168 2024.
Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual Ilm for video understanding. arXiv preprint arXhv:2312.067200 2023.
Enxin Song. Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18221 188332, 2024.
Yixuan Su, Tian Lan, Huayang Li. Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint 1rXiv:2305.16355 2023.
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma. Yuxuan Wang, and Chao Zhang. saloon Speech-cnhanced andiolvisual large language models. arXiv preprint ar 2466 /5704, 2024.
Guangzhi Sun, Yudong Yang, Jimin Zhuang, Changli Tang, Yixuan Li, Wei Li. Zejun Ma, and Chao Zhang. video-salmonn-ol: Reasoning-enhancedd enhanced audio-visual large language model. arXiv preprint arXiv:2502. 1177,, 2025.
13

---

Published as a conference paper at ICLR 2026
Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li. Zejun Ma, and Chao Zhang. video-salmonn 2: aoptioning-enhanee audio-visual large language models. arXiv preprint 1xXiv:2506.152200 2025a.
Yunlong Tang, Daiki Shimada, Jing Bi, Minggian Feng, Hang Hua, and Chenliang Xu. Empowering Ilms with pseudo-untrimmed videos for audio-visual temporal understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7293-7301, 2025b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2t.i::23/2.///000 1805, 2023.
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634 2024a.
Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng, Xiaohan Zhang. Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv 2024b.
preprint ar
Shenggiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal I1m. In Forty-first International Conference on Machine Learning. 2024a.
Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang. Wenhu Chen, et al. Scimmir: Benchmarking scientific multi-moda information retrieval. arXiv preprint xrXiv:2401.13475 2024b.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.202// 2025a.
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765. 2025b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.155 15115.. 155, 2024.
Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu, Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pp. 3480-3491, 2022.
Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li, Zejun Ma, and Chao Zhang. Acvuhench: Audio-centric video understanding benchmark. arXiv preprint ar Xiv: 1,951, 2025.
Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios. In European Conference on Computer Vision, pp. 146-164. Springer, 2024.
Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large Assion-language models from detailed video description to comprehensive video understanding. arXiv preprint (rXXi::501.078800 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-disciplin multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501. 2025.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang. Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint ar0iv:2408. 13257, 2024a.
14

---

Published as a conference paper at ICLR 2026
Yuanhan Zhang. Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, arXiv preprint arxiv:2410.02713, 2024b.
15

---

Published as a conference paper at ICLR 2026
A MORE DETAILS ON JOINTAV BENCH
A.1 TASK DEFINITION
We construct taxonomy of 15 tasks requiring audio-visual joint reasoning ability based on the benchmark's requirements. The detailed task descriptions with examples are presented in Table Since single-scene tasks and multi-scene tasks require focus on movie details (e.g., visual cues, temporal relationships), question templates are designed to ensure the generated questions both capture critical information and comply with task specifications.
Table 5: Problem Categories and Definitions of JointAVBench's Taxonomy
Task Name
Code
Category Description
Question Example
Speech-hased
Which objects are mentioned only dialogue but not elearly shown in the video, and when does the first object appear in the dialogue?
Locate the temporal position of an object in dialogue
Timepoint
STL
Localization
Where's 's the character that says I'm goema give you a compliment nowwith contemptuous tone located in the video?
Speaker Spatial
Locate the spatial position of character the video
SPI
Localization
Sounding Object Grounding
Licate the spatial position of a sound amitting
What the spatial position of the object that produced the loud bung in the scene?
SOOG
object in the movie
Sound Event Recognition
SOER
Infer what action occurred that caused the sound
What makes hight hight-ptchedd bright sound?
Speaker Emotion Recoynition
What's the emotion of the speaker that wears a brown jacket over blue shirt?
SPER
Ideniiy the emotions of characters
Musical Tone Inference
MPTI
Determine the overall tose of a soene
What is the overall atmosphere of the scene?
what order were the following items mentioned in the video? (a) 'Did you see this guy smoking pat? (b) The police officer bolding an object. (c)
Vision-Speech Sequence Recognition
Determine the chronological order of element appearances
VSSR
"Excuse sir.'
Cross-scene
Identify associations between elements across different scenses
Which dialogue the remaining parts most relevant to what the man does in 18.56s-35. 16s?
CSA
Association
In what order were the following items mentioned in the video? (a) The girl is seen adjusting an oxygen mask on child (b) The woman
Order different audio-visual details across di feeeeennt movie segments
Multi plol Ordering
MPO
trimming flower stems. (c) The man says, "This isn't 't funny anymore."
Temporal
Identify approximate temporal position plot segments
When did the waman in green tulle outfit reveal her success in the music industry?
Grounding
PTG
Audio Function
Analyze the purpose of sound effects and buck ground music in movie segments
How does the video depict the movement of the man in the beige suit?
Anulysis
AFA
Plot Development
Predict future plot developments based on existing plot elements
Which of the following options is most likely to occur after this video ends?
PDP
Prediction
Audio-Visual Dotail
What was the man doing when the police officer asked they were smoking!
AVDM
Test the model's long-term memory capability
Memory
Musical Emotion
Idcutify cmotional changes and trends througbout the movie
How does the cmotional tone evolve from the beginning the middle of the movie?
MESI
Shift Inference
Character Relationship
Infer complex relationships between churacters based on the overall plot
What the relationship between the man in the brown jacket and the police oflicer?
CRI
Inference
A.2 MORE STATISTICS
To further analyze the descriptive characteristics of audiovisual events in our dataset, we performed lexical frequency analysis on both captions and QA pairs, visualized through the word cloud in Figure 8c The results highlight frequent visual descriptors (e.g.. "position"" "object", "location") and auditory terms (e.g., "sound", "speaker", "speech"). Furthermore, we examined the distribution of question lengths (in words), illustrated in Figure Sd which confirms the conciseness of our formulated questions with minimal redundancy. Figure 8a prepeesnnts the quantitative distribution of QA pairs across different tasks, revealing that single-scene tasks exhibit particularly rich audiovisual
16

---

Published as a conference paper at ICLR 2026
360
Scana Type
353
Single
Dificully
331
278
Multi
Num of Ore
230
200
Madwilyy
150
121
1:49
110
133
100
Infomation
066
80%
100%
STL
SPL.
500G
SDBR
SPER
MPT
VSUR
CBA
MPO
PTG
ApA
PDP
AvoM
MESI
CRR
Human Retings
(a)
The number MCQs across lasks.
(b) The distribution of human rating scores.
tone located say du "it iems shown
fol1own
Towing
FORL
Litems
auuund scone
800
moosphere
scene
character
spatt
pptiaa
steree
position 1ioo
Mcos
vi
ded
wheres
located
video-
when
post
tonon
cleadly
Showng
object
.900
overa
aatmsppherr
man
relat
mochlp
woman
252
emotion
speaker
O
der
200
wha t emotion
121
71
tens mention
ment-loned
presuci
video
40
5-9
10-14
15-19 20-24
25-29
30:34
35-39 60-44
45-49
>50
Num
(c) The word cloud presentation of our benchmark.
(d) The distribution of word count.
WorldSense JoinsAvBunch
35
Percantage of bokks
30
15
50
100
150
200
250
300
330
400
Task Data Size
(c) A comparison of task distribution
Figure More details JointAVBench. The number 1-5 Figure (b) are human ratings, represents the lowest score and 5 represents the highest score.
correlations and simplicity in question designs. We also present the comparison of our task sample size with that of WorldSense (Hong al., 2025) in Figure [e This comparison indicates that our benchmark contains more data samples per task compared with WorldSense, which also proves our benchmark's s credibility.
A.3 MANUAL CHECK
To ensure the quality of JenntAVBench, engage group of annotators to evaluate and correct the questions according to the criteria specified in Section 3.3 The annotation process consists two main steps: First, annotators rate the generated MCQs; then, for incorrect MCQs where the correct answer appears in the distractors, they replace the designated correct answer with the appropriate distractor. Through this process, maintain high-quality MCQS while discarding only those that fail
17

---

Published as a conference paper at ICLR 2026
to meet our quality standards. After human verification, discard 17.4% of the data for incorrect answers. Other data are filtered based on the other judging criteria, resulting in 2,853 MCQs.
Figure 8b presents the human evaluation scores of the automatically generated MCQs. The results demonstrate that our pipeline generates high-quality questions, with the majority of MCQs receiving the highest rating across all evaluation criteria. Specifically, find that our QAs are relatively difficult, with the difficulty criteria having high ratings.
A.4 ANNOTATION INFORMATION
To ensure that our manual data check process is rigorous, we have selected professional annotators. Our annotation team provided by a professional crowdsourcing labeling company. Each of our annotators has bachelor's degree from a highly ranked university, is proficient in English, and has passed qualification test on sample tasks before participating.
To assess the reliability of our annotations, we randomly selected 400 samples and had them inde- pendently annotated by two annotators. We calculated Cohen's Kappa coefficient measure the agreement. The resulting Cohen's 's Kappa was 0.713, which is interpreted as "Substantial Agreement" according to established benchmarks (Landis & Koch 1977). This strong level of agreement validates the clarity of our annotation guidelines and the reliability of our final benchmark.
A.5 EXPERIMENT DETAILS
To ensure comparability between MLLMs, we selected similar model parameters and video frames. For open-source models, we selected 7B as the parameter size (we selected VLter5VL2.5-8B (Chen et al. 2024c) since it does not provide 7B model). We also selected 32 frames as the maximum number frames per video to ensure that the frame number did not affect performance. For closed- source models, we adhered to the official model settings. Notably, for GPT-4o (Hurst et al. 2024), we only input video frames alongside the question text, and therefore, the model unable to access timestamps. During the experiment, we randomly shuffled the options to ensure that the distribution of correct answer prefixes was uniform and free from biases. For Gemini2.5-Flash we use the setting: temperature=1.. .0, max temperature=2.0, top hpppp=0.95, top k=64, thinking-True. For Gemini2.5 P-Pro we use the setting: temperature= .0, max www.teeeertaure22...... top-p--t-- 1op-p-0.95, top4 thinking-True.
All experiments conducted NVIDIA H-100 GPUs and can be reproduced using single H-100 (80G) GPU. During the automatic generation process, we used the official API (model name: "qwen2.5-72b-instrurr) from Qwen ensure long-context capability and generation stability.
A.6 CASES
Figure 9 presents detailed examples from benchmark. The questions and options in Join- tAV BAVBench are designed to incorporate both audio and video modalities and to evaluate audio-visual joint reasoning capabilities. We present the task names, few frames of the video, questions, options, and correct answers in the cases.
B DETAILS ON GENERATION PIPELINE
B. VIDEO CAPTION GENERATION
We generate visual captions using Qwen2.5-VL LL Bai et al. 2025 at a rate of frame per second (fps) for each identified scene, regardless of its quality level (low high). To ensure the captions capture both static and dynamic scene information, we carefully design a video captioning prompt as shown in Figure 10
B.2 AUDIO CAPTION GENERATION
To ensure the audio captions contain rich information about diverse audio signal types, generate separate captions for each type. We then utilize an LLM with carefully designed Chain-of Thought (CoTT) prompts reduce caption hallucination.
18

---

Published as a conference paper at ICLR 2026
Vocal Traits, Sound Event, and Music Description. We find that directly using general- lanioolanguage models (ALMs) to generate overall audio captions often overlooks important details and may produce inaccurate results. Therefore, employ Qwen2.5-0mni (Xu et al. 2025a), an open-source multimodal model, to separately generate descriptions of vocal traits, sound events, and music components. Notably, current ALMs cannot reliably distinguish between sound events and music. We address this limitation by generating both sound event and music descriptions initially, then separating them during post-processing. The detailed prompt templates are provided in Figure I0
Subtitle Transcription. For dialogue transcription, our primary objectives are accurate speech recognition and precise timestamp generation. Since general ALMs underperform in timestamp estimation, we utilize Whisper-v3 (Radford et al. (003)) an advanced and widely utilized automatic speech recognition (ASR) system, ensure transcription quality.
Audio Caption Refinement. The initial audio descriptions contain hallucinations (i.e., factually incorrect or repetitive outputs). We employ Qwen-2.5 (Yang et al.] (202) to perform three refinement steps: (I) distinguishing between background music and sound events, (2) aligning vocal characteris- ties with dialogue transcripts, and (3) removing redundant content. The detailed prompt engineering for this process is illustrated in Figure IT
B.3 QA PATR GENERATION
We utilize Qwen2.5 Yang et al. 2024) to generate QA pairs following predefined question templates. For single-scene and multi-scene tasks, we only provide descriptions for high-quality scenes. For full-scene tasks, we include all scene descriptions regardless of quality to ensure no details are omitted. After generating multi-scene QA pairs, verify the interval between questions to ensure they require information from multiple scenes, using the prompt shown in Figure [6] identify their information intervals. Additionally, we require the model to generate brief justification for each answer while generating QA pairs.
B.4 QUALITY CONTROL
In the quality control stage, employ extensive CoT techniques ensure that Qwen2.5 Yang et al. 2024) achieves optimal performance in identifying potential hallucinations.
During the general check, we utilize only the QA pair and its explanation filter out unqualified QA pairs. This stage includes four checks: modality, format, content, and speculation checks. The details of each check are as follows: (i) modality check assesses whether the modality clues used in the QA pair are derived from dual modalities; (ii) format check evaluates whether the answer corresponds to the question format (e.g.. the answer explains two items, but the question asks about only one item); (iii) content check verifies whether the answer can be logically inferred from the question based on the explanation; (iiii) speculation check examines whether the answer relies excessively on speculation rather than concrete evidence. The prompts used in this stage are shown in Figure [3
For the specific check, we design task-specific prompts based on the definition of each task. The prompts for each specific check are presented in Figure [14 and Figure [5] Note that for better adaptation to different tasks, the prompt for the sequence check varies slightly between VSSR and MPO, and the ambiguity check varies slightly between SPL and SOER.
B.5 DISTRACTOR GENERATION
Since generating distractors requires additional information from the video, we generated them after filtering the QA pairs. In this process, we designed generation prompt incorporating various error types to ensure option diversity and complexity, as illustrated in Figure [6] This includes common error categories such incorrect details and temporal/spatial misplacement.
19

---

Published as a conference paper at ICLR 2026
Table 6: AV Correlation Scores for Different Datasets
Dataset
Automatic Score
Human Score
MusIe-AVQA
56.7%
54.5%
OmmiBench
100%
94.0%
AV-Odysey
100%
99.5%
LongVALE
76.2%
75.0%
WorldSense
62.9%
60.5%
JoincAVBench (ours)
93.5%
94.5%
Table 7: Examples of failure cases.
Task
Question
Groundtruth Answer
Model Answer
Where's 's the character that says 'Oh, my God' with vihrant shocked tone located center C. Standing behind the lovescat D. On the right
SPL.
video? A. left B.
A
On the right.
What's the emotion the speaker that a vibrant green tulle outfit with hair rollers and dramatic makeup? A.
SPER
D
A. Surptised.
Suprissd Amused C. Confused D. Excited
In what order were the following items mentioned in the video? (a) 'When do you need to have the van back". (b) The man talked about cars with joyful tone (c) The man driving car along a lined with trees A. (c) (b) B. (a) (b) (c) C. (b) (c) (a) D. (c) (a) (b)
MPO
D
The correct answer is A.
When did the woman in the green tulle outii reveal her success in the music indusiry" 7 A. 51.05.-80.07777 B. 126. 126.38s-174.51s 86.466- 126.388 45.844-51.055
PTG
A
The correct
answer is
C MORE EXPERIMENTS
C.I EVALUATION OF AV CORRELATION FOR PREVIOUS WORKS
We evaluate the AV correlation ratio (the proportion of questions that truly require auditory and visual information answer) for previous works in Table However, since some benchmarks do not evaluate the ratio themselves, we utilize Qwen-2.5 Yang et al.) 2024) to judge the correlation. We utilize our modality check prompt (shown in Figure [3 and sample 1,000 QA pairs from each dataset to judge. We also recruit human volunteers to verify the AV correlation ratio by asking them assess whether the randomly sampled 100 data items from each benchmark require both auditory and visual information to answer. The overall comparison is described in Table[6] indicating that our automatically calculated score has high correspondence to human scores. This score indicates that our benchmark can assess joint audio-visua reasoning ability.
C.2 ERROR STUDY
We'd like to provide some failure cases and deep analysis in this section. The failure cases are selected from the worst-performing tasks.
Based on the failure cases, we find that:
Models fail to understand vocal traits. In the first example, the model fails to find the spatial information based speech and vocal traits. In the second example, the model can't 't find vocal traits based on visual information. These two examples indicate that the ability align visual information with vocal traits remains low in current models and needs to be improved.
Models fail to understand temporal information. In the third example, the question tests the model's ability to understand the storyline and arrange the detailed information. And the fourth example tests the model's temporal grounding ability from long video. These two examples showcase that future works should focus on increasing the model's 'ss ability to understand temporal relationships in misual scenarios.
20

---

Published as a conference paper at ICLR 2026
D LIMITATIONS
We acknowledge that the JointAVBench has several limitations in its generation and experimental evaluation. First, the dataset exclusively derived from SF20K, which may introduce biases in data distribution. Second, our designed taxonomy, while comprehensive, may not encompass all dimensions of audio-visual joint reasoning capabilities. Nevertheless, we have rigorously ensured that the included tasks cover critical aspects and effectively assess the target abilities. Third, due to computational constraints, our experiments were limited to selected representative MILLMs rather than an exhaustive evaluation. We intend to address these limitations in future work through dataset expansion and more extensive benchmarking.
E BROADER IMPACTS
We constructed JointAVBench facilitate research and development in omni-L.LMs and video understanding. We anticipate that this dataset may yield both positive and negative societal impacts. JointABBench offers several potential benefits, including: (1) enabling development human-like agent systems, (2) advancing video understanding tools, and (3) creating assistive software for people with disabilities. However, the dataset also presents certain risks, such as privacy concerns and copyright issues. We believe thorough discussion of these benefits and challenges will lead to more comprehensive understanding of the dataset's societal implications.
F DECLARATION OF LLM USAGE
During our research, we use LLMss as a major dataset construction tool, including dataset generation, quality control, and experiments. During our paper writing, use L.MSs polish paper and correct the defects in our paper.
G SAFEGUARDS
To ensure our benchmark excludes unsafe content, we adopted two key measures. First, we used the publicly released SF20K dataset (Ghermi al., 2024) the foundation, which provides pre-filtered safe content. Second, we employed the official Qwen API during benchmark generation, whose built- in safety mechanisms automatically screen both input prompts and output responses for potentially unsafe video recommendations.
H LICENSE
The JointAVBench dataset is released under the CC BY-NC-SA 4.0 license. Subsequent research using this dataset must comply with the license terms.
21

---

Published as a conference paper at ICLR 2026
Speech-based Timepoint Localization
Sounding Object Grounding
Question:
Question:
Which objects are mentioned only in the dialogue but not clearly shown in the video, and when does the first object appear the dialogue?
Where's the sounding object located in the video?
A. Near the sink.
B. By the exit door.
The frisbee, mentioned at around 329s The pig in blanket, mentioned at around 329s
the hallway.
Near the stalls.
The green box, mentioned around 502s
Correct Answer:
dead porcupine, mentioned around 445s Correct Answer:
Sound Event Recognition
Musical Tone Inference
Question:
Question:
What makes the footsteps sound?
What overall atmosphere scene?
A. The man getting up from the table
Busy and chactic
B. The closing as the woman enters the cabin
Calm relaxed
C. The woman approaching the man
C. Somber and focused
D. The woman pacing back and forth in the cabin
D. Bright and cheerful
Correct Answer: C
Correct Answer: C
Plot Temporal Grounding
Cross-scene Association
Question:
Question:
At what point in the video does the boy in the orange sweater explain to the man in the dark turtleneck that the money he brought is his own, not his father's?
Which dialogue the remaining parts most relevant to what the man does in 18.56s-35. 56s-35.166s?
A. "Excuse me."
A. 668. 04s-710.883
B. 'I'll take care of them."
B. 778.088 08s-816.425
C. "This is true."
C. 759. 388-772.000 772.00s
D. "Stop."
D. 744.29s-751.62
Correct Answer:
Correct Answer:
Charater Relationship Inference
Musical Emotion Shift Inference
Question:
Question:
What the relationship between the man with the
What is the emotional shift depicted in the scenes
blood-stained hands and the woman in the red jacket?
leading up to and including the car ride after the "debs"?
A. The man and the woman are father and daughter.
A. From celebratory and joyful to tense and anxious.
B. The man and the woman are brother and sister.
B. From calm and serene to lively and energetic.
C. The man and the woman are uncle and niece.
C. From celebratory and jayful to reflective and tender.
D. The man and the woman are husband and wife.
D. From reflective and tender to celebratory and joyful.
Correct Answer:
Correct Answer:
Figure 9: Additional cases of JointAVBench. The first and second row represents single-scene tasks, the third represents multi-scene tasks, and the last fourth row represents full-scene tasks.
22

---

Published as a conference paper at ICLR 2026
Prompt for video captioning
Please generate a clear, concise, and detailed cuption for the provided movie video clip. Ensure the description is entirely based on the visual content of the viden, without any speculation, assmptions, uncertain information. Focus capturing the most essential visual eloments while avoiding unccessary repetition or overly verbose language. Describe the scene as follows: Scene Setting: Brielly describe enviromment, location. Only include detuils that are clearly visible in the video.
day, lighting any nctable objects or elements in background.
Characters and Actions: Highlight the appearance, clothing, and key actions of any characters present. Focus on their most significant movements, gestures, and interactions. Do not infer emotions, intentions, buckstory unless explicitly shown through visual Scene Dynamics: Descrihe any importunt changes in the scene. Include the sequence of events and the pacing of the scene to convey unfolds time. Only describe what is visually evident.
4. Emotional Tonc: Convey the mood or atmosphere through the most impactful visual cues, such as facial expressions, body language. envirommental details. Only describe emotions that are clearly expressed through visible actions or expressions.
5. Key Events: Highlight significant events or actions that occur within the scene, locusing on their narrative importance or impact on the characters, Only include events that are explicilly shown in the video. Important Notes:
Strictly factual: Ensure the description is entirely hased on the visual content of the video. Do not include any information that is not explicidly shown or clearly visible.
Avoid speculation: Do not infer or assume anything about characters' thoughts, feelings, or motivations unless they are directly expressed through visible actions or expressions.
Acknowledge uncertainty: If certain details are unclear or ambiguous, do not include them in the description. Focus only on what is clearly visable.
Exclude text: If there subtitles text displayed the bottom the video, do not include them in the caption. Focus only the visual elements of the scene.
Be concisc: Avoid unnceessary details repetition. Focus on the critical visual elements that define the secne. Use vivid but economical language to paint a clear and engaging picture of the scene for someone who cannot see the video.
Prompt for sound event & music captioning
Descrihe the audio by focusing ONLY on the following detectable elements:
1. Sound Evenis (if clearly present):
Non-speeci sounds: (e.g.. crumpling, fontsteps, door closing, glass hreaking).
Non-verbal vocalizations: (o.g. laughter. sighing, coughing, crying, humming, screaming).
-Characteristies (only unambigueus):
Pitch (high/low), ), timbre (brigh/mufiled), rhythm (steadylerratic), volume (loud/sal)).
Do NOT guess sound sources (e.g., no "paper crumpling" just "a crumpling sound").
uncertain, omit the detail entirely.
2. Background Music (if clearly present):
Instruments (c.g.
piano, strings, electric guitar).
Moodie.g. g.. cheerful, lense. melancholic).
Avoid technical lerms (BPM, key, scales).
Rules:
Do NOT describe lyrics vocal melodics.
the music is amhiguous (e.g, genre unclear), only state ohservable fentures.
Critical Constraints:
Do NOT describe speech. spoken words, conversation content, or verbal interactions.
Never use speculative language (e.g. "likely." "probably." "seems like"). >
a sound cannot he confidently identified, skip it.
No timestamps, durations, or speaker demographics (e.g.. "a child langhing" + just "laughter").
Prompt for vocal traits captioning
Please analyze the provided speech audio and generate a strictly factual description following these requirements:
Output format for each utterance:
Speech Content: [Exact dialogue content]
Emottio: [Obeerved emuttional lone] (eg. huppy, sad, angry. fearful, surprised, disgusted, excited)
Speaker traits: [Directly discemible characteristies like agelgender if evident from voice]
Rules:
(1) Only describe emotions cleatly conveyed through vocal tone
(2) Note speaker characteristics ONLY when immediately apparent from voice (e.g." "child-like voice") sudio contains
(3) add interpretations beyend
(4) Process each ulterance sepurately
(5) Non speech audio (music or sound only): output Speec- audio: skip analysis]" and stop
Figure 10: Prompts for generating omni-modal caption.
23

---

Published as a conference paper at ICLR 2026
Prompt for cleaning and separating sound event caption and music caption
You are an expert in audio data cleaning. Your task is to clean and organize audio captions by separating them into two distinct categories: [music] and [sound event].
Please analyze the provided sudio caption and separate into two parts:
[musicl Include descriptions related to music, such as buckground music, musical instruments, and any emotional or atmossperic content (e.g.,
sad). music described, output "None".
[scund event: J: Include descriptions of Non-masical sounds (e.g.. actions, engine noises, wind) and Non-verhal vocal sounds (c: laughter, sighing. coughing) (EXCEPT for any are described (after filtering out Crumpling sounds), ourput "None". Do not include any content related specch, dialogue, or verbal communication in either section. Remember not to generate any explanations notations after the separated outpur Outpul Format:
"Crumpling sound" these should be completely omitted). sound
[must péescription of music and emotional characteristics if any; None [scund e eveat] idescription of sound events; or None
Example: jexamplec
Prompt for cleaning vocal traits caption
are an expert speech subtitle analysis, have access to follawing:
Vocal traits: The text that deseribes the emotion and other characteristics of speoch sentence by sentence, each utterance contains spooch content, emotion, and speaker traits.
Subtitle: The transcription of speech. Each line the subtitle represents single phrase.
Your task to evaluate the prvvided vocal traits by comparing them with the corresponding subtidle. The goal to determine whether contain usable information based on their aligmment structured appeouch.
the
subtitle. Your analysis should rigorous
Please follow the guidelines helow:
1. Comparison with Subtitle:
Directly compare the Speech Content text with the provided subtitle.
Discard any speech content marked with neutral emotion (e.g.. "neutral tone", "neutral mond") immediately. regardless of subtitle
alignment.
For non-ncutral speech content:
If the speech content contains phrases that overlop or align with the subritle (e-g., shared keywords or contextual similarity), procedd to the next stage.
alignment is found, the vocal traits unusable.
Ouipui of Emmtiional Information:
For each utterance:
Only the vocal traits align with the subritle and are non-neutral:
Replace any specch content in the vocal
queted reforeneed dialogue) with the matching phrase from the subtitie. Preserve all other emotionaltomal descriptors (e.g. "sad mood." "English accent").
Format the ourput as coherent description combining the subtitle content and emotional features.
alignment exists, output !Unavallabee".
Always prefix the final outpui with "OOuipul]' or "Unavailable]".
Provide the final cleaned atterances in the following format:
[Output] jutterances with emotionaltonal information; if subtitle roughly matches or (Unavailable] if NO utterance is available Example: jexample)
Input:
Vocal Traits: [vrcal
Subtitle: {sublitle)
Please follow the guidelines to clean the vocal traits text, and provide the final output the specified format, Remember to output
'[Unavall labblllee only when utterance available.
Figure 11: Prompt for audio caption refinement.
24

---

Published as a conference paper at ICLR 2026
Prompts for generating single-scene QA pains
Your task to generate a question answer pair hased on the instructions. The question must utilize both visual and andio information (o.g., speech, sound hints or details about the answer. The answer should be precise and concise, and include an explanation justifying the answer. If the provide sufficient information to generale valid question answer pair, respond with "Linavailablee"" Format the output as follows:
music). Generate question-answer pair based this analysis, ensuring the question does not contain any
material
[Quest Moneetoon]..... cor.
[Answes]
[explanation -.....
Please follow the instructions (instrucsttttiion Example: ( (examples)
Here are input material: (ingut.modaliy }
Please follow the instructions and refer to the examples provided assist with your question design.
Prompts for generating multi-scene QA pairs
Your task to generate five different question-answer pains based on the instractions, Each question must integrate information from both the audio and video modalities, ensuring that neither modality alone can provide the answer. Impurtantly, the information used to formulate each question should he derived from few consecutive segments of the material, rather than from single segment the cntire content.
generated question- answer pairs should be unique avoid overlapping content. The questions should designed without revealing hints or details about the answers. The answers should be precise and cuncise, accompanied by briel explanation that justifies combined audio-visual information the segments. Use the video description enhance your undorstanding of the material.
the response based
Format the output as follows:
:Output
[Ques:ien:....
[answer] -....
EExplanation
follow instructions below:
Generate five different question-answer pairs that (instruction)
Example: (ode:(examg (examples)
Here the input material: (segments
Please follow these instructions and refer to the examples provided to guide your question design.
Prompls for generating Jull scene QA puins
Your task is to generate live different question answer puirs based on the instructions. The questions must utilize both visual and audio information, cannot answered by information from only one modality. other. The questions should also be derived based on the whole movie, not just few segments. Generate the question-anse answer pairs based on this analysis, ensuring each question does not contain uny hints or details about the answer. The answers should be precise and concise, and include explanation justifying understand the material.
generated questions should different
for each question. You the video description to help you hetter
Format the output follows:
:Output idl
[Queatt lon):ttto....................
[Ans
[Fxplan lanttlaattt.. ion) .....
Please follow the instructions below:
Generate five different question-answer pairs that {instruction)
Example: ( (exmples)
Here are the input material: (ociial:ssemmman (segments jifo]
follow the instructions and refer the examples provided assist your question design.
Figure 12: Prompts for generating QA pairs
25

---

Published as a conference paper at ICLR 2026
Prompt for modality check in general check
You are multimddll understanding ussistant. You have access to the following:
Question: question related the clip.
2. Answer; An answer provided to the question.
Explanation: An explanation supporing the unswer.
Your task is evaluate whether the question modalities, Please strictly base your judgment on the information explicitly required to answer the question, as well as the content of the modalities beyond is explicitly stated in the
be answered using modality (either video audin) if it requires both
peovided answer explanation. Avoid making assumptions about question, unswer, or explanation.
Please follow
steps complete your evaluation:
1. Information Analysis:
Analyze the question to identify the specific visual and auditory details required to answer Does the question require visual details (e.g., objects, actions, settings) anditory details (c.g., speech, sound effects, music)? Extract the visual and the auditory information from the explanation to determine which modalities are used to support the answer.
Modality
Based on the analysis of the question and explanation, determine if the required information can he obtained entirely from one modality audin) both audio visual modalities are necessary.
(either
Determine if the question can be answered using only the extracted video text.
Determine the question can be answered using only the extracted madio text. the question requires information from both the video text and the audio be answered, then considered feasible use both modalities.
Conclusion:
Provide your final determination: Output [YFS] if the question explicifly requires information from hoth videa and andio moda answered correctly, or the answer and explanation rely on information from both modalities. Otherwise, outpur [YES], Here is the
modalities lities to he
question: qquestion]
Here the answer: (
Here the explanation: (explanation)
Please complete the Information Analysis, Modality Assessment, and Conclusion stages with the special answer token YES) or [NO].
Prompt for checking furmal content and speculation in in general check
You are quality evaluation assistant. You have access to the following:
Question:
question related given context.
2. Answer: An answer provided to the question.
Explanation: An explanation supporting the unswer.
Your task evaluate the quality the question-answe pair performing two checks: format cheak and content check. Please strictly base your judgment on the information explicidly provided in the question, answer, and explanation. Avoid making assumptions beyond
stated. Please follow these steps to complete your evaluation:
1. Format Check:
pieces information asking for. Check if the answer addresses all the pieces of information requested in the question.
Analyze the question to determine how many
the question asks for only ooe piece information and the answer fully addresses it, proceed tn the content check, only addresses
the question asks for multiple picces of information but
them, ourpur '[NOJ' and stop.
2. Content Check:
Analyze the explanation to determine if reasenuble logically sound.
Check if the answer can be derived from the explanation and the answer is correct hased on the context of the question.
the explanation is reasonable and answer is correct supported by the explanation, output [[YES]'.
the explanation is unreasonable the answer cannot be derived from the explanation, output 'NOO]: Speculation Check:
Analyze the explanation desermine the answer relics heavily speculation rother than concrete evidence or logical reasoning.
If the explanation provides clear, evidence-base based reasoning or logical steps to derive the answer, proceed to the tinal output. relies assumptions, guesses, unsupported claims, output [NO] and stop.
Final Output:
both format chock content check pass, outpot [YES]'. eitber check fuils, output '[NO]'.
Here the question: (question) Here the unswer: (amswer) Here is the explanation: (explanation) Pleuse complete the Formal Check, Content Check, and Final Output stages with the special answer token 'YEES'' or '[NO]'.
Figure 13: Prompts for general checks
26

---

Published as a conference paper at ICLR 2026
Prompt for sequence check
You quality control assistant for video-base bused yuestion-asswerig pairs. Your task is validate whether given QA about element ordering in videos is correct und properly sourced. Follow this 4 4-stage process:
Stage Element Extraction
Extract all elements to be sorted from the question (Format: (alfelementi], (b]jelement2) etc.) not, immediately ourpat "[NO]"
Verify exactly elements
Stage 2: Occurrecce Localization
For each extracted element:
Seurch through all video segments lind the first
Record for each element:
Segment ID of first appearance
Modality type (video capion//thiiillkpeechh emotion)
any element cannot found - Ourput "[NO] (unverifiable element: nnmmm)
Stage 3: Element Validation
Verily the located elements meet these criteria:
Unique Segment Check:
All elements must appear in different segments
any segment shared Output "[NO] (co-occurring clements: [elementi] [element2] in segment
2. Modality Diversity Check:
Elements must come from 2 different modalities
all same modality Output "[NO] (single modality: |modality -ypel)"
Stage 4: Order Verilication
Sort elements by their first appearance segment ID (ascending)
Compare against the provided answer:
orders
Output "[YES]"
If orders differ Output "[Corrected]" with proper order and explanation
Output Format:
[val Ivaiddating] :4-stage analysis.
[Output:]] [VES/NO/Comectel)
[Correctied:
(c) (if (papliicalle)
[Explanation (if Correctod):
(a) [element]]]: appears in segment [X] (Imodality)
(b) [elemen1]]: first appears in segment [Y] ([modality])
(c) [element]]: first appears in segment [2] (Imodality)
Provided Information:
Video Text: [segments infa)
Question: (question)
Answer: {answer)
Explanation: {esplanation)
Please perform the stages above earefully in [Validating] and provide the final output the specified format.
Prompt For ambiguity check
You are QA evaluation assistant tasked with filtering incorrect low-quality question-anse wnswer pairs based video and audio context.
Follow this structured evaluntion:
Phase Specificity Check
Check the 'answer' is overly generic (e.g., fails distinguish between objects/agents),
'question' asks "Where occurs in wooden house), mark as "[NO]" (lacks specificity).
Example:
sound source located?" the 'answer" is "in wooden house" (while entire video
Output: Proced only "[PASS]", clse, output "[NO]".
Phase Sound Source Ambiguity (Video Context)
Using the "Video Caption', verify if other objects the scene could plausibly produce the sound mentioned in the 'question".
Example: the 'question' asks "What caused the splashing sound?" and the VViden Caption" only describes "person pool," but other water-related objects exist, mark as "[NO]".
OOutput Proceed only "IPASSI":
output "INO]".
Phase 3: Cross-Modality Dependency
Determine the 'answer' can be derived solely from VVideo Caption' "question' + commonseuse (ignoring 'Sound Event').
Exaupple: the 'question' asks "Where the splashing sound coming from and the "Video Caption" mentions "a beach with
commonsersee suggests "ocean" - mark as "[NO]" (audio not needed).
Soundd Event' is required (e.g., distinguish between similar objects), mark "[PASS]".
Outppt: If "[PASS]" in all phases, output "[YES]": clse,"[NO] "[NO]".
Final Output:
Only "[YES]" or "[NO]" based the above chocks.
Provided Information:
(segments info)
Here the question-answer pair to he analyzed:
Question: [question)
Answer: {answer)
Expllaation: (explanation)
Perform the analysis stop-by-step and output either "[YES]" "INO]" based evaluation.
Figure 14: Prompts for sequence check and ambiguity check.
27

---

Published as a conference paper at ICLR 2026
Prompt for sound event check
You QA evaluation assistant tasked with filtering incorrect question-answeer pairs based on video and sound event information.
Follow this phased approsch:
Phase OlI Screen Sound Check
If the Answer descrihes an off screen sound event (e.g. "An off screen object"), oulput [NO].
Phase Sound Event Presenee Validation
Extract the sound event mentioned in the Question, Answer, Explanation.
Check this sound event exists the provided Audio Text. If not, output [NO].
Phase Contextual Consistency with Video
Using only the Video Caption fignore Explanation),, verily the sound event logically lits the scene. sound
Example:
"glass breaking" the Video Caption lacks glass related objectsfacionss output [NO].
Do not speculate; reject if the video lacks supporiing evidence.
Phase 4: Final Judgment
all phuses pass, oulput [YES]. Otherwise, output [NO].
Here is the provided information: (segments_info info)
Question: [question]
Answer: {answer)
Explanation: (explanation)
Perform the four-phase analysis and ourput either '[ESS' or '[NO]'.
Prompt for music check
You are QA pair evaluation assistant. Your task is to determine whether given question-answer pair is valid based on the provided vidco description and music content. Follow these steps:
1. Phase 1: Music Information Validation
Extract the music related information used
"Answer" and 'Explanation" of the QA puir.
Check if this music information appears in the provided 'Music Content".
the music information is not found in the "Music Content', oulput "INOJ" (invalid QA pair).
Phase 2: Visual Information Cmss-Check
the music information is valid (from Phase 1), analyze whether the emstion/atmosphere described in the music can also be inferred from the "Vidco Caption' (e.g.. character expressions, scene mood, events).
Examppe:: If the music mentions a "sad atmosphere" and the video shows "a characier crying." the music info can be inferred visually - 'NOO'; music describes a "warm atmosphere" and video caption mentions "hright lighting." the music info can inferved visually '[NO];; If the music's 'sssicsssin emmotionalltmoosherii cues can be derived from the video alone, output '[NO].
Phase 3: Final Judgment
the pair passes both Phose and Phase (i.e., music info is valid and cannot be inferred visually), output [YESS'.
Otherwisc, output 'NOO'.
Here the provided information: [segments_info)
Question: [question]
Answer: (answer)
EExplanation: (explanation)
Perform the three-phase unalysis and output either '(YES]' or 'INO,.
Prompt for vocal traits check
are an assistant tasked with validating question-answer (QA) pairs generated from video content, specifically focusing on the use speech emettion data (Speech Content, Emotiio,, Speaker Truits). Your goal is to filter out incurrect or weakly supported QA paizs by following phased approach:
Phase I: Extract Utilized Speech Emotion Information
From the Question and Explamation (if provided), identify:
Speech Content: Exact phrases/words from the audio used
the question.
Emmotion: The emotion label (c.g.. "angry." "Joyful") tied to the specch content.
-Speaker Traits: Any speaker characteristics (e.g.. "deep voice." " "child") referenced.
Output: List only the explicitly used compunents. If nome are used, stop and output "[NO]".
Phase Verify Grounding Provided Speech Emotion Text
Check if the extracted Speech Content, Emotion, and Speaker Traits from Phase appear verbatim unambiguously in the provided
speech emotion text.
Example: If QA pair "confused" emetion for "What?", but the speech emotion lacks this pairing, it fails.
Outppt: If any extracted component is missing. output 'INO]'
Phase Assess
Based Emotion Inferrability
For the Speech Content and Emotion pair used in the QA pair, determine if the emotion could be directly inferred from the text alone
(e.g.. "I'm furious" - "angry"). ).
Disqualify: Obvious cases (c.g.. sarcasm-free explicit statements, clear interrogativee).
Outpur: If inferrable, output '[NO]'.
Phase Check Vidco-Based Emation Redundancy
Determine if the Emotion used in the QA pair could also be elearly deduced from the video caption (e.g.. "sbe frowns" - "sad").
Note: Assume video captions describe visible emotions unless stated otherwise.
Output: If deducible, output "[NO]-
Final Decision
Only output 'YES|' if phases are passed (i.e., the QA pair uses non-inferrable, video-independent speech emotion data with explicit grounding).
For any phäse failure, output 'INO)".
Provided Information: (segments
Question:: [question)
Answer: (answer)
Explanation: (explanation)
Perform the analysis step-by-step and output either "IYESI" "INOJ" based on your evaluation.
Figure 15: Prompts for audio check.
28

---

Published as a conference paper at ICLR 2026
Prompt for interval identification
You are an expert in finding all the continuous segments needed to answer a question answer pair. Your task is to identily the minimal continucus sequence movie segments (neither the first nor last segment) that contains all the necessary information to answer the given question.
You will be provided with:
1. A question answer pair
explanation how answer derived
Complete information about all movie segments (limestamps, video descriptions, audio deseriptions, and subtilles)
Instructions:
Carefully analyze the question and answer explanation understand what information required
2. Examine all movic segments scqueutially to locate where the relevant information begins
Determine where last necessary piece information appears
4. Select the earliest segment where required information starts (start segment) and the latest segment where required information ends (end segment)
Ensure:
1. The selected segments form a continucus sequence
The sequenee
from the very first the very last segment
3. All information aceded to answer the question is contained within this sequence
The sequence as compact as possable
Output Format:
Provide your response in this exact format:
[Start]: I: peegment number, [End]] psegment number; [Rat onale]: :hrief explanation of why these segments choseng
Important Notes:
the answer roquires information that only appears disjoint segments, seloct smallest continueus sequence that contains all
relevant segments
start segments must different (cannot be the same segmentt
Never choose segment 0 and the final segment at the same time
Input:
Here the question: [question)
Here is the answer: [answer)
Here the explanation: (explanation)
Here are the segments information:: sugmennss.inff)
Please follow guidelines and the inpat material identify the start segment and end segment question answer pair, Make
sure that the generated output follow output formar.
Prompt for generating distractors
You are an expert in generating multi-choice question. Your task is to generate distractors based on the guidelines. Given the following background information. question, correct answer, and rationale, generate three incorrect answer options (distractors) that closely mimic the correct answer in terms of length, format, and style. The distracters should appear reasonable to someone doesn't I fully grasp concept must contain subtle emoors (factual, logical, contextual).
Please follow these guidelines to genertt distractors:
Seelective Modification **; Alter specific elements such as character actions, dialogue, objects, or settings to create plausible yet incorrect options.
2. *MMintain Plausibilitty** * Ensure each distractor could feasibly occur within the context of the video, making them appear credible
based visual and audio cues.
3. **Inoorporate Diverse Misdirections***
Action Confusion Modify or swap character actions or events ways that the context are incorrect.
*Dialogne Adjustmeent**: * Propose believable alterations to dialogue audio cues that didn't 't actually occur.
**Objoct or Setting Misdirection "* Suggest plausible but incorrect details about objects, settings. or visual elements.
*SSpeech Emotion Alteration**; Medify the deseribed emotional of speech content while keeping the words themselves accurate.
**Sound Event Manipulation"* **; Change spocific sound effects or cavironmental audio cucs to similar but incorrect versions that could plausibly the context.
*MMuical Aumosphere Shift*; Adjust the described mood or emational impact of background music to different but related atmosphere.
4. ** **Inoopprate Partial Tru Use true audio-visual details or partial truths within the distractors to add complexity, casuring these elements do not directly answer the question but make the distractors more compelling.
** Avoid Obvious Falsities**; Shift the context details significantly without creating options that are blatantly wrong or unrelated the video.
**Ensure Distinct Incornettness*+ + Craft distractors and listened to the video, challenging their attention to detail.
be clearly identifiable as incorrect by someome who has closely watched
Requirements for Distractors:
1. Plausibility: Each distractor should seem conrcc at first glance, matching the tonc and structure of the correct answer.
Variety: Errors should vary (e minor inaccuracies, Ilipped terms, oversimplilliationn, or common misconceptions). Consistency: Maintain same verb tense, technicality. and formatting as the correct Format the output as follows: [Distractos 1] jIncorrect but plausible option,
[Distractor 2] JIncurrect but plausible option;
[D1 stactor 3] [Incorrect hut plausible option;
Provided Information:
Background infromation: (segments info)
Question: {question)
Correct Answer: { [answer]
Answer Rationale: (esplanation)
Now generate three high quality distractors for the given question correct answer the specified format. NOT provide option rationale.
Figure 16: Prompts for interval check and distractor generation.
29


Reviews:
[{"review_id": "9wa6m4chBy", "reviewer": "Reviewer_1yxd", "summary": "The JointAVBench benchmark proposed a comprehensive evaluation tool that covers 5 cognitive dimensions (temporal, spatial, emotional, plot, and long-form), 4 audio types (speech, sound events, music, and vocal traits), and 3 scene spans (single-scene, cross-scene, and full-scene). It addresses the shortcomings of existing datasets in terms of multimodal dependency, audio diversity, and scene complexity.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Achieves a 100% Audio-Visual Correlation Ratio (AV Corr. Ratio), where every question requires the integration of both audio and visual information to answer, avoiding the flaw that single-modality can solve questions in existing benchmarks.\n- Conducts a comprehensive evaluation of three types of mainstream models (Omni-LLMs, Video-LLMs, Audio-LLMs) and analyzes performance differences across audio types, scene spans, and cognitive dimensions, providing in-depth insights for model optimization.", "weaknesses": "- The benchmark exclusively uses the SF20K short-film dataset (1,046 films after filtering) as its video source, but this choice introduces severe scene bias and data homogeneity: Films are professionally produced, narrative-driven content—they do not reflect real-world audio-visual scenarios (e.g., surveillance footage, live streams, industrial monitoring, daily vlogs) where Omni-LLMs are likely to be applied. \n- The \"semi-automated pipeline\" (omni-caption generation → QA creation → quality control) relies heavily on LLMs (Qwen2.5-Omni, Qwen2.5-VL) but fails to validate the accuracy and absence of hallucinations in LLM outputs—this undermines the entire benchmark’s credibility.\n- Meanwhile, the authors use Qwen2.5-Omni and Qwen2.5-VL, and having these models serve as both annotation models and evaluation models will introduce bias. This causes the answers to be more inclined towards the understanding and responses of these two models, leading to biased evaluation.\n- The dataset adopts movies, which focus heavily on verbal dialogue. This results in greater bias in the evaluation of natural sounds and music, making the dataset unsuitable to be used as a benchmark for evaluation.\n- No information on annotator qualifications (e.g., whether they have expertise in audio-visual analysis) or inter-annotator agreement (e.g., Cohen’s Kappa coefficient). If two annotators disagree on 30% of QAs, the \"accepted\" data is subjective and unreliable.\n- The manuscript uses \"official codebase with default configurations\" for open-source models  but \"official APIs\" for closed-source models . However, it does not disclose API parameters (e.g., max tokens) or whether closed-source models used larger parameter sizes  This makes performance comparisons meaningless—higher accuracy could stem from larger model size, not better joint reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSPHGiB8FH", "reviewer": "Reviewer_Qjdf", "summary": "This paper introduces JointAVBench, a new benchmark designed to evaluate the joint audio-visual reasoning capabilities of Omni-LLMs. JointAVBench is organized according to a comprehensive, multi-dimensional taxonomy. With the proposed three-stage semi-automated generation pipeline, the data acceptance rate during human verification reached 71.8%, significantly improving data collection efficiency. An extensive evaluation of current Omni-LLMs, Video-LLMs, and Audio-LLMs on JointAVBench highlights significant performance gaps and specific areas of weakness.", "questions": "Given the submission deadline, the exclusion of some recent models from the evaluation on JointAVBench is understandable. Nevertheless, the paper would be significantly strengthened if the authors could include performance metrics for models like Qwen3-Omni, video-SALMONN-2+, and Gemini-2.5-pro, as this would provide a more comprehensive reflection of the current capabilities of Omni-LLMs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. The proposed semi-automated benchmark generation pipeline is reasonable and and effectively lowers data labeling costs.\n- The comparative analysis of various models reveals the limitations of existing models, which in turn reflects the value of the benchmark.", "weaknesses": "The benchmark's exclusive reliance on pre-existing datasets raises two methodological concerns. Firstly, it compromises the integrity of the evaluation, as the test data may be contaminated (i.e., previously seen by models) or decontextualized. Secondly, this approach circumvents the foundational challenge of raw data acquisition and curation. The fidelity of any benchmark is fundamentally contingent on its source data, and a robust construction pipeline should therefore incorporate an automated or semi-automated mechanism for this essential process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGampxRSy1", "reviewer": "Reviewer_77mz", "summary": "The authors introduce a benchmark for evaluating models that jointly process audio and vision. The dataset consists of 2.8k QA pairs organized into fine-grained categories. Unlike existing benchmarks, it explicitly includes diverse audio types—speech, music, vocal traits, and sound events. In addition, the data are annotated along cognitive dimensions (e.g., temporal, spatial, emotional) and by scene type (single, multiple, or full, indicating how much of the video is needed to answer a question). The evaluation reveals that models struggle disproportionately on certain question types, and even the strongest model assessed achieves only 56% accuracy on average, highlighting substantial room for improvement.", "questions": "Could you provide a breakdown of each question type's size in the benchmark? (as broken down in Table 2)\n\nTypos\n- line 481: \"JointAVBench is\" - missing space.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The dataset is automatically generated and then manually inspected, an important step that strengthens its reliability as a benchmark.\n- I appreciate the fine-grained categorization of question types, which enables more detailed error analysis of current models.\n- The analysis in Table 4, comparing joint modality performance with single-modality performance, is particularly insightful and underscores an important dimension for evaluating “omni-models.”", "weaknesses": "- Although the dataset’s fine-grained design is valuable, the effective size of each subset is necessarily small given the overall scale of 2.8k examples, which limits the strength of conclusions about model performance on specific question categories.\n- While the dataset has been manually inspected, reporting human performance from an independent set of annotators would provide a useful reference point for comparison with model capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TfGH1j9j6d", "reviewer": "Reviewer_EKth", "summary": "This paper proposes yet another audio-visual understanding benchmark called JointAVBench. The benchmark is trying to capture audio-visual correlation in videos, especially focusing on questions that cannot be answered by visual information alone. The authors also mention the idea about multi scene-spans and emphasize that their benchmark exhibits multiple scenes.", "questions": "See weaknesses about the definition of scenes", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Paying attention to the role of audio in video understanding is needed in the research community. \n\n2. They provided webpage for the benchmark and their experiments spans a range of audio-visual LLMs, which might be useful information for the research community.", "weaknesses": "1. Technical Contribution:\n\nI have significant doubt on the technical contribution of this paper/benchmark. Given that it is mainly focused on audio in video, a clear discussion of how important the audio modality in these tasks is needed with supporting evidence in experiments. Please also compare to the following papers:\n\n[1] Yang et al. \"Audio-centric Video Understanding Benchmark without Text Shortcut\", https://arxiv.org/pdf/2503.19951.\n[2] Hong et al. \"WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs\", https://arxiv.org/abs/2502.04326\n\nwhich also introduce audio-visual benchmark with audio-centric data. In particular, [1] also includes mainly questions that cannot be answered with visual modality alone, and also uses both speech and audio events, so the first two points are already being done previously unfortunately. \n\n2. Ambiguity in the definition of __scene__:\n\nRegarding the 3rd contribution, I am unsure why datasets like video-MME are not multi-scene. I don't see a clear difference between the proposed dataset and videos in video-MME. \n\n3. Careless Writing:\n\nPlease correct the question marks on page 3 for those citations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)