OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
Authors: Caorui Li
Yu Chen
Yiyan Ji
Jin Xu
Zhenyu Cui
Shihao Li
Yuanxing Zhang
Zhenghao Song
Dingling Zhang
Heying
Haoxiang Liu
Yuxuan Wang
Qiufeng Wang
Jiafu Tang
Zhenhe Wu
Jiehui Luo
Zhiyu Pan
Weihao Xie
Chenchen Zhang
Zhaohui Wang
Jiayi Tian
Yanghai Wang
Zhe Cao
Minxin Dai
ke wang
Runzhe Wen
Yinghao Ma
Yaning Pan
Sungkyun Chang
Termeh Taheri
Haiwen Xia
Christos Plachouras
Emmanouil Benetos
Yizhi LI
Ge Zhang
Jian Yang
Tianhao Peng
Zili Wang
Minghao Liu
Junran Peng
Zhaoxiang Zhang
Jiaheng Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Reasoning
MLLM Benchmark
Text
Audio
Video
Primary area: datasets and benchmarks
TL;DR: OmniVideoBench is a benchmark for comprehensive evaluation of synergistic audio–visual understanding in Omni MLLMs.

Abstract: Recent advances in multimodal large language models (MLLMs) have demonstrated substantial potential in video understanding. However, existing benchmarks fail to comprehensively evaluate synergistic reasoning capabilities across audio and visual modalities, often neglecting either one of the modalities or integrating them in a logically inconsistent manner. To bridge this gap, we introduce OmniVideoBench, a large-scale and rigorously designed benchmark dedicated to assessing synergistic audio-visual understanding, with a strong emphasis on modality complementarity and logical consistency. Specifically, OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces, derived from 628 diverse videos ranging from several seconds to 30 minutes, and manually verified to guarantee complete correctness and uniqueness. Moreover, OmniVideoBench encompasses 13 carefully designed question types, covering temporal reasoning, spatial localization, counting, causal inference, summarization, and beyond, thereby capturing the essential challenges of video understanding. Evaluation of multiple MLLMs on OmniVideoBench reveals a pronounced gap between model performance and human reasoning, with open-source models lagging significantly behind their closed-source counterparts, underscoring the inherent difficulty of genuine audio-visual reasoning. We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.

Published as a conference paper at ICLR 2026
OMNIVIDEOBENCE: TOWARDS AUDIO-VISUAL UNDERSTANDING EVALUATION FOR OMNI MLLMS
Caorui Li!!2, Yu Chen Yiyan Ji
Jin Xu', Zhenyu Cui', Shihao Li', Yuanxing Zhang', Zhenghao Song", Dingling Zhang', Ying He Haoxiang Liu Yuxuan Wang", Qiufeng Wang", Jiafu Tang', Zhenhe Wu', Jiehui Luo", Zhiyu Pan', Weihao Xie", Chenchen Zhang", Zhaohui Wang', Jiayi Tian?, Yanghai Wang', Zhe Cao', Minxin Dai', Ke Wang', Runzhe Wen', Yinghao Ma², Yaning Pan Sungkyun Chang", Termeh Taheri", Haiwen XialI, Christos Plachouras,, Emmanouil Benetos", Yizhi Li!3, Ge Zhang", Jian Yang', Tianhao Peng", Zili Wang', Minghao Lin Junran Peng Zhaoxiang Zhang' Jiaheng Liu
Nanjing University Southeaast University Allibaba Group *Kuaishou Technology
MM-A-P University of Science and Technology Beijing "Central Conservatory of Music Huazhong University of Science and Technology "Queen Mary University of London
10 Fudan University PPPeking University 12707A
13 University of Manchester caoruilieseu, edu. cn liujiaheng@nju. edu. cn
14Chinese Academy of Sciences
ABSTRACT
Recent advances in multimodal large language models (MILMs) have demon- strated substantial potential in video understanding. However, existing bench- marks fail comprehensively evaluate synergistic reasoning capabilities across audio and visual modalities, often neglecting either one of the modalities or in- tegrating them in a logically inconsistent manner. To bridge this gap, we in- troduce OmniVideoBenchh dedicated assessing synergistic audio-visual understanding, with a strong em- phasis on modality complementarity and logical consistency. Specifically, Om- ni viVeeoBBenc comprises 1000 high-quality question-answer((
large-scale and rigorously designed benchmark
pairs, each an- notated with step-by-step reasoning traces, derived from 628 diverse videos rang- ing from several seconds to 30 minutes, and manually verified to guarantee com- plete correctness and uniqueness. Moreover, VideoBenn encompasses 13 carefully designed question types, covering temporal reasoning, spatial localiza- tion, counting. causal inference, summarization, and beyond, thereby capturing the essential challenges of video understanding. Evaluation of multiple MLLMs on Omni ConniVideoBencl reveals a pronounced gap between model performance and human reasoning, with open-source models lagging significantly behind their closed-source counterparts, underscoring the inherent difficulty of genuine au- dio-visual reasoning. We will release Omni VideoBeBee to foster the development MLLMs with stronger and more generalizable reasoning capabilities.
INTRODUCTION
Multimodal large language models (MLLMs) have recently made impressive progress in bridging vision, language, and audio (Yin al. 2024 Song et al., 2025: Cheng et al. 2025). While early benchmarks primarily focused on ieage-text text alignment or visual reasoning (Xu et al. 2025c Chen et al. 2024c Yue al. 2024a), the integration of video and audio presents quite different chal-
lenge: models must jointly process long temporal sequences, dynamic scene transitions, and com-
Equal contribution. This work conducted during the author's internship Nanjing University.
Corresponding Author.
https://gi thub. wwtp:///.........mmoviieeeeeeee.

---

Published as a conference paper at ICLR 2026
Reasoning Steps
Where
FIGHT
NO ONE
ALONE' poster relative to person who deployed Culein-Anbar
V: Find the he 'NO ONE FIGHT ALONE poster.
On the left wall.
A: Find who located Cutein-Anbar down.
LINE
B. On the back wall.
C. On the right wall.
D. On the front wall.
@ V: Confim the positional relationship between the two from global perspective.
"...I deployed Culein-Anbar at last second..."
Answer:
Gemini-2.0-Flash:
Reasoning Steps
What will the young don't prevent him?
Steven
A: Find who is Steven.
Now,.., STEVEN, you stay out ehose cookles..."
A. Tell the secret of cookie. Get some cookie to eat.
C. Tell everyone girlfriend surprise.
cookie.
V: The ring box and his serious expression indicate that he wants to surprise his girlfrendd Steven prevents young man..
"...Can just talk to you in the kichen for a second..."
Answer: D
Gemini-2.DDFlaase A
Figure 1: Examples in Omni VideoBench ("V"" presents vision and "A" presents audio), and we present the atomic reasoning traces for these examples.
plementary acoustic cues. Despite rapid advances, evaluation of MLLMs audio visual reasoning remains underdeveloped. Existing benchmarks (Li et al., 2024a Hong et al. 2025) often (i) focus on short video clips that underrepresent long-term temporal dependencies, (ii) emphasize single modality (e.g., vision) while treating audio as auxiliary or optional. As a result, current evalua- tions fail capture the challenges inherent comprehensive video understanding, where audio and vision must be integrated consistently and logically to support robust inference.
To address these limitations, introduce VenneVhde,,,nch high-quality benchmark designed for evaluating audio-visual reasoning abilities in MLLMs. Specifically, first, we collect 628 diverse videos spanning up to 30 minutes across 8 major categories and 68 subcategories, covering realistic contexts such as news, sports, documentaries, vlogs, and ego-centric recordings. Then, we construct 1,000 high-quality question-answer pairs based on these videos, and each pair is annotated with step-by-step reasoning chains shown in Figure where these reasoning steps explicitly indicate modality and evidence information. This design not only strengthens the reliability the evaluation but also provides unique signal for analyzing how models reason, rather than just the final answers.
Based on our Omni VideoBench, we conduct extensive evaluations of both closed-source and open- source MILMs, and several insightful findings follows:
OmniVideoBench poses significant challenges for Omni-Modal Language Models. Cur- rent MILMs have not achieved a passing score (<60%) Omni Vinioiideeeeech The best- performing model, Gemini-2.0-P 2ro-Pro, only achieves an accuracy of 58.90%. Except for the newly proposed Qwen3-Omni, the performance of open-source models is close to random.
Omni-understanding abilities on long videos have significant improvement room. Al- though some leading models (such as Gemini-2.5-pro) demonstrate relatively robust perfor-
2

---

Published as a conference paper at ICLR 2026
(1) Collection Annotation
(2) Filtering
Using single modality
Initial QA
Video Input
AlHiAuddi Input
Text
Common -sense QA
Invalid QA pairs
bibli
VLM Filtering
(3) Refinement
@ LLM
① Collection
Filterlng
Final Data
Manual Annotation
33
Manual Modification
@ Manual Refinement
Reasoning Steps
QA
The complete pipeline of data collection, annotation, and refinement, where filtering and refinement serve as two key processes for quality assurance.
Figure 2:
mance long videos, other models (e.g., Semini-2.0-Flasst struggle on long video understanding.
still
Performance varies a lot for videas with different audio signals. Comini-2.5-PPe only achieves 38.46% accuracy on videos with music signal, while the results on sound and speech 57. 57.72% and 61.66%, respectively.
Performance on different task types differs a lot. For example, Gemini-2.5-Pro achieves accuracy below 50% the background and music understanding task, which requires low- semantic acoustic cues (e.g., musical style, tempo changes), and the accuracy results on the relationship reasoning and summarization tasks more than 80%.
2
OMNIVIDFOBENCH
2.1 OVERVIEW
Omni VideoBench is benchmark for evaluating the audio-visual collaborative reasoning of MILMs. The main task in the evaluation requires a model to process a video, its audio, and as- sociated text generate textual answer supported by explicit reasoning steps. This process sesses the model's ability to synthesize information across modalities, from recognizing objects to comprehending complex scene dynamics and context. This section details the benchmark's 's design principles, annotation protocols, and dataset statistics.
2.2 VIDEO COLLECTION
OnniVVideoBench is composed real-world videos sourced from YouTube and Bilibil These videos feature rich audiovisual content; therefore, comprehensive understanding necessitates the accurate processing and integration of both audio and visual modalities for reasoming.
Regarding video richness, primarily focus two dimensions: type and duration. For type di- versity, we categorize videos into eight broad classes: Vlog, News, Cartoon, Sports, Documentary, TV, Ego, and Others. Each class is further subdivided into nearly seventy fine-grained subcate- gories, which facilitates video retrieval and ensures broad coverage. Video categories are unevenly distributed. News and documentary videos have dense audio that nearly covers visual content, mak- ing them unsuitable for audio-visual reasoning tasks; thus, we manually controlled the video type distribution. For duration diversity, restrict video lengths the range of several seconds to 30 minutes, so as to evaluate reasoning across varying temporal scales.
Building upon this foundation, we established set of rigorous video collection criteria that only ensure the quality of the videos themselves, like resolution, but also guarantee the richness and diversity of their audio and visual content. To further avoid data overlap with existing training sets
com/
http:///www.hiibili.ccmm
3

---

Published as a conference paper at ICLR 2026
(e.g., popular TV shows), restrict the selection to recent publications. The detailed collection principles are provided in Appendix
2.3 DATA ANNOTATION
After collecting high-quality videos, carried out manual annotation. Compared with automated annotation, automated methods cap the evaluation ceiling by the capabilities of the annotating model, whereas manual annotation produces questions that are closer real-world needs.
In Figure 2 we first designed multiple-choice questions consisting of the question stem, the cor- rect answer, and several distractors, facilitate convenient evaluation of model performance. At this stage, we obtained approximately 2,500 QA pairs. We categorize the tasks into 13 types: Fine- grained Perception, Spatial Reasoning, Attribute Comparison, Background & Music Understanding, Counting, Temporal Understanding, Summarization, Sentiment Analysis, Causal Reasoning, Rela- tionship Reasoning, Reference Reasoning, Ego Reasoning, and Hypothetical Reasoning. In this design, each question is required to rely audio-visual reasoning, and the answer must be both correct and unique with no alternative plausible interpretations in the video. Moreover, we require that questions should not depend on video resolution frame rate. Cases where the target object is extremely small, blurred, and barely recognizable to the human eye, or where the relevant event oc- curs only within instant, are excluded. In addition, established the following rules to minimize the interference caused by extrancous textual information.
Questions should avoid redundant information. We minimize unnecessary details in the question text, such as the gender, clothing, or exact speech of characters, as long as doing so does not affect the correctness or uniqueness of the answer. This serves two purposes: reducing textual cues the model could exploit and increasing question difficulty to better test its audio-visual understanding.
The length of answers is capped. To prevent the answer text itself from providing excessive cues to the model, which could reduce the extent to which the evaluation reflects its under- standing and reasoning over audio and visual modalities, we impose a limit on answer length. This constraint ensures that the results more faithfully capture the model's multimodal com- prehension and reasoning capabilitics.
The format of options must be consistent. Here, "format" refers aspects such as length, tone, style, and variation patterns. If these features are inconsistent, they may provide the model with unintended cues for reasoning. For instance, when three options are consider- ably longer than the remaining one, when three options adopt a casual tone while the other is markedly formal, such discrepancies undermine the assumption that each option should have an equal probability of being chosen, thereby compromising the fairness of the evaluation.
Negative options must be relevant to the question. We require that all distractors appear in the video and maintain relevance to the question. Without this constraint, the model could easily eliminate distrastors, greatly reducing the need for reasoning.
Options should maintain a consistent semantic distance. We formalize semantic distance as the number of differing semantic units between options. Let option o, be represented as a set of semantic units S. The semantic distance between two options o, and oj is defined as:
d(o;; 02) SS,A,,
where A denotes the symmetric difference, capturing the distinct semantic units between two options. To prevent models from exploiting unbalanced textual cues rather than performing genuine audio-visual reasoning, we require that all distractors have consistent distances from one another and from the correct option.
2.4 QUALITY ASSURANCE
We employed advanced MLLM (i.e., Gemini 2.0 Flash), with strong audiovisual perception and comprehension capabilities. as well as long-context processing ability, to filter out questions that could be resolved using only a single modality. If the model successfully selected the correct swer with a plausible explanation while relying solely on unimodal information, the corresponding question removed. After this filtering stage, approximately 1,500 questions were retained.
4

---

Published as a conference paper at ICLR 2026
Table Dataset statistics divided into video-level and annotation information.
Video Statisties
Annotation Statisties
#Major Categories
8
#Task Types
13
#Suhcategories
68
Avg. Question Len.
14.68 words
Avg. Duration
384.24
Avg. Answer Len.
4.92 words
Min. Resolution
480p
Avg. Reasoning Steps
5.68 762:147:91
Max. Resolution
1080p
Audio Types (Sp:So:Mu)
Tokeo lncune
Sham
Coura Kepia
Madian
Hale
Par
Long
20
110
TV 173
Ubolos
Cartoon
Drewn
(c) Duration Distribution
22
Peceprann
76.2%
Cenrial
Mustc
SpuRd
(a) Video Catagories
(b) QA Task Distribution
(d) Audio Type Distribution
Figure 3: (a) OmniVideoBench covers 8 major categories and 68 subcategories. (b) Om- ni niVideoBench comprises 13 task types. The above part shows the video duration distribution across different tasks, while the durations categorized into four groups: "Short" for less than 1 minute, "Medium" for 1-5 minutes, "Long" for 5-10 minutes, and "Ultralong" for more than 10 minutes. The lower part illustrates the distribution of three types of audio (i.e., Speech, Sound and Music). (c) Distribution of video durations across four time intervals. (d) Distribution of three audio types.
Subsequently, we employed a large language model, DeepSeek- V3.1 (Liu et al. 2024a), with strong reasoning capabilities filter out questions that could be answered solely based on textual informa- tion. Such cases primarily fall into two categories: first, questions that involve classical, well- well-known, or universally shared knowledge or objects, which can be answered without reference to the video content; and second, questions where the phrasing of the question, options, or answers provides unintended textual cues. For the former, we directly discarded the questions. For the latter, our an- notators reviewed the reasoning process generated by the model and revised the textual formulations to eliminate such biases. After this stage of refinement, 1103 questions were retained.
Another group of annotators conducts the final refinement stage, thoroughly reviewing all questions to identify and remove those with incorrect, non-unique, or mismatched answers. After this valida- tion, annotators enriched each question with step-by-step reasoning chains, where each step consists of three elements: modality, evidence, and inference. The modality specifies whether the step relies on audio visual information; the evidence denotes the specific information extracted from the video; and the inference describes the reasoning derived from that information. We required each step to be atomic, meaning that it should involve only one modality and capture minimal unit of evidence, such as a spoken sentence, an action, or the appearance of a character. This design ensures that the reasoning process is both detailed and comprehensive. Through this process, we obtained 1000 high-quality QA pairs with explicit step-by-step reasoning chains, forming a robust dataset for multimodal audio-visual reasoning.
2.5 DATASET STATISTICS
As shown in Table I our OnniVideoBench dataset consists of 628 real-world videos with audio tracks, spanning 8 major categories and 68 subcategories. The videos are of high quality and diverse in content, with average duration of 384.6 seconds, average resolution of 480p. about 2k ASR- transcrribed tokens per video, and roughly three speakers per video. On the annotation side, Omni Bede contains 1000 audio-visual reasoning QA pairs across 13 task types, with an
5

---

Published as a conference paper at ICLR 2026
Benchmark
Modality
Qwenz.5-Omni
Multiple Domains
Video
Type
Audio Type
Video Duration
Answer
Type
AVQA Yang et
2022
V+A
R
10
MC
Music-AVQA Li
2022
V+
R+S
Mu
60
CLS
AYTRUSTRENCH
howarary ot
2025
V&A
R
So
10:60
MMAU [akshi et
21724
A
71.0
Sp+
MC
(Kateeski
R+S
AV-Odyisey (Migg 0E 2024
I+A
R
Sp+Sc+Muu
MC
AVHBench (аapente Stng
2024
V+A
74.7
R+S
10
CLS
OmniBench eL al. 20144
I+A
56.
R
So+So
MC
Duily Ommi
21745
V+A
47.5
R
Spe So+
30
MC
WorldSense
Hong 01 sl 2025
V+A
48.3
R
Sp+So+Mu 15-656
MC
OmniVideoBench (Ours)
V+A
29.3
R Sp So+ 4-1955
MC
Table 2: Comparisons between different benchmarks and datasets. I, A for modality represent video, image and audio. Qwen2.5-0mni represents the performance of Qwen2.5-0mni--7 on these benchmarks. Multiple Domains signifies whether the video includes diverse domains. R and S in Video Type denote real-world and synthetic data. Sp. So, and Mu represent Speech, Sound, and Music for Audio Type, respectively. Video Duration represents the duration in seconds. MC, CLS for Answer Type indicate Multiple Choice and Classification from fixed vocabulary, respectively.
average question length of 14.68 words and an average answer length of 4.92 words. Each QA pair annotated with step-by-step reasoning chains averaging 5.68 steps. The reasoning process covers both modalities, with 54% of steps grounded in vision and 46% in audio. There are 762, 147, 91 QA pairs related to Speech, Sound and Music, respectively, highlighting the complementarity of modalities in multi-step reasoning. Moreover, we provide more detailed statistics in Figure
2.6 DATASET COMPARISON
As shown in Table 2 we compare OmniVideoBench with representative audio-video benchmarks. While AV- Odyssey (Gong et al. 2024 and OmniBench (Li et al.
BOmaiVioeeBechh
2024c) operate single images, Omni Vice tar-
Ascoracy (56)
60
+ Dhill-OOmi
gets substantially more challenging videos with durations ranging from few seconds 30 minutes. Recent bench-
49
marks (Chowdhury et al., 2025 Radevski et al.] 2025
Sakshi et al.) al.]| [202]] (Sung-Bin al.[[2024] have begun emphasize audio-vidco coordination; however, they typ- ically focus on specific capabilities short clips. Some also evaluate audiovisual consistency, yet their tasks re- main primarily centered on shorter videos and hallucina- tion detection. Omni VideoBench, by contrast, expands scope to diverse video types, broader temporal spans, and fine-grained cross-modal reasoning, capturing richer dependencies between audio and vision. Some other audio-visual benchmarks (Zhou al. 2025 Hong et al. 2023), do not fully achieve a natural integration addresses this limitation effectively. For instance, disabling audio causes Gemini-2.0--Flast per- formance to plummet to the near-random level, indicating that visual-only cues are insufficient. Furthermore, Figure 4 shows that widely used models such Qwen2.5-0mni-i perform closer to random guessing on our benchmark, indicating that Omni VideoBench presents significantly greater challenges than existing multimodal datasets.
Gomild-1 S.aak
U-1-1
Grmini
Figure 4: Performance comparison of selected models on OmniVideoBench and Daily-Omni. "Red line" denotes random guessing.
the two modalities, while
3 EXPERIMENTS
3.1 BASELINE MODELS
We evaluate open-source MLLMs (i.e., Qwen3-Omni series (Xu al. 2025b), series (Xu et al. 2025a), Seichuan-Ommi-ee (Li et al., 2025). HumanO (Zhao et al. MiniCPM-o Yao et al. 2024), Lad2 (Cheng al. 2024 1 (Fu al.
Qwen2.5 5-Omni
2025
2025

---

Published as a conference paper at ICLR 2026
Audio Type
Video Duration
Models
Avg.
Music
Sound
Speech
[0,1] min
(1,5) min
(5,10) min
(10330) min
Omni-Modal Language Models (With
Visual and Audio)
Gemini-3.0-Pro
52.81
55.17
64.13
62.42
66.18
57.02
59.76
61.80
Gemini-2.5-Pro
38.46
57.72
61.66
57.83
64.43
55.02
58.90
Demini-330--Faan
49.45
50.34
$6,69
58,43
$5.10
55.90
52.29
55.10
Gemini-2.5-Flash
39.56
57.04
53.17
55.42
55.10
47.37
52.11
52.40
Demmin-22.0-Flas
29.67
40.27
43.21
49.40
43.15
41.05
34.87
41.50
Owen3-Ommi-3005
37.36
34.67
39.26
45.78
37.03
35.11
38.40
OmniViinci-BB
30.77
32.67
32.15
38.55
34.11
30.13
27.10
32.10
Baichuan-Omaa 0mni-1.55
24.18
31.33
31.36
28.92
31.78
28.38
32.44
30.70
HumanOmmi-7B
20.87
31.08
31.61
36.57
29.36
29.60
29.25
30.50
VITA-1.5-7B
25.27
28.57
31.49
31.33
27.41
30.57
33.97
30.50
MiniCPM-o
27.47
28.57
30.24
31.43
28.49
34,53
26.15
29.70
Swen2.5-Omni-7R
23.07
25.33
30.70
41.57
27.41
25.33
26.72
29.30
VideoLLaMMA27B
26.37
30.67
29.25
32.00
28.20
29.60
28.29
29.20
Omni-Modal Language Models (Visual Only)
Gemini-2.0-FFlah
25.27
36.67
30.99
33.73
35.86
32.75
22.48
31.30
Owen2.5-Omni-7BB
27.47
26.67
26.22
28.31
27.11
24.45
25.95
26.40
Visual Language Models (Visual Only)
Qwen2.5-VL 32.97
32.00
31.49
38.55
31.20
29.26
30.53
31.80
Qwen2.5-VLL 29.67
31.33
29.51
25,90
30.03
31.88
30.15
29.80
Qwen2.5-VL--72)
26.37
29.33
29.91
33.13
30.03
31.88
24.43
29.50
Baseline LLMs
DeepSeck-V3.1
28.57
26.17
27.28
30.91
27.57
25.00
26.44
27.60
Table 3: Results of different models. The table reports accuracy videos across three audio types and four duration ranges. Boldface highlights the best performance within each column.
Omni Vinci-99 YYe et al., 2025) and various closed-source MLLMS (i.e., Gemini-2.5-Pro. Gemini- 2.5-Flash (Comanici al. 2025), and Gemini-2.0- (.-Faa-,,.0-Flash). We also evaluate the Qwen2.5-VL se- ries (Bai et al. 2025) and DeepSeck-V3.1 (Liu et al. 2024a).
3.2 HUMAN PERFORMANCE
We invited 10 qualified annotators, including graduate students experienced in multimodal search and experts trained in music-related analysis. Before the evaluation, 50 questions were randomly selected for 10 testers answer simultaneously. The fact that the difference in results was no more than 5 questions indicates that the manual evaluation has minimal deviation and is viable. We consolidated questions requiring musical knowledge and assigned them two music experts. The remaining questions were divided to ensure roughly equal distribution of speech and sound audio samples per set, which were then evenly assigned high-level personnel such as graduate students to obtain persuasive results. Manual responses had no time constraints, yielding final ac- curacy rate of 82.69%. This demonstrates that existing models still fall significantly short human capabilities.
3.3
MAIN RESULTS
In Table we present evaluation results on OmniVideoBench and have the following observations:
Open-source models still lag significantly behind closed-source models. Gemini-2.5-PP achieves the best performance across most tasks. This underscores the urgent need for current open-source models to improve in multiple areas, including fine-grained perception, cross- modal reasoning, and speech awareness.
MILMs show performance degradation when dealing with music-related audio. We ob- serve that models exhibit lower accuracy in responding to music-dominated videos compared to those containing human voices ambient sounds, phenomenon particularly pronounced in open-source models. Unlike human voices conveying explicit semantic content or ambi- ent sounds often corresponding to specifie visual events, music primarily encodes abstract

---

Published as a conference paper at ICLR 2026
1.0
Gemini-2.5 2.5 PPr
Gemini-2. 5-Flash
Gemini-2 0-Flash
Owen3-Omni
Accuracy
0.4
0.2
0.0
Attr
BacâMu
Caus
Coun
Ego
Fine
Hypo
Ref
Rela
Senti
Spatl
Summ
Tempo
Figure 5: Performance Comparison of some Open-Source and Closed-Source Omni Models 13 Tasks in Omni VideoBench. Here, "Attr": Attribute Comparison, "Bac&Mu": Background and Music Understanding, "Caus": Cause and Effect Reasoning, "Coun": Counting. "Ego": Ego Rea- soning, "Fine": Fine-grained Perception, "Hypo": Hypothetical Reasoning, "Ref": Referential Reasoning. "Rela": Relationship Reasoning, "Senti": Sentiment Analysis, "Spati": Spatial Rea- soning, "Summ": Summarization, "Tempo": Temporal Sequencing Understanding.
emotional and atmospheric information. Current MLLMs demonstrate limited capability to translate such implicit cues into effective reasoning. indicating that cross-modal alignment for emotional and atmospheric understanding remains urgent challenge to be addressed.
Current MILMs still have room for improvement in long videos. Although some leading models like Gemini-2.5-PP demonstrate relatively robust performance on long videos, most MILMMs (e.g., Gemini-2.0-Fla. Qwen3-Omni) still struggle in long videos, which highlights the widespread challenge in understanding long videos.
3.4 FURTHER ANALYSIS
Performance of Models on Tasks across Different Types. Figure 5 presents a fine-grained com- parison of model accuracy the 13 reasoning categories in OmniViieoBench. Several consistent patterns emerge. (1). Closed-source MLLMs demonstrate superior performance across nearly all task types. Gemini-2.5-Pro achieves the highest accuracy on 11 of 13 tasks, demonstrating par- ticularly strong performance in Relationship Reasoning, Spatial Reasoning, Referential Reasoning, and Cause and Effect Reasoning. These tasks require long-term sequence integration and multi-step cross-modal reasoning, highlighting Gemini's strengths in long-context modeling and multimodal fusion. (2). MILMSs understanding of audio remains limited relatively superficial surface-level information. Whether open-source or closed-source models, Background and Music Understand- ing remains the most challenging task, with even Gemini-2.5-Pro achieving accuracy below 50%. This is probably because such tasks require linking low-semantic acoustic cues (e.g., musical style, tempo changes) with high-level reasoning, while current models struggle to master the capability. In contrast, Relationship Reasoning and Summarization relatively easier. This may be because they rely more on recognizing language within audio and visual observation capabilities, and less cross-modal abstraction abilities.
Effect of ASR Transcripts for Visual Only MILMs. To further investigate the role of audio in- formation in MLLMs' reasoning performance, we evaluate several models using both the automatic speech recognition (ASR) transcripts generated by the
model (Liu et al. 2025a)
8

---

Published as conference paper at ICLR 2026
Visual Only
Visual ASR
Visual Audio
Music
Sound
Speech
-0
Vienal Only
40
Accuracy
Viuua ASR
30
10
Quen25-0mmn-18 Qum225-L-78 Qva225-V1-728 Gemini-2-Flaah
Accuracy (%)
(a) Accuracy rates of selected MLLMs under differ- ent inputs.
(b) Accuracy of Gemini-2.0-FFan videos with dif- ferent audio types.
Figure 6: Accuracy comparison of MILMs with and without ASR transcripts on Omni VidhoBench.
and silent video frames as inputs. The results are shown in Figure 6 The observations are as fol- lows: (I). Open-source models demonstrate weaker integration capabilities for audio information compared to their understanding of textual information. In Figure 6a. all tested models demonstrate significantly improved accuracy after extracting ASR text information compared receiving only visual inputs. However, the Owen2.5-Ommi- simultaneously. performed even worse than the Qwen2.5-VL-77 model with equivalent parameters. This highlights common challenge faced by most open-source Omni-Modal Language Models: in- sufficient cross-modal reasoning capabilities for audio-visual information. (2). cross-modal video reasoning, audio comprehension capabilities remain irreplaceable by ASR. In Figure 6b] although ASR can help MILMs achieve decent performance on certain tasks requiring speech recognition ca- pabilities, its effectiveness is extremely limited for tasks demanding deeper and more abstract audio comprehension such as the videos whose audio type Music or Sound.
model, which processes both visual and audio inputs
Effeet of Different Numbers of Frames. We conduct experiments Qwen2.5-0mni-7. and 1aina--Omni330---
with total frame counts fixed at 32, 64, 128, and 256, respectively, and observe that both models benefit from more frequent time sampling. In Figure 7a the total frame counts increase, accuracy steadily improves, likely because richer temporal coverage provides more complete motion cues and reduces the risk of missing key events. As shown Figure 77b this improvement becomes more pronounced for longer videos. The consistent gains across different video durations further indicate that dense frame sampling not only captures fine-grained visual dynamics but also strengthens cross-modal alignment. This highlights the importance of dense temporal information and long-context processing for achieving robust audiovisual reasoning.
Open-ended QA MCQ. To inves- tigate whether the multiple-choice ques- tion (MCQ) format overstates model per- formance, we additionally evaluated sev- eral representative models on open-ended question-answering (QA) tasks, where no predefined answer options are provided. In this setting, models must directly gen- crate textual responses, eliminating both the possibility of random guessing and any lexical cues potentially present in candi- date options. In Table the accuracy of all models drops significantly compared to their perfor- mance on multiple-choice questions. For instance, the Gemini-2.5-Pro. which leads MCQ bench- marks, experiences relative accuracy decline exceeding 14 percent in open-ended scenarios, while open-source models exhibit even steeper drops.
Table Comparison of performance on Open-ended Question Answering (QA) and Multiple- Choice Ques- tions (MCQ) across various models.
Models
Open-ended QA
MCQ
Gemini-2, (.0ni--.--FFlss)) Qwen2.5-0mni-7B
27.06
41.50
17.25
29.30
4 RELATED WORKS
Omni- Understanding MLLMs. The development of MLLMs (Chen et al. 2022 Awadalla et al.]
2023
Liu al. 2023
Peng al. 2025 Yang al. 2023] began with foundational focus

---

Published as a conference paper at ICLR 2026
Qren225-0mm--77
-
min
(5, 10j min 30]
Frames
Prames
128 Frzmos
256 Frames
- Svve33-0mal-000-A30
Aeturag (Ng
35
64
Accrray (5)
Accuracy
30
32
64
128
256
Qwen25--mml-78
Accuracy (%s)
-30
20
Duration (minutes)
Qwea2-Omml:30-AA3
(b) Accuracy of Sreen--mmn-30B-AA3 on questions with videos of varying durations across different numbers of frames.
(a) Performance of Qwen2.5-0mmi-77 and Qwen3- Omni-30B-ARl at different numbers frames.
Figure 7: Performance of selected models when inputting videos with different numbers of frames.
integrating the two primary modalities of vision and language or audio and language. A recent paradigm shift aims develop Omni-modal MLLMs capable of processing and generating infor- mation across an arbitrary combination of modalities "'Any-toAAny") This approach positions the LLM as a central cognitive engine, unifying diverse data types like image, audio, video, and text within its semantic space (Liu et al. 22024b] Yuan et al.) (025). This has driven move from inte- grating pre-trained unimodal components towards developing "natively multimodal" architectures trained from the ground up, exemplified by models like GPT-4o (Hurst et al. (202)). This ambi- tion showcased by state-of-the-ar models (Xu et al. 2025a Zhao et al. 2025 Li et al. 2024bb 2025 Yao et al. 2024 Sun et al.) 2025 Liu et al. 2025ct Wu et al. (025a), which pioneer end- to-end streaming capabilities for simultaneously processing video and audio to generate text and speech. At the forefront of this paradigm, proprietary models like Gemini series [Team] 2024 Co- manici et al. (025) demonstrate pinnacle performance, powered by a natively multimodal design and massive context window that together unlock superior understanding of complex, interwoven data streams.
MILM Benchmarks. The landscape of MLLM evaluation has matured significantly, evolving from foundational perception benchmarks (Liu et al., 2022c] Li et al. [202aa] Yu et al.
2222a b
Chen al.
2024a Jiang al. 2025) to more sophisticated Trameworks (He al. [2025] Du al.
2025
Wu et al. 2025b). Recent efforts probe deeper cognitive abilities, with MLLM-
(Ge
al.
2025) assessing a hierarchy of cognitive skills. MMMU (Yue et al.] (023) and MMMU-
Pro (Yue al. 2024b) challenging models with expert-level, multi-disciplinary reasoning under
stricter protocols like vision-only inputs. Simultaneously, evaluation has specialized into high-stakes
domains such as finance (Gan al., (024) and medicine (Chen al., 2024b). For video, some benchmarks (Wang et al., 2019; Li et al. 2021] 2023] Fang et al. [0024] Wu et al. 2024) now focus on the critical challenge of long long-context temporal understanding (Liu et al. 2025b), revealing key
limitations in current models.
5 CONCLUSION
We presented OmniVideoBench, a large-scale benchmark for evaluating audio-visual collaborative reasoning in MLLMs, with diverse videos, carefully verified QA pairs, and explicit reasoning notations. Experiments show that both open- and closed-source models still struggle with modality complementarity. long-form temporal reasoning, and music understanding, underscoring large gap from human- performance. We hope this benchmark will drive future research toward more ro- bust and generalizable multimodal reasoning systems.
10

---

Published as a conference paper at ICLR 2026
ETHICAL STATEMENTS
This work fully adheres to the ICLR Code of Ethics in all aspects of research conduct. The processes of data collection, usage, annotation, and benchmark construction strictly comply with ethical stan- dards regarding privacy, consent, and responsible Al practices. Videos in OmniVideeBende are strictly limited to academic research purposes. Any form commercial use prohibited. All video copyrights remain the property of their original owners. To the best of our knowledge, this study does not involve any data, methodologies, or applications that raise ethical concerns. The authors confirm that they have reviewed and followed the ICLR Code of Ethics throughout the entirety of this research.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we have made the following comprehensive efforts. We provide detailed descriptions of the video collection, filtering, annotation, and reasoning- con- struction pipeline in Sec. 2 and Appendix [ We also present complete statistics of the dataset and explicit definitions of task types, reasoning modalities, and evaluation metries in Sec. 2.T and Appendix A Furthermore, the prompts used in the experiments documented in detail in Ap- pendix C Extensive replication experiments demonstrate only minor variations across runs, con- firming the stability and full reproducibility of the reported results. We promise to release the exper- imental code for evaluation the future to facilitate verification and benchmarking.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. 62506161). This work was supported in part by the Beijing Major Science and Technology Project (No. 7251 125100000842003).
REFERENCES
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani S. Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. ArXiv, 0ass/3308.03990 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 2025.
Feilong Chen, Duzhen Zhang, Minglun Han, Xiuyi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey vision-language pre-training. Machine Intelligence Research, 20:38-56, 2022.
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang. Xiangdi Meng, Tianyu Liu, and Baobao Chang. Pca-bench: Evaluating multimodal large language models in wwweppetooonceooo www.rpiiiiiooooiiiiooooooeoeoeee actionn chain. ArXiv, abs/2402.155221, 2024a.
Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tian-Xin Li, Haodong Duan, Ziyan Huang, Yan-Cheng Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang. Eric J. Seibel, Junjun He. and Yu Qiao. Gmai Emmai-mmeench: A comprehensive multimodal evaluation benchmark towards general medical ai. ArXiv, abs/2408.0336. 2024b.
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. arXiv preprint arXiv:2405. 1647:33305.66433, 2024c.
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can milm think like holmes for complex video reasoning? ArXiv, abs/2505.21374, 2025.
11

---

Published as a conference paper at ICLR 2026
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-tempone modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024.
Sanjoy Chowdhury, Sayan Nag. Subhrajyoti Dasgupta, Yaoting Wang. Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual lims. arXiv preprint arXiv:2501.02133 2025.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. Xrxiv preprint arXiv:2507.0626/ 2025.
Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502. 14739, 2025.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbenchhvideo: A long-form multi-shot benchmark for holistic video understanding. ArXiv, abs/2406.145/1 2024.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint 250.:250...095.,...
Ziliang Gan, Yu Lu, Dong Zhang. Haohan Li, Che Liu, Jian Liu, Ji Liu, Haipang Wu, Chaoyou Fu, Zenglin Xu, Rongjunchen Zhang, and Yong Dai. Mme-finance; A multimodal finance benchmark for expert-level understanding and reasoning. ArXiv, abs/241 1.03314, 2024.
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, Chenghao Zhu, Ziyue Lin, Dingjie Song, Xidong Wang. Anningzhe Gao, Zhang Zhiyi, Jian- quan Li, Xiang Wan, and Benyou Wang. MLLM-benchh Evaluating multimodal LLMs with per-sample criteria. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Papers), pp. 4951-4974 Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-1877 //cl//acl/ wtw.eaannneooooooo//0020000 6. doi: 10.18653/v 12065555///222...mooooo.gg naacl-long.n 256. URL htt.l//tttp.//caaaaahllllgg///e
naacl naacl- (oacll1oorg256))
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang. and Xiangyu Yue. Av-odyssey bench: Can your multimodal Ilms really understand audio-visual information? ArXiv, abs/2412.02611 2024.
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng. Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, et al. Can large language models detect errors in long ofain-of-thoughh reasoning? arXiv preprint arXiv:2502. 19361, 2025.
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense; Evaluating real-world omnimodal understanding for multimodal llms. ArXiv, abs/2502.04322 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang, Chaoyou Fu, Peng Gao, and Hongsheng Li. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. ArXiv, abs/2502.09621,, 2025.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench: Benchmarking multimodal large language models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299- 13308, 2024a.
12

---

Published as a conference paper at ICLR 2026
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19086-19096, 2022.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video under- standing benchmark. 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22195-22206 2023.
Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohith Krishnan Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, Tamara L. Berg, Mohit Bansal, Jingjing Liu, Lijuan Wang, and Zicheng Liu. Value: A multi-task benchmark for iideo-and-language understanding evaluation. ArXiv, 00s/2106604432, 2021.
Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, Song Chen, Xu Li, Dawei Pan, Shusen Zhang, Xin Wu, Zheng Liang. Jun Liu, Keer Lu, Yaqi Zhao, Yan-Bin Shen, Fan Yang. Kaicheng yu, Tao Lin, Jianhua Xu, Zenan Zhou, and Weipeng Chen. Baichuan -omii technical report. ArXiv, abs/2410.08565, 2024b.
Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Gu- osheng Dong, Dawei Pan, Chong Li, Yuanbo Fang, Dong-Ling Kuang, Mingrui Wang, Chenglin Zhu, Youwei Zhang. Hongyu Guo, Fengyu Zhang, Yuran Wang, Bowen Ding, Wei Song. Xu Li, Yuqiu Huo, Zheng Liang, Shusen Zhang, Xin Wu, Shuai Zhao, Lin-Xiao Xiong, Yozhen Wu, Jia-Reng Ye, Wenhao Lu, Bowen Li, Yan Zhang, Yaqi Zhou, Xin Chen, Lei Su, Hongda Zhang, Fuzhong Chen, Xu Dong, Na Nie, Zhiying Wu, Bin Xiao, Ting Li. Shunya Dang, Ping Zhang, Yi- jia Sun, Jincheng Wu, Jinjie Yang. Xionghai Lin, Zhi-Xing Ma, Ke- Wu, Jia Li, Ai-Min Yang. Hui Liu, Jianqiang Zhang, Xiaoxi Chen, Guangwei Ai, Wentao Zhang, Yicong Chen, Xiaoqin Huang. Kun Li. Wenjing Luo, Yi qun Duan, Lingling Zhu, Ran Xiao, Zhengquan Su, Jiani Pu, Dian Wang, Xu Jia, Tianyu Zhang, Mengyu Ai, Mang Wang, Yu Qiao, Lei Zhang, Yanjun Shen, Fan Yang, Miao Zhen, Yijie Zhou, Mingyang Chen, Fei Li, Chenzheng Zhu, Keer Lu, Yaqi Zhao, Hao Liang. Youquan Li, Yanzhao Qin, Lin-Lin Sun, Jianhua Xu, Haoze Sun, Mingan Lin, Zenan Zhou, and Weipeng Chen. Baichuan-omni- 1.5
technical report. ArXiv, abs/2501. 15368, 2025.
Yizhi Li, Ge Zhang, Yi Ma, Ruibin Yuan, Kang 7hu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhen Yang, Xiangzhou Wang. Zhaoxi- ang Zhang. Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin. Omnibench: Towards the future of universal omni-language models. ArXiv, abs/2409. 15272, 2024c.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1wX11:2412.19437 2024a.
Alexander H Liu, Andy Ehrenberg. Andy Lo. Clément Denoix, Corentin Barreau, Guillaume Lam- ple, Jeln-Malo Delignon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Mud- direddy, et al. Voxtral. arXiv preprint ar mxXxx:2507.13666 2025a.
Haotian Liu. Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tun- ing. ArXiv, abs/2304.00485,, 2023. URL htt.uttsr/pppp..eettcccccooooott org/
CorpusID:25817977
Jiaheng Liu, Zehao Ni, Haoran Que, Tao Sun, Zekun Wang, Jian Yang. Jiakai Wang, Hongcheng Guo, Zhongyuan Peng. Ge Zhang, et al. Roleagent: Building, interacting, and benchmarking high-quality role-playing agents from scripts. Advances in Neural Information Processing Sys- tems, 37:49403 49428, 2024b.
Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context lan- guage modeling. arXiv preprint arXiv:2503. 17407, 2025b.
Yuan Liu. Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model
13

---

Published as a conference paper at ICLR 2026
an all-around player? In Computer September 29-October 4, 2024, Proceedings, Part VI. pp. 216-233, Berlin, Heidelberg, 2024c. 13.
18th European Conference, Milan, Italy,
Springer-Verlag. ISBN 978-3-031-726574 doi: 00.1007/978--031--220
Zuyan Liu, Yuhao Dong, Jiahui Wang. Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. ArXiv, abs/2502.043288, 2025c.
Yingzhe Peng. Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang. Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl, 2025. URL / abs/2533.
http://axiv..oo org
07536
Gorjan Radevski, Teodora Popordanoska, Matthew B Blaschko, and Tinne Tuytelaars. Dave: Diag- nostic benchmark for audio visual evaluation. arXiv preprint vrXiv:2503.09321, 2025.
S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint 4rXiv:2410.19160 2024.
Shezheng Song. Xiaopeng Li, Shasha Li, Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, Weimin Zhang, and Meng Wang. How to bridge the gap between modalities: Survey on multimodal large language model. IEEE Transactions on Knowledge and Data Engineering, 37(9):3311-533 2025. doi: 10.1109/KKDE..2255337799
Wei Sun, Linhan Cao, Yu Shan Cao, Weixia Zhang, Wen Wen, Kaiwei Zhang, Zijian Chen, Fang- fang Lu, Xiongkuo Min, and Guangtao Zhai Engagement prediction of short videos with large multimodal models. ArXiv, abs/2508.02516 2025.
Kim Sung-Bin, Oh Hyun-Bin Bin, JungMok Lec, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. Avhbench: A cross-modal hallucination benchmark for audio-visual large language models. arXiv preprint arXi::2400. 18325, 2024.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. ArXiv, abs/2403.05530, 2024.
Xin Eric Wang. Jiawei Wu, Junkun Chen, Lei Li, Yuan fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for hideo-and-language research. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4580 4590, 2019.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. ArXiv, abs/2407. 2024.
Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga, 2025a. https:
//arxiv.or//abe /at////ss//11111. .1111111133122
Yongliang Wu, Yizhou Zhou Zhou Ziheng, Yingzhe Peng. Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: reinforcement learn- ing perspective with reward rectification, 2025b. URL httppp:://axxxw.rr////a// 05629
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503. 2025a.
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma. Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang. Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu. Jingren Zhou, and Junyang Lin. Qwen3-omni technical report. arXiv preprint arXiv:2509. 17765, 2025b.
14

---

Published as a conference paper at ICLR 2026
Weiye Xu, Jiahao Wang, Weiyun Wang. Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li. Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modie large language models. arXiv preprint arXiv:2504.15211 2025c.
Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu, Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM Inter- national Conference on Multimedia, 2022.
Xu Yang, Yongliang Wu, Mingzhuo Yang. Haokun Chen, and Xin Geng. Exploring di- verse in-context configurations for image captioning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023. NeurIPS 2023, New Orleans, LA. USA. December 10 16, 2023, 2023. URL http:: nips. wwwp:mii.pppp.peeeeeeeeee wiwes/ wites//appeeeee23333333 / 80
html
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qi-An Chen, Huarong Zhou, Zhensheng Zou. Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level mllm on your phone. ArXiv, abs/2408.01880 2024.
Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang. Ligeng Zhu, Yuanhang Su, Sean Lin, An-Chieh Cheng, Zhen Wan, Jinchuan Tian, et al. Omnivinci: Enhancing architecture and data for omni-modal understanding Ilm. arXiv preprint arXiv:2510. 15870, 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li. Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12): nwae403, November 2024. ISSN 2095-5138. doi: 10. 10.109//////wwa4403. URL https:
//doi.org/10. 10tp:////100/////////aeeeee p
_eprint: https///caaemmiiccoo.cc///////ta
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024a.
Weihao Yu, Zhengyuan Yang, Linfeng Ren, Linjie Li, Jianfeng Wang. Kevin Qinghong Lin, Chung- Ching Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. Mm-vet v2: challenging benchmark to evaluate large multimodal models for integrated capabilities. ArXiv, abs/2408.0070 2024b.
Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al. Yue: Scaling open foundation models for long-form music generation. arXiv preprint 2503 08638, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. 2024 IEFF/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 9556-9567, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu- pro: A more robust multi multi-discipline multimodal understanding benchmark. In Annual Meeting of the Association for Computational Linguistics, 2024b.
15

---

Published as a conference paper at ICLR 2026
Jiaxin Zhao, Qize Yang. Yi-X Yi-Xing Peng. Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Weixuan chen, Xihan Wei, and Liefeng Bo. Humanomni: A large vision-speech language model for human-centre video understanding. ArXiv, abs/2501.15111, 2025.
Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities. ArXiv, abs/2505. 17662, 2025.
16

---

Published as a conference paper at ICLR 2026
A
FULL VIDEO CATEGORY TAXONOMY
Table 5 shows that videos OnniVideoBench span 8 major categories and 68 subcategories.
Table Full taxonomy of the video dataset.
Main Category
Subcategories
Vlog
Cooking Cuisine; Travel & Outdoor; Art; Animals; Daily Life at Home; DIY & Handcraft; Gardening: Fitness; Sports; Interviews;
Party Games; Makeup & Beauty; Fashion & Styling; Hiking Trekking
News
Politics; Economy; Society; Technology; Education; Healthcare; Mil- itary; Law & Justice; Sports; Culture; Entertainment; Weather; Disas- ter; Transportation
Cartoon
2D Animation; 3D Animation
Sports
Basketball; Football (Soccer); Volleyball; Badminton; Table Tennis; Swimming: Figure Skating; Skiing; Gymnasties; Wrestling & Judo;
Track & Field; Esports; Others
Documentary
Nature & Wildlife: History & Archaeology: Society & Humanity: Pol- itics & Military; Science & Engineering; Medicine & Health; Crime & Law; Art & Culture; Education & Growth; Economy; Environment
Climate: Food & Culinary Culture; Religion Belief
TV
Short; Dramas & Web Series; Variety; Stage Plays; Dance; Mime;
Movies
Others
Live; Advertisement; Course Replay; Short Video
Ego
First-person: People; First-person.Pets
B DETAILED PRINCIPLES OF VIDEO COLLECTION
To ensure objective and reliable evaluation of MLLMs, the videos included in the benchmark must satisfy multiple requirements, ensuring diversity in both type and duration. The content should provide rich information across audio and visual modalities, while maintaining complementarity between the two. In other words, the benchmark avoids cases where the visual content can be fully inferred from the audio alone, where the audio is redundant given the visual stream. Furthermore, since many existing video training datasets overlap with the sources of our benchmark--ic ex- ample, clips from Friends-evaluation may otherwise reduce simple "answer memorization." To mitigate this unfairness, we additionally consider the publication year of videos when constructing the dataset. The detailed principles for video collection are as follows:
Video publication date. Given that most existing training datasets are constructed from YouTube videos, similar to ours, or contain overlapping content such as identical TV shows, we restrict our selection to videos published after June 2024. We use the most recent videos possible to mitigate unfairness and potential overestimation issues arising from the model having already been exposed to similar content during training.
Rich dynamic visual information. The distinguishing feature of videos compared to im- ages lies in their rich dynamic visual information. A prerequisite for evaluating model's ability to understand visual information in videos is that the vidcos themselves contain suf- ficient dynamic content to be captured and analyzed. Consequently, vidcos lacking diverse dynamic visual information scenes or perspectives throughout, or those that remain largely static with minimal motion confined to a small comnr of the frame.
excluded, such those consisting of only several static
Effective audio information. In some videos, the audio is completely unrelated to the visual content, such as when only an independent background track is added. We consider
17

---

Published as a conference paper at ICLR 2026
such audio to be invalid. To fairly evaluate the model's capability in audio-visual collabo- rative reasoning, the whether speech, environmental sound, or music must align with the visual content.
Absence of subtitle. We excluded videos with embedded subtitles, such practices vey most of the audio information visually, enabling models to "cheat" through vision alone. Likewise, videos containing large text overlays these overlays often directly reveal information about characters' speech, mental states, or ongoing events, thereby undermining the assessment of the model's genuine understanding and reasoning abilities.
regarded as undesirable, since
Video resolution. To ensure video quality, we require minimum resolution of 480p, and the visual content must free from issues such distortion blurriness that would hinder comprehension.
C PROMPTS USED IN THIS WORK
C.1 PROMPT FOR OVERALL EVALUATION
Instruction:
You are given video. Based on content of the video, answer the follow-
ing question:
Question: {Question}
Options:
A: (Option A} B: (Option B} C: {Option C} D: {Option D}
Task:
Answer with the option's 's letter directly(e.g.. A, B, C, or D).
If your access the video content is limited, least option that is more likely than the others must be chosen. Mustn's 't give any other reason for can not choose!
C.2 PROMPT TO SELECT QUESTIONS THAT CAN BE ANSWERED WITHOUT RELYING ON OPTIONS
Role:
You are an impartial judge.
# Instruction: Your task is NOT to answer the question, but to determine whether the ques-
tion is inherently DEPENDENT the multiple-choice options in order to be answered.
Task:
We aim to convert this multiple-choice question into an open-ended question. The video content is NOT provided here, but you should assume you have fully watched the video and know everything about it. Your job is ONLY to decide whether the question itself *requires* the options to an- swerable.
# Guidelines:
If the question can still be reasonably answered #*without needing the options** (even if the exact wording might change slightly), return "No".
the question cannot be answered at all without the options (c.g., it explicitly asks "Which of the following..." ), return "Yes".
Question: {Question}
# Answer: (Answer)
Respond ONLY with "Yes" "No".
18

---

Published as a conference paper at ICLR 2026
C.3 PROMPT FOR MULTIPLE-CHOIC QUESTIONS WITH STEP-BY-STEP REASONING
# Instruction: You are given video. Based on the content of the video, answer the follow-
ing question:
Question: {Question}
Options:
A: (Option A} B: {Option BJ C: {Option CJ D: {Option D}
Task:
Note that you should first reason step by step, and then you should give your final choice in
A, B, C, or D.
Your answer format should be follows:
Step X: [Reasoning step
The final choice is:
[bbox((Answer (Answer with the option's 's letter directly(A, B, C, or D).)).
D CASE STUDY
Table 6: Error Distribution Across Models
Error Category
Gemini2.0-Flash
Qwen3.0-Omni
Qwen2.5-0mni
Audio Understanding
17.77%
25.6%
29.2%
Temporal Reasoning
6.15%
39.9%
28.2%
Multimodal Fusion
61.03%
23.0%
25.6%
Visual Understanding
12.03%
8.3%
9,9%
Counting
1.9%
2.0%
4.7%
Spatial Understanding
1.12%
1.2%
2.4%
Q: When the and in the picture discussing ice cubes, did they nolice Superman behind them?
Task:
causal reasoning
"raaoonnng_ster [tspps:
Based on the audio and visual cues in the video, can determine resson why and waman noticed Superman:
Audio Understanding Error
"modality:: "vision".
"evidence": "they notice Superman
0.37."
The
glass woman are discussing
"inference:: "gat Superman"
cubes.
says,
Locate the correct event.
"Yes, that's what happens when water cold." This indicates they
"modelity" "vision",
are focused the lce cubes.
*evidence": "Superman just turned around and took step.. "inference:: "get the point.
2. Immediately after the woman says,
Locate the correct event.
'No. what I'm saying is... there loud, crashing sound.
3. The camera cuts the man in the holding sandwich. He startled by the noise,
robe,
Misidentifying the content and source of the audio!
puodality" "audio",
X
"evidencs": "Superman made sound
drops his hiss--endwichh and and-tums tums around ehocked expression.
slepped wooden floor.!
"nferencc" 'Because Superman made
sound is clearly the noise of the
sound when he stepped the wooden Heoor
sandwich talling and hitting the floer. them notice Superman was the sound of
Thärefore, noise
Incorrect answer.
I}
"answer": B. Secause Superman made
him dropping his sandwich.
sound stepped on the wooden floor."
Model's answer: Bocause Superman
made too much noise while eating."
Figure 8: A case about Audio Understanding Error.
19

---

Published as a conference paper at ICLR 2026
Q: What is the fourth animal seen in Safari Zone?
Task: temporal underslanding
"reascning_slepe"
Based on the sequence of animals shown Safari Zane, we can identily the fourth one by observing the order of apoearaneer
Temporal Reasoning Error
"modallty": "audio",
*ovidence": *At 4:50, woman said 'We
going the Satan Zonel."
"ineerence:: "This indicates that they
would to see the animals the Safari
The video shows large bird's nest
Zone afterwards."..
on the ground, which not gn animal.
Incorrect temporal and sequential
Amodaiity"
"vision, "evidence":
X
brown bear
camera pans show group of elephants in an endosure.
appear the scene.".... ..ch
understanding !
X
"modelity" : "vision", *evidence": 'At 5:32, several siberian ligers appears the
3.
The view shifts to linn and lloness resting grassy slope.
..ch
The camera then shows a group cheetahs, with one lying down and standing
X
"vision", "evidence": 5:3, two white lions appear in the scene.".....
"modality":
Incorrect answer.
Therefore, the fourth animal seen Safari
"modullty": "vision", *ovidence": "At 6:04, a jackal appear in the scena. .....
cheetah.
Model's answer:" D. Chestai"
"answer": C. Jackal"
Figure 9: A case about Temporal Reasoning Error.
Q: When a blogger presents the process searching for skier, which of the following is correct?
Task: summarization
"reasoning.co L
Let's analyze the video content to determine the correct answer.
Multimodal Fusion Error
"modelity":
"sudio", "audio",""""uuuo"
"evidence": 'The blogger said he will
The where blogger presents process searching for skier occurs from 2:47 to 2:50.
search skier..
"inferenee "locale this point.").
Locate the correct event.
"modelity":
"vision",
Analyze the options The video shows search for "skier" and displays clipa.
"evidence": 'The blogger is showing the
Correct number of whole clips.
search
"inference
"locate this point.").
The blogger does not mention than them not aption is incorrest.
"modality": "vision",
This
Couldn't reason with cross-modality information!
*evidence": "There are 10 clips searched
overall.".
'inference::
point.").
Model's answer: The longth of second viewed cli is 8 seconds."
monsiilyy "vision',
evidence": "thore clips highly
relevent with the video."
x
Incorrect answer.
"inference:: "there videos found
that were not highly relevant."
"answer": B. There were clpps found that were not highly relevant."
Figure 10: A case about Multimodal Fusion Error.
To better understand the underlying causes of model successes and failures on Omni BideoBBee,,,,h we conducted detailed analysis using three representative models: Qwen2.5-Omni,, Qwen3-Omni- 30A3B, and Gemini2.0-Flash. We aligned each model's incorrect reasoning chain with the Om- niVideoBench ground-truth chains using DeepSeek-V3.. and categorized the resulting discrepan- cies into 44 concise error types. These types were further grouped into six high-level categories: temporal reasoning error, audio comprehension error, multimodal fusion error, visual compre- hension error, counting error, and spatial reasoning error. A human verification study covering 50 randomly sampled errors per category confirmed high level of agreement with the automatic
20

---

Published as a conference paper at ICLR 2026
Q: While Peggy is complaining her friends, where is the person with hair the color of the jacket she made for her husband located in relation to Peggy?
Task: spatial understanding
"reasoning_seee [
scene where Peggy is complaining to her friends, determine the location of the pereon with hair the color of the jacket she made for her husband.
Visual Understanding Error
"modalityy: : "vision", "evidense": "In 1:16, the jacket appears in water. The main
jacket
is kind of orange and the sleeves are led.
Locate the correct event.
Peggy is the woman with brown hair and glasses, sitting in the middle the three women.
"nference": 'The heir person either
ora
incorrect recognition of jacket and hair colors.
"audio", "avidence": "In 3:16, Paggy is complaing to her friends.", "inferenze": friends appear.") "
"modality:
Fie bionde-haieed woman is sitting an the right the table Peggy's 's perspective.
X
Tharefore, the person with hair color of the jacket is located to the right Pegay
"modality": : "vision",
X
only person with the red hair is shown, sitinng acoss from Peggy.".
"evidence":
Incorrect answer.
Model's answer: 'C. To the right Pegay"
"nneerece": "The hair color of woman acrces from Peggy matches the color of jacket.")
"answer:: AAA Across from Poggy.'
Figure 11: A case about Visual Understanding Error.
annotations. The quantitative comparisons appear in Table 6 and some cases are shown as figs.
[I
As shown in Figure 12 the first three categories constitute the majority of observed failures for open-source models, indicating that the core bottlenecks current MLLMs lie in the intersection of temporal modeling, non-speech audio interpretation, and pross-moda integration. These capa- bilities align closely with the design focus of Omni VideoBench, emphasizing long-range temporal dependencies, general acoustic semanties, and robust multimodal alignment.
3mmh
NGY SAAEET
SUON
2A.2%
21
Figure 12: Error distribution of Gemini2.0-Flashh Qwen2.5-Omni and Qwen3-Omni.
Temporal Reasoning Error. By analyzing the distribution of error types across different tasks, as shown in Table 7 we find that models frequently struggle with capturing long-range dependencies and sequential relationships, especially when relevant evidence is distributed across multiple clips. Difficulties are further amplified when visual and auditory streams are not perfectly synchronized, requiring the model to integrate misaligned cues over extended durations. These trends suggest that current MILMs lack sufficiently expressive mechanisms for hierarchical temporal modeling and consistent aggregation of temporal evidence.
Audio Understanding Error. By analyzing the distribution of error types across different dio types, as shown in Table g we find that non non-speech audio-ssin such as music and environmental sounds-reman major challenge because it lacks stable symbolic anchors comparable those
21

---

Published as a conference paper at ICLR 2026
Table Error distribution by video duration (percentage) for Gemini2.0-Flann Qwen2.5-0mni, and Qwen3-Omni.
Tresporal Reawnting
Cantting
Spattal Undersanniieg
06.1 mae
16.3
33%
36.8
9.5
16.3
6.2
GerniniZ
ean
(10.30g 19l
27.3
$3.2
33.4
1641
11.5
5-3
(1.5 maa
20,3
15.8
14.3
12
(1.00]
04.31 use
263
7:41
361
78
18
man
(5.00)
31.5
(00.30g 198
231
25.6
256
10.3
31
15
in spoken language. Models often fail to map continuous acoustic features (e.g., rhythm, timbre, intensity) to higher-level semantic interpretations involving events, emotions, or actions. Attribution mistakes also common, including misidentifying background music speech incorrectly signing off-screen sounds to on-screen entities. This reflects the speech-centric nature of existing audio encoders and insufficient grounding in general acoustic semantics.
Table 8: Error distribution by audio type (percentage) for Demiii2200FFlase Qwen2.5-0mni, and Qwen3-Omni.
Audie Type Audie Undeestoeiigg (9)
Tergeral (Sssning:()
Melisndal Frsine
Unlentaniag (S)
Counting
Sprtiel Uwlerstending
Masic
$05
59
16.3
23
F75
5.5
Soasd
$3.0
12.9
23.1
156
25.1
Masic
15.9
418
36.6
53
144
Qven2&0uui
Soaed
18.
18.1
56
47
00
Made
314
761
313
91
78
D6
Owea3-Clnnd
Multimodal Fusion Error. Multimodal fusion failures manifest primarily in two ways: (1) im- perfect alignment between visual and auditory cues and (2) modality neglect, where the model over- relies on single modality while disregarding complementary another. For example, a model may rely solely on visual cues to infer an answer while failing to incorporate crucial auditory evidence. Such issues highlight weaknesses in cross-moss moddll attention robustness and balanced multimodal reasoning.
corrective information present in
Tea
Figure 13: Proportion of error types across different tasks for Gemini2.0-Flash, Qwen2.5-Omni and Qwen3-Omni.
Counting and Spatial Understanding Error. Although models demonstrate comparatively weak performance on Counting and Spatial Understanding tasks, only a limited portion of errors are explicitly attributed to these categories. Further inspection, as shown in Figure 13 reveals that most failures stem from earlier deficits in fine-grained perception and precise temporal localiza- tion. These tasks require detecting multiple small entities, identifying subtle spatial orientations, or localizing objects at specific moments in long video sequences. Errors in early perception or temporal alignment therefore cascade into incorrect counting or spatial reasoning outcomes.
22

---

Published as a conference paper at ICLR 2026
E USE OF LLMS
Large language models (LLMs) were utilized in this work solely as research tools to assist with data quality control and ancillary writing support. Specifically, employed advanced multimodal LLMs (e.g., Gemini-2.0-Flasi and DeepSeek-V3) to help filter out questions that could be answered using only a single modality and to identify potential textual biases during dataset refinement, as described in Sec. 2.4 In addition, LLMs were used to perform minor language polishing of the manuscript after the main content was written by the authors. All experimental design, dataset con- struction, analysis, and conclusions were conceived and executed by the authors without automated decision-making. No confidential, private, or sensitive data were provided to any external L.LM services.
23


Reviews:
[{"review_id": "WAP71LavDr", "reviewer": "Reviewer_adN5", "summary": "The paper presents OmniVideoBench, a large-scale, carefully curated benchmark evaluating synergistic audio-visual reasoning in MLLMs. OmniVideoBench consists of 1,000 manually verified QA pairs with explicit step-by-step reasoning traces, based on 628 diverse long-form videos across 8 major categories and 68 subcategories. The benchmark emphasizes logical consistency, modality complementarity, and covers 13 question types relevant to real-world video understanding. The authors systematically filter and annotate the data to ensure questions demand audio-visual integration. Baseline experiments on both open- and closed-source MLLMs reveal a substantial gap to human-level reasoning, particularly regarding music, long videos, and abstract audio understanding.", "questions": "Please check the Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The benchmark features 1,000 high-quality, manually verified QA examples, each annotated with step-by-step reasoning chains with human proofreading. This explicit annotation supports analysis of both model answers and reasoning processes.\n2. Long Videos span 8 major categories and 68 subcategories, ensuring comprehensive coverage and evaluation of a wide range of real-world scenarios.\n3. The authors conducted extensive and comprehensive evaluations and experiments.", "weaknesses": "1. Missing Comparisons with Key Recent Audio-Visual Benchmarks: Several highly relevant, recently released benchmarks should be compared and discussed [1,2,3]. \n2. It will be great to see the error analysis on the properties of the questions/items themselves—e.g., what makes some reasoning chains or audio-visual interactions particularly difficult?\n3. More qualitative examples and error cases could be included in the appendix to provide deeper insights into model behavior and help readers better understand the paper.\n\n\n\n\n\n\n\n\n[1] Sung-Bin, Kim, et al. \"Avhbench: A cross-modal hallucination benchmark for audio-visual large language models.\" arXiv preprint arXiv:2410.18325 (2024).\n[2]Chowdhury, Sanjoy, et al. \"Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms.\" arXiv preprint arXiv:2501.02135 (2025).\n[3[ Sakshi, S., et al. \"Mmau: A massive multi-task audio understanding and reasoning benchmark.\" arXiv preprint arXiv:2410.19168 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B8gD35LbTb", "reviewer": "Reviewer_rRon", "summary": "This paper introduces OmniVideoBench, a large-scale benchmark designed to evaluate the collaborative audiovisual reasoning capabilities of multimodal large language models. The benchmark comprises 1,000 manually annotated high-quality question-answer pairs (QA) across 628 videos, each featuring explicit step-by-step reasoning chains indicating modalities and evidence. OmniVideoBench spans 8 primary video genres, 68 subcategories, and 13 distinct task types (e.g., temporal, spatial, causal reasoning), structured to comprehensively evaluate modal complementarity and logical consistency. The paper evaluates both open-source and proprietary MLLMs on OmniVideoBench, revealing that model performance lags significantly behind human capabilities, particularly on tasks requiring genuine multimodal integration.", "questions": "1. Can the authors provide quantitative statistics on the efficacy of the automated filtering steps in Section 2.4? Specifically, what is the rejection or retention rate at each filtering stage, and what percentage of QA pairs end up truly requiring both modalities?\n2. Please clarify how \"semantic units\" ($S_i$) are operationalized for the semantic distance metric. Is this manual phrase decomposition, or is some NLP toolchain applied? This is crucial to evaluating distractor design reproducibility.\n3. Will human benchmark results (e.g., accuracy, agreement rates) on OmniVideoBench be reported?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper ensures the richness and coverage of the dataset, comprising 1,000 distinct QA pairs that span a wide range of real-world scenarios, video durations, and audio types. It also contains 13 different task types covering diverse reasoning skills.\n2. The annotation protocol ensures that all questions require true audio-visual integration and stepwise reasoning, with multi-layered filtering to weed out unimodal or bias-prone items.", "weaknesses": "1. The paper does not provide statistical results comparing it with other relevant datasets, which fails to highlight the contribution of the proposed dataset.\n2 From Table 1 and Figure 3, it is evident that the vast majority of QA pairs relate to Speech (76.2%) versus Sound (14.7%) and Music (9.1%), creating considerable class imbalance.\n3 The benchmark's positioning relative to several directly analogous or recently proposed audio-visual (AV) benchmarks is incomplete. Recent works, such as AVHBench (Kim et al., 2025) and DAVE (Radevski et al., 2024), are not cited or discussed, nor are specialized audio-visual QA datasets, including AVQA (Yang et al., 2022) and MusicAVQA (Li et al., 2022).\n4. While human annotation is used in construction, the paper does not report human baseline accuracy or response variability for the main test set.\n5. While Figure 1 gives some specific sample breakdowns, the paper lacks a deeper set of qualitative analyses of successful versus failure cases, especially for (a) long video cases and (b) music understanding tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IhHSlKhEfk", "reviewer": "Reviewer_iHcc", "summary": "This work proposes OmniVideoBench, a new benchmark designed to address key limitations in existing video datasets—namely, the lack of systematic evaluation of audio–visual co-reasoning and inconsistencies in logical task composition. The benchmark is constructed from 628 real-world videos ranging from a few seconds to around 30 minutes, covering three types of audio (speech, sound, and music). It includes 1,000 multiple-choice questions spanning 8 major categories, 68 subcategories, and 13 task types, each accompanied by human-annotated step-by-step reasoning chains.\n\nExperimental results show that current models struggle significantly on this benchmark. While the best model, Gemini-2.5-Pro, only achieves 58.90% accuracy, and most open-source models perform near random. Besides, Long video understanding remains a major challenge for most models. Notably, performance drops sharply for music-dominated audio even for the strongest models. \n\nOverall, OmniVideoBench combines long temporal structure with audio–visual complementarity, providing a more realistic and comprehensive testbed for advancing multimodal video reasoning research.", "questions": "1. What is the core distinction between this work and WorldSense that is mentioned in Table 1?\nTable 1 suggests that the two benchmarks share similar characteristics across several dimensions—including modality coverage, domain diversity, video type, audio type, and answer format. A more explicit comparison would help clarify the novel contributions of OmniVideoBench.\n2. Qwen2.5-Omni performs better on WorldSense than on OmniVideoBench. Does this imply that OmniVideoBench presents a more difficult challenge?\nBeyond video length, are there other factors or task-level distinctions that contribute to the increased difficulty of OmniVideoBench? Providing such an analysis would help justify the benchmark’s added value.\n3. What is the significance of emphasizing long videos?\nTable 1 shows that SOTA model performance is comparable between the 5–10 minute and 10+ minute video subsets. Additionally, WorldSense already includes videos of up to 10 minutes. Could the authors clarify what unique challenges OmniVideoBench introduces with longer videos?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Broad Coverage and Task Diversity。 The dataset spans 8 high-level categories, 68 subcategories, and 13 task types, with video durations ranging from 4 to 1955 seconds. It also explicitly includes an Ultralong category for videos longer than 10 minutes.\n* Step-by-Step Human-Annotated Reasoning Chains. Each question is accompanied by a human-labeled step-by-step reasoning chain, in terms of modality, evidence, and inference triples.\n* Data Quality Control. The authors take multiple steps to ensure data quality, including a three-stage filtering pipeline and the exclusion of videos with large-scale on-screen subtitles that might leak answers or bias model predictions.\n* Comprehensive Evaluation. The benchmark is evaluated using a wide range of models, including both open-source and close-source of varying scales.", "weaknesses": "* Limited Per-Task Coverage. The dataset contains 1,000 questions spread across 13 distinct task types, resulting in fewer than 100 examples per task on average. This limited coverage may constrain the robustness of task-specific evaluation and generalization analysis.\n* Unbalanced Audio Category Distribution. The distribution of audio types is heavily skewed—Speech accounts for over three-quarters of the dataset, while Music constitutes only 9.1%. This imbalance may bias models and limit insights into performance under underrepresented audio conditions.\n* Lack of Human Baseline. Although the conclusion emphasizes a large gap between human and model performance, no human baseline is reported in the experiments. \n* Dataset unavailable. The authors did not provide the dataset link in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NUE3KcDHJh", "reviewer": "Reviewer_xiBm", "summary": "In this work, the authors propose OmniVideoBench, a benchmark for audio–visual understanding in videos. It consists of 1000 QA pairs from 628 videos ranging from several seconds to 30 minutes, within 13 question types, covering temporal reasoning, spatial localization, counting, causal inference, summarization, etc. They also evaluate multiple MLLMs on OmniVideoBench to performance investigation.", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novelty\n\nCurrently,  the video understanding community mainly focuses on understanding and reasoning from the visual modality. It ignores the fact that video is the combination of visual and audio clues. This OmniVideoBench provides a bench for comprehensively evaluating reasoning capabilities across both modalities. Hence, it somehow shows the value of this bench for multimodal video understanding.\n\n* Clarity\n\nThe paper is well-written with good structure. Hence, the clarity is basically good.\n\n* Significance\n\nThis paper focuses on evaluating audio–visual understanding capacity of MLLMs, which is an important and practical problem for video understanding. Hence, the significance is basically OK for video research community.", "weaknesses": "* Question Type \n\nThe authors choose 13 types for QA pairs. Please further explain why to choose these types. Is it sufficient to evaluate audio–visual understanding in videos? \n\n* Method Insight\n\nIt woule be more interesting to investigate or indicate how to design MLLMs to boost the tasks in this benchmark.\n\n* Small Size\n\nThe authors collected only 628 original videos. The small number of videos would restrict the generalization of this benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)