WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs
Authors: Jack Hong
Shilin Yan
Jiayin Cai
Xiaolong Jiang
Yao Hu
Weidi Xie
Venue: ICLR 2026 Poster (2026)
Keywords: OmniModality
Multimodal LLMs
Benchmark
Real-World Understanding
Primary area: datasets and benchmarks
TL;DR: We introduce WorldSense, the first benchmark to assess models' omni-modal understanding ability.

Abstract: We introduce WorldSense, the first benchmark to assess the multi-modal video understanding, that simultaneously encompasses visual, audio, and text inputs. In contrast to existing benchmarks, our WorldSense has several features: (i) collaboration of omni-modality, we design the evaluation tasks to feature a strong coupling of audio and video, requiring models to effectively utilize the synergistic perception of omni-modality; (ii) diversity of videos and tasks, WorldSense encompasses a diverse collection of 1,662 audio-visual synchronised videos, systematically categorized into 8 primary domains and 67 fine-grained subcategories to cover the broad scenarios, and 3,172 multi-choice QA pairs across 26 distinct tasks to enable the comprehensive evaluation; (iii) high-quality annotations, all the QA pairs are manually labeled by 80 expert annotators with multiple rounds of correction to ensure quality. Based on our WorldSense, we extensively evaluate various state-of-the-art models. The experimental results indicate that existing models face significant challenges in understanding real-world scenarios (65.1% best accuracy). By analyzing the limitations of current models, we aim to provide valuable insight to guide development of real-world understanding. We hope our WorldSense can provide a platform for evaluating the ability in constructing and understanding coherent contexts from omni-modality.

Published as a conference paper at ICLR 2026
WorldSense: EVALUATING REAL-WORDD OMNIMODAL UNDERSTANDING FOR MULTIMODAL LLMs
Jack Hong'. Shilin Yan I, Jiayin Cai', Xiaolong Jiang', Yao Hu', Weidi Xie
Shillnn
Xiaohongshu Inc. Shanghai Jiao Tong University
(jaaackhong, tattoo. (attoo..ys]/egmailn com weidilsjtu. edu.t
Project Page: https: https://jeacckhonggg..ccc github io..iioo/////// Wor/oo///orrre dSense
ABSTRACT
We introduce WorldSense, the first benchmark to assess the multi-modal video understanding, that simultaneously encompasses visual, audio, and text inputs. In contrast to existing benchmarks, our WorldSense has several features: (i) col- laboration of omni-modality, we design the evaluation tasks to feature a strong coupling of audio and video, requiring models effectively utilize the synergis- tic perception of omni-modality; (ii) diversity of videos and tasks, WorldSense encompasses diverse collection of 1,662 audio-visual synchronised videos, sys- tematically categorized into 8 primary domains and 67 fine-grained subcategories to cover the broad scenarios, and 3,172 multi-choice QA pairs across 26 distinct tasks to enable the comprehensive evaluation; (iii) high-quality annotations, all the QA pairs are manually labeled by 80 expert annotators with multiple rounds of correction to ensure quality. Based ate various state-of- thetthr models. The experimental results indicate that existing models face significant challenges in understanding real-world scenarios (65.1% 1% best accuracy). ). By analyzing the limitations of current models, we aim to provide valuable insight to guide development of real-world understanding. We hope WorldSense can provide platform for evaluating the ability in constructing and understanding coherent contexts from omni-modality.
WorldSense, we extensively evalu-
INTRODUCTION
The ability comprehend and reason about multimodal inputs-ranging from visual and textual to auditory, tactile, and beyond-is fundamental for both human and artificial agents to navigate and interpret the world. For example, when driving car, human driver integrates visual information (e.g., recognizing road signs, traffic lights, and obstacles), auditory cues (e.g., hearing the honking of another car or a siren approaching from behind), and tactile feedback (e.g., the feel of the steering wheel, the vibrations of the road, or the responsiveness of the brakes) to make real-time decisions and ensure safe navigation. This seamless multimodal integration enables intelligent agents to process complex, dynamic environments and respond to subtle cues-an ability that is essential for both human perception and development of embodied agents designed to interact naturally in the world.
In the recent literature, the development of Multi-modal Large Language Models (MLLMs) (Ope- nAI, 2023; Hurst et al., 2024; OpenAI; Team et al., 2023; 2024b; Zhang et al., 2023; Ma et al., 2024: Fang al., 2023) have led to remarkable progress sification (Liu et al., 2024c), captioning (Alayrac et al., 2022; Dai et al., 2023; Liu et al., 2024b), question-answering (Tang et al., 2024; Panagopoulou et al., 2023; Liu al., 2024f), OCR (Mathew et al., 2021; Zhang et al., 2024b), segmentation (Lai et al., 2024; Xia et al., 2024; He et al., 2024a), autonomous driving (Nie al., 2025; Sima et al., 2025; Chen et al., 2024a) and more. However, multi-modal analysis primarily focuses on visual-language information, leaving out crucial modal- ities like audio, which results in an incomplete evaluation of their multimodal capabilities. While
a series of tasks, for example, clas-
Project leader. Corresponding author.

---

Published as a conference paper at ICLR 2026
Question: What is the man in the video doing with the fruit in his hand?
Showing color of the blueberries
Displaying the strawberries Counting the number of blueberries
Displaying the size of the blueberries
Thi hure
Bigar than
Question:
Which country associated with the lively and energetic music with the highest pitch that appears in the video?
A. Israel
B. Greece
C. England
D. Ghana
and anooionii maai
elegent and remariiccceooo mUsE
Ivyyy and energetic mask
Figure 1: Examples in WorldSense. WorldSense highlights the importance of tightly coupled audio- visual perception for real-world understanding, where neither modality alone provides sufficient context for correct answer. In the first example, the video shows a man holding a fruit. However, vi- sual information alone reveals the object, and only audio clarifies the action. In the second example, identifying cultural elements and locating the "lively and energetic" music segment requires both visual and auditory cues. WorldSense offers a platform to evaluate MLLMs' real-worls perception and
understanding capabilities.
some benchmarks have started incorporating both visual and audio modalities, they still exhibit eral limitations. For example, OmniBench (Li et al., 2024d) and AV-Odyssey Bench (Gong et al., 2024) mainly emphasize image evaluation, whereas other benchmarks (Geng al., 2024; Li al., 2022; Yang et al., 2022) either restrict to captioning tasks or are limited to simple scenarios, or suffer from low-quality, monotonous questioning patterns.
This paper presents WorldSense, the first comprehensive benchmark designed to evaluate Multi- modal Large Language Models (MLLMs) perceiving, understanding, and reasoning with omni- modal information in real-world settings. The benchmark is defined by three key features: (i) Omni- modal integration. The benchmark emphasizes the joint processing audio and visual modalities, illustrated in Figure 1. Each question requires both modalities for accurate response removing either results in failure failure-enabling rigorous assessment of a model's capacity for integrated sensory understanding. (ii) Diverse videos and task coverage. The benchmark includes 1,662 synchro- nized audio-visual videos spanning 8 domains and 67 fine-grained suhcategories. It features 3,172 multiple-choice questions across 26 cognitive tasks, ranging from basic perception high-level rea- soning. This diversity supports systematic evaluation of multimodal comprehension across broad task spectrum. (iii) High-quality annotations. All question-answer pairs are curated by 80 expert annotators and undergo multiple validation rounds, including human review and automated MLLM verification. This ensures annotation accuracy and benchmark reliability. Through these method- ological advancements, WorldSense sets new standard for evaluating MILMS in real-world mul- timodal reasoning, advancing the field toward more human- understanding.
We conduct extensive evaluations for broad spectrum of MILMs, including open-source video models, video-audio models, and proprictary systems. Results reveal significant limitations in current models' ability to reason over omni-modal inputs in real-world contexts. Specifically, open-source video-audio models, despite processing both modalities, achieve only 25% accu- socy-comprrabl to random guessing. In contrast, proprietary models such Gemini 2.5 Pro reach up to 65.1% % accuracy. However, when restricted to single modality (audio or video), existing model's performance drops greatly, highlighting the critical role of integrated modality processing.
2

---

Published as a conference paper at ICLR 2026
200
Seiiy Maye
Senkuing siare
O Count
532 127 125-300
109 155
101
Orean Dore
Matcovan Py
smoligeee
Qadyy
cuantig
World Nour
Over shrarra
Traefee
Gvers
[b] QA Task Distribution
The Thilees
Cree
mus recd Gare
1400
(a) Domain Distrbbtion
(c) Signa's Distribution
Distribution
Figure 2: Distribution of WorldSense. (a) Videos in WorldSense spans 8 primary categories with 67 fine- fine-grained subcategories. (b) QA pairs are structured across 26 tasks. (c) Acoustic signals distribution. Individual videos may contain multiple audio categories, leading to overlapping counts in statistical analysis. Consequently, the cumulative sum of audio instances exceeds the total video count. (d) Video duration distribution. The average duration of videos is 141.1 seconds.
We further conduct ablation studies to dissect modality contributions. Visual inputs are essential, while
raw signals-yields additional gains over text transcriptions, due to pre- served paralinguistic cues, e.g., prosody, intonation, acoustic context. These findings affirm the com- plementary nature of audio-visual information and the necessity of their joint modeling for robust real-world understanding. Failure case analysis reveals persistent limitations in current MLLMs, motivating future directions for improving multimodal reasoning.
To summarize, we have made the following contributions: (i) we present WorldSense, the first benchmark tailored for evaluating MILMs' ability omni-modal video understanding, character- ized by integrated audio-visual inputs, diverse content, and high-quality question- -answirnnnnswrring anno- tations; (ii) we have conducted extensive evaluation of existing MLLMs, showing that most open- source models perform near chance, and even the best proprictary model achieves only 65% ac- curacy- exposing significant gap in real-world omni-modal reasoning: (iii) through ablation and failure analysis, we identify the key factors influencing performance, including raw audio and visual cues, and provide actionable insights to guide future omni-modal understanding design.
2 RELATED WORK
Multimodal Large Language Models. Current Large Language Models (LLMs) are capable of processing multimodal information, including visual, text, and audio. Early works, such as (Zhang et al., 2023; Liu et al., 2024b; Zhu et al., 2023; Driess et al., 2023; Wang et al., 2024d; Pi et al., 2023), successfully combine vision and text modalities. Subsequent research extends temporal under- standing (Wang et al., 2024f; Hurst et al., 2024; Team et al., 2024a; Liu et al., 2024e; Wang et al., 2024b;e; Li al., 2024a; Fang et al., 2024b; Xu al., 2024; Zhang al., 2024a: Tong et al., 2024; Chen et al., 2024c; Lu et al., 2024a; Liu et al., 2024a), while parallel efforts (Tang et al., 2023; Chu et al., 2023; 2024) focus on audio processing. Recently, researchers shift attention to models (Cheng et al., 2024; Sun et al., 2024; Team et al., 2024a; Lu et al., 2024b; Team et al., 2024b) capable of simultaneously processing text, vision, and audio inputs. Despite the growing interest in the mod- els which can perform the omnimodality understanding, the absence of comprehensive evaluation benchmark restricts the development. To address this limitation, we introduce our WorldSense to evaluate models' capabilities in perceiving and understanding real world omnimodal scenarios.
Multimodal Benchmarks. The development of MLLMs has been driven by benchmarks, evolving from static image understanding (Zhang al., 2024c; Liu et al., 2025; Li al., 2023; 2024b; Fu
3

---

Published as a conference paper at ICLR 2026
et al., 2024a; Yue et al., 2024) temporal comprehension (Li al., 2024c; Liu et al., 2024d; Song et al., 2024; Zhou et al., 2024; Fang et al., 2024a: Fu et al., 2024b; He et al., 2024b; Wang et al., 2024c; Xu et al., 2017; Yu al., 2019; Lin et al., 2024; Chandrasegaran et al., 2024). However, these benchmarks largely overlook the crucial role audio real-world perception. While several audio- visual benchmarks have been proposed, they face significant limitations. AV-Odyssey Bench (Gong et al., 2024) and OmniBench (Li et al., 2024d) focus on static images, Music-AVQA (Li et al., 2022) and AVQA (Yang al., 2022) are domain-specific with monotonous questions, and Long- VALE (Geng et al., 2024) limits its assessment to captioning capabilities alone. Given that existing benchmarks fail to provide comprehensive evaluation MLLMs' real-world understanding capa- bilities, we introduce WorldSense to address this critical gap in the field.
3 WorldSense
In this section, we first introduce the design principles in Section 3.1, followed by a description the data collection (Section 3.2) and annotation processes ( (3.3). We then compare statics of WorldSense with previous benchmarks in Section 3.4, and finally present our evaluation methodology 3.5.
3.1 DESIGN PRINCIPLE
As for multi-modal evaluation, base on the audio-visual synchronized videos, which capture temporal events, motion patterns, and audio-visual correlations. To curate the benchmark, we adhere the following three principles, ensure rigorous and comprehensive evaluations for MLLMs.
Comprehensive Domain Coverage. To capture the diversity of real-world scenarios, we construct hierarchical taxonomy starting from broad human-centric domains, refined into 67 fine-grained subcategories. This structure ensures wide ecological coverage, enabling robust assessment of multi- modal understanding across varied contexts.
Diverse Acoustic Modalities. Real-world audio can be broadly classified into speech, environmen- tal events, and music. The benchmark includes all three types, enabling evaluation across a spectrum of acoustic complexity-from from linguistic content to non-verbal and abstract auditory cues.
Multilevel Cognitive Assessment. We design a three-tiered evaluation framework targeting: recog- nition (detection of basic audio-visual elements), understanding (comprehension of multimodal relationships), and reasoning (high-level inference tasks such as causal inference or abstract think- ing). The benchmark includes 26 tasks aligned with these levels, encouraging holistic evaluation of perceptual and cognitive capabilities in multimodal settings.
3.2 DATA COLLECTION & CURATION
We primarily source video content from Video (Farré et al., 2024), large-scale dataset comprising high-quality YouTube videos that exhibit strong audio-visual correlations across diverse real-world scenarios. To enrich the benchmark's coverage of musical content, supplement with selected videos from MusicAVQA (Li et al., 2022), ensuring a more balanced representation of auditory modalities.
Our data collection employs a systematic filtering pipeline to ensure high-quality videos with rich visual-audio semantics and temporal dynamics, following three main steps in Figure 3(a): (i fil- tering videos according to predefined taxonomic categories delineated in Section 3.1; (ii) selecting clips based on pre-computed audio-visuu correlation and dynamic content metrics from about 8,000 initial videos; and (iii) human expert review for video quality and real-world relevance. This rigor- ous selection and processing results in 1,662 high-quality video segments with strong audio-visual correlations across various real-world scenarios.
3.3 ANNOTATION PROTOCOL
Question-Anan Answerigg (QA) Annotation. A team of 80 professional annotators is engaged in creat- ing high-quality multiple-choice QA pairs for each video by thoroughly reviewing both visual and audio content. The questions are designed to require integration of multiple modalities, enabling effective assessment of MLLMs multimodal understanding.
4

---

Published as a conference paper at ICLR 2026
Audio-Visual
Correlation 8,000 Clips
1,662 Clips
FineViDeo
Dynomic
Content
Source Videos
Video Domain Filter
Video Quality Filter
Human Review
WorldSense
Data Collection & Curation
Human Review
Invalid QA
4p
Q
QA Revision
MLLM Verification
Valid QA
Manual Annotation
Poirs
Quality Check
WorldSense
(b) QA Annotation Quality Control
Figure 3: Data collection and QA annotation pipelines. (a) Data collection and curation process.
(b) QA annotation and quality control pipeline.
Quality Control. To ensure QA quality, implement rigorous quality control process combin- ing expert review and automated checks, as illustrated in Figure 3(b). Professional quality control experts evaluate each QA pair based three essential criteria: (i) linguistic clarity and coherence, (ii) multimodal necessity for correct answers, and (iil) appropriate difficulty. Questions that fail to meet these standards are returned for revision.
We also use MLLMs for automated verification. Vision-language models like Qwen2-VL VLWang al., 2024b) verify that questions require multiple modalities for correct answers. Furthermore, multimodal MLLMs capable of processing video, audio, and text, such as Video-LLaMA2(Chee))) al., 2024) and OneLLM (Han al., 2024) are used to assess question difficulty, with questions answered correctly by all models being flagged for manual revision as too simple.
This dual verificatiin system, combining expert review and automated testing, ensures that all ques- tions in our benchmark are of high-qualit and well-formulated, that requires multi-modal compre- hension, and present significant challenges for the models.
3.4 DATASET STATISTICS
As summarized in Table 1, our proposed WorldSense benchmark contains 1,662 video clips with synchronized audio across 8 categories and 67 subcategories, averaging 141.1 seconds in length, including 3,173 multiple-choice questions on three cognitive levels.
WorldSense features diverse audio types such speech, environmental sounds, and music. Un- like existing benchmarks that use static images (e.g., AV-Odyssey Bench (Gong et al., 2024), Om- niBench (Li et al., 2024d)) or feature weak audio-vishe visual correlations (e.g., Video-MME (Fu al., 2024b)), WorldSense is the first to comprehensively evaluate MLLMs' real-world multimodal un- derstanding. It distinguishes itself through: (i) open-domain videos with multi-task evaluation, (ii) original audio-visual content with complete transcriptions, and (iii) carefully crafted questions re- quiring true audio-visure integration, establishing comprehensive benchmark for real-world multi- modal understanding assessment.
3.5 EVALUATION PARADIGM
In our evaluation framework, each test instance consists of vidco clip with synchronized audio and a multiple-choice question. Models must process these multi-modal inputs and select the correct answer from several options. Performance is measured by accuracy, comparing the model's selection to the ground-truth answers. A model's success is determined by its ability to accurately align with the correct answer. We employ matching-bass based approach to extract answers.
To rigorously assess the necessity of multimodal integration in real- worldwrldd understanding, we conduct ablation studies across various modality configurations. This approach not only evaluates overall

---

Published as a conference paper at ICLR 2026
Table 1: Statistics. V. for modality represent audio, video, and image. Len. refers the mean video duration in seconds. A and M for Anno. indicate automatic and manual annotation genera- tion. QA Tokens represents the average token count in QA pairs, while Sub. Tokens denotes the mean number of subtitle tokens. Multi-task represents whether the dataset encompasses more than two question categories. Open-domain signífies whether video content spans diverse domains. Sub./Au.. ispecifies the availability of audio signals or subtitle transcriptions. A-V Correlations indicates whether answering questions requires integration of omnimodal information.
Benchmarka
Modality
#Videos
Len.s)
#QA Pairs
Amno
Tokens
Tokens
task
Open domain
Aud.
Correlations
MSRVIT-QA (Nu ol., 2017)
V
2,950
72,821
X
X
X
X
ActivityNet-O QA (Yu al. 2019)
V
500
111.4
8,000
M
10.2
X
X
X
MVBenchi 2024c)
3,641
X
V
V
X
MovieChat (Song al., 2024)
V
130
500.0
1,950
M
X
X
x
Videu-Bench Bench (Ning et al., 2023)
V
5,917
$6.0
17,036
ARM
21.3
EgoSchema (Mangalam et al., 2025)
5,03
180.0
X
x
X
Video-MMEE (Fu ol., 2024b)
V
900
1017.9
2,700
M
35.7
3086.5
MMBench- Video (Fang et al., 2024a)
(09
105.4
1,998
19.3
(Yang 2022)
A+V
$7,000
57,335
14.3
X
X
/
Music-AVQA (L. al., 2022)
A4V
9,258
60
45,867
M
8.6
X
x
x
Omnißendh (Li al., 20246)
Ae1
X
1,142
M
37.8
AN-Odyssey (Gong
A-1
X
X
4,555
19.5
X
VongVALE VALE (Geng al., 2024)
A+V
8,400
235
X
A&M
X
A+V
141.1
37.2
986.2
model performance but also quantifies the models' reliance on individual modalities, highlighting the critical role of multimodal collaboration in real-world comprehension tasks.
EXPERIMENTS AND FINDINGS
4.1 SETTINGS
To comprehensively assess the multi-modal understanding ability, we evaluate three types of MLLMs: (1) open-source audio-visual models, such as Unified-IO-2 (Lu et al., 2024b), OneLLM (Han al., 2024), and VideoLLaMA2 (Cheng et al., 2024); (ii) open-source MILMs, such as Qwen2-VL (Wang et al., 2024a), LLaVA-One Vision (Li et al., 2024a), Intern (Chen et al., 2024b), LLaVA- Video (Zhang et al., 2024d), and so on; (iii) proprietary MLLMs, such as Claude 3.5 Sonnet (Anthropic, 2024), GPT 4o (Hurst et al., 2024), Gemini 1.5 Pro (Team et al., 2024a), and Gemini 2.5 Pro (Comanici et al., 2025). For all evaluations, we strictly adhere to each model's 's official implementation guidelines and the recommended pre- rrepprocessig procedures. Video frame extraction follows the official configurations specified by corresponding MLLMs, while pro- prietary models are evaluated according their API specifications and recommended input formats. Model performance assessed through direct comparison between model outputs and ground-true -ruth.
4.2 RESULTS ON WorldSense
Main Results. We present comprehensive evaluations of WorldSense in Table Our analysis reveals several significant insights regarding the capabilities of MLLMS in real-world understanding.
First, current open-source video models are limited in their performance as they process only visual information. This restriction highlights significant gap in their ability to perform complex, multi- modal understanding tasks, as evidenced by their maximum performance score of only 54.0%. The results underscore the inadequacies of relying solely visual processing, emphasizing the need to integrate audio inputs for more comprehensive understanding in practical applications.
Second and surprisingly, most of existing open-source audio-visual MLLMs perform even worse, achieving accuracy rates comparable to random guessing and notably below video-only MLLMs. This counter-intuitive finding reveals that despite having access to both modalities, these models struggle with effective visual integration, suggesting that multimodal processing capability alone does not guarantee better performance without sophisticated integration mechanisms.

---

Published as a conference paper at ICLR 2026
Table Overall performance on WorldSense. We evaluate three types MLLMs on WorldSense, understanding.
showing the significant limitations of existing MLLMs on real-world
LLM Tech Culture & Daily
Film & TV
Perfor- mance
Methods
Size Science
Polities
Life
Games
Sports
Music
Avg
Open-Source Video-Audio MLLMs
Unified-10-2 (Lu 2024b)
19.3
22.8
23.1
25.6
25.8
24.1
22.9
25.3
23.3
Unified-10-2 XI. al., 2024h)
26.5
24.4
22.5
23.5
24.7
28.0
25.7
24.2
24.7
Unified-10-2 XXL (Lu al., 2024h)
27.1
31.7
23.9
23.7
25.5
23.7
25.7
27.3
25.9
OneLLM (Hum al., 2024)
26.7
25.1
19.0
22.7
27.0
23.7
22.4
19.8
22.8
Vide (Cheng et al., 2024)
7B
29.4
25.4
21.8
24.5
26.2
24.6
25.5
27.1
25.4
VITA-1.5 (Fu et 2025)
38.2
35.9
34.3
39.8
41.2
32.6
34.7
39.9
36.9
Qwen2.550mmi (Xu et al., 2025a)
47.8
49.8
43.6
43.8
48.3
39.1
43.5
47.3
45.4
video-SALMONN 2+ (Tang et al., 2025)
7B
57.1
54.4
48.9
50.9
49.1
51.1
44.9
51.0
50.9
Qwen3-Omni (Xu et al., 2025b)
7B
58.7
60.5
54.5
53.8
55.4
46.8
48.8
52.2
54.0
video-SALMONN 2+ (Tang al., 2025)
72B
59.0
63.1
54.0
59.9
58.1
54.1
51.9
54.4
56.5
Open-Source Video MLLMs
Video-LLLaV (Lin et al., 2023)
23.6
20.8
19.1
17.3
23.6
17.2
20.8
20.1
20.3
L (Grattafiori et al., 2024)
27.5
25.7
28.9
25.9
27.7
21.1
29.0
26.8
27.1
Qwen2-VL (Wang al., 2024a)
33.5
29.0
28.4
33.6
30.3
32.3
34.7
38.5
32.4
mPLUG-OwL (Ye et al., 2024)
37.5
31.4
31.0
34.1
33.3
33.2
32.1
30.5
32.9
LLaVA-OneVision
al., 2024a)
38.9
36.3
37.6
37.8
37.9
36.3
39.1
37.7
InternVL2.5 (Chen et al.. 2024b)
43.7
40.9
34.6
39.7
37.8
36.2
39.4
41.1
39.1
LLaVA-Video (Zhang et al., 2024d)
41.6
40.6
42.1
40.4
39.7
37.0
40.9
40.2
Proprictary MLLMs
Claude 3.5 Sonnet (Anthropic, 2024)
43.7
31.7
30.6
36.5
30.7
31.9
36.6
33.9
34.8
GPT 4o (Hurst et al., 2024)
48.0
44.0)
38.3
43.5
41.9
41.2
42.6
42.7
42.6
Gemini 1.5 Pro (Team et al., 2024a)
53.7
47.2
50.3
50.4
52.4
46.8
40.2
42.0
48.0
Gemini 2.5 Flash (Comanici et al., 2025)
51.8
50.2
54.1
51.2
39.6
50.6
51.6
51.5
52.3
Gemini 2.5 Pro (Comanici et al., 2025)
64.9
66.0
65.8
68.1
69.7
65.7
63.5
61.3
65.1
Recigg
Oeger Bustrwe
Rempyshe
Sezrgen
boogene
Coarting
Terond
Fownten
Genin 15
SPT 4g
Guade 3.5 Sonnet
Quee2 5 Onei
VITA 15
VideoLLaMA2
Interi1.2.5
LLsV/A Video
OreLLi
UeffieT7PP-X XXL
Figure 4: Fine-grained results on task category. We present performance across all tasks.
Third, among proprietary MLLMs, vision-only models GPT-4o and Claude 3.5 Sonnet demonstrate performance comparable to the leading open-source video MLLMs. Gemini 2.5 Pro, capable of processing both audio and visual information, achieves the highest accuracy of 65.1%. However, this performance still falls considerably short of requirements for reliable real-world applications, indicating substantial room for improvement.
These comprehensive results illuminate several critical insights: (i the fundamental importance of audio-visual collaborative understanding in real-world scenarios; (ii) the current significant gap in models' capabilities for effective multimodal integration, and (iii) the need for more sophisticated approaches to combining and reasoning about multiple modalities. These findings point to crucial directions for future research and development in MLLMs.
Breakdown Results. We conduct a fine- tineggrinee analysis of model performance across different audio types and task categories, shown in Figure 4 and 5, highlighting the limitations of MLLMs.

---

Published as a conference paper at ICLR 2026
Speech Ewer? Madi
401.
283
26.8
25.2
25/6
1611
25.9
Voua
5433
24.5
Uscevionity
23.2
233
223
80
Gerve 15
Qven25-Ors
VITA-1
WMooLaaMM
CWNLLM
Unfeed1O44
UnfiedEOR-LL UnriedTOC
Figure 6: Error distribu- tion. Sampled 5 error cases per task.
Figure 5: Fine-grained results on audio signals. Existing models exhibit inconsistent performance across audio types.
Table 3: Impact of vision information. We evaluate MLLMs' performance under different input configurations: audio-only input, audio combined with either video captions video frames.
Tech Science
Culture Politics
Daily Life
Film & Perfor- mance
Methods
Modality
Games
Sports
Music
Avg
TV
Audio
23.0
25.4
24.2
26.7
27.7
23.7
25.0
27.1
25.2
Unified 10-2 (I. et al., 2024h)
Caption
21.5
21.1
20.7
17.1
19.9
19.0
22.9
23.7
20.9 4.3
Video
19.3
22.8
23.1
25.6
25.8
24.1
22.9
25.3
23.3-1.9
Audio
21.7
22.4
22.4
22.1
24.7
25.0
25.9
24.7
23.4
Unified-10-2 XL (Lu et al., 2024b)
Caption
19.9
19.8
20.8
19.2
20.2
15.9
21.7
25.5
20.7
Video
26.5
24.4
22.5
23.5
24.7
28.0
25.7
24.2
24.7.4..
27.5
28.7
23,9
23.2
25.8
21.1
26.2
30,2
25.9
Unified-10-2 XXL (Lu et. 2024b)
Caption
24.0
26.7
23.0
18.9
18.7
20.7
25.9
29.4
23.7-
Video
27.1
31.7
23.9
23.7
25.5
23.7
25.7
27.3
25.9.00.0
Audio
25.7
26.1
19.3
21.9
25.8
25.9
21.5
22.4
23.0
OncLLM (Han 2024)
Caption
29.0
25.9
29.1
33.0
29.2
28.6
28.645.6
Video
26.7
25.1
19.0
22.7
27.0
23.7
22.4
19.8
22.8_o.2
Audio
23.8
23.4
21.3
22.4
24.7
19.8
27.1
27.9
23.8
VideoLLaMA2 (Cheng al., 2024)
Caption
30.0
30.0
25.6
29.9
28.5
25.0
29.7
29.9
28.5
Video
29.4
25.4
21.8
24.5
26.2
24.6
25.5
27.1
25.4.
Audio
30.2
35.6
36.3
30.9
32.2
32.2
31.4
33,3
32,9
VITA-1.5 (Fu et al., 2025)
Caption
39.2
39.8
37.2
37.5
37.5
35.2
34.9
38.4
37.5
Video
34.3
41.2
32.6
34.7
39.9
36.9;4.0
Audio
40.0
38.2
36.0
33.5
31.1
30.5
32.3
33.3
34,9
Qwen2.5-Omm (Xu et al., 2025a)
Caption
40.0
37.9
38.9
33.55
36.7
37.8
37.7
38.9
37.9
Video
47.8
49.8
43.6
43.8
48.3
39.1
43.5
47.3
45.4410.5
Audio
33.0
34.6
Gemini 1.5 Pro (Team al., 2024a)
Caption
49.5
$2.1
41.8
42.9
46.4
41.8
39.6
36.7
43.649.
Video 53.7
47.2
50.3
50.4
52.4
46.8
40.2
42.0
48.0
First, models consistently underperform on audio-related tasks (e.g., audio recognition, audio count- ing) compared to other task types, demonstrating significant challenges in audio understanding. Sec- ond, spatial reasoning and counting tasks present notable difficulties for current models, pattern consistently observed across multiple benchmarks. Third, emotion-related tasks prove particularly challenging. likely due to their requirement for integrating subtle and complex multimodal cues, including facial expressions, vocal tones, and contextual speech content. This underperformance in emotional understanding suggests significant gap in current MLLMs' training data and capabili- ties, highlighting an important area for future development.
Additionally, performance varies across audio types. While Gemini 1.5 Pro performs best overall, shows notably lower accuracy on event-related questions compared to speech or music tasks, possibly due the complex nature of environmental sounds. Other models also exhibit inconsistent performance across audio types, underscoring a general limitation for audio understanding.
4.3 ROADMAP TOWARDS REAL-WORLD UNDERSTANDING
Given the substantial performance gap revealed in above evaluation, we conduct an in-depth inves- tigation into potential approaches enhance the MILMMs performance.
8

---

Published as a conference paper at ICLR 2026
Table Impact of audio information for Video-Audio MLLMs. We conduct experiments across three input configurations: video-only, video with subtitles, and video with original audio.
Speech
Event
Music
Uninel--n-2 2024b)
23.1-1.1
26.9
13.5
15.0
23.3-a1
Unified 30-2
2024b)
13.0-130
24.8
24.7-au
United 10.2 XXL
20248)
15.6-124
25.7-23
26.2
14.2-22.
17.2-141
OseLLM (Han 2024)
23.2-107
12.4
19.3
22.348.
12.4
22.8.3822
VideLLLMM22 (Cheng
2024)
17.1
25.5-a.4
25.2,x.1
16.1
24.5444
17.7
27.0.a3
26.
25.4..0
VIUA-L.5 (I's 2025)
37.6
19.1-11
36.2-3/4
36.4
38.2.14
38.3-1.1
37.1-1
39.3.3.8
36.5-13
Ommn: (Kn 2025at
36.7
18.2.ns
64.8.
37.6
37.7um:
43.8u83
40.7
39.2
45.2.6o
Gemini Pro (Team 2004g)
34.3
39.6.a
49.2.12.
33.0
389,69
4677.163
35.4
392
498,114
34.4
39.3,m
48,0,150
Table 5: Impact of audio information for Video MLLMs. We provide video-only MLLMs with
the subtitles and compare the performance with models with only video input.
Speech
Event
Music
Overall
Methods
Vidco
Subtitle
Vidco
Subvitle
Video
Subtitie
Vidco
Subxitle
Va 2023)
20.3
15.4
19.8
14.4
6.4
19.5
16.4
20.3
16.0 4.3
LLaMA3.2 (Grattafioti al., 2024)
2..3+
29.6+2.00
28. 2-.1
28.8 1.7
Qwen2-VL (Wang et al., 2024a)
31.8
41.1+9.3
30.9
39.4.8.5
34.2
41.347.6
32.4
41.2-8888
mPLLUG-OW3 (Ye at.,. 2024)
33.0
69.2+6.2
32.3
38.3+6.0
34,6
39.2+4.66
32.9
38.7 38.7
LLaVA-OneVision (Li et al., 20224)
37.7
44.0-6.3
36.3
42.7+6.4
39.7
45.7
37.7
43.9
VL2.5
202/b)
48.3-9.3
47.9+9.3
47.1+7.9
47.8+8.7
LLaVA-Vidoo (Zhang et al., 2024d)
40.5
45.9-6.44
38.9
44.645.7
42.3
47.746.44
40.2
45.6445.4
GPT 4o (Hurst et al., 20244
42.8
$1.148.3
40.9
50.2+9.3
43.6
49.9+6.3
42.6
50.147.5
Vision Information. We investigate the impact of visual information through different input con- figurations:
audio-only, audio with video captions, and audio with video frames. As shown in Table 3, visual information generally improves performance, with Gemini 1.5 Pro's accuracy in- creasing from 34.6% (audio-only) to 48.0% (+video). However, impact varies across models, with Unified102 showing inconsistent gains and even degradation with captions.
These findings suggest two important insights: (1) visual information is crucial for enhancing multi- modal understanding when properly integrated, and (2) current models' ability to effectively utilize visual information remains limited.
Audio Information. We examine the impact of audio information through three configurations: video-only, video with subtitles, and video with original audio.
The results in Table 4 reveal intriguing patterns in how different forms of audio information in- fluence model performance. For Gemini 1.5 Pro, accuracy increases from 34.4% (video-only) to 39.3% with subtitles, and further 48.0% with original audio. Other models, such OneLLM and Qwen2.5-0mni, show similar improvements. These results demonstrate that both subtitles and acoustic features (including tone, emotion, and environmental sounds) contribute valuable informa- tion, beyond what subtitles alone can capture, emphasizing the importance of complete acoustic cues in omni-modal real-world understanding.
Interestingly. UnifiedIO2 demonstrates performance degradation when integrating either subtitles or audio, with subtitles causing a notable accuracy decline, suggesting difficulties in multimodal processing. Conversely, Video-I.LaMA2 improves with both modalities but performs better with subtitles than original audio, indicating stronger reliance on textual rather than acoustic information.
We further evaluate video-only MLLMs by providing transcribed subtitles, as shown in Table 5. Nearly all models show significant improvements with subtitle integration, reinforcing the impor- tance of audio information. However, the performance gain is less pronounced in music-related questions, as subtitles cannot effectively capture inherent acoustic features such as melody, rhythm, and harmony.
These evaluations highlight several critical findings: (i) original audio contains rich information be- yond what subtitles capture, particularly for music; (ii) current models show significant limita- tions multimodal processing. These insights suggest important directions for improving MLLMs' ability to integrate acoustic and textual information for comprehensive scene understanding.
Failure Analysis and Future Improvement. We perform error analysis on 130 samples of Gemini 1.5 Pro (5 random samples per task) through manual review, identifying three main error types: Au-

---

Published as a conference paper at ICLR 2026
-|1-1-
had for and holf hours"
-1|1---
"sound guzheng music"
Question: At approximately what time does the meat start rocsting in the video?
Question: How does the rhythm the music change in the first half the video?
A. From heavy to joyful B. From passionate to soothing
A. 11:00::000
10:00:00
12:00:00 10:30:00
From joyful heavy From soothing to passionate
clock in the shows the time 10:30. Therefore, the correct onswer is D.
The music begins with gentiee, flowing melody, As the video progresses, the tempo increases, and the plucking of the strings becomes intense. Therefore, the correct answer is VideoLLoMAZ: D
Gemin/ 1,5 Pro:
VideoLLaMA2: C
LLaVA-Videe:
LLaVA-Video: D
Error Category: Vision Understanding & Reasoning Error
Error Category: Audio Understanding Error
Error Reason: Gimini Pro incorrectly identified the clock time. in the picture and calculated the start time.
Error Reason: Gimini 15 misinterpreted musical changes
Figure 7: Failure Case. We present two error examples.
dio Understanding Errors (misinterpreting audio information), Visual Understanding Errors (missing visual details), and Reasoning Errors (faulty logical steps). As shown in Figure 6, most errors stem from audio understanding deficiencies and reasoning failures. The reason for poor accuracy and limitation of existing models can be summarized as follows: (i) Inadequate Audio Understanding. Existing models fail to understand audio information correctly and show significantly weaker audio processing than visual understanding. (ii) Limited Cross-Modal Integration. Models often pro- cess modalities independently rather than performing true multimodal integration and suffer from insufficient omni-modal information integration. (iii) Insufficient Complex Reasoning Ability. Despite correct perception, MILMs still conduct error reasoning. leading to incorrect conclusions.
Figure showcases two failure cases of Gemini 1.5 Pro, VideoLLaMA2, and LLaVA-Vided The left case involves vision understanding and reasoning error, where Gemini 1.5 Pro incorrectly iden- tified the clock time as 10:30 instead of the actual 10:00 displayed, leading to the incorrect answer. This reflects deficiencies in basic visual perception and properly correlating visual information with the question. The right case demonstrates an audio understanding error, where models misjudged the rhythm pattern change in guzheng music, interpreting it as changing from soothing to intense (option D) rather than the correct passionate to soothing (option B). This case indicates that tasks involving interpretation of emotion and rhythm patterns remain challenging for existing MLLMs.
We also raise several key strategies to enhance models' understanding of omni-modality informa- tion: (i) Coupled Multimodal Training Data. Using naturally coupled, interleaved multimodal data, for example, audio, visual, language content, would enhance models' capability to leverage cross-mes modal dependencies. (ii) Architectural Improvements. Enhanced attention mechanisms fa- cilitating deep multimodal integration could emphasize early fusion between modalities, rather than processing them as separate streams for late fusion. (ii) Advanced Modal Alignment Techniques. Progressive alignment strategies that gradually enhance the model's ability to align information across modalities could lead to more effective utilization of multimodal inputs. (iiii) Reasoning strengthening. Incorporating diverse reasoning-focure focusee data can strengthen logical inference capa- bilities, enabling more coherent and accurate conclusions.
CONCLUSION
In this paper, we propose WorldSense, the first benchmark designed evaluate MLLMs' omni- modal understanding in real-world scenarios. Distinguished by its emphasis on joint omnimodal comprehension across diverse real-world contexts, WorldSense encompasses rich video categories and carefully curated question-answer pairs that necessitate the integration of visual and acoustic information. Through extensive experiments, we expose significant limitations in current MLLMs' ability to process and coherently integrate omnimodal information. Our analysis demonstrates the importance of omnimodal collaboration in real-world understanding. We hope that WorldSense can serve as foundational benchmark for advancing human-like omnimodal understanding capabilities.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Man-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
Anthropic.
Introducing the next generation of Claude. https:/ /www anthropic....... com/ news/ news/claude 3- Family, Ly, 2024. Accessed: 2024-10-22.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Byzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video- language understanding. arXiv preprint arXiv:2411 1.04998, 2024.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with Ilms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 14093-14100, IEEE, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-source suites. arXiv preprint urXiv:2404. 16821, 2024c.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 2024.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.0799 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407. 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Oni Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.
Wenliang Dai, Junnan Li, D Li, AMH Tiong, Zhao, W Wang, B Li, P Fung, and S Hoi. Instruct- blip: Towards preprint arXiv:2305.0663 2, 2023.
vision-language models with instruction tuning. arxiv 2023. arXiv
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint
2023.
Rongyao Fang, Shilin Yan, Zhaoyang Huang, Jingqiu Zhou, Hao Tian, Jifeng Dai, and Hongsheng Li. Instructseq: Unifying vision tasks with instruction-conditioned multi-modal sequence gener- ation. arXiv preprint arXiv:2311.7888 2023.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406. /4565, 2024a.
Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jan Kautz, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. Vila2: Vila augmented vila. arXiv preprint arXiv:2407.17455 17453, 2024b.

---

Published as a conference paper at ICLR 2026
Miquel Farré, Andi Marafioti, Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Finevideo. htttps://huginngeee
co/ datasetss/////ss wtw.ggggogaoee/////eeeeee/eeeeee
2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a. URL 2306. 13394.
abs/
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405. 21075,, 2024b.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.019577 2025.
Tiantian Geng, Jinrui Zhang, Qingni Wang. Teng Wang, Jinming Duan, and Feng Zheng. Long- vale: haan---dddo-lnnguge-eeeee benchmark towards time-aware omni-modal perception of long videos. arXiv preprint arXiv:241..19772. 2024.
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, et al. Av-odyssey bench: Can your multimodal Ilms really understand audio-visual information? arXiv preprint arXiv:2412.02611, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and al. The Ilama 3 herd of models, 2024. URL https: /ttptips://aaxi.../// /aaaaaiiiivv.... /ttp///ii.....aaa///////a.oataaa 217833.
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26584 26595, 2024.
Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun- He, Jin-Peng Lan, Bin Luo, and Xuan- song Xie. Multi-modal instruction tuned ilms with fine-grained visual perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 13980-13990, 2024a.
Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos. arXiv preprint arXiv:2406.08407, 2024b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.211 210.2216.. 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9999999 9579-9589, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li. Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0 03326. 2024a.
Bohao Li, Rui Wang, Guangzhi Wang. Yuying Ge, Yixiao Gc, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13299- 2024b.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-nong Wen, and Di Hu. Learning to answer questions in dynamic audio visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108-19118. 18, 2022.
12

---

Published as a conference paper at ICLR 2026
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-22206, 2024c.
Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, et al. Omnibench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409. 15272, 2024d.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for milms to achieve streaming video understanding. arXiv preprint (0ti::2211.033222 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Huan Liu, Lingyu Xiao, Jiangjiang Liu, Xiaofan Li, Ze Feng, Sen Yang, and Jingdong Wang- Revisiting milms: An in-depth analysis of image classification abilities. arXiv preprint arXiv:2412.16416 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pp. 216-233. Springer, 2025.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476, 2024d.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04460 2024c.
Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang. Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for Ivims. arXiv preprint arXiv:2406. 11833, 2024f.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseck-vl: towards real real-world vision-language understanding. arXiv preprint arXiv:2403.055: 005555555...... 2024a.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem. and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22455, 2024b.
Feipeng Ma, Hongwei Xue, Guangting Wang, Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Visual perception by large language model's weights. arXiv preprint arXiv:2405.20 20339, 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long- form video language understanding. Advances in Neural Information Process- ing Systems, 36:46212 12246222244244, 2023.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
13

---

Published as a conference paper at ICLR 2026
Ming Nie, Renyuan Peng, Chunwei Wang. Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Rea- son2drive: Towards interpretable and chain- based reasoning for autonomous driving. In European Conference Computer Vision, pp. 292-306 Springer, 2025.
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:231 1.16103, 2023.
OpenAl. Gpt-4v(ision) system card, 2023.
OpenAl. Gpt-4 technical report. arxiv 2303.08774. View Article, 2(5), 2023.
Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: framework for aligning x-modal instruction-aware representations to Ilms and emergent cross-modal reasoning. arXiv preprint arXiv:2311.18799 2023.
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, et al. Detgpt: Detect what you need via reasoning. arXiv preprint arXiv:2305.14167, 2023.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph vi- sual question answering. In European Conference on Computer Vision, pp. 256-274. Springer, 2025.
Enxin Song. Wenhao Chai, Guanhong Wang. Yucheng Zhang. Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221-18232 2024.
Guangzhi Sun, Wenyi Yu, Changli Tang. Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint wxXv:2440.11504 2024.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint ar 1xxi:23100.13200 2023.
Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn 2: eenhoninng-ehanneeed audio-visual large language models. arXiv preprint ar arXiv:2506. 152200 2025.
Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al. Mtvqa: Benchmarking multilingual text-centric visual question answering. arXiv preprint 1985 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Dean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.0 055300 2024a.
Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, al. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint arXiv:2404, 12387, 2024b.
Shengbang Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer. Xichen Pan, et al. Cambrian-L 1: A fully open, vision-centric exploration of multimodal llss. arXiv preprint arXiv:2406. 16860, 2024.
14

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024a.
model's
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing langugge model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng. Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, al. Lvbench: An extreme long video understanding benchmark.
arXiv preprint arXiv:2406.0800 08035, 2024c.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024d.
Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scal- ing multi-modal Ilms to 1000 images efficiently via hybrid architecture. arXiv preprint arXiv:2409.0 02889, 2024c.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024f.
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Gen- eralized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3858-3869, 2024.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Pro- ceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.202/5 2025a.
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang. Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint 25X9::2509.1777 17765, 2025b.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models. arXiv preprint arXiv:2407.1558 2024.
Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering videos. In Proceedings of the 30th ACM inter- national conference on multimedia, pp. 3480-3491, 2022.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint 2rXiv:2408.04844 2024.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qu: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong A versatile large vision language
Duan, Bin Wang, Linke Ouyang, et al. mitemlm-xcomposee model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320 2024a.
15

---

Published as a conference paper at ICLR 2026
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. clama-adapted Efficient fine-tuning language models with zero- init attention. arXiv preprint 233::23333.16199 2023.
Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, and Yoshua Bengio. Ver: Visual caption restoration. arXiv preprint arXiv:2406.06462. 2024b.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal IIm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408. 13257, 2024c.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.0271: 2024d.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.0422 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 1055 2023.
A APPENDIX
A.I THE USE OF LARGE LANGUAGE MODELS (LLMS)
Authors use large language models (LLMs) solely as writing assistant for text refinement and language polishing.
A.2 QUALITY CONTROI.
Experienced annotators. Our annotation team consists of 80 professional annotators with extensive QA annotation experience. These annotators are proficient in English, and have participated in sereval QA data annotation projects.
Sufficient annotation training. We conducted one-week training program with 200 videos (ex- cluded from final benchmark) until annotators achieved high proficiency (only 10% requiring mod- ifications).
Annotation instruction. Each annotator received comprehensive instruction with task expla- nations, question formulation guidelines, QA creation instructions, annotated examples, and cross- modal inference requirements.
Our review process identified and revised similar QA pairs. Through professional annotators, thorough training, detailed guidelines, and rigorous quality control, ensure high-quality notations.
A.3 IMPLEMENT DETAILS
For open-source MLLMs, we strictly follow their official implementations and recommended pre- processing pipelines to ensure fair comparison. For GPT 4o and Claude 3.5 Sonnet, we sample 16 frames uniformly from each video, while for Gemini 1.5 Pro, we utilize the official API for raw video file uploads. We conduct all the experiments on a NVIDIA A100 GPU.
A.4 EVALUATION PROMPT
Following previous works (Fu et al., 2024b; Li et al., 2024c), we adopt the format of "whole video frames + whole subtitles/audios (optional) question with prompt" prompt. We show the evalua- tion prompt across three input configurations: video-only input, video with subtitles, and video with audio content as following.
16

---

Published as a conference paper at ICLR 2026
Evaluation Prompt
Carefully watch this video and pay attention ion every detail. Based on your observations, select the best option that accurately addresses the question.
These the frames of video. Select the best answer the following multiple-choice question based on video. Respond with only the letter (A, B, C, or D) of the correct opticn.
Question: 0
(Option1
[pption2
[Option3]
[Option4]
Answer:
Evaluation Prompt with Subtitles
Carefully watch this video and pay attention every detail. Based on your observations, select the best option that accurately addresses the question.
These are the frames a video. This video's sustitles are listed below:
(subtitles)
Select the best answer to the follewing multiple-choice question based on the video. Respond with only the letter (A, B, C, or D) of the correct option.
Question: (}
(Option1
[Option2
Option3]
(Option4)
Answer:
Evaluation Prompt with Audios
Carefully watch this video and pay attention to every detail. Based your observations, select the best option that accurately addresses question.
These are the franes of a video and the corresponding audio. Select the best answer to the follewing multiple-choice question based the video. Respond with only the letter (A, B, C, D)
of the correct option.
Question: 0
(Option1]
Option2
Option3
(Option4)
Answer:
17

---

Published as a conference paper at ICLR 2026
A.5 LIMITATION
While our WorldSense represents significant advancement in evaluating multimodal understanding capabilities of MLLMs, the multiple-choice format inevitably constrains the assessment of models' generative capabilities. Real-world understanding often requires open-ended responses, explana- tions, and adaptability beyond selecting from predefined options. Our WorldSense may not ade- quately evaluate how models perform tasks requiring nuanced reasoning creative problem- solving. We will add open-ended questions and expand the evaluation paradigm to better assess real-world multimodal understanding.
B BROADER IMPACTS & ETHICS STATEMENT
Our work on WorldSense has several potential positive impacts on society and AI development, while also presenting certain risks that warrant careful consideration. WorldSense contributes to advancing MLLMs' ability to understand and interact with the real world through multiple modali- ties. This progress could benefit various applications, including assistive technologies, educational tools, human-AI interaction systems, safety systems, and so on. We also acknowledge potential risks and challenges. The development of more capable systems might raise privacy concerns. Advanced multimodal understanding capabilities could potentially be misused for surveillance or monitoring purposes. We believe that open discussion of these impacts is crucial for the responsible development of multi-modal large language models.
Our research on WorldSense adheres to strict ethical principles and guidelines. We acknowledge several important ethical considerations: (1) Data Collection and Privacy. All video content in WorldSense has been collected from publicly available sources with appropriate licensing agree- ments. We have con ducted thorough reviews and implemented comprchensive data processing procedures to ensure privacy protection, including the removal of any personally identifiable infor- mation. (2) Potential Biases. While acknowledging that inherent biases may exist in any dataset, we have undertaken systematic efforts to ensure diverse representation across our video content and question-answer pairs, encompassing various domains, cultures, and contexts. Nevertheless, we rec- ognize that completely eliminating bias remains significant challenge, and users should carefully consider these potential limitations when utilizing our dataset. (3) Intended Use. WorldSense is specifically designed to advance research in omnimodal real-world understanding. While we ac- tively encourage the use this benchmark for academic and research purposes, we strongly caution against any applications that could potentially result in harmful or discriminatory outcomes. Users are expected to adhere to ethical guidelines and responsible practices.
B. LICENSE
The WorldSense dataset released under the CC BY-NC-SA 4.0 License. Authors bear all respon- sibility in case of violation of rights and confirmation of the data license.
B.2 DATASHEETS
B.3 MOTIVATION
For what purpose was the dataset created?
To evaluate MLLMs capabilities in real-world omnimodal understanding.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
The authors of this paper.
Who funded the creation the dataset?
Xiaohongshu Inc.
Any other comments?
No
18

---

Published as a conference paper at ICLR 2026
B.4 COMPOSITION
What do the instances that comprise the dataset represent (e-g., documents, photos, people, countries)?
Videos along with captions and question/answer pairs.
How many instances are there total (of each type, if appropriate)?
WorldSense contains 3,172 question-answer pairs and contains 1,662 videos in total.
Does the dataset contain all possible instances or is it sample (not necessarily ran- dom) of instances from a larger set?
Videos of WorldSense sampled from Vine and Music AVQA. All QA pairs re-annotated manually.
What data does each instance consist of??
Each instance contains one video with its corresponding audio, a question about the video content and the corresponding answer, the category of the video, the fine-grained video understanding capability examined by the question, and the class of audio content. Each instance also contain the auto-generated subtitles sourced from YouTube.
Is there label target associated with each instance?
Yes. We provide the ground-truth answer for each question.
Is any information missing from individual instances? N/A.
Are relationships between individual instances made explicit (e-g., users' movie rat- ings, social network links)? N/A.
Are there recommended data splits (e-g., training, developneen/vaaliaation,, testing)?
No, WorldSense is designed for evaluation only.
Are there any errors, sources of noise, redundancies in the dataset?
No.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? WorldSense self-contained.
Does the dataset contain data that might be considered confidential (e.g,, data that is protected by legal privilege or by doctor - patient confidentiality, data that includes the content of individuals' non-public communications)? N/A.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
N/A.
B.5 COLLECTION PROCESS
How was the data associated with each instance acquired?
See main paper for details.
What mechanisms or procedures were used to collect the data (e.g., hardware appa- ratuses or sensors, manual human curation, software programs, software APIs)?
Humans are required to propose question and corresponding answer based on the video. MI.LMs, such as Qwen2-VI, Video-LLaMAN and OneL.LM are utilized perform quality control.
If the dataset is sample from a larger set, what the sampling strategy (e.g.. deterministic, probabilistic with specific sampling probabilities)?
Yes, sample the videos from Fine Video and Music-AVQA. See main paper for details.
19

---

Published as a conference paper at ICLR 2026
Who was involved in the data collection process (e.g., students, crowdworkers, tractors) and how were they compensated (e.g., how much were crowdworkers paid)? The authors and contractors are involved in the data collection process and are paid a fair wage.
Over what timeframe was the data collected?
The dataset is collected in 2024.
Were any ethical review processes conducted (e.g,, by an institutional review board)?
All videos in our benchmark are human-selected based on appropriate value propositions and undergo second manual quality check ensure there are no ethical violations.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e-g,, websites)?
We obtained video data from Fine Video and Music-AVQA.
Were the individuals in question notified about the data collection?
We didn't collect the data from the individuals. The data was collected from public web sources instead.
Did the individuals in question consent to the collection and use of their data? N/A.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?
N/A.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., data protection impact analysis) been conducted?
N/A.
Any other comments?
No.
B.6 PREPOCESTIN//CCEENNNICCLELLEEE
Was any preprocesing/deeaing/bbbee of the data done (e.g., discretization buck- eting, tokenization, part-of-speech tagging, SIFT feature extraction, removal of in- stances, processing of missing values)?
We firstly select videos based on pre-designed categories, and then clip the video based on visual-audio correlation and dynamic scores.
Was the "raw" data saved in addition to the preprocesea//ceenedddaaeeee data (e.g., to support unanticipated future uses)?
N/A.
Is the software that was used to preprocess/cdean/label the data available? We use the open-source models.
Any other comments? No.
B.7 USES
Has the dataset been used for any tasks already?
Yes. We have used the dataset to evaluate video question answering in real- world.
Is there repository that links to any or all papers systems that use the dataset?
No.
What (other) tasks could the dataset be used for?
It also can be used to evaluate the video understanding capability of VLMs.
Is there anything about the composition of the dataset or the way was collected and preprocssedd/ceenne//aae that might impact future uses?
No.
20

---

Published as a conference paper at ICLR 2026
Are there tasks for which the dataset should not be used?
N/A.
Any other comments?
No.
B.8 DISTRIBUTION
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
Yes, the dataset will be made publicly available.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? We host it on the webpage, GitHub, and Huggingface.
When will the dataset be distributed?
It's availale and open to the public now.
Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
We release our benchmark under CC BY-NC 4.0 license.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
Do any export controls or other regulatory restrictions apply to the dataset or to indi- vidual instances?
No.
Any other comments?
No.
B.9
MAINTENANCE
Who will be supportinghootingnnaiiiiiinngg the dataset? The authors will be supporinghhosinn/manttannnn the dataset.
How can the ovner/curatorr/maager of the dataset be contacted (e.g., email address)? No.
Is there an erratum?
Currently, we do not have erratum. We will update if we find errors.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
Yes. We will make announcements on GitHub there any update.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? N/A.
Will older versions of the dataset continue to be supprttadhottemmaintiiie?? Yes.
If others want to extend/augmmenthhiid on/contribute the dataset, there a mech- anism for them to do so?
post issues submit pull requests GitHub. We will review and verify contributions, and update the dataset if the contribution is useful.
Yes. Contributors
Any other comments?
No.
21


Reviews:
[{"review_id": "yboAMMPRLl", "reviewer": "Reviewer_ZaHB", "summary": "The paper introduces WorldSense, a new benchmark designed to evaluate how MLLMs understand and reason about real-world video-audio-text inputs. WorldSense integrates both audio and video to test true omni-modal understanding.\nThe dataset includes 1,662 synchronized videos across 8 domains and 67 categories, with 3,172 multiple-choice QA pairs spanning 26 different tasks. These questions require both audio and visual cues to answer correctly. The authors also evaluate various open-source and proprietary MLLMs, showing that even advanced models like Gemini 2.5 Pro only achieve 65.1% accuracy, exposing significant gaps in real-world multimodal reasoning.", "questions": "- How do you ensure that questions cannot be answered from text transcripts alone?\n- Will the benchmark be publicly released, and under what license?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A novel audio-video benchmark focused on omni-modal understanding.\n- Expert curated by annotators and manual QA design for better quality.\n- Paper is well-written.", "weaknesses": "- Although, the dataset is diverse, the distribution of question difficulties across categories or cognitive levels is not very clear. Some tasks might be more perception-heavy than reasoning-heavy, which could bias model comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5alQqqTm2t", "reviewer": "Reviewer_Fj9V", "summary": "This paper propose WorldSense, an omnimodal benchmark including video, audio, and text. WorldSense requires joint reasoning over synchronized visual and audio inputs to evaluate the ability of existing MLLMs. It contains 3,172 MC QA pairs, spanning various domains and subcategories. The paper evaluates open-source and proprietary MLLMs on WorldSense, showing that current MLLMs are having trouble in integrating audio and visual data effectively in realistic setting.", "questions": "1. How does WorldSense handle temporal reasoning—do questions depend on sequential context or only short clips? An ablation study on image / a few frames would elaborate this.\n2. This is beyond the scope of this paper. But is it possible to extend WorldSense to an open-ended or generative benchmark? This would make it more aligned with user-end usage.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Solid motivation. The paper focus on omnimodal reasoning, requiring the models to utilize visual and audio inputs together to answer the question. This designed correlations make it distinct from most existing MLLM benchmarks.\n2. High-quality human-reviewed annotations. Unlike recent benchmarks, WorldSense is reviewed and revised by human expert, instead of relying solely on LLMs. This is a guarantee for the quality of the benchmark. It is very rare these days.\n3. Comprehensive experiments. The paper conducted comprehensive experiments on existing models, including open-source ones and proprietary ones, providing a benchmarking foundation for future research.", "weaknesses": "1. Video caption is not a good representative for \"text\" modality. This makes the \"omni\" a little overclaim. Given real-world constraints, audio–video may already suffice for evaluating omnimodality, as seen in emerging “world models.”\n2. The question types are restricted in multiple-choice QA. This is common in existing benchmarks, but given the ability of MLLMs, free-form answers can yield deeper insights and align with user-end usage.\n3. The benchmark is currently focusing on perception and recognition. Given the lengths of the dataset (141.1s) and the requirement for multiple modalities, it may be possible to curate a subset for higher-level reasoning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lZgMmlJ2Qz", "reviewer": "Reviewer_zBjg", "summary": "This paper proposes WorldSense, a benchmark designed for omni-modal video understanding. Specifically, to answer questions in WorldSense, an MLLM’s response must rely on both video and audio information. Experiments on the proposed WorldSense benchmark reveal the limitations of current MLLMs in omni-modal reasoning.", "questions": "1. #82–83: There’s a typo ,“THe”.\n2. #129–130: Pay attention to the spacing between the image title and the main text. It currently looks a bit confusing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Requiring both video and audio modalities for accurate responses to each question in WorldSense facilitates a more comprehensive evaluation of current MLLMs in omni-modal reasoning.\n2. Experimentally, the performance drop of current video-audio MLLMs indicates that the fusion between modalities is ineffective or even detrimental.", "weaknesses": "1. While WorldSense emphasizes real-world omni-modal perception, understanding, and reasoning, the benchmark primarily consists of QA pairs. I believe that interactive question answering would be more practical for real-world scenarios. Moreover, isn’t the term \"omni-modal\" somewhat overstated, given that the benchmark only includes video, audio, and text modalities?\n2. Lack of analysis on why the fusion of open-source audio and video models failed. While I wouldn’t tend to reject the paper for this reason, providing such an analysis would offer the community deeper insights than merely presenting the conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bul2PCO4zy", "reviewer": "Reviewer_E5me", "summary": "This paper introduces WorldSense, a new benchmark for evaluating the omnimodal understanding capabilities of multimodal LLMs. The benchmark simultaneously incorporates visual, audio, and text inputs to assess a model's ability to comprehend and reason about real-world scenarios. The paper also presents a detailed evaluation of existing multimodal LLMs on the WorldSense benchmark.", "questions": "- About the evaluation settings.  In Line 316, the paper mentions that the models are tested \"following the recommended pre-processing procedures\". What are the specific test settings for each model (such as the frame rate for video frame extraction, the maximum number of extracted frames, etc.)? Could the different pre-processing settings between models lead to incomparable results?\n\n- The audio-visual LLMs evaluated in the paper seem a bit weak on WorldSense. Could the authors provide an analysis of recent powerful omni models like Qwen3-Omni and Video-SALMONN 2? This would establish a more robust and current baseline, and also help to verify if the significant challenges highlighted by the benchmark are persistent even for the latest generation of models.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The WorldSense Benchmark seems well annotated and comprehensively considers visual and auditory information, which is meaningful for evaluating a model's ability to understand video.\n\n- The experiments are sufficient and convincing.", "weaknesses": "- In terms of task design, although WorldSense includes a list of tasks, the paper does not clearly show the special design targeted at the integration of audio and visual information. To me, it seems many tasks either assess audio understanding independently or visual understanding independently, rather than evaluating the understanding of audio and video as a whole.\n\n-  There are already some benchmarks for evaluating audio-visual video understanding, such as AVUT [1] and DailyOmni [2]. The paper did not discuss or compare with them. What are the main differences between WorldSense and these benchmarks?\n\n[1] Yang et al, Audio-centric Video Understanding Benchmark without Text Shortcut, arXiv preprint arXiv:2503.19951\n\n[2] Zhou et al, Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities, arXiv preprint arXiv:2505.17862", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)