MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos
Authors: Xuehai He
Weixi Feng
Kaizhi Zheng
Yujie Lu
Wanrong Zhu
Jiachen Li
Yue Fan
Jianfeng Wang
Linjie Li
Zhengyuan Yang
Kevin Lin
William Yang Wang
Lijuan Wang
Xin Eric Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Video Understanding
Benchmark
Primary area: datasets and benchmarks
TL;DR: We build a multi-discipline multi-faceted multimodal video understanding benchmark towards world model evaluation.

Abstract: Multimodal Language Language Models (MLLMs) demonstrate the emerging abilities of "world models"---interpreting and reasoning about complex real-world dynamics. To assess these abilities, we posit videos are the ideal medium, as they encapsulate rich representations of real-world dynamics and causalities. To this end, we introduce MMWorld, a new benchmark for multi-discipline, multi-faceted multimodal video understanding. MMWorld distinguishes itself from previous video understanding benchmarks with two unique advantages: (1) multi-discipline, covering various disciplines that often require domain expertise for comprehensive understanding; (2) multi-faceted reasoning, including explanation, counterfactual thinking, future prediction, etc. MMWorld consists of a human-annotated dataset to evaluate MLLMs with questions about the whole videos and a synthetic dataset to analyze MLLMs within a single modality of perception. Together, MMWorld encompasses 1,910 videos across seven broad disciplines and 69 subdisciplines, complete with 6,627 question-answer pairs and associated captions. The evaluation includes 4 proprietary and 11 open-source MLLMs, which struggle on MMWorld (e.g., GPT-4o performs the best with only 62.5% accuracy), showing large room for improvement. Further ablation studies reveal other interesting findings such as models' different skill sets from humans. We hope MMWorld can serve as an essential step towards world model evaluation in videos.

Published as a conference paper at ICLR 2025
MMWORLD: TOWARDS MULTI-DISCIPLINE MULTI- FACETED VIDEO UNDERSTANDING EVALUATION
Xuchai Het, Weixi Feng", Kaizhi Zheng', Yujie Lu', Wanrong Jiachhn Jianfeng Wang'. Linjie Li", Zhengyuan Yang', Kevin Lin',
Jiachen Li,, Yue Fan?,
William Yang Wang", Lijuan Wang', Xin Eric Wang!
UUCSC UCSB, Microsoff
Currespondence: xhe89, (erili:wwnnnnesseso.
Frumes
Futura Prediction
Chack Chart
Basertean
Footbar
How SELEEEEE Trovsee moxm LowRt
Smatth Marageennt
Brawing bes Parrog
envnaement
Typer Explanation
Phussos
Unierna
Coeeierry
Business
Spons
Hoa does Derig srilee book
Soungg!
Science
Hellt
Truks
siyect Variueee
ccestorig 25 bebac
Type: Thinking
Moniaine
Cergratti
Asiinal Bevvnio
Mastamiaiioo Viggin segranciin
Engineering
Tecch &
Game
Man Medial
Disiia Menig
Aspinaty
lecger maness ERORT te
Type: Domain Experise
suen rooee
densedo
meeica
Aoee sar
ssng fupt
Clans
Figure 1: MMWorld covers seven broad disciplines and 69 subdisciplines, focusing on the evalua- tion of multi-faccted reasoning beyond perception (c.g., explanation, counterfactual thinking, future prediction, domain expertise). On the right are four video samples from the Science, Tech & Engi- neering, Embodied Tasks, and Game disciplines.
ABSTRACT
Multimodal Language Language Models (MLLMs) demonstrate the emerging abilities of "world "naielssiittttettiing and reasoning about complex real-world dynamics. To assess these abilities, posit videos the ideal medium, they encapsulate rich representations of real-world dynamics and causalities. To this end, introduce MMWorld, new benchmark for multi-discipline, multi- faceted multimodal video understanding. MMWorld distinguishes itself from pre- vious video understanding benchmarks with two unique advantages: (1) multi- discipline, covering various disciplines that often require domain expertise for comprehensive understanding; (2) multi-faceted reasoning. including explana- tion, counterfactual thinking, future prediction, etc. MMWorld consists of a human-annotated dataset to evaluate MLLMs with questions about the whole videos and synthetic dataset analyze MLLMs within single modality of perception. Together, MMWorld encompasses 1,910 videos across seven broad disciplines and 69 subdisciplines, complete with 6.627 question-answe pairs and associated captions. The evaluation includes proprietary and 11 open oounesource MILMM,, which struggle MMWorld (e.g., GPT-40 performs the best with only 62.5% accuracy), showing large room for improvement. Further ablation studies reveal other interesting findings such models' different skill sets from humans. We hope MMWorld can serve as an essential step towards world model evaluation in videos.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Foundation models, such as Large Language Models (LLMs) (OpenAl 2023c Touvron et al.
2023a Jiang et al. 2023 Anil et al. 2023) and Multimodal LLMs (MLLMs) Team et 2023
Lin ct al., 2023a Li ct al. [023c]Mass et al. 2024 Chen et al. 2023). have demonstrated remark-
able abilities in text and image domains, igniting debates about their potential pathways to Artificial General Intelligence (AGI). This raises a critical question: how well do these models understand the dynamics of the real world? Are they equipped with an inherent World Model (LeCun) [022] Chen
et al. 2024 Ha & Schmidhuber 2018 Xiang et al. (2024) that can understand and reason about the underlying principles and causalities of the dynamic, multimodal world?
Videos, with their rich, dynamic portrayal of the real world, ideally suited for evaluating the "world modeling" capabilities of MLLMs. Existing video understanding benchmarks (Li et a 2023d Ning et al.) 2023b) Pâträucean et al. 2023: Li et al., (023d), however, fall short in two key perspectives for such evaluations. First, as LeCun et al. [SSCum][2022 discussed, the world model should be able to (I) estimate missing information about the state of the world not provided by perception, and (2) predict plausible future states of the world. Evaluation of such capabilities re- quires multi-faceted reasoning beyond perception level, including explaining the video dynamics, counterfactual thinking of alternative consequences, and predicting future activities within videos. Moreover, the multi-discipline nature of the multimodal world necessitates a grasp of diverse funda- mental principles-raree expertise across a variety of disciplines imperative for thorough evaluation of a model's world understanding towards AGI [Morris et al. 2023 Yue et al. (023).
from physics and chemistry to engineering and business. Hence, domain
Therefore, we introduce MMWorld, a multi-discipline multi-faceted multimodal video understand- ing benchmark to comprehensively evaluate MLLMs' abilities in reasoning and interpreting real- world dynamics ! MMWorld encompasses a wide range of disciplines and presents multi-faceted reasoning challenges that demand combination of visual, auditory, and temporal understanding. consists of 1,910 videos that span seven common disciplines, including Art & Sports, Business, Science, Health & Medicinee Embodied Tasks, Tèch & Engineering, and Games, and 69 subdisci- plines (see Figure [ such as Robotics, Chemistry, Trading, and Agriculture, thereby fulfilling the objective of breadth in discipline coverage. The dataset includes a total of 1,559 question-answer pairs and video captions annotated and reviewed by humans, Mcanwhile, for multi-faceter rea- soning, MMWorld mainly contains seven kinds of questions focusing on explanation (explaining the phenomenon in videos), counterfactual thinking (answering what-if questions), future predic- tion (predicting future events), domain expertise (answering domain-speeiffe inquiries), temporal understanding (reasoning about temporal information). and etc. Four video examples with these questions from different disciplines are depicted in Figure [ To serve as a comprehensive bench- mark, MMWorld comprises two datasets: a human-annotaue the whole video and a synthetic dataset designed to analyze MLLMs' perception within single vi- sual or audio modalities. We evaluate 15 MLLMs that can handle videos image sequences MMWorld, including both open-source (e.g.. Video-LLaVA-7B (Lin et al., (023a)) and proprietary models (GPT-4o AOpenAl 2024) and Gemini (Team et al. (023))
dataset for evaluating MLLMs on
We summarized the contributions and key findings as follows:
We introduce MMWorld, a new benchmark designed to rigorously evaluate the capabilities of Multimodal Large Language Models (MLLMs) in world modeling through the realm of video understanding. MMWorld spans a broad spectrum of disciplines, featuring a rich array of question types for multi-facetee reasoning.
In addition the human-annotated dataset, we develop an automatic data collection pipeline, streamlining video content selection and question-answa generation, and con- struct a well-controlled synthetic dataset to analyze MLLMs within single visual or audio modalities.
Note that the term "world model" in MMWorld is broadened from its traditional use in reinforcement learn- ing to more generalized sense. MMWorld is not a sufficient testbed for world model evaluation. but we believe overcoming the unique challenges presented in MMWorld is essential and necessary towards comprehensive world modeling.
2

---

Published as a conference paper at ICLR 2025
Comparison between MMWorld and previous benchmarks for real-world video under- standing on variety of criteria. facti-faceted include Explanation (Explain Counterfactual Thinking (Counter.), Future Prediction (Future.) and Domain Expertise (Domain.) MM- World is the first multi-discipline and multitask video understanding benchmark that covers wider reasoning questions, and also included first-party data annotations.
Table 1:
Benchmmarks
Discipline
Multi-
Task
Explain.
Counter,
Reasoning
Puture.
Donain,
Pirst-Party
Annotation
MovieQA (Trasswi
TVQA ar A018
Activity 22 euret 2019b
MSVI>-QA Xu
MSRVTT-QA (A
2016
Sports- QA
Srai 2025
VilleX Walr A01
VALUE af X049
Video-Belic Ni al. 2023u
MVBench L
Perception ESTCIA masan
2023
VideoMME te 20.9
MMBench Viokor
2024
TempCompast
VILMA Resen ot dla 2014
VITATECS
NExT-QQA (Xnn
CVRR (Khataa ara 2024
Causal VI0UA
MMWorld (Ourk)
We observe that existing MLLMs still face substantial challenges posed by MMWorld. Even the best performer, GPT-4o, can only achieve 62.54% overall accuracy, and four MLLMs particularly trained on videos perform worse than random chance.
Although there stll clear gap between open-source and proprietary models, the open- source model Liteo-LLaVA-.. achieves the best Embodied Tasks. It outperforms GPT- 4V and Gemini Pro on Embodied Tasks by a large margin and performs similarly on Art & Sports, where spatiotemporal dynamics play more erucial role in video understanding. This is further validated with its leading results on Temporal Understanding question type.
our study comparing MI.LMs with average humans (non-experts), we notice some cor- relation between question difficulties as perceived by humans and MLLMs. However, MLLMs present different skill sets than humans that they can answer reasonable amount of difficult questions that humans completely fail but also struggle easy questions that humans excel at. This indicates different perception, cognition, and reasoning abilities between MLLMs and humans.
RELATED WORK
2.1 MULTIMODAL LARGE LANGUAGE MODELS (MLLMS)
Emerging MILMMs
Recent advancements in Large I Language Models (LL.Ms) OpenAl
2023a
Google 2023
Touvron et al. 2023a Chiang et al. 2023 Touvron et al. 2023b Bai al.
2023a
have paved the way for several multimodal counterparts in the vision-and languuu domain (Dai
al. 2023 Liu et al., 2023bla Li et al. 2023a Zhu al. 2023b Zheng et al. 2023 Bai al. 2023b), and recently released GPT-VV (OpenAl 2023b). followed by Gemini Vision family (Tem al. 2023). As LLMs have been applied world modeling and simulation (Wang al.) 2024a),
MLLMs now extend their capabilities beyond text and image inputs. Pretrained on large-scale, diverse datasets, these models are equipped with commonsense, domain-specific knowledge, and broad generalizability.
VideoChat (Li et al. 2023ch leverages the QFormer (Li ct al. 2023b) to map visual representa- tions to LLM (Chiang et al.
2023), and performs multi-stage training pipeline. Otter (Li et al.) 2023a) proposes to conduct instruction finetuning based on Openflamingo (Awadalla et al. 2023) PandaGPT (Su et al. 2023) employs the ImageBind (Han et al. 2023) as the backbone and fine- tunes it. The mPLUG-OW Ye et al., (2023) introduces an abstractor module to perform visual and language alignment. Videol.LaMA (Zhang et al. 2023a) introduces frame embedding layer and

---

Published as a conference paper at ICLR 2025
also leverages ImageBind to inject temporal and audio information into the LLM backend. Chat- UniVi (Jin et al. 2023) uses clustering to do feature fusion. LWM (Liu et al. 2024b) collects a large video and language dataset from public books and video datasets and trains world model that is capable of processing more than millions of tokens.
These MILMS demonstrate emerging abilities in multi-disciplinaryy world knowledge and excel multi-faceted reasoning tasks, such as inverse dynamic veedicion-ppredictinn intermediate steps between previous and next states, a crucial auxiliary task for next-state prediction (Devlin 2018 Lu et al. 2019; Paster et al. 2020) in real-world scenarios. In response to the emerging capabilities of MLLMs, we propose MMWorld evaluate their ability understand real-workd dynamics, underlying principles, and causalities, with the ultimate goal of achieving world modeling.
Benchmarking MLLMs
evaluaae MILMMs, there is flourishing of analysis (Liu et al. 2024a
Zhang et al.] 2023b Jiang et al., 2022; Lu et al.) 2024] Fan et al.] 2024 Cui et al. 2023 Guan aaal|[20000 [024]] Yu et 2023; Fu al. (023a) and the establishment of innovative benchmarks such as VisIB-Bench [Bitton et al. 2023 which evaluates models with real-world
cosructiio-followingg
ability given image inputs, MMMU Yue al., 2023) designed to access models on college-level
image-question pairs that span among different disciplines, and VIM (Lu et al. (002)) which chal- lenges the model's visual instruction following capability.
However, these recent analyses and benchmarks only cover the image input. Recently, video bench- marks such as Perception Test (Päträucean et al., 2023) is proposed to focus on perception and skills like memory and abstraction. However, it uses scenarios with few objects manipulated by a person, which limits the variety of contexts. In contrast, MMWorld operates in an open-domain scenario with diverse scenes; MVBench (Li et al. 2023d), TempCompass (Liu et al. 2024c) centers on temporal understanding, while MMWorld not only includes temporal reasoning but also eval- uates other multi- multi-faceted reasoning abilities such as counterfactual thinking and Noomain-speciinn expertise; EgoSchema Mangalam et al. (2023) focuses on natural human activity and behavior, but it does cover the broad range of disciplines that MMWorld does. MILMS that perfectly solve MMWorld would unlock the ability to perform multifaceted, multidisciplinary reasoning and the potential serve world model.
2.2 VIDEO UNDERSTANDING BENCHMARKS
Previous video benchmarks, as shown in Table I focus on video understanding tasks, including activity-focused on web videos (Yu al. 2019a), description-based question answering (Zeng al., 2017]. video completion (Fu et al. 2023b). and video infilling (Himakunthala et al. 2023). Recently, Video-Bench (Ning al. 2023b) introduces benchmark by collecting videos and annotations from multiple existing datasets. Mementos (Wang et al.) (2224b) builds a benchmark for MLLM reasoning for input image sequences. STAR (Wu et al. 2021) builds benchmark for situated reasoning in real-world videos. CLEVER (Yi et al. (2200) builds benchmark containing videos focusing on ob- jects with simple visual appearance. None of these benchmarks match the multi-discipline coverage that MMWorld provides. MMWorld, in contrast, presents new benchmark designed to encom- pass interdisciplinary coverage, task diversity, and multifaceted reasoning capabilities aopabilites-iinludiinn future prediction, counterfactual thinking, and more underpinned by original human annotations and integrated domain knowledge.
THE MMWORLD BENCHMARK
The MMWorld benchmark is built on three key design principles: multi-discipline coverage, multi- faceted reasoning, and temporal reasoning. It spans various disciplines that require domain expertise and incorporates diverse reasoning skills such explanation, counterfactual thinking, and future prediction. The benchmark consists of two parts: a human- annottted dataset and a synthetic dataset. The human-annotated dataset serves as the main testbed to evaluate MLLMs from multiple perspectives. The synthetic dataset is divided into two subsets, each designed to assess MLLMs' perception behavior based visual and audio inputs, respectively.
4

---

Published as a conference paper at ICLR 2025
Hou eary anirndt appert
WILE Fach BOD del The HAOOT
Torporal Undessarrding
Undersarding
Expalso
Ai Decatos hus no les louling
10.5%
What VOC stbdiies Moseiaai
Shat Tuppen FEe
balbn th blas rellint
11.%%
Futur
Eqlassioe
Q How EVB actianyy chonge For spining CONN??
video7 deddop coftee 52bl be palia which moke
dipped
Thanking
Figure 2: The questions in MMWorld are designed to evaluate seven primary understanding and reasoning abilities of models. Each question is annotated with all relevant categories. The figure showcases one example question for each reasoning category, based its main category.
3.1 MANUAL DATA COLLECTION
We collect videos from YouTube with the Creative Licence in seven disciplines: Art & Sports (18.5%), Business (12.0%), Science (20.4%). Health & Medicine (12.0%). Embodied Tasks (12.0%), Tech & Engineering (12.9%), and Game (12.2%). For Art & Sports, 29 videos are col- lected from the SportsQA dataset (Li et al. 2024). And for Embodied Tasks, 24 videos are sourced from IKEA Assembly (Ben-Shabat et al.] 2021). RT-1 (Brohan et al. 2022). and Ego4D (Grauman datasets to increase video diversity.
ct al.
2022
Our manual benchmark collection takes two stages. In the first stage, we conduct a detailed exami- nation of each of the seven primary disciplines to identify comprehensive range of subdisciplines for inclusion in our benchmark. Our selection of videos is driven by three key principles:
1. The first principle, multi-discipline coverage, emphasizes the requirement for domain knowl- videos that inherently demand an understanding of specialized content across vari- ous disciplines; The second principle, multi-faceted annotation, involves collecting videos that enable the creation of question-answer pairs from multiple perspectives to evaluate world model properties comprehensively: 3. The third principle, temporal information, prioritizes the inclu- sion of videos that provide meaningful content over time, as understanding temporal information crucial for grasping world dynamics. This allows models to engage temporal reasoning and answering questions in MMWorld requires implicit temporal reasoning, e.g., the model needs to understand temporal information explain "why does the robot need to do the step shown in the video". We also design "temporal understanding" question type to explicitly test models' ability to reason about temporal information (more examples can be found in Section
the Appendix).
During the second stage, our team began the task of annotating questions, answers, and options. All annotators were asked to carefully watch the collected vidcos and create questions with corre- sponding answers and options, ensuring that understanding the video content and applying temporal reasoning were necessary to determine the correct answers. We also ensured that the clarity, cor- rectness, and grammatical accuracy of the questions and answers were verified using GPT-4o, and that the questions could not be correctly answered without video input. We craft questions that primarily test seven aspects of multimodal video understanding also from the perspective of multi- faceted reasoning: 1) Explanation: Questions ask the model to elucidate the underlying logic or purpose within the video; 2) Counterfactual Thinking: Tests the model's ability to hypothesize and consider alternative outcomes; 3) Future Prediction: Aims to predict future events based on the current scenario, challenging the model's foresight; 4) Domain Expertise: Evaluates the model's depth of knowledge in specific fields, such as how to assemble coffee table; 5) Temporal Un- derstanding: Assesses the model's capability to reason about temporal sequences and dynamics; 6) Attribution Understanding: These questions focus on identifying cause-and-effect relationships within the video, including tasks like counting: 7) Procedure Understanding: Tests the model's 's abil- ity to comprehend and explain procedural tasks shown in the video. The detailed distribution and examples are shown in Figure 2 For quality control, we ensure each annotation is cross- cross-checked by at least two professional researchers to ensure accuracy and prevent annotation errors.
5

---

Published as a conference paper at ICLR 2025
ASR Module
Audio-based ed QA generaatio
Van sddice
Vileo Trensingh
Video Filering
C. wyy the angonss: at the begeniig
A: mply preciica
You Tube
BM
Alto commicnnall petting...
Query
generstion
(Nd Paness
CA generation
Video Summal Dasion
C. coult democataded
veply
Erboodied
somiher of Alas?
tasks
A amiiiing
Figure 3: Schematic diagram of the synthetic data generation pipeline in MMWorld. It starts with generating subdiscipline-specine queries, followed by video retrieval from YouTube-8M (Abu-EI-) Haija et al.) (016) and YouTube. Keyframes are extracted for visual-based QA generation, and videos are transcribed using an ASR module for audio-based QA generation.
3.2 AUTOMATED DATA COLLECTION
Understanding real-world dynamics requires models process both audio and visual modalities. To evaluate MLLMs' perception abilities in these modalities, we designed an automated data collection pipeline. This pipeline collects targeted videos and generates QA pairs based on either audio or visual information, ensuring the model's 's capabilities are assessed independently for each modality. By using information from a single modality to generate QA pairs, our pipeline ensures that the synthetic data remains unbiased regarding input modality.
The synthetic data generation pipeline is illustrated in Figure 3 to gather videos with Creative Commons licenses from YouTube and the extensive YouTube-8MM dataset (Abu-El-Haija et al., 2016). This method ensures a diverse and comprehensive collection of video data, which is important for the robust evaluation of multimodal video understanding models.
We employ systematic approach
Video Collection and Processing We start with the video Query Generator. We start with the same seven disciplines as the manually collected dataset. For each discipline, set of subdisciplines is de- fined to encapsulate a wide spectrum of topics, ensuring a diverse and comprehensive dataset. Once the queries are generated, the Video Mapping and Filtering step is initiated. We perform mapping of videos to YouTube-8M and online videos, constrained by a strict time limit of two minutes per query, keeping only the most pertinent videos that satisfy the predefined criteria. Simultaneously, the works in conjunction with the video transcripts to extract key terms and concepts. This iterative process refines the search parameters and enhances the semantic richness of the dataset by identify- ing and encoding the salient themes present in the videos. The Video Summarization module utilizes Query-focused video summarization techniques based on Katna" and UniVTG (Lin al. 2023b). This module selects ten representative frames from each video, distilling the essence of the content while preserving the narrative context. This summarization facilitates efficient storage and quicker processing times, which are crucial for large-scale analysis.
QA Generation
The final stage our pipeline is the QA / Caattion Generation module, where we leverage the capabilities of GPT-4V to generate accurate and contextually relevant questions and answers, as well as captions, based on the video frames and transcripts. This step not only provides rich annotations for each video but also equips the dataset with multimodal dimension that supports various downstream tasks such as video QA, captioning, and more.
Quality of the Synthetic Dataset Human evaluators were engaged to ascertain the reasonableness of automatically generated questions and answers, ensuring that the synthetic dataset maintains a high standard of quality and relevance. The findings from this human evaluation phase detailed in Section D of the Appendix, offering insights into the dataset's efficacy and the realism of its constructed queries and responses.
Finally, the statistics of automated curated data, which used for the ablation study, are shown in Table 2 The laxonomy of our dataset is shown in Figure [ We note that only portion of the subdisciplines are shown due to space concerns. Please refer to the Appendix for full information.
http:://giittbbb.oo/////eeee///

---

Published as a conference paper at ICLR 2025
Table 2: Key Statistics of the MMWorld Benchmark. The main subset the subset. Synthetic Subset contains generated QA pairs focused exclusively on the audio content, while Synthetic Subset II contains QA pairs focused exclusively on the visual content of the video.
Statistics
Main Subset
Synthetic I
Synthetic II
#DDisipline//Suudddiiiinne
7/61
7/51
7/54
#Videos
417
746
747
#QA pairs
1,559
2,969
2,099
Avg Video Lengths (s)
102.3
103.4
115.8
Avg #Questions per Video
4.05
3,98
2.81
Avg #Options
3.90
4.00
4.00
Avg Question Length
11.39
15.12
17.56
Avg Option Length
7.27
6.01
5.19
Avg Answer Length
6.42
6.71
5.67
Avg Caption Length
27.00
71.87
82.33
Unique Words in Questions
1,913
2,528
2,279
# Unique Words in Answers
2,292
2,981
2,657
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
In our study, we compare MILLM's performance on the MMWorld benehmark, including GPT-
4o (OpenAl, 22244. GPT-4V (OpenAl 2023b), Gemini Pro (Team et al. 2023). Claude-3.5-
Sonnet Anthropic (2024). Video-Chat (Li et al., 2023c).
Video-ChatGPP Maaz et al.) 2024).
Video-LLaMA (Zhang et al. 2023a). Video- Laec--LLatee VA (Lin al., 2023a). ChatUnivi (Jin et al. 2023). mPLUG-OW Ye et al. 2023), Otter (Li et al. 2023a), Image -LM (Han et al. 2023),
al. 2023). For proprietary model, we adhere to the default settings provided by their official APIs. They both take ten image frames extracted from the video content the input. The Gemini Pro is set to process visual input and configured with safety settings to filter range of harmful content. The configuration thresholds are set to 'BLOCK NONE', For PandaGPT, 'temperature' to 0.5. For VideoChat, we set 'max_ frames' to 100. For X-Instruct-BLIP. the model implemented using four image frames. We use GPT-4-32K the judge for judging whether the model answer is correct when it can not mapped to the option letter using the rule-based method. For others, we all use the default setting. All inferences are run on a NVIDIA A6000 workstation. The detailed implementation is given in the Appendix.
PandaGPT (Su et al. 2023), LWM (Liu et al. 2024b), and X-Instruct-BLIP (Panagopoulou
set 'top p' 0.7 and
4.2 EVALUATION STRATEGY
Our dataset contains multiple-choice questions and captions corresponding each video, supporting tasks such as video question answering and video captioning. In our evaluation setup, we focus on video question answering by measuring a model's accuracy in selecting the correct answer from the provided options. This method is straightforward to quantify and provides objective assessment. However, one challenge is reliably mapping the model's predictions one of the predefined choices.
To address this, we employ two mapping strategies. We employ two mapping strategies. The first method employs automated scripts parse the models' predictions and compare the parsed results with the ground truth, similar to the approach used in (Yue et al. 2023); The second method involves models freely generating answers, which are then evaluated by GPT-4. Given the question, correct answer, and model's prediction, GPT-4 returns True False judgment. This approach is based on
recent works in model evaluation (Maaz et al. 2024 Hsu et al. 2023 Hackl et al. 2023 Liu et al. 2023ch.
We validated the second GPT-4- evaluation approach with human evaluators, showing an error rate of only 4.76% across 189 examples, demonstrating its reliability evaluator. Detailed results for human evaluation and both evaluation strategies are provided Appendix. All results presented in the main paper based on the second evaluation approach.

---

Published as a conference paper at ICLR 2025
Table 3: MLLM accuracy across diverse disciplines (averaging over three runs). GPT-4V and Gem- ini Pro lead at most disciplines and achieve the best overall accuracy. The best open-source model Video-LLaVA outperforms them on Embodied Tasks and perform similarly on Art & Sports. All data are annotated by humans.
Model
Art& Sports
Business
Science
Health& Medlcine
Embodied Tasks
Tech& Engineering
Game
Average
Random Choice
25.03
25.09
26.44
25,00
26.48
30.92
25.23
26.99
Proprictury MLLMs
GPT-Ho OpenAl 2024
47.87 1147
91.14
cam
73.78
1iss
83.33
62.94
:34
75.53 =261
80,32 1230
62.54 19.79
Clande 1 Sonnet Amihropic 2024
54.58
1045
63.87
59.85
aM2 54,51
30.99
58.87 saa
59.44 1008
54.54
64:0
GPT-4V OpenA| 21230
36.17
$1.59
66.52
7366
58.48
27
61.35
73.49
52.30
Gemini Pio
[eand & Ml. 2023
37.12
ases
76.69
2ate
62.81 1155
76.74
1120
43.59
69.86 =2m
66.27
1240
51.02
11.35
Open-source MLLMs
Video-LLaVA-71B al. 202.3a
35.91 1e0
51,28
e03 56.30 s0. 32. 62.64 soo
63.17 21:11
2:10
49.00 2/:
14.9
Video-Chat-7k
(L ml 203.30
39.53 1035 51.05
gire
30.81
1=:1 46. 18
gns
40:56
=::1
39.36 s/1
44.98 2015
40. 11 =*:
ChatInivi-7B c11
24.47
60.84
52.00
61.11
46
=>>
56.74
52.61
39.47
mPLUU--Ww-778 1S50 70733
29.16 21x: 64. 14.10
2:
47:41
112h 60.07 1:3 23.78 23:7
41.84 231
62.25 2m1
11:50
Yidco-ChatGiPT-755
Mili 21.
2024)
2024
26.84 1027 39.16
sue
36:45
1131 53.12
36,60
:12
41.49 $1.4
36.55 1227
33.27
13777
PandGGPT-7B (Su et 11.
25.33
42.66
39.41
38.54
35.43
41.84
40.16
32.48
ImageRind-I.1 Hamrer er
11233
24.82 20:5
42.66
2ns 32.15
1:11 30.21 g/
46.85 2114
41.49 2::0
41.37 2ne
31.75 2014
Cinstract-BIIIP.. Pananppannioi cl
2023
21.08 1627 15.85
22.52
2:1 28.47
18.41
22.34
gaxt
26.10 gum
21.36 2215
LWM-IM.JAX (L
12.04
17.48
15.41
aam 20.49
ams 25.87
21.
231
11.65
15.39
Ouar-7B (Liet al 20258
17.12 11.7
18.65
sam
9.33
1836
6.94
1008
13.29
15.96
21.44
15.26 1057
14.99
18.77
Video LI 1... 13 Zhang al.
2023a
6.15 som
21.21
22.22
31,25
15.38 2014
19.15
81.24
24.90
6:0 14.03 54:09
Table 4: Results of different MLLMs on
reasoning. All data are annotated by humans.
Model
Explanation
Counterfactual Thinking Proprictary MLLMss
Future Prediction
Domain Expertise
Attribation Understanding
Temporal Understanding
GPT-4o OpenAAl
2024
56.68
75.88
82.48
69.05
65.10
40,90
GPT-4V OppnAi 2175h
44,90 2417
50 2010
78.59 2:
61.00
2017
59.61
27.17 2m
Soudo-333--0mmm Anpnropic
2024
$1.91 10:2
62.75 1806
71.78 1840
66.79 sa4e
40,00 18:55
25.77 1o44
Gemini Pro TTemm
48.58
65.49
65,45
53.87
43,92
24.65
Open- tource MLLMMs
Vidco-LLaVA (Lin al. 2023g
26.46.. 18:11
42.55 1005
64.96 11:1
47. 87.86 :3.5
36.86 11:9
34.45 21.39
41.66
43.73
45.74
40.95
30.59
25.77
Chhoo-hh
32.13 20.0
$1.0
33.69 2:/
2010
241s
ImngeBind-11M--76 Hai 21
20155
29.51 10:77
26,86 20:55
50.61 24:00
33.93 2017
34,90 21:4
19:89 gnst
PundaGiPT-78 (Su et Ai
2024
29.55
37.45
46,47 2:m
33.93
26.27
28.01 200
ChatUnivi-78
33.91
45.95
1BM
22.97 1x01
Video-LLaMA-21113 Ziangg at.l.
2023g
10.55 10:29
23.92 1007
25.30 s111
16.31 41/0
8.63 +115
6.16 11m0
X-Instner-Bl IP-77 (Phaponiiiio mi 2023
23.05
15,29
27.25 1414
21.07
24.31 2011
11.20 g00
LWM-IM-JAX (La
11.62
14:34
17.98 29:3
Otter-BB eT 20233
16.91 20M
10.98 2014
15.82 2m
13:10 2010
17.65 2010
9.52 2m
aPLUG-ORR-78
To 2023
35.20 11:7
49.61 1121
55.47 11.55
47.74 11.7
24.71 12:0
20.17 1009
4.3 MAIN EVALUATION RESULTS ON HUMAN-ANNOTATED DATA
We show in Table 3 the main evaluation results of different MLLMs. Among these, GPT-4o emerges as the top performer, followed by Sauude-3.5-Somne Video-LLaVA also demonstrates strong re- sults, primarily due to the extensive training data which consists of 558K LAION-CCSBB image- text pairs and 702K video-text pairs from WebVid (Bain et al.] 202T). Its superior performance may also be attributed to the adoption of CLIP ViT-L/14 trained in LanguageBind (Lin et al. (023a) as its vision model and the inclusion of a large volume of image-video-tex pairings within the train- ing data. On the other hand, models like Otter and LWM perform poorly across most disciplines, possibly due to their weaker backbone and architecture used, Otter uses the LLaMA-7B language encoder and CLIP ViT-L/14 vision encoder, both of which are frozen, with only the Perceiver resampler (Awadalla et al. (023) module fine-tuned, which may lead to the lower performance. Additionally, four MI.MM perform even worse than random, highlighting the challenging nature of MMWorld.
Study on Multi-faceted Reasoning Table 4 illustrates the multi-faceted reasoning performance of each MLLM. GPT-40 emerges the strongest model across all facets. Notably, in temporal understanding the open-sourced Video-LLaVA outperforms all other models except GPT-4o, likely due to its extensive training on high temporal resolution video data, enhancing its spatio-temporal reasoning abilities. This is further reflected in its high scores on Embodied Tasks (the best) and Art & Sports, both of which involve dense spatio-temporal information, as shown in Table 3
8

---

Published as a conference paper at ICLR 2025
Business
92
85
84
80
ChatUntel
90
0.7
Embodied ed Tasks
60
87
80
0.6
Game
89
70
0.5
LAN
ImageBind LUM
& Mediaiee
Health
71
78
77
64
40
0.4
MPLUG-OW!
Octer
Science
83
71
90
PendeGrP
0.2-
Video Chat
Sports
Arts
26
40
0.1
Video
Engineering
Tech
60
59
61
69
30
Fasy
Medium
Hard
Expert
KimstuuciBLP
Easy
Medium
Hard
Expert
(a) Accuracy of MLLMs at difficulty levels for average (b) GPT-4V results by disciplines at different difficulty humans.
levels for average humans.
Figure 4: Model performance at different difficulty levels for average humans. Average human difficulty levels are defined by 3 turkers' performance per question: Easy (3/3 correct answers), medium (2/3 correct), hard (1/3 correct), and expert (0/3 correct).
Study on MLLM Performance at Different Difficulty Levels for Average Humans
Figure 4a indicate some correlation between the difficulty levels as perceived by humans and the per- formance of MLLMs. The difficulty levels are defined based on the average human performance. MLLMs generally follow trend where accuracy decreases as the difficulty level increases, which aligns with human performance patterns. However, the correlation is not perfect, suggesting that while models and humans share some common ground in understanding question difficulty, there are also notable differences their capabilities. The data reveals that MILMs exhibit different skill sets compared to humans. As highlighted in Figure 4b] models like GPT-4V correctly answer expert-level questions that humans often get wrong, particularly in disciplines such as Business and Health & Medicine, where humans often struggle, yet they sometimes falter on easier questions, likely due to the lack of contextual understanding. Notably. discrepancies in disciplines like Art & Sports and Tech & Engineering highlight areas where MLLMs performance does not align with hu- man results, suggesting different perception, cognition, and reasoning abilities in handling abstract concepts. These differences suggest that MI.LMs can complement human capabilities, offering po- tential for enhanced task performance by combining the data-driven insights of models with human intuition and contextual knowledge.
Error Analysis To gain deeper insights into the limitations of current open-sourced MLLMs and provide guidance for developing next-generation models, we prompted models to explain their reasoning, particularly when errors occurred. We grouped and identified common patterns into seven distinct categories. We conducted a comparative test by posing the error-inducing questions for GPT-4V other MILMs, GPT-4V used representative model due its strong performance and its ability to highlight errors common across MILMSs.
Our analysis revealed that Video-LLaVA exhibited the lowest frequencies among open-source MLLMs Its superior performance, particularly in reducing Visual Perception Errors (PE), Hallu- cination Errors (HE), and Reasoning Errors (RE), also be linked to its of the CLIP ViT- L/14 model in LanguageBind (Zhu et al.) 2023a). In contrast, mPLUG-Owl showed higher rates of Visual Perception Errors, possibly due to its reliance on weaker video embedder architectures. Furthermore, VideoChat outperformed Video-LLaMA due to its GMHRA (Li et al. (202c)) module for temporal aggregation, demonstrating the importance of effective temporal aggregation in reduc- ing errors. Common trends across all models included frequent hallucination errors and lack of domain-specific knowledge, highlighting the need for accurate, noise-free training data and sug- gesting that techniques like Reinforcement Learning from Human Feedback (RLHF) (Ouyang al.) 2022) could help mitigate these issues. While current MILMs demonstrate strong multi muuli-disciplinary world knowledge, they could benefit from enhanced domain- speciffc expertise, potentially through eetrieval-based methods. Detailed gualitative examples and further analysis are provided the Ap- pendix.

---

Published as a conference paper at ICLR 2025
14
frequeeccy
AUE
VPF
RE
LDK
Videttaal
Mideo Chatoey
imspedrns LLM
Fandsort
Chartimin
Vndeosiaate
Kinstrucs suu
UWM
Othe
Histo
Figure 5: The frequency of different error types across various MLLMs. For each error type, 100 examples were evaluated. Error types are abbreviated as follows: QUE (Question Understanding Error), AUE (Audio Understanding Error), VPE (Visual Perception Error), HE (Hallucination Er- ror), RE (Reasoning Error), LDK (Lack of Domain Knowledge), and RA (Reject to Answer).
Table 5: Performance on Synthetic Subset I (Audio) and II (Visual). Synthetic Subset I contains QAs based solely on the audio content, while Synthetic Subset II focuses exclusively on the visual content of the video. We evaluated four MLLMs processing both audio and visual inputs along with Gemini Pro (for the audio setting, only providing the question).
Madel
Audio
Sportp
Vissal
Asdioo
Vicual
Audia
Sciencz
Vesaa
flcatheMaaastre Vassl
Audio
Emboatia Tade:
Azdio
Vosel
Ardio
Tishiklingruztnng
Visua
Audio
Gam::
Visual
Audio
Vaial
Chicse
36,47
9215
-4336
99'3
+12X)
"35.01
-9296
Challsi Ingees
10.65
63:23
93.19
52.88
18
$4.59
34.76
90.89
20.15
42.61
34.17
46.41
29.03
45:46
31.82
43.44
Vitamllama PAREE S15 10237
10.15
50.23
38.18
15.17
51
31.34
10.00
32.74
33.13
9003
11:18
30.55
20:49
21:30
10.88
90.43
Oner [iet J000
14:22
16.22
15:77
14:24
16.12
17:00
19.2
13:19
10.96
12.50
15.63
12.43
6.65
10.44
12.95
13:41
29:43
77.36
31.53
23.57
29.96
65.81
4.4 STUDY ON MODALITY OF PERCEPTION ON SYNTHETIC DATA
We conducted ablation studies evaluate how well MILMS can perceive the world when limited to a single modality (audio or visual) using the synthetic dataset of MMWorld. In these experi- ments, we isolated scenarios where only one modality-eile ble the results, which assess the models' ability to interpret spoken language, background noises, and other audio elements without visual context, as well as their visual perception without any audio input. For the visual perception test, Gemini Pro performed the best, demonstrating its strong ability to process visual information. Interestingly, Video-Chat exhibited better audio per- ception than ChatUnivi, despite its poorer visual perception. This may be attributed to its use of the Whisper (Radford et al. 2022) speech recognition model. It also explains that in Table 3 Video- Chat outperforms ChatUnivi in the Art & Sports discipline, which requires greater understanding of music, voice, and background audio. However, in other disciplines such as Science and Health & Medicine, Video-Chat's performance is significantly worse
audio visual-was available. Ta-
5 CONCLUSION
Our MMWorld Benchmark represents a significant step forward in the quest for advanced multi- modal language models capable of understanding complex video content. By presenting diverse array of videos across seven disciplines, accompanied by questions that challenge models demon- strate explanation, counterfactual thinking, future prediction, and domain expertise, we have created a rigorous testing ground for the next generation AI. While using LLMs for data generation can introduce hallucination issues, these challenges manageable and are commonly addressed (Wang et al. 2024c Shen et al. 2023). Another potential risk is the misuse of MLLMS for surveillance or privacy invasion. The ability of models to understand video content and perform reasoning could be exploited to monitor individuals without their consent, leading to serious ethical and legal concerns regarding privacy.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sami Abu-El-Haija, Nisarg Kothari, Joonscok Lec, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaynarrasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint drXiv:16090086755 2016.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy Meier-Hellsterm, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo- pher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa De- hghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag. Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycherihh, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Mar- cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yun- han Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang. Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Anthropic. Introducing the next generation of Claude. news/claude 3-family Ly 2024. Accessed: 2024-07-29.
fttts::/www.anthiie
com/
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint UrXiv:2308.01399 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vasiunnl model with versatile abilities. arXiv preprint arXiv:2308. wtf.:2338..16666, 12966, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond, 2023b.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Yizhak Ben- Sen-Shabat, Xin Yu, Fatemeh Salch, Dylan Campbell, Cristian Rodriguez- Hong- dong Li, and Stephen Gould. The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pp. 847-859, 2021.
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gard- ner, Rohan Taori, and Ludwig Schimdt. Visit-benchi A benchmark for vision- lsngua-l instruction following inspired by real-world use. arXiv preprint arXiv:2308. 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-worke control scale. arXiv preprint arXiv:22/2.06811 2022.
11

---

Published as a conference paper at ICLR 2025
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Kr- ishnamoorthi, Vikas Chandra, Yunyang Xiong. and Mohamed Elhoseiny. Minigpt-v2: large lan- guage model unified interface for vision-language multi-task learning, 2023.
William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision lasiun-language models provide promptable representations for reinforcement learning. arXiv preprint arXiv:2402.0200. 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL hetps:
/ //lmsys.org/bbll...... http:///////oo//////////////////
Chenhang Cui, Yiyang Zhou, Xinyu Yang. Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:231 03287, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.00 06500, 2023.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: (010:0v8100000000 2018.
Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chae Kuo, Xinze Guan, and Xin Eric Wang. Muffin or chihuabua? challenging large vision-language models with multipanel vqa, 2024.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-vidde A long-form multi-shot benchmark for holistic video understanding. arXiv preprint ar arXiv:2406. 14515, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 01223300..3399 13394, 2023a.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang. Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi modal lims in video analysis. arXiv preprint arXiv:2405.21073 2024.
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng-Yane Fu, Jong-Chyi Su, William Yang Wang, and Sean Bell. Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Google. Bard chat based ai tool from google, powered by palm 2. https://bard.ooogle.com 2023.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 18995-19012, 2022.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision- language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 1803. 10122, 2018.
Veronika Hackl, Alexandra Elena Müller, Michael Granitzer, and Maximilian Sailer. Is gpt-4 reliable rater? evaluating consistency in gpt-4 text ratings. arXiv preprint arXiv:2308.02573 2023.
12

---

Published as a conference paper at ICLR 2025
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-lin Multi-modality instruction tuning. arXiv preprint 0rXiv:2300.03905. 2023.
Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, and William Yang Wang. Let's think frame by frame with vip: A video infilling and prediction dataset for evaluating video chain-of-thought, 2023.
Ting- Yao Hsu, Chich- Yang Huang, Ryan Rossi, Sungchul Kim. C Lee Giles, and Ting-Hao K Huang. Gpt-4 as an effective zero-shot evaluator for scientific figure captions. arXiv preprint arXiv:2310. 15405, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Kenan Jiang, Xuehai He, Ruize Xu, and Xin Eric Wang. Comclip: Training- free compositional image and text matching. arXiv preprint arXiv:2211.38554 2022.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified vi- sual representation empowers large language models with image and video understanding, arXiv preprint arXiv:2311.08044 2023.
Ilker Kesen, Andrea Pedrotti, Mustafa Dogan, Michele Cafagna, Emre Can Acikgoz, Letitia Par- calabescu, Iacer Calixto, Anette Frank, Albert Gatt, Aykut Erdem, al. Vilma: A zero-shot benchmark for linguistic and temporal grounding in video-language models. arXiv preprint arXiv:2311.07022 2023.
Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Jameel Hassan, Muzammal Naseer, Fed- erico Tombari, Fahad Shahbaz Khan, and Salman Khan. Complex video reasoning and robustness evaluation suite for Imme. arXiv preprint arXiv:2405.036690 2024.
Yann LeCun. path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1), 2022.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint 1rXiv:/809.011 2018.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726 2023a.
Haopeng Li, Andong Deng, Qiuhong Ke, Jun Liu, Hossein Rahmani, Yulan Guo, Bernt Schiele, and Chen Chen. Sports-qa: A large-scale video question answering benchmark for complex and professional sports. arXiv preprint arXiv:2401.0/505. 2024.
Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12500 2023b.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint 2xXiv23355063555 2023c.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi molti-modal video under- standing benchmark. arXiv preprint arXiv: 231 I. 17005, 2023d.
13

---

Published as a conference paper at ICLR 2025
Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang. William Yang Wang, et al. Value: A multi-task benchmark for video-and- language understanding evaluation. arXiv preprint arXiv:2106004632, 2021.
Shicheng Li, Lei Li. Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. arXiv preprint arXiv:2311. 2023e.
Bin Lin, Bin Zhu, Yang Ye. Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.00122 2023a.
Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jin- peng Wang. Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 2794-2804, 2023b.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In Proceedings of the International Conference on Learning Representations, 2024a.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:240..08266 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844 2023b.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint ar3iv:2303. 16634, 2023c.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXiv preprint DeXiv:2403.00411 2024c.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for assionandd-lnguagg tasks. Advances in neural information processing systems, 32, 2019.
Yujie Lu, Xiujun Li, William Yang Wang, and Yejin Choi. Vim: Probing multimodal large language models for visual embedded instruction following, 2023.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Wang, Yejin Choi, and Yuchen Lin. Wild- vision arena: Benchmarking multimodal 1lms the wild, February 2024. URL https:
//huggin www.iiigaacc...cc cot.ceeccccee www.cccppppppppccotooooo Wi
arena/
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. Advances in Neural Information Process- ing Systems, 36:46212-444 1-244442444444 2023.
Meredith Ringel Morris, Jascha Sohl-dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Alek- sandra Faust, Clement Farabet, and Shane Legg. Levels of agi: Operationalizing progress on the path to agi. arXiv preprint arXiv:2311.02446 2023.
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-basse large language models. arXiv preprint arXiv:2311. 2023a.
14

---

Published as a conference paper at ICLR 2025
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-basse large language models. arXiv preprint arXiv:2311.16100 2023b.
OpenAI. Gpt-4: Technical report. arXiv preprint
2023a.
OpenAI. Gpt-4v(ision) system card. http://wwenn.ccomm/errrr/ggggsss 2023b.
OpenAl. Gpt-4 technical report, 2023c.
OpenAl. Hello gpt-4o. 2024-07-29.
40/| 2024. Accessed:
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Artemis Panagopoulou, Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to Ilms and emergent cross-modal reasoning. arXiv preprint arXiv:2311. 2023.
Keiran Paster, Sheila A Mcllraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02419, 2020.
Viorica Păträucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang. Carl Do- ersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimezak, Raphael Koster, Junlin Zhang. Stephanic Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multi- modal video models. In Advances in Neural Information Processing Systems, 2023. URL net forum 11=HYEGXF
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, C. Mcl.eave, and I. Sutskever. Ro- bust speech recognition via large-scale weak supervision. International Conference on Machine Learning, 2022. doi: 00448550/rrxiv2221..000
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in mn-the-wild jailbreak prompts on large language models. arXiv preprint arXiv: 2308.03825, 2023.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang. and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305. 16355, 2023.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4631-4640, 2016.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. /8005 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 13374 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2XXi:2307700928 2023b.
Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen. Can language models serve as text-based world simulators? arXiv preprint arXiv:2406.
4664068685 06485, 2024a.
15

---

Published as a conference paper at ICLR 2025
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for aido-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4581-4599 2019.
Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jachong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for mul- timodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529 2024b.
Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: Evaluating safeguards in LLMs. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 896-911, St. Julian's 's, Malta, March 2024c. Association for Computational Linguistics. URL https://aclanthool 2024. 1024.findingg win.iiinnnnnnal..... 61
org/
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Jiannan Xiang. Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Pandora: Towards general world model with natural language actions and video states. 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pp. 9777-9786, 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Pro- ceedings of the 25th ACM international conference on Multimedia, pp.
2017.
Jun Xu, Tao Mei. Ting Yao, and Yong Rui. Msr-vtt:
large video descrip- tion dataset for bridging video and language. In IEEE International Confer- ence on Computer Vision and Pattern Recognition (CVPR). IEEE International Conference Computer Vision and Pattern Recognition (CVPR), June 2016. URL
msr vLt ott.::/:wwa..aaa...ae...oaaaaaa.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint 2rX44:2304.14178, 2023.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: collision events for video representation and reasoning. ICLR, 2020.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 2023.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In AAAI, pp. 9127- 9134, 2019a.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.165//1 2023.
16

---

Published as a conference paper at ICLR 2025
Kuo-Hao Zeng, Tseng-Hung Chen, Ching Yao Chuang. Yuan- Liao, Juan Carlos Niebles, and Min Sun. Leveraging video descriptions to learn video question answering. Proceedings of the AAAI Conference Artificial Intelligence, 31(1). Feb. 2017. doi: 10. 10.1609/ani.v3ll...11..
BtR.hhtts::///33 www..aa.aa...... http:////ia....///a/aaaaaaaaaaaa
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding, arXiv preprint wrXiv:23306028858 2023a.
Xinlu Zhang. Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v(ision) as generalist evaluator for vision- language tasks, 2023b.
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-ane aaiio-and-language genera- tion via generative vokens. arXiv preprint arXiv:2310.02239, 2023.
Bin Zhu, Bin Lin, Munan Ning. Yang Yan. Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n- modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023a.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023b.
Minigpt-4: En-
17

---

Published as a conference paper at ICLR 2025
Ethics Statement In line with the ICLR Code of Ethics, acknowledge responsibility to adhere to ethical principles throughout the entirety of our research. Our work does not involve hu- man subjects, and the datasets we used are available in the submitted supplementary material and do not raise any concerns regarding privacy or security issues. The evaluation of models in this paper focuses publicly available multimodal larger language models, and sensitive per- sonally identifiable information was involved in this process. While our work benchmarks existing multimodal large language models via multi-discipline, multi-faceted world model evaluation, recognize the potential risks of unintended bias and faimess issues in these models, which may have inherited biases from their training data. We encourage future research address these concerns in the development of more inclusive and fair models. There are no conflicts of interest or sponsorship influencing this research, and our work fully complies with legal and ethical standards.
Reproducibility Statement To ensure the reproducibilityy of work, we provide extensive de- tails on our methodology. datasets, and evaluation setup in the main paper and the Appendix. The datasets used are available in the supplementary material, and their collection and annotation steps are described in Section 3 of the paper. We also provide detailed descriptions of the experimen- tal setup, including hyperparameters, model architectures, in the main paper and Appendix. All evaluation protocols and metrics are explained to facilitate replication of our results.
A OVERVIEW OF THE APPENDIX
This Appendix organized as follows:
Section
contains additional experimental results;
Section
contains the implementation details;
Section
D
contains the settings and results from human evaluations;
Section
contains the error analysis;
Section
contains the data examples from MMWorld;
Section
contains additional data statistics of MMWorld;
B ADDITIONAL RESULTS
B. RESULTS ACROSS DIFFERENT SEED FOR EACH MODEL
In Table we show detailed results using three different seeds for each evaluated models.
B.2 RESULTS FROM AMAZON TURKERS
Table presents the evaluation results from three sets of Amazon Turkers across various disciplines. The results indicate that there is slightly variability in performance across different human evalua- tors.
B.3 RESULTS FOR THE TWO DIFFERENT EVALUATION STRATEGIES
In Table 8 we give additional evaluation results for different MILMSs evaluated in this paper. For closed-source models, the evaluation pipeline is the one used in the main paper, which involves utilizing GPT-4V as a judger. The process consists of presenting GPT-4V with the question, corresponding answer generated by the baseline model, and the set of possible options. GPT-4V then assesses whether the genee--gnnerttdd answer is accurate within the given context; Another is open-ended generation where employ two-step methodology. We first prompt each model to do open-ended generation. Subsequently, we prompt the model to align its generative response with one of the predefined options: 'a', 'b', 'c", or 'd'.
18

---

Published as a conference paper at ICLR 2025
Table 6: Detailed results (%) of model performance, measured as accuracy percentages across di- verse disciplines for three runs. The random choice baseline involves shuffling candidate answers for each video question before consistently selecting answer 'a'. GPT-4V and Gemini Pro utilize 10
image frames extracted from the video content.
Model
Ar& Sports
Business
Seience
Health& Medicine
Emhodied TTaskss
Techk Engineering
Game
Average
4osee-
OpenA]
2024
47.10
92.31
75.11
K1.25
65.03
72.34
78.31
62.22
GPI-4o-seed
UIERAT
2124
46.38
90.91
69.77
84.38
65.73
75.53
83.13
61.77
GPT-4e-sod3
OpenA1
Claxde-3.--seed
Anthoop
2124
54.32
64.34
59.11
53.12
30.77
59.57
59.04
54.27
Claode-35-seed
2022
54.32
63.61
61.33
54.17
30.77
58.51
59.04
54,52
Claude-3.5.seed
GPI-4V-eed
0 MSBAT 217-h0
36.90
79.72
64.00
73.96
$1.75
60.64
71.08
$1.64
GPT-IV-seed
20236
35.48
83.92
68.44
73.96
58.04
60.61
75.90
52.79
GPT-4Vseed
OpEn/31
$1.12
Gemini Pro-seed
1 Team 23.
2023
40,90
79.72
60.44
78.12
43.36
71.28
65.06
52.92
Gemini Pro-seed
35.10
75.52
63.11
75.00
44.06
71,28
69.88
50,16
Gemini Pro-seed
Taam
74.83
77.08
67.02
1in.e
20760
34,58
$1.05
37.11
31.29
61.34
57.45
56560
35.94
Vidoo-LLaVA-seed
$2.45
$7.45
Vidoo-LLVVA-seel
3
LJR 20
20,30
36.39
50.35
$5.56
33.33
62.94
59.57
44.58
44.52
Video Chat-sead
[L
Hi
201
39.48
$1.05
30.67
46.88
39.86
39.36
44.58
40,03
Vido-Char--eedd
48
$1.05
41.26
39.36
Videa CeadeeCateeeed
La
A0235
39.61
$1.05
31.11
45.83
40.56
39.36
44.58
40.15
mPLLUG-Owlseed
Y0 1cctll
31.35
65.73
45,78
61.46
28.67
48.94
65.06
41,05
mPLUG-Owl-sodd
mPLUG-Owlsseed
12
27.48
61.54
$2,00
60.42
20.98
39.36
63.86
38.23
ChatUnivi-seed
Jin 21
217-1
24.13
60.14
$2.00
62.
48.95
56.
56.63
39.77
Clattnivissedd
ChatUnivi-eedd
Jm emeea
21751
24.13
59, 44
$2.89
58.
45.45
58.32
50.60
38.87
Tidoo-hauGPT-scU
2024
26.71
34,97
34.67
$3.12
37.76
41.49
34,94
32.59
Vidaa-ChutiiT-eeedl
11 al
207.10
27.74
41.96
36,89
53.12
39.80
43.62
76
34.64
PandGPTTsscd
Su St 2025
26.06
44.06
38.22
41.67
35,66
39,36
42.17
32.97
Pund:GPTswed2
PundaGPT-seedd
Sa mm
202.
25.16
38.45
43.11
39.58
36.36
45.74
33.73
32.58
ImageBind-LLas L.M-iinn-LLLM-qeu
11am
2023
24.77
41.96
30.67
31,25
46.85
43.62
40.95
31.62
ImugeRind-11
Bing:Bind-LLM-ece
20755
24.65
44.06
33.33
28, 12
48.25
40.43
42.17
31.94
X-Instruct BLIP-seed 1
2023
21.42
14.69
22.22
29.17
16.78
21,28
26.51
21.23
X-Instruct-B
20.77
16.78
X-Instruet-BB Bittruct-BLiin
[P-eedd 3 [Paliggpoiooo 2025
21.03
16.08
21.33
28. 12
18.18
23.40
26.51
21.23
I.W--sed
20744
11.35
18.18
24.88
24.47
15.20
LWM-sood
2
20246
12.13
17.48
15.56
19.79
24.48
22.34
8.43
15.14
LWM-seed 3
L
ct
2024b
12.65
16.78
14.22
21.88
28.67
19.15
15.66
15.84
Otter-sead
20230
18.45
19.58
8.33
15,96
14.45
15.84
Ofter-soad
L1
20230
17.29
17.48
13.99
18.09
15.14
Otter-seed
3
L1
20233
15.61
18.88
9.78
6.25
11.19
13.83
15.66
13.98
Vidxo-LLaMA
Ziaan
Video-LLMMA
Zhang e 20253
6.58
20.28
20.44
31.25
13.99
17.02
32.53
14.05
Vidbo-LL.MM/-seee
7hang et 20233
6.32
21.68
22.22
33.33
16.78
19.15
24.10
14.37
Table 7: Performance (%) of different set of turkers
Art&
Health& Medicine
Embodied Tasks
Tech&
Model
Business
Science
Game&
Average
Sports
Engineering
Turker Set
25.224
39,860
32.444
40.625
51.049
50.000
40.964
33.227
Turker Set 2
30.452
46.154
35.556
42.708
53.846
51.064
46.988
37.652
Turker Set 3
26.710
41.958
36.889
46.875
53.147
42.553
38.554
34.830
B.4 ADDITIONAL EVALUATION RESULTS USING OPEN-SOURCED EVALUATOR AS THE EVALUATOR
In addition to GPT-4V, we also experimented with using the open-sourced Video-LLaVA model as an evaluator. The average accuracy of various models evaluated with this method shown in Table 10 The rankings are consistent with those obtained using GPT-4V in the main paper, high- lighting the versatility our benchmark, which supports multiple evaluator options beyond GPT-4V.
B.5 ADDITIONAL TEMPORAL REASONING EXPERIMENTS
To better understand the impact of temporal coherence on reasoning tasks of different models, we conducted two experiments focused on temporal reasoning. These experiments were designed to
19

---

Published as a conference paper at ICLR 2025
Table 8; Performance (%) of different MLLMs across different disciplines.
Healthak Medicine
Emnoodic Tasks
Techd
Sports
Rusiness
Science
Engineering
Average
Chat (Open-ended)
20233
27.181
9.091
18.137
10.417
29.371
19.179
22.887
Video-Chat [L ol. 2025g
39.355
48.951
31,863
55,833
161
38.298
Video-LLaMAX Coprmuie Zhang et al. 2023a
5.419
27.972
24,020
31.250
11.816
15.957
16.096
Video-LLaMA Zhang
27.355
31.373
48,958
16.084
28.723
28.729
ChatUnivi (Open-eniee) Dia cl 2023
21.161
61.538
42.157
61.438
30.070
37.234
32.646
CharUnnivi Jin et 2021
12.387
58,042
50,000
60.417
30.070
43,617
29.072
(Open-entzd)
Ofter (I. 20231]
17.677
16.783
12,255
5.208
17.483
15.957
15,876
Imagel5ind ended)) [Han al. 2023
3.355
3,497
14,706
10.417
21.678
18.085
8.179
InageBind-II.M (Han 1023
23,742
51,471
33.333
PandsGIT (Opee-Cix)s]] (St 2023
22.581
16.084
24,020
21,875
19.580
21.277
21.718
PandaGPT Su
20255
27.613
39,706
25,000
40.559
21.277
31.615
LWM (Oper (Lar ct 2024b
16.000
20.979
14.706
16.667
19.580
20.213
16.976
LWM [Lu et al, 20240
16.387
18.182
18.157
19.792
22.378
21.277
17.938
X-Instrag--BLl (: (Pussgoooalou
X Instrct-BBIIP Panagopoulod er 2023
19.355
13.287
22.549
29.167
18.881
14.894
19.519
Table 9; Performance (%) of MLLMs on temporal reasoning tasks under different conditions.
Model
Original Videos
Shuffled Videos
Reduced Video Frames
GPT-4o (OpenA 2024
40.90
35.11
32.19
GPT-4V OpenAA
2023b
27.17
22.04
22.33
S.ude-3.3-Sonmee (Anthropic
2024
25.77
21.58
19.45
Gemini Pro Team of al.
202.
24.65
20.19
18.97
Li al.
2123a
34.45
18.47
28.50
Video-Chat-7B (Lr al. 2023c
25.77
21.50
20.19
Video-ChalGPT-7755 (Maaz el al.
(Maaz
2024
23.53
21.62
20.17
ImageBBind-LLMM-1 (Hann el
2023
19.89
16.19
PandaGPT-77 (Su al. 2022
28.01
24.35
22.57
ChatUnivi-7B (Jim
2025
22.97
19.41
17.14
Video-LLaMA-2213181 (Zhang 2023a
6.16
5,02
4.58
X-Instruct-BLIP-7B [Panagopoulou al. 2023
11.20
9.88
8.95
LWM-IM-JAX (Liu et 20246
7.00
5.75
5.56
Otter-7B (Li al. 2023a
9.52
3.25
7.93
mPLUG-OWL-7B Ye 2023
20.17
18.19
16.59
analyze model performance under varying temporal constraints, including reduced video frames and shuffled video frames.
Reduced Video Frames: Videos were processed by reducing the number of frames to 1/5 of the original. This setting evaluates the models' ability reason with limited temporal information.
Videos were processed by shuffling their frames. This setting tests the models' ability to reason when the temporal order of the frames is disrupted.
Shuffled Video Frames:
The results of these experiments summarized in Table From Table there is significant per- formance drop when videos are either reduced in frame count or shuffled. These findings highlight the sensitivity of models to temporal coherence and emphasize the necessity of maintaining suffi- cient temporal information for accurate reasoning. Notably, proprietary models such as GPT-4o and GPT-4V demonstrate better resilience under these settings compared to most open-source models.
C IMPLEMENTATION DETAILS
We use the optimum number of video frames and report the performance in the main paper. The numbers of the sampled frames are 10 for GPT-4V/o and Gemini Pro, 8 for Video-LLaVA, 32 for ChatUniVi. For closed-source models, for both Gemini Pro and GPT-4V, we use the default settings provided by their official APIs. We use Katna ( to extract key video frames as input to these two models. The Gemini Pro is set to process visual input and configured with safety settings to filter range of harmful content. The configuration thresholds are set to 'BLOCK NONE', For
20

---

Published as a conference paper at ICLR 2025
Table 10: Performance of different models across evaluations using Video-LLaVA as the evaluator.
Model
Accuracy (%)
Video-Chat-7B (Li al. 2023c
41.96
ChatUnivv-7B (Jin et al.
2023)
39.81
Ye mPLUG-OWW-7B
2023
38.01
PandaGPT-7B (Su et al. 2023
31,66
ImageBind-LLLM-715 (Han et al.
2023)
31.65
X-Instruct-BLIP (Panagopoulou al.
2023
22.02
LWM IM-JAX (Liu et al. 2024b)
16.81
Otter-7B (Li et al. 2023a)
12.08
-MA-- 3B (Zhang et al.
2023a)
10.84
Table 11: Category-wise and overall error rates
Category
Incorrect/Total
Error Rate (%)
Sports & Arts
5/62
8.06
Health & Medicine
2/7
28.57
Science
1/52
1.92
Robotics
0/12
0.00
Business
0/10
0.00
Tech & Engineering
1/46
2.17
Overall
9/189
4.76
PandaGPT, set 'top-p' to 0.7, and 'temperature' to 0.5. For VidcoChat, set framees to 100. For LWM, we use the LWM-Chat-IN variant. For X-Instruct-BLIP the model is implemented using four image frames. For Otter,
the video variant. We use GPT-4-32K for judging whether the model answer is correct when can not mapped to the option letter using the rule-based method. The prompt provided to GPT is structured follows: "I will present response from
the judge
model alongside several answer options. Your task is to evaluate the response and determine which of the following options it most closely aligns with, denoting the most similar option by its corresponding letter (a, b, or d)."
Query Generation in Synthetic Data Generation Pipeline For the discipline of Science, queries are generated for subdisciplines such as Geography, Chemistry, Wildlife Restoration, Mycology, Na- ture. Physics, Weather, Zoology, Math, Botany, Biology. and Geology. In the Tech & Engineering discipline, our queries span across Electronics, Animal Behavior, Mcchanical Engincering, Energy & Power, Architecture, Agriculture, Nature, Physics, Robotics, Woodworking, and Gardening. The Sports & Arts discipline encompasses a broad range cultural and physical activities, including Music, Drawing and Painting, Football, Volleyball, Aerobic Gymnasties, Basketball, Instrument, Baking, Dance, Woodworking, Graffiti, Anatomy, and additional Music-related topics. Embodied Tasks are represented through queries for Assembly, Ego-motion, and Single Object Manipulation, focusing on the interaction between agents and their physical environment. The Health & Medicine discipline is segmented into Pharmacy, Public Health. Clinical Medicine, and Basic Medical Sci- ence, reflecting the multifaceted nature of healthcare and medical studies. The Business discipline stratified into fundamental areas such as accounting, finance, management, marketing, and eco- nomics, each representing key facets of the commercial and economic world. Lastly, the Game discipline consists of Role Playing Game, First Person Shooting game, Racing Game, Adventure Game, Real-Time Strategy Game, Tower Defense game, and Fighting Game
Each generated query retrieves relevant video content, which is then filtered and processed to align with the specific needs of our research objectives. Videos that meet criteria in terms of con- tent, length, and quality are downloaded and incorporated into our dataset, forming the basis for subsequent analysis and model training.
21

---

Published as a conference paper at ICLR 2025
Example task from this Batch
0:14 2:29
Dane Farber
Match the video.
Read the quastion provided carafullyy Based on the content of the video, select the most accurats answer from the options glven. There only one correct ansuer for each question.
chamotherapy say texanes
trestments?.
newearh inalages may to paradgm tweament aeceeee Maadtee curcer.
Figure 6: The interface of using Amazon Mechanical Turk to do human evaluation.
Table 12: Comparison Human Evaluation on subset of 75 videos.
Art& Sports
Hcalth& Medicine
Embodied Tasks
Tech&
Model
Business
Science
Average
Engincering
Human Evaluation
31.183
59.782
42.103
48.858
56.429
50.134
43.758
GPT-4V (OpenAl 2023b
30.309
89.203
68.731
80.059
38.432
69.108
48.793
Gemini- Pro (leam el al. 2023
28.745
80.909
69.425
80.023
50.987
80.479
48.083
D HUMAN EVALUATION
D.1 QUALITY DATA
We hired Amazon Mechanical Turk do human evaluation on the data with the results shown in T Workers were required to have completed more than 1000 Human Intelligence Tasks (HITs) and have an HIT approval rate greater than 95% qualify for our tasks. We show in Figure the human evaluation interface the generated data. Each worker was compensated 0.20 for complet- ing an assignment. This amount was determined based on the estimated time and effort required to complete each task. We set the number of unique workers per task to to collect diverse perspec- tives while avoiding redundancy. Workers were given hour to complete each assignment. This time frame was chosen to enable thoughtful responses from workers.
We also hired students from campus to do human evaluation on subset of the data. The results are shown in Table T2 The performance of the human evaluators did not surpass that of GPT-4V and Gemini-Pro This outcome underscores the challenging nature of the dataset, which often necessi- tates specialized domain knowledge that our evaluators- These results highlight the complexity of the questions and the potential necessity for discipline- specific understanding achieve high accuracy
non-experts-foundd demanding.
D.2 QUALITY OF USING GPT AS THE JUDGER
For comprehensive assessment GPT-4V's accuracy when using it as the judger, we devised human evaluation protocol also resort Amazon Mechanical Turk, visualized in Figure 7 The evaluators present a series of statements derived from the video, and GPT-4V is tasked with selecting the most accurate answer from set of multiple-choice questions. Through this interface. human evaluators can efficiently gauge GPT-4V's performance across different types of questions- when using it as the judger.
22

---

Published as a conference paper at ICLR 2025
Caeabers
anotist
is up wato
hott odet banwe
The
hare fard bel
O
Comtter
A.tharee bolterr to iata diee
implle eay aciue
Stat
bedeove Indioal upges bersom wwweeeeeeeee
Figure 7: Human evaluation interface for GPT judger.
The results obtained from this human evaluation process are shown in Table across 189 examples, there are only incorrect ones with the error rate of 4.76%, validating the effectiveness of using GPT-4V as the judger.
E ERROR ANALYSIS
In this section, we delve into the analysis errors from evaluated MLLMs. We summarized error types follows:
Question Understanding Error (QUE): Models misinterpret the question's 's intent, such as misun- derstanding how pendulum's period would change if condition in the scenario is altered.
Audio Understanding Error (AUE): Models fail to interpret audio cues correctly, shown by their failure to recognize blue and red lines stock chart.
Visual Perception Error (VPE): There a misinterpretation of visual content, leading to incorrect assumptions about the visual data presented in the video.
23

---

Published as a conference paper at ICLR 2025
Why does the need step shown the video?
Type: Explanation
Type: Procedure Understanding
Which step demonstrated in the video?
He needs to flip the tabletop to expose mounting holes for attaching the legs.
A: The first step of assembling a coffee table, which is flipping the desktop so the mounting holes face up.
Type: Counterfactual Thinking
Q: What would happen if the skipped the step shown in the video?
What will the next step assembling?
Type: Future Prediction
A: The desktop of the coffee table would be upside down, making impossible to mount the legs.
A: The next step involves securing the legs to the now-exposed mounting holes.
Figure 8: Examples from MMWorld in the Embodied Tasks discipline.
Q: What would happen if the ground was dry?
Type: Counterfactual Thinking
Type: Domain Expertise
Q: Why would snail moving relatively fast in this video?
A: The mucus produced by the snail reduces friction on the ground.
A: The snail will travel slower than depicted in the video.
Type: Explalnation observed phenomenon in the video?
Q: What might be the reason for the
Q: What will the ground feel like after the video has ended?
Type: Future Prediction
A: The snail looking for a plant to climb.
A: Wer with mucus.
Figure 9: Examples from MMWorld in the Tech & Engineering discipline.
Hallucinations (HE): Models generate content or details that are not present in the actual data, essentially "hallucinating" information.
Reasoning Error (RE): Models demonstrate lack of logical reasoning, leading to incorrect con- clusions based on the given data.
Lack of Domain Knowledge (LDK): Models show an inability to answer questions that require specific domain expertise, indicating a gap in their knowledge.
Reject to Answer (RA): An example of this error was observed when the model was asked to select an answer regarding the outcome of an experiment involving liquid nitrogen. Instead of choosing option, the model provided misunderstanding or a cautious approach due to the potential for the question to be interpreted as pertaining sensitive topic, which trigger content filters focused safety and compliance policies.
unrelated response concerning light bulb, indicating either
some cases of Question Understanding Error, Audio Under- standing Error, Visual Perception Error, Hallucinations, Reasoning Error. Lack of Domain Knowl- edge, and Reject Answer respectively from MI.LMs evaluated on MMWorld.
We show in Figure 18
19 20
24

---

Published as a conference paper at ICLR 2025
What would happen stick in the experiment was removed?
Type: Counterfactual Thinking
Type:
Q: What scientific principle demonstrated in video is crucial for understanding the concept chain reaction?
Domain
Expertise
A: The chain would break apart from the middle in chaotic manner.
A: Tension and Friction.
Type: Explaination
Why the popsicles flying through the air?
What the people in the video concerned with?
Type: Future Prediction
tension and friction holding them together being released.
A: Cleaning up the sticks and reassembling the line.
Figure 10: Examples from MMWorld in the Science discipline.
OO CXIT
Whed
Q: What does tell if the price is below the blus and red lines?
Type: Counterfactual Thinking
Type: Domain
Q: What does the blue and red lines in the video represents?
Expertise
A: The chain would break apart from the middle in chaotic manner.
The average price last 20 and 50 days..
Figure 11: Examples from MMWorld the Business discipline.
F DATA EXAMPLES
F.I MAIN SUBSET
We present additional examples from the main subset of MMWorld in Figures
9
10
and 3
Furthermore, Figures 14 15 and dedmoonsratee how Q&A pairs in MMWorld are carefully tailored specific disciplines, including Sports & Arts, Science, and Business. Each example is designed to reflect the unique reasoning and understanding required within its respective discipline.
F.2 SYNTHETIC AND SYNTHETIC II
We present in Figure 17 additional examples from Synthetic and Synthetic II of MMWorld. The examples correspond to various disciplines: Business, Health & Medicine, Science, and Gaming,
25

---

Published as a conference paper at ICLR 2025
Q: What would happen if she not wearing shoes?
Type: Counterfactual Thinking
Type: Domain Expertise
Q: What knowledge is required to find this video useful?
A: She will likely not stand upright her toes.
A: Knowledge of Step Counting.
Type: Explaination
Q: What is the purpose of the timer in the bottom left of video?
What ballet dancer likely after the video faded white?
Type: Future Prediction
A: used number steps that dancer takes.
A: She took off her dance shoes.
Figure 12: Examples from MMWorld in the Arts & Sports discipline.
Q: What has been changed in the video?
Q: Why does the robot need to the step shown in the video?
Type: Explanation
Type:
Temporal Understanding
A: The bottom drawer has been closed.
A: Becasue the bottom drawer is empty and closing can aviod block way.
Figure 13: Examples from MMWorld of explicit temporal understanding and implicit temporal understanding (e.g., in explanation).
respectively. For each discipline, the first example showcases an audio-only generated QA from Synthetic while the second example represents visual-only generated QA from Synthetie II. These examples highlight the multi-disciplinary reasoning capabilities evaluated in our benchmark, even for the synthetic dataset.
G ADDITIONAL DATA STATISTICS
For human annotated dataset, the length of each video was capped at approximately two minutes. The statistical distribution of the disciplines within the dataset for this part is as follows:
Sports Arts: The subset that consists of 77 videos, showcasing a vibrant collection that covers wide range of topies from athletic endeavors various forms of artistic expression.
26

---

Published as a conference paper at ICLR 2025
Q: What might be the reason to start with the eyes when drawing a face?
A: want start with the short distances which are easier measure compared longer distances. And then you relate the rest of everything to the distance of the eyes.
Q: What would happen if you started with the nose?
A: would more difficult relate rest of the face and maintain proportional accuracy.
Q: What will likely happen to the cheesecake video?
A: The cheesecake will be eaten.
Q: influenced the making of this cheesecake?
A: Spanish burnt Basque cheesecakes.
Q: How will the canvas look the experiment is forever?
A: The center of the canvas will become solid color.
Q: According to the video, what happens when the person takes their foot off the pedal?
A: The hi-hat opens.
Q: How many players are in the video?
One.
SOON
HuTtaus T04NAGOON
Q: What players perform before performing transition flight from high bar low bar?
A: Giant circle forward with turn on one arm before handstand phase.
Figure 14: Examples from the Sports & Arts discipline, illustrating tailored Q&A pairs.
Science: A subset of 75 videos, which delves into the empirical world of scientific inquiry, spanning a multitude of specializations from fundamental physics to advanced biological studies.
27

---

Published as a conference paper at ICLR 2025
Q: How does the girl actively change her spinning speed?
A: By stretching out or contracting her legs and arms.
Q: What would happen if the woman let go of the rope in the middle of the video?
A: She would fall onto the platform, lose balance, and eventually fall.
Q: What is the physical law that makes the girl spin faster when she contracts her legs and arms?
A: Conservation of angular momentum.
Q: What is the reason that the girl the right is floating in the air?
A: The woman on the left is pulling a rope connected to the girl.
does the solution color change time?
A: Purple
Green + Light brown.
What will happen once the stirrir turned off?
A: The solution will settle, no further change.
Figure 15: Examples from the Science discipline, illustrating tailored Q&A pairs.
Tech & Engineering: Encompassing 54 videos, this segment captures the cutting-ed edge ad- vancements and foundational concepts that drive innovation and infrastructure in the mod- ern world.
Embodied Tasks: With 50 videos, the dataset provides a focused insight into the dynamic field of Embodied Tasks, highlighting the intersection of AI, mechanics, and automation.
Health & Medicine: This essential discipline is well-represented with 50 videos, offering perspectives on medical breakthroughs, healthcare practices, and life sciences.
Business: This discipline includes 50 videos, reflecting on the multifaceted nature of com- merce, from economics management sciences.
Game: This discipline includes videos, reflecting various aspects of gaming.
Altogether, the MMWorld Benchmark's diversity visually encapsulated in Figure 22 which de- lineates the distribution of videos across 61 subdisciplines. The horizontal bar chart provides quantified representation the dataset's range, reflecting the careful curation process that has gone into ensuring breadth across various knowledge areas.
MMWorld also has additional annotations such as "Requires Audio", "Requires Video", and "Ques- tion Only". The world we live in is rich with both audio and visual information, and effective world modeling requires an understanding of how these modalities interact and convey meaning. To achieve this, annotated additional attributes such "Requires Audio". "Requires Video". and "Question Only" during data collection. These annotations help determine whether correctly answering question necessitates audio information, visual cues from the video, be ad-
28

---

Published as a conference paper at ICLR 2025
ONE
OUNCR
DENCEE
is primary goal showing the coins with both sides?
A: To encourage people to purchase the coins.
What does it tell if the price below the blue red lines?
A: The price is moving downward.
Q: What do the blue and red lines in the video represent?
A: The average price over the last
days.
What might be possible subsequent action taken by the person the video after drawing the paper?
A: Proceeding to enact a scene related to shopping.
Q: What process is shown in the video?
A: The process of making coins.
Figure 16: Examples from the Business discipline, illustrating tailored Q&A pairs.
dressed based solely on the question itself. By doing so, ensure that benchmark tests the full spectrum of multimodal comprehension, reflecting the complex, sensory-rich environment which real-world understanding takes place. The statistics of these annotations are shown in Figure 23
29

---

Published as a conference paper at ICLR 2025
hightrr increasing amount of printed money in the United States? A: 2000-2010.
tractor versus TV. What reasoning does he use to explain which is better?
Q: The speaker mentions borrowing
A: Borrowing for tractor is better because generates Income to repay the debt, while TV does not produce income.
Q: Based on video, what substance surrounds the organelles outside nuclous, organelies suroundded by eytoplasm, appeers jolly-ike substance within the cell.
According to the speaker, what eukaryotic lost its nucleus? call longer able to regulato its activities reproduce properly
functions.
DOPSI
nee
SUGAR
Why phenomenon
A: Adding releases CO2 rapidly, creating pressure that inflates the balloon.
According the does adding the egg the upper egg?
A: Adding salt creates rough surface, increasing friction and providing micro- support paints that stablize the upper egg, preventing from sliding.
Q: From the video, a player continues using the technique shown in the A site, what is likely to happen dose--uartters battle?
The sttacking team can leverage reduced visibiityy created surprise staying concealed.
According to the what you Macbeth shot your nightclub stormy night between and AM?
A: There is chance you will spewn on top of Mount Chillad and see UFO flying away.
Figure 17: Examples from Synthetic Subset and Synthetic Subset of MMWorld.
30

---

Published as a conference paper at ICLR 2025
Q: How many eggs can frog lay at once?
Lack of Domain Knowledge
A: I'm sorry, but the images provided are of mushrooms in forest setting, which habits of frogs.
provide any information egg-laying
Q: How do the pulleys move when the hands are off the pulley system?
Reasoning Error
When hands are removed from the pulley system, shown series of images, there is no force being applied to of the pulleys to cause movement. Therefore, all four pulleys remain static.
Figure 18: Error pattern of lack domain knowledge and reasoning error. In the first case, the model does not give the correct answer because of lacking the domain knowledge. In the second case, the model makes the wrong reasoning.
What would happen pendulum period if the professor sit straight on the ball instead?
Question Understanding Error
The pendulum period would not afford
What would happen
skipped step man skipped the step of rolling the blue tube floor to uncover the
the video?
Visual Perception Error, Hallucinations
box, he would likely have encountered difficully in accessing the contents of the box. step is important because helps the the red box. ensuring that the box is visible and accessible.
Figure 19: Error pattern of question understanding, visual perceptron error, and hallucinations. In the first case, the model does not understand the question correctly where the model fails to ac- curately discern the query regarding the pendulum's period. In the second scenario, the model crroncously identifies objects within the visual input, leading to the hallucination of non non-cxistent elements, such red box.
31

---

Published as a conference paper at ICLR 2025
Sequabning
Elgantoogg
Reject Answer
Q: What if replace the boiling water with iced water? The question will trigger the content filter and there will
Audio
Q: What does the blue and red lines in the stock chart represents?
Understanding Error
The blue and red lines levels, respectively. The support level where the stock price tends stop falling, while the resistance level is where the stock price tends to stop rising. In the video, the blue line indicates level support. while the red indicates level of resistance.
stock chart represent the support and resistance
Figure 20: Error pattern of reject to answer and audio understanding error. In the first case, the model will reject to answer because of the dangerous action showing in the video. In the second case, the model does not understand the audio correctly as the answer to the question given in the audio of the video.
Q: How many capacitors are there in the final circuit?
Visual Perception Error
There several capacitors the final circuit. We see least them in the video.
Visual Perception Error, Lack of Domain Knowledge
What could the three gases order?
A: I'm unable to provide assistance as the request requires information to be extracted from images/idees which is beyond my current capabilitiss
Figure 21: Error pattern due to visual perception inaccuracies and insufficient domain knowledge.
The first case demonstrates visual perception error where the model incorrectly identifies the num- ber of capacitors present. The second case showcases compound error where the model not only fails to discern colors indicative of different gases but also lacks the domain knowledge neces- sary to infer their identity correctly.
32

---

Published as a conference paper at ICLR 2025
Gardening
Basketball And Fainting
Drawing
Single Object Manipulation
Electroincs
Physics
Architecture
Accounting
Instrument
Mechanical Engineering Energy & Power
Botany
Geography
First Person Shooting Game
Wealth Management
Clinical Medicine
Mycology
Anatomy
Materials
Music
Taxation
Trading
Zoology
Football
Gymnastic
Role Playing Game
Quantitative Finance
Supply Chain Management
Chemistry
Basic Medical Science
Weather
Fighting Game
Robotics
Adventure Game
Wildlife Restoration
Commercial Promotion
Nature
Agriculture
Math
Animal Behavior
Graffiti
Marketing
War
Baking
Tower Defense Game
E-commerce
Robotics Behavior
Ego-mation
Slock Chart
Management
Valleyball
Finance
Investment
Economics
Moosic
Biology
Assembbly
Pharmacy
Racing Game
Public Health
Diagnostic
Geology
Manual Collected Subset
Aerobic Gymnastics
Dance
Synthetic Subset
Woodworking
Electronics
Synthetic Subset II
Real-time Strategy Game
10
20
30
40
50
60
70
Number Of Videos
Figure 22: The number of videos per subdiscipline in MMWorld. Each horizontal bar indicates the
quantity of videos corresponding to subdiscipline, showcasing the dataset's 's diversity and coverage across various domains of knowledge. Synthetic Subset I is collected with audio-only data and Synthetic Subset II is collected with visual-only data.
33

---

Published as a conference paper at ICLR 2025
Requiring Audio
Requiring
Question Only
Attribute
Requiring
Domain Knowledge
Requiring
Visual
Information
200
400
600
800
1000
1200
Count
Figure 23: The distribution statistics of questions in the MMWorld benchmark by annotations.
34


Reviews:
[{"review_id": "72RBbUFnQ0", "reviewer": "Reviewer_MwCy", "summary": "This paper introduces MMWorld, a new benchmark for evaluating multimodal large language models' (MLLMs) ability to understand and reason about video content across multiple disciplines. The benchmark consists of 1,910 videos spanning 7 broad disciplines and 69 subdisciplines, with 6,627 question-answer pairs. The key innovation is its focus on multi-faceted reasoning (explanation, counterfactual thinking, future prediction, domain expertise) and multi-discipline coverage. The authors evaluate 15 MLLMs (both proprietary and open-source) on this benchmark and provide detailed analysis of model performance and error patterns.", "questions": "As comments in Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1: The benchmark makes a novel contribution by filling an important gap in evaluating MLLMs' world modeling capabilities through video understanding across diverse disciplines.\n\n2: The evaluation framework is comprehensive, covering seven broad disciplines and 69 subdisciplines while testing multiple reasoning types including explanation and counterfactual thinking.\n\n3: The methodology demonstrates rigorous attention to detail, with well-documented data collection processes and multiple evaluation strategies.\n\n4: The research provides valuable insights by identifying significant performance gaps in current MLLMs and revealing interesting differences between human and MLLM capabilities through detailed error analysis.", "weaknesses": "1. The dataset, while diverse, remains relatively similar compared to other modern benchmarks. Please include the recent works and discuss the difference between MMWorld and recent works, such as VideoMME, MMBench-Video.\n\n2. The evaluation methodology shows potential bias through its heavy reliance on GPT-4V as a judge and in synthetic dataset generation.\n\n3. The temporal reasoning aspect, though claimed as a key feature, could be more thoroughly explored and evaluated in the benchmark.\n\n4. How to get the final results with the three subsets (Main Subset, Synthetic I, Synthetic II) in Table 3?\n\n5. As the evaluated models cannot handle the audio modality, is the model's performance on the audio-related questions underestimated?\n\n6. As the current VLMs are not the world model and the current questions in this benchmark are also limited to basic QA, limiting complex reasoning tasks. I think the author may reconsider the name of the benchmark to avoid over-claiming.\n\n7. More in-depth analysis of the errors from the perspective of current VLM's architecture, training data, or training techniques would improve the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcWvqHBuLa", "reviewer": "Reviewer_eShE", "summary": "This paper proposed a comprehensive benchmark for MLLMs, covering 7 major disciplines, 69 subdisciplines, and multi-faceted reasoning tasks. The authors conducted experiments on 15 models (including both open-source and closed-source models) and analyzed the results.", "questions": "1. Based on the results in Table 3, it appears that MLLMs perform significantly better in the Business discipline compared to other disciplines. Does this imply that the QA pairs in the Business discipline are relatively simpler, or does it indicate that MLLMs are stronger in the Business discipline than in other disciplines?\n2. The authors emphasize that they have introduced a new dataset. I did not see an anonymous dataset link in the paper. Where can I find it?\n3. Typos:\n • “an characteristic” in Figure 1.\n • “%%” at line 215.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This benchmark encompasses 7 major disciplines and 69 subdisciplines, offering comprehensive coverage and highlighting its advantages over existing benchmarks.\n* The authors introduced a new dataset, ensuring its quality through human annotation.\n* The experiments are conducted on a large scale, covering mainstream MLLMs and providing analysis across different difficulty levels.", "weaknesses": "* Although the authors conduct an error analysis with six different categories, the error sample size is relatively small. \n* The analysis of experimental results across different models lacks depth. For example, what specific differences exist between model and human reasoning? It would be helpful to compare with human performance, as in business discipline, the model achieves 90% accuracy, showing a strong understanding ability. It is not surprising that MLLMs perform better than average humans. It would be meaningful to see whether the model has reached a level close to or beyond human expert performance, especially in business discipline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gGISJKRmGA", "reviewer": "Reviewer_ZqHq", "summary": "This work proposes a novel benchmark to evaluate commonsense knowledge and reasoning capabilities of the existing multimodal LLMs that are capable of processing videos or sequences of images. The proposed benchmark is classified as *multi-discipline* since it includes videos from a diverse set of subjects, and it is also classified as *multi-faceted* because it also includes different types of tasks where some tasks are focusing on commonsense reasoning abilities. The authors also collected a synthethic dataset to evaluate either auditory or vision processing capabilities of these models. The experiments reveal that (i) most of the existing models fail to excel in the proposed benchmark, (ii) even the proprietary models lack deficiency in embodied tasks, (iii) and they fail to process auditory information sufficiently.", "questions": "- The hexagon figure (Fig.4) is really difficult to interpret: I think this kind of figures makes sense if one compares 3 models at most. The score normalization makes things even worse: So we cannot see how well the best model performs in this way. I would definitely remove it from the paper, and put the Table 8 into the main text. Also note that the figure is embedded as \"PNG/JPEG\" instead of \"PDF\" (but removing it from the paper would make things much more better).\n- Fig. 6 is also hard to interpret. I would rather group some selected models instead of error categories, so one can observe which model produces what kind of errors more or less.\n- Is there any example for auditory questions in the appendix? Can you please point? Can we see some synthethic questions/answer pairs generated using vision and audio side-by-side?\n- How much diverse is the language, i.e. questions and answers?\n- One suggestion: Please embed all figures as PDFs.\n- It would be good to see many more examples in the appendix in a more compact format by avoiding large text with conversation bubbles.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the most valuable aspect of this paper, the proposed dataset contains videos from a diverse set of subjects, and questions have a variety focusing on commonsense knowledge and reasoning. This unique combination could position the benchmark as an important resource for the near future, as recent models fails to thrive in this video question answering benchmark.", "weaknesses": "- Presentation could be improved (see the questions field).\n- The paper actually has a main contribution, which is human annotated dataset. There are also two synthetic datasets to evaluate the performance of individual modality processing performances of the models that are capable of processing both vision and audio. This aspect complicates this work a little bit, where I think it would better to have a separate work which carefully examines how well the models process each modality.\n- The related work could be improved there are a lot of recent related work which are missing (e.g. [TempCompass](https://arxiv.org/abs/2403.00476), [VILMA](https://openreview.net/forum?id=liuqDwmbQJ), [VITATECS](https://arxiv.org/abs/2311.17404v1), also older but related since nearly half of the questions focus on explainability [Next-QA](https://arxiv.org/abs/2105.08276))\n- The imbalanced nature of the subtasks: 48% of the questions belong to the explanation category/task. Also, the main task figure gives the impression of that these tasks could be fluid. For instance, the *explanation* example also requires *spatiotemporal reasoning*, and *future prediction* requires also some *domain-expertise*. Maybe, rather than assigning just single category, they could have multiple categories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ld6Jg3czCN", "reviewer": "Reviewer_Q3it", "summary": "This paper introduces MMWorld, a new benchmark designed for multi-disciplinary, multi-faceted multimodal video understanding. MMWorld comprises 1,910 videos spanning seven broad disciplines and 69 sub-disciplines, along with 6,627 question-answer pairs and corresponding captions.", "questions": "- Regarding multi-discipline evaluation, if a model underperforms in specific domains, is it due to a lack of general domain knowledge, specific domain ability, or both? Designing experiments to analyze this would be valuable. For instance, comparative testing with models pre-trained on general knowledge versus those fine-tuned with domain-specific datasets could help isolate the impact of each knowledge type on performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- MMWorld is the first video benchmark to cover videos beyond the general visual domain.\n- The paper includes several analyses to assess the strengths and limitations of current video large language models (LLMs).", "weaknesses": "- Regarding multi-faceted abilities, almost all are already evaluated in previous datasets, such as CVRR [1] and Tempcompass [2]. This raises the question of the necessity for another benchmark to assess these capabilities, especially as the data in MMWorld also comes from YouTube, which may reduce its novelty and significance.\n\n- For multi-discipline evaluation, the use of GPT-4V to generate questions raises concerns about ensuring expertise-level quality, as seen in MMMU, which uses college exam questions to maintain a rigorous standard. If the generated questions don’t meet this level, it blurs the distinction between domain-specific and general-domain questions. If no substantial difference exists, a model that performs well in general-domain benchmarks (e.g., CVRR or Tempcompass) may also perform well in MMWorld, potentially limiting the unique insights this new dataset could offer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H4WfSDfIbj", "reviewer": "Reviewer_dopE", "summary": "The paper introduces MMWorld, a new benchmark designed to evaluate multimodal large language models (MLLMs) across multiple disciplines in their understanding of real-world dynamics. MMWorld includes three datasets: a manually annotated dataset for multimodal reasoning and two automatically generated datasets focused on audio and visual understanding, respectively. Comprehensive baseline results and analyses demonstrate the challenges presented by MMWorld, with a detailed examination of primary errors and valuable insights.", "questions": "For the first weakness. I hope the authors can provide the followings:\n\n1. The instructions given to human annotators for reviewing and modifying generated QAs.\n2. The criteria used for accepting or rejecting generated QAs.\n3. Any measures taken to ensure diversity of knowledge sources beyond GPT-4.\n4. The percentage of questions that were significantly modified or entirely rewritten by human annotators.\n\nFor the second weakness. I suggest that the authors include a comparison that explicitly compares MMWorld to NExT-QA and Causal-VidQA, especially on the types of reasoning tasks.\n\nFinally, I find that the average scores (the right most column) in Table 3 are much below than the average of the 7 disciplines. Any explanations on how to obtain the average scores?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.To my best knowledge, MMWorld is the first video understanding dataset that contains diverse question types necessitating knowledge from multiple disciplines.\n\n2.The baselines and analyses are comprehensive, covering 15 advanced MLLMs.", "weaknesses": "1.\tAccording to the results in Table 3, GPT-4o already achieves high performance (80%~90%) on some disciplines, significantly surpassing other competitors. However, the QAs are generated and the model predictions are evaluated by GPT-4 families. This raises a concern that the dataset is likely biased to the knowledge of specific models (GPT-4). It would be better to show how the human annotators are instructed to amend the generated QAs.\n\n2.\tThe paper seems forget to discuss and compare with two related benchmarks: NExT-QA and Causal-VidQA. The two benchmarks also emphasize explanation, temporal dynamics, counterfactual and future predictions, respectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)