MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos

## Abstract

Abstract Multimodal Language Language Models (MLLMs) demonstrate the emerging abilities of "world models"—interpreting and reasoning about complex real-world dynamics.
To assess these abilities, we posit videos are the ideal medium, as they encapsulate rich representations of real-world dynamics and causalities.
To this end, we introduce MMWorld, a new benchmark for multi-discipline, multi-faceted multimodal video understanding.
MMWorld distinguishes itself from previous video understanding benchmarks with two unique advantages: (1) multi-discipline , covering various disciplines that often require domain expertise for comprehensive understanding; (2) multi-faceted reasoning , including explanation, counterfactual thinking, future prediction, etc.
MMWorld consists of a human-annotated dataset to evaluate MLLMs with questions about the whole videos and a synthetic dataset to analyze MLLMs within a single modality of perception.
Together, MMWorld encompasses 1,910 videos across seven broad disciplines and 69 subdisciplines, complete with 6,627 question-answer pairs and associated captions.
The evaluation includes 2 proprietary and 10 open-source MLLMs, which struggle on MMWorld (e.g., GPT-4V performs the best with only 52.3% accuracy), showing large room for improvement. Further ablation studies reveal other interesting findings such as models’ different skill sets from humans. We hope MMWorld can serve as an essential step towards world model evaluation in videos.

## 1 Introduction

Foundation models, such as Large Language Models (LLMs) [OpenAI, ; Touvron et al., ; Jiang et al., ; Anil et al., ] and Multimodal LLMs (MLLMs) [OpenAI, ; Team et al., ; Lin et al., ; Li et al., ; Maaz et al., ; Chen et al., ], have demonstrated remarkable abilities in text and image domains, igniting debates about their potential pathways to Artificial General Intelligence (AGI). This raises a critical question: how well do these models understand the dynamics of the real world? Are they equipped with an inherent World Model [LeCun, ; Chen et al., ; Ha and Schmidhuber, ; Xiang et al., ] that can understand and reason about the underlying principles and causalities of the dynamic, multimodal world?

Videos, with their rich, dynamic portrayal of the real world, are ideally suited for evaluating the "world modeling" capabilities of MLLMs.
Existing video understanding benchmarks [Li et al., ; Ning et al., ; Pătrăucean et al., ; Li et al., ], however, fall short in two key perspectives for such evaluations.
First, as LeCun et al. [LeCun, ] discussed, the world model should be able to *(1) estimate missing information about the state of the world not provided by perception, and (2) predict plausible future states of the world*. Evaluation of such capabilities requires multi-faceted reasoning beyond perception level, including explaining the video dynamics, counterfactual thinking of alternative consequences, and predicting future activities within videos.
Moreover, the multi-discipline nature of the multimodal world necessitates a grasp of diverse fundamental principles—ranging from physics and chemistry to engineering and business.
Hence, domain expertise across a variety of disciplines is imperative for a thorough evaluation of a model’s world understanding towards AGI [Morris et al., ; Yue et al., ].

Therefore, we introduce MMWorld, a multi-discipline multi-faceted multimodal video understanding benchmark to comprehensively evaluate MLLMs’ abilities in reasoning and interpreting real-world dynamics (^1^11Note that MMWorld is not a sufficient testbed for world model evaluation, but we believe overcoming the unique challenges presented in MMWorld is essential and necessary towards comprehensive world modeling.). MMWorld encompasses a wide range of disciplines and presents multi-faceted reasoning challenges that demand a combination of visual, auditory, and temporal understanding.
It consists of 1,910 videos that span seven common disciplines, including *Art & Sports*, *Business*, *Science*, *Health & Medicine*, *Embodied Tasks*, *Tech & Engineering*, and *Games*, and 69 subdisciplines (see Figure 1) such as Robotics, Chemistry, Trading, and Agriculture, thereby fulfilling the objective of breadth in discipline coverage. The dataset includes a total of 1,559 question-answer pairs and video captions annotated and reviewed by humans.
Meanwhile, for multi-faceted reasoning, MMWorld mainly contains seven kinds of questions focusing on *explanation* (explaining the phenomenon in videos), *counterfactual thinking* (answering what-if questions), *future prediction* (predicting future events), *domain expertise* (answering domain-specific inquiries), *temporal understanding* (reasoning about temporal information), and etc. A video example with these four questions from the Health & Medicine discipline is depicted in Figure 1. MMWorld comprises two datasets: a human-annotated dataset for evaluating MLLMs on the whole video and a synthetic dataset designed to analyze MLLMs’ perception within single visual or audio modalities. We evaluate 12 MLLMs that can handle videos or image sequences on MMWorld, including both open-source (e.g., Video-LLaVA-7B [Lin et al., ]) and proprietary models (GPT-4V [OpenAI, ] and Gemini [Team et al., ]).

We summarized the contributions and key findings as follows:

- •
We introduce MMWorld, a new benchmark designed to rigorously evaluate the capabilities of Multimodal Large Language Models (MLLMs) in world modeling through the realm of video understanding. MMWorld spans a broad spectrum of disciplines, featuring a rich array of question types for multi-faceted reasoning.
- •
In addition to the human-annotated dataset, we develop an automatic data collection pipeline, streamlining video content selection and question-answer generation, and construct a well-controlled synthetic dataset to analyze MLLMs within single visual or audio modalities.
- •
We observe that existing MLLMs still face substantial challenges posed by MMWorld. Even the best performer, GPT-4o, can only achieve a 52.30% overall accuracy, and four MLLMs particularly trained on videos perform worse than random chance.
- •
Although there is stll a clear gap between open-source and proprietary models, the best open-source model Video-LLaVA-7B outperforms GPT-4V and Gemini on Embodied Tasks by a large margin and performs similarly on Art & Sports, where spatiotemporal dynamics play a more crucial role in video understanding. This is further validated with its leading results on the Temporal Understanding question type.
- •
In our study comparing MLLMs with average humans (non-experts), we notice some correlation between question difficulties as perceived by humans and MLLMs. However, MLLMs present different skill sets than humans in that they can answer reasonable amount of difficult questions that humans completely fail but also struggle at easy questions that humans excel at. This indicates different perception, cognition, and reasoning abilities between MLLMs and humans.

**Table 1: Comparison between MMWorld and previous benchmarks for real-world video understanding on a variety of criteria. Multi-faced include Explanation (Explain.), Counterfactual Thinking (Counter.), Future Prediction (Future.) and Domain Expertise (Domain.) MMWorld is the first multi-discipline and multitask video understanding benchmark that covers wider reasoning questions, and also included first-party data annotations.**
| Benchmarks | $\begin{array}[]{l}\textbf{Multi-}\\<br>\textbf{Discipline}\\<br>\end{array}$ | $\begin{array}[]{l}\textbf{Multi-}\\<br>\textbf{Task}\\<br>\end{array}$ | Multi-Faceted Reasoning | $\begin{array}[]{l}\textbf{First-Party}\\<br>\textbf{Annotation}\\<br>\end{array}$ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $\begin{array}[]{c}\texttt{Explain.}\\<br>\end{array}$ | $\begin{array}[]{c}\texttt{Counter.}\\<br>\end{array}$ | $\begin{array}[]{c}\texttt{Future.}\\<br>\end{array}$ | $\begin{array}[]{c}\texttt{Domain.}\\<br>\end{array}$ |  |  |  |  |
| MovieQA [Tapaswi et al., ] |  |  | ✓ |  |  |  | ✓ |
| TVQA [Lei et al., ] |  |  | ✓ |  |  |  | ✓ |
| ActivityNet-QA [Yu et al., ] |  |  |  |  |  |  | ✓ |
| MSVD-QA [Xu et al., ] [Xu et al., ] |  |  |  |  |  |  | ✓ |
| MSRVTT-QA [Xu et al., ] |  |  |  |  |  |  | ✓ |
| Sports-QA [Li et al., ] |  |  |  | ✓ |  | ✓ | ✓ |
| VaTeX [Wang et al., ] |  | ✓ |  |  |  |  | ✓ |
| VALUE [Li et al., ] |  | ✓ |  |  |  |  |  |
| Video-Bench [Ning et al., ] |  | ✓ |  |  | ✓ | ✓ |  |
| MVBench [Li et al., ] |  | ✓ |  | ✓ | ✓ |  |  |
| Perception Test [Pătrăucean et al., ] |  | ✓ | ✓ | ✓ | ✓ |  |  |
| MMWorld (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

## 2 Related Work

### 2.1 Multimodal Large Language Models (MLLMs)

#### Emerging MLLMs

With recent breakthroughs [OpenAI, ; Google, ; Touvron et al., ; Chiang et al., ; Touvron et al., ; Bai et al., ] in Large Language Models (LLMs), several counterparts in the vision-and-language domain have been proposed [Dai et al., ; Liu et al., , ; Li et al., ; Zhu et al., ; Zheng et al., ; Bai et al., ], and recently released GPT-4V [OpenAI, ], followed by Gemini Vision family [Team et al., ].
Many MLLMs have expanded their capabilities beyond handling only text and image inputs. VideoChat [Li et al., ] leverages the QFormer [Li et al., ] to map visual representations to LLM [Chiang et al., ], and performs a multi-stage training pipeline. Otter [Li et al., ] proposes to conduct instruction finetuning based on Openflamingo [Awadalla et al., ]. PandaGPT [Su et al., ] employs the ImageBind [Han et al., ] as the backbone and finetunes it. mPLUG-Owl [Ye et al., ] introduces an abstractor module to perform visual and language alignment. VideoLLaMA [Zhang et al., ] introduces a frame embedding layer and also leverages ImageBind to inject temporal and audio information into the LLM backend. Chat-UniVi [Jin et al., ] uses clustering to do feature fusion. Observing their emerging abilities in multimodal video understanding, we propose MMWorld to evaluate these models’ skills in understanding the dynamics of the real world.

#### Benchmarking MLLMs

To evaluate MLLMs, there is a flourishing of analysis [Liu et al., ; Zhang et al., ; Jiang et al., ; Lu et al., ; Fan et al., ; Cui et al., ; Guan et al., ; Yu et al., ; Fu et al., ] and the establishment of innovative benchmarks such as VisIB-Bench [Bitton et al., ] which evaluates models with real-world instruction-following ability given image inputs, MMMU [Yue et al., ] designed to access models on college-level image-question pairs that span among different disciplines, and VIM [Lu et al., ] which challenges the model’s visual instruction following capability.
However, these recent analyses and benchmarks only cover the image input, which hinders the evaluation of MLLM’s performance as a world model. Recently, video benchmarks such as Perception Test [Pătrăucean et al., ] is proposed to focus on perception and skills like memory and abstraction. However, it uses scenarios with a few objects manipulated by a person, which limits the variety of contexts. MVBench [Li et al., ] centers on temporal understanding, while MMWorld not only includes temporal reasoning but also evaluates other multi-faceted reasoning abilities.

### 2.2 Video Understanding Benchmarks

Previous video benchmarks, as shown in Table 1, focus on video understanding tasks, including activity-focused on web videos [Yu et al., ], description-based question answering [Zeng et al., ], video completion [Fu et al., ], and video infilling [Himakunthala et al., ]. Recently, Video-Bench [Ning et al., ] introduces a benchmark by collecting videos and annotations from multiple existing datasets. LWM [Liu et al., ] collects a large video and language dataset from public books and video datasets and trains a world model that is capable of processing more than millions of tokens. However, modeling
millions of tokens is extremely difficult due to high memory cost, computational complexity, and lack
of suitable datasets. Mementos [Wang et al., ] builds a benchmark for MLLM reasoning for input image sequences. STAR [Wu et al., ] builds a benchmark for situated reasoning in real-world videos. CLEVER [Yi et al., ] builds a benchmark containing videos focusing on objects with simple visual appearance.
Our contribution, in contrast, presents a new video understanding benchmark designed to evaluate models on several pivotal components crucial for a comprehensive world model. These components encompass interdisciplinary coverage, task diversity, and multifaceted reasoning capabilities—including future prediction, counterfactual thinking, and more—underpinned by original human annotations and integrated domain knowledge.

## 3 The MMWorld Benchmark

The MMWorld benchmark is built on three key design principles: multi-discipline coverage and multi-faceted reasoning. It spans various disciplines that require domain expertise and incorporates diverse reasoning skills such as explanation, counterfactual thinking, and future prediction. The benchmark consists of two parts: a human-annotated dataset and a synthetic dataset. The human-annotated dataset serves as the main test bed to evaluate MLLMs from multiple perspectives. The synthetic dataset contains two subsets, focusing on evaluating MLLMs’ perception behavior from both visual signals and audio inputs, respectively.

### 3.1 Manual Data Collection

We collect videos from YouTube with the Creative Licence in seven disciplines:
Art $\&$ Sports (18.5%), Business (12.0%), Science (20.4%), Health $\&$ Medicine (12.0%), Embodied Tasks (12.0%%), Tech $\&$ Engineering (12.9%), and Game (12.2%). For Art $\&$ Sports, 29 videos are collected from the SportsQA dataset [Li et al., ]. And for Embodied Tasks, 24 videos are sourced from IKEA Assembly [Ben-Shabat et al., ], RT-1 [Brohan et al., ], and Ego4D [Grauman et al., ] datasets to increase video diversity.

Our manual benchmark collection takes two
stages. In the first stage, we conduct a detailed examination of each of the seven primary disciplines to identify a comprehensive range of subdisciplines for inclusion in our benchmark.
Our selection of videos is driven by three key principles:

- •
The first principle, multi-discipline coverage, emphasizes the requirement for domain knowledge—selecting videos that inherently demand an understanding of specialized content across various disciplines.
- •
The second principle, multi-faceted annotation, involves collecting videos that enable the creation of question-answer pairs from multiple perspectives to evaluate world model properties comprehensively.
- •
The third principle, temporal information, prioritizes the inclusion of videos that provide meaningful content over time, as understanding temporal information is crucial for grasping world dynamics. This allows models to engage in temporal reasoning.
Therefore, answering questions in our dataset requires implicit temporal reasoning, e.g., the model needs to understand temporal information to explain “why does the robot need to do the step shown in the video”. We also design a “temporal understanding” question type to explicitly test models’ ability to reason about temporal information (examples can be found in Section F in the Appendix).

During the second stage, our team embark on the task of question annotation. We craft questions that primarily test seven aspects of multimodal video understanding also from the perspective of multi-faceted reasoning: 1) Explanation: Questions ask the model to elucidate the underlying logic or purpose within the video;
2) Counterfactual Thinking: Tests the model’s ability to hypothesize and consider alternative outcomes;
3) Future Prediction: Aims to predict future events based on the current scenario, challenging the model’s foresight;
4) Domain Expertise: Evaluates the model’s depth of knowledge in specific fields, such as how to assemble a coffee table;
5) Temporal Understanding: Assesses the model’s capability to reason about temporal sequences and dynamics;
6) Attribution Understanding: These questions focus on identifying cause-and-effect relationships within the video, including tasks like counting;
7) Procedure Understanding: Tests the model’s ability to comprehend and explain procedural tasks shown in the video. The detailed distribution and examples are shown in Figure 2.

Figure: Figure 2: The questions in MMWorld primarily evaluate seven understanding and reasoning abilities of models. We give one example for each category.
Refer to caption: x2.png

### 3.2 Automated Data Collection

Understanding real-world dynamics requires models to process both audio and visual modalities. To evaluate MLLMs’ perception abilities in these modalities, we designed an automated data collection pipeline. This pipeline collects targeted videos and generates QA pairs based on either audio or visual information, ensuring the model’s capabilities are assessed independently for each modality. By using information from a single modality to generate QA pairs, our pipeline ensures that the synthetic data remains unbiased regarding input modality.

The synthetic data generation pipeline is illustrated in Figure 3. We employ a systematic approach to gather videos with Creative Commons licenses from YouTube and the extensive YouTube-8M dataset [Abu-El-Haija et al., ]. This method ensures a diverse and comprehensive collection of video data, which is important for the robust evaluation of multimodal video understanding models.

Figure: Figure 3: Schematic diagram of the synthetic data generation pipeline in MMWorld. It starts with generating subdiscipline-specific queries, followed by video retrieval from YouTube-8M [Abu-El-Haija et al., ] and YouTube. Keyframes are extracted for visual-based QA generation, and videos are transcribed using an ASR module for audio-based QA generation.
Refer to caption: x3.png

#### Video Collection and Processing

We start with the video Query Generator. We start with the same seven disciplines as the manually collected dataset. For each discipline, a set of subdisciplines is defined to encapsulate a wide spectrum of topics, ensuring a diverse and comprehensive dataset.
Once the queries are generated, the Video Mapping and Filtering step is initiated. We perform mapping of videos to YouTube-8M and online videos, constrained by a strict time limit of two minutes per query, keeping only the most pertinent videos that satisfy the predefined criteria. Simultaneously, the works in conjunction with the video transcripts to extract key terms and concepts. This iterative process refines the search parameters and enhances the semantic richness of the dataset by identifying and encoding the salient themes present in the videos. The Video Summarization module utilizes Query-focused video summarization techniques based on Katna(^2^22 [https://github.com/keplerlab/katna](https://github.com/keplerlab/katna)) and UniVTG [Lin et al., ]. This module selects ten representative frames from each video, distilling the essence of the content while preserving the narrative context. This summarization facilitates efficient storage and quicker processing times, which are crucial for large-scale analysis.

#### QA Generation

The final stage in our pipeline is the QA / Caption Generation module, where we leverage the capabilities of GPT-4V to generate accurate and contextually relevant questions and answers, as well as captions, based on the video frames and transcripts. This step not only provides rich annotations for each video but also equips the dataset with a multimodal dimension that supports various downstream tasks such as video QA, captioning, and more.

#### Quality of the Synthetic Dataset

Human evaluators were engaged to ascertain the reasonableness of automatically generated questions and answers, ensuring that the synthetic dataset maintains a high standard of quality and relevance. The findings from this human evaluation phase are detailed in Section D of the Appendix, offering insights into the dataset’s efficacy and the realism of its constructed queries and responses.

Table: Table 2: Key Statistics of the MMWorld Benchmark. The main subset is the human-annotated subset. Synthetic Subset I contains generated QA pairs focused exclusively on the audio content, while Synthetic Subset II contains QA pairs focused exclusively on the visual content of the video.

Finally, the statistics of automated curated data, which is used for the ablation study, are shown in Table 2.
The taxonomy of our dataset is shown in Figure 1. We note that only a portion of the subdisciplines are shown due to space concerns. Please refer to the Appendix for full information.

## 4 Experiments

### 4.1 Experimental Settings

In our study, we compare MLLM’s performance on the MMWorld benchmark, including GPT-4V [OpenAI, ], Gemini Pro [Team et al., ], Video-Chat [Li et al., ], Video-LLaMA [Zhang et al., ], ChatUnivi [Jin et al., ], mPLUG-Owl [Ye et al., ], Otter [Li et al., ], ImageBind-LLM [Han et al., ], PandaGPT [Su et al., ], LWM [Liu et al., ],
and X-Instruct-BLIP [Panagopoulou et al., ]. For both Gemini Pro and GPT-4V, we adhere to the default settings provided by their official APIs. They both take ten image frames extracted from the video content as the input. The Gemini Pro is set to process visual input and configured with safety settings to filter a range of harmful content. The configuration thresholds are set to ‘BLOCK_NONE’. For PandaGPT, we set ‘top_p’ to 0.7 and ‘temperature’ to 0.5. For VideoChat, we set ‘max_frames’ to 100. For X-Instruct-BLIP, the model is implemented using four image frames. We use GPT-4-32K as the judge for judging whether the model answer is correct when it can not mapped to the option letter using the rule-based method. For others, we all use the default setting. All inferences are run on a NVIDIA A6000 workstation. The detailed implementation is given in the Appendix.

**Table 3: MLLM accuracy across diverse disciplines (averaging over three runs). GPT-4V and Gemini Pro lead at most disciplines and achieve the best overall accuracy. The best open-source model Video-LLaVA-7B outperforms them on Embodied Tasks and perform similarly on Art & Sports.**
| Model | Art& | Business | Science | Health& | Embodied | Tech& | Game | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sports | Medicine | Tasks | Engineering |  |  |  |  |  |
| Random Choice | 25.03 | 25.09 | 26.44 | 25.00 | 26.48 | 30.92 | 25.23 | 26.31 |
| Proprietary MLLMs |  |  |  |  |  |  |  |  |
| GPT-4o [OpenAI, ] | 47.87 $\pm$1.47 | 91.14 $\pm$0.87 | 73.78 $\pm$2.88 | 83.33 $\pm$1.47 | 62.94 $\pm$3.47 | 75.53 $\pm$2.61 | 80.32 $\pm$2.05 | 62.54 $\pm$0.79 |
| Claude-3.5-Sonnet [Anthropic, ] | 54.58 $\pm$0.45 | 63.87 $\pm$0.40 | 59.85 $\pm$1.28 | 54.51 $\pm$1.28 | 30.99 $\pm$0.40 | 58.87 $\pm$0.61 | 59.44 $\pm$0.68 | 54.54 $\pm$0.29 |
| GPT-4V [OpenAI, ] | 36.17 $\pm$0.58 | 81.59 $\pm$1.74 | 66.52 $\pm$1.86 | 73.61 $\pm$0.49 | 55.48 $\pm$2.70 | 61.35 $\pm$1.00 | 73.49 $\pm$1.97 | 52.30 $\pm$0.49 |
| Gemini Pro [Team et al., ] | 37.12 $\pm$2.68 | 76.69 $\pm$2.16 | 62.81 $\pm$1.83 | 76.74 $\pm$1.30 | 43.59 $\pm$0.33 | 69.86 $\pm$2.01 | 66.27 $\pm$2.60 | 51.02 $\pm$1.35 |
| Open-source MLLMs |  |  |  |  |  |  |  |  |
| Video-LLaVA-7B [Lin et al., ] | 35.91 $\pm$0.96 | 51.28 $\pm$0.87 | 56.30 $\pm$0.76 | 32.64 $\pm$0.49 | 63.17 $\pm$1.44 | 58.16 $\pm$1.00 | 49.00 $\pm$3.16 | 44.60 $\pm$0.58 |
| Video-Chat-7B [Li et al., ] | 39.53 $\pm$0.06 | 51.05 $\pm$0.00 | 30.81 $\pm$0.21 | 46.18 $\pm$0.49 | 40.56 $\pm$0.57 | 39.36 $\pm$0.00 | 44.98 $\pm$0.57 | 40.11 $\pm$0.06 |
| ChatUnivi-7B [Jin et al., ] | 24.47 $\pm$0.49 | 60.84 $\pm$1.51 | 52.00 $\pm$0.73 | 61.11 $\pm$1.96 | 46.15 $\pm$2.06 | 56.74 $\pm$1.33 | 52.61 $\pm$2.84 | 39.47 $\pm$0.42 |
| mPLUG-Owl-7B  [Ye et al., ] | 29.16 $\pm$1.62 | 64.10 $\pm$1.84 | 47.41 $\pm$3.29 | 60.07 $\pm$1.30 | 23.78 $\pm$3.47 | 41.84 $\pm$5.09 | 62.25 $\pm$3.16 | 38.94 $\pm$1.52 |
| PandaGPT-7B [Su et al., ] | 25.33 $\pm$0.54 | 42.66 $\pm$3.02 | 39.41 $\pm$2.67 | 38.54 $\pm$3.07 | 35.43 $\pm$0.87 | 41.84 $\pm$2.79 | 40.16 $\pm$4.65 | 32.48 $\pm$0.45 |
| ImageBind-LLM-7B [Han et al., ] | 24.82 $\pm$0.16 | 42.66 $\pm$0.99 | 32.15 $\pm$1.11 | 30.21 $\pm$1.47 | 46.85 $\pm$1.14 | 41.49 $\pm$1.50 | 41.37 $\pm$0.57 | 31.75 $\pm$0.14 |
| X-Instruct-BLIP-7B [Panagopoulou et al., ] | 21.08 $\pm$0.27 | 15.85 $\pm$0.87 | 22.52 $\pm$1.11 | 28.47 $\pm$0.49 | 18.41 $\pm$1.44 | 22.34 $\pm$0.87 | 26.10 $\pm$0.57 | 21.36 $\pm$0.18 |
| LWM-1M-JAX [Liu et al., ] | 12.04 $\pm$0.53 | 17.48 $\pm$0.57 | 15.41 $\pm$0.91 | 20.49 $\pm$0.98 | 25.87 $\pm$1.98 | 21.99 $\pm$2.19 | 11.65 $\pm$3.01 | 15.39 $\pm$0.32 |
| Otter-7B [Li et al., ] | 17.12 $\pm$1.17 | 18.65 $\pm$0.87 | 9.33 $\pm$0.36 | 6.94 $\pm$0.98 | 13.29 $\pm$1.51 | 15.96 $\pm$1.74 | 15.26 $\pm$0.57 | 14.99 $\pm$0.77 |
| Video-LLaMA-2-13B [Zhang et al., ] | 6.15 $\pm$0.44 | 21.21 $\pm$0.66 | 22.22 $\pm$1.45 | 31.25 $\pm$1.70 | 15.38 $\pm$1.14 | 19.15 $\pm$1.74 | 24.90 $\pm$5.93 | 14.03 $\pm$0.29 |

### 4.2 Evaluation

Our dataset includes multiple-choice questions and captions corresponding to each video, enabling tasks such as video question answering and video captioning. We focus on video question answering by evaluating a model’s performance based on its accuracy in selecting the correct answer from the provided options. One challenge lies in reliably parsing the model’s response to map it to one of the predefined choices. To address this, we employ two mapping strategies. We employ two mapping strategies. The first method employs automated scripts to parse the models’ predictions and compare the parsed results with the ground truth, similar to the approach used in [Yue et al., ]. The second method involves models freely generating answers, which are then evaluated by GPT-4. Given the question, correct answer, and model’s prediction, GPT-4 returns a True or False judgment. This approach is based on recent works in model evaluation [Maaz et al., ; Hsu et al., ; Hackl et al., ; Liu et al., ]. We validated this method with human evaluators, showing an error rate of 4.76% across 189 examples, confirming the effectiveness of GPT-4 as an evaluator. Detailed results for human evaluation and for these two different strategies are provided in Appendix B. In the main paper, all results are evaluated using the second approach.

### 4.3 Main Evaluation Results

We show in Table 3 the main evaluation results of different MLLMs. Among these, GPT-4V emerges as the top performer, closely followed by Gemini Pro. Video-LLaVA also demonstrates strong results, primarily due to the extensive training data which consists of 558K LAION-CCSBU image-text pairs and 702K video-text pairs from WebVid [Bain et al., ]. For instruction tuning, datasets were gathered from two sources: a 665K image-text instruction dataset from LLaVA v1.5 and a 100K video-text instruction dataset from Video-ChatGPT [Maaz et al., ]. This superior performance may also be attributed to Video-LLaVA’s adoption of CLIP ViT-L/14 trained in LanguageBind [Lin et al., ] as its vision model and the inclusion of a large volume of image-video-text pairings within the
training data. On the other hand, models like Otter and LWM perform poorly across most disciplines, possibly due to their weaker backbone and architecture used. Otter uses the LLaMA-7B language encoder and a CLIP ViT-L/14 vision encoder, both of which are frozen, with only the Perceiver resampler module fine-tuned, which may contribute to its lower performance.
Additionally, some MLLMs perform even worse than random, highlighting the challenging nature of MMWorld.

Figure: Figure 4: Results of different MLLMs on multi-faceted reasoning. The detailed performance numbers can be found in the Appendix.
Refer to caption: x4.png

### 4.4 Study on Multi-faceted Reasoning on MMWorld

Figure 4 illustrates the multi-faceted reasoning performance for each MLLM. GPT-4V emerges as the strongest model across Future Prediction, Domain Expertise, and Attribution Understanding. Closed-source models like GPT-4V and Gemini Pro perform similarly on counterfactual thinking and outperform all others. However, for temporal understanding, Video-LLaVA performs the best. This may be due to its extensive training on large amounts of video-language data, which enhances its spatio-temporal reasoning abilities. This can be also observed in its high scores on the Art & Sports and Embodied Tasks, which involve dense spatio-temporal information, as shown in Table 3. Video-LLaVA’s performance is comparable to GPT-4V and Gemini on explanation tasks, likely because of its two-stage training process and exposure to a large amount of instruction-tuning data in the second stage, which includes similar instructions.

### 4.5 Study on MLLM Performance at Different Difficulty Levels for Average Humans

Figure 5(a) indicate some correlation between the difficulty levels as perceived by humans and the performance of MLLMs. MLLMs generally follow a trend where accuracy decreases as the difficulty level increases, which aligns with human performance patterns. However, the correlation is not perfect, suggesting that while models and humans share some common ground in understanding question difficulty, there are also notable differences in their capabilities. The data reveals that MLLMs exhibit different skill sets compared to humans. As highlighted in Figure 5(b), models like GPT-4V can correctly answer expert-level questions that humans often get wrong, particularly in disciplines such as Business and Health & Medicine, where humans often struggle, yet they sometimes falter on easier questions, likely due to the lack of contextual understanding. Notably, discrepancies in disciplines like Art & Sports and Tech & Engineering highlight areas where MLLMs’ performance does not align with human results, suggesting different perception, cognition, and reasoning abilities in handling abstract concepts. These differences suggest that MLLMs can complement human capabilities, offering potential for enhanced task performance by combining the data-driven insights of models with human intuition and contextual knowledge.

Figure: (a) Accuracy of MLLMs at difficulty levels.
Refer to caption: x5.png

### 4.6 Study on Modality of Perception

We conduct ablations to evaluate MLLMs ability to perceiving the world on the synthetic dataset of MMWorld. With our synthetic dataset, we considered scenarios where only one modality—either audio or visual—is available. Table 4 shows the results which evaluates the model’s ability to interpret spoken language, background noises, and other audio elements without the aid of visual context and the model’s perception ability to operate without any audio input. For the visual perception test, Gemini Pro performed the best, demonstrating its strong ability to process visual information. Interestingly, Video-Chat exhibited better audio perception than ChatUnivi, despite its poorer visual perception. This may be attributed to its use of the Whisper [Radford et al., ] speech recognition model. It also explains that in Table 3, Video-Chat outperforms ChatUnivi in the Art & Sports discipline, which requires a greater understanding of music, voice, and background audio. However, in other disciplines such as Science and Health & Medicine, Video-Chat’s performance is significantly poorer.

**Table 4: Performance on Synthetic Subset I (Audio) and II (Visual). Synthetic Subset I contains QAs based solely on the audio content, while Synthetic Subset II focuses exclusively on the visual content of the video. We evaluated four MLLMs processing both audio and visual inputs along with Gemini Pro (for the audio setting, only providing the question).**
| Model | Art&Sports | Business | Science | Health&Medicine | Embodied Tasks | Tech&Engineering | Game | Average |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Audio | Visual | Audio | Visual | Audio | Visual | Audio | Visual | Audio | Visual | Audio | Visual | Audio | Visual | Audio | Visual |  |
| Random Choice | 31.59 | 30.14 | 31.18 | 26.58 | 36.98 | 32.89 | 38.74 | 32.64 | 32.81 | 31.25 | 27.23 | 32.60 | 32.01 | 30.78 | 32.44 | 30.91 |
| \hdashlineVideo-Chat [Li et al., ] | 33.98 | 32.48 | 46.47 | 41.46 | 41.86 | 39.15 | 45.95 | 36.81 | 32.81 | 46.88 | 37.48 | 35.91 | 32.98 | 46.70 | 38.82 | 39.07 |
| ChatUnivi [Jin et al., ] | 30.03 | 43.22 | 30.19 | 52.85 | 38.75 | 54.59 | 34.76 | 50.69 | 20.14 | 40.63 | 24.17 | 46.41 | 29.98 | 45.44 | 31.82 | 48.44 |
| Video-LLaMA [Zhang et al., ] | 30.15 | 30.23 | 36.18 | 33.17 | 31.33 | 31.34 | 30.90 | 32.78 | 33.13 | 30.05 | 31.18 | 30.55 | 20.49 | 27.20 | 29.08 | 30.47 |
| Otter [Li et al., ] | 14.22 | 16.82 | 16.77 | 14.24 | 16.12 | 17.00 | 19.82 | 13.19 | 10.94 | 12.50 | 15.63 | 12.43 | 6.65 | 10.44 | 12.83 | 13.41 |
| Gemini Pro [Team et al., ] | 20.88 | 61.38 | 29.43 | 77.35 | 30.62 | 74.26 | 30.14 | 81.53 | 22.57 | 70.31 | 18.83 | 66.22 | 29.96 | 65.01 | 24.45 | 69.97 |

### 4.7 Error Analysis

To gain deeper insights into the limitations of MLLMs, we prompted the models to explain the reasoning behind their choices, particularly when errors occurred. Through this analysis, we identified common error patterns and summarized them into seven distinct categories. We conducted a simple test where the same questions that triggered errors in GPT-4V were also posed to other MLLMs. The frequencies of each type of error are presented in Figure 6, as annotated by human evaluators. Detailed qualitative examples of these errors and further analysis are provided in the Appendix.

Figure: Figure 6: The frequency of different error types across various MLLMs. For each error type, 10 examples were evaluated. Error types are abbreviated as follows: QUE (Question Understanding Error), AUE (Audio Understanding Error), VPE (Visual Perception Error), HE (Hallucination Error), RE (Reasoning Error), LDK (Lack of Domain Knowledge), and RA (Reject to Answer).
Refer to caption: x7.png

## 5 Conclusion

Our MMWorld Benchmark represents a significant step forward in the quest for advanced multi-modal language models capable of understanding complex video content. By presenting a diverse array of videos across seven disciplines, accompanied by questions that challenge models to demonstrate explanation, counterfactual thinking, future prediction, and domain expertise, we have created a rigorous testing ground for the next generation of AI. While using LLMs for data generation can introduce hallucination issues, these challenges are manageable and are commonly addressed [Wang et al., ; Shen et al., ]. Another potential risk is the misuse of MLLMs for surveillance or privacy invasion. The ability of models to understand video content and perform reasoning could be exploited to monitor individuals without their consent, leading to serious ethical and legal concerns regarding privacy.

## Appendix A Overview of the Appendix

We host the project website on [https://mmworld-bench.github.io/](https://mmworld-bench.github.io/). The benchmark and code implementations can be found at [https://github.com/eric-ai-lab/MMWorld](https://github.com/eric-ai-lab/MMWorld). The link to Croissant metadata record documenting the dataset/benchmark available for viewing and downloading is available at [https://github.com/eric-ai-lab/MMWorld/blob/main/data/croissanta_hf_data.json](https://github.com/eric-ai-lab/MMWorld/blob/main/data/croissanta_hf_data.json).
This Appendix is organized as follows:

- •
Section B contains additional experimental results;
- •
Section C contains the implementation details;
- •
Section D contains the settings and results from human evaluations;
- •
Section E contains the error analysis;
- •
Section F contains the data examples from MMWorld;
- •
Section G contains additional data statistics of MMWorld;
- •
Section H contains the datasheet of MMWorld;
- •
Section I contains the author statement, licence, and maintenance plan.

## Appendix B Additional Results

### B.1 Results Across Different Seed for Each Model

In Table 5, we show detailed results using three different seeds for each evaluated models.

**Table 5: Detailed results of model performance, measured as accuracy percentages across diverse disciplines for three runs. The random choice baseline involves shuffling candidate answers for each video question before consistently selecting answer ‘a’. GPT-4V and Gemini Pro utilize 10 image frames extracted from the video content.**
| Model | Art& | Business | Science | Health& | Embodied | Tech& | Game | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sports | Medicine | Tasks | Engineering |  |  |  |  |  |
| GPT-4o-seed 1 [OpenAI, ] | 47.10 | 92.31 | 75.11 | 81.25 | 65.03 | 72.34 | 78.31 | 62.22 |
| GPT-4o-seed 2 [OpenAI, ] | 46.58 | 90.91 | 69.78 | 84.38 | 65.73 | 75.53 | 83.13 | 61.77 |
| GPT-4o-seed 3 [OpenAI, ] | 49.94 | 90.21 | 76.44 | 84.38 | 58.04 | 78.72 | 79.52 | 63.63 |
| Claude-3.5-seed 1 [Anthropic, ] | 54.32 | 64.34 | 59.11 | 53.12 | 30.77 | 59.57 | 59.04 | 54.27 |
| Claude-3.5-seed 2 [Anthropic, ] | 54.32 | 63.64 | 61.33 | 54.17 | 30.77 | 58.51 | 59.04 | 54.52 |
| Claude-3.5-seed 3 [Anthropic, ] | 55.10 | 63.64 | 59.11 | 56.25 | 31.47 | 58.51 | 60.24 | 54.84 |
| GPT-4V-seed 1 [OpenAI, ] | 36.90 | 79.72 | 64.00 | 73.96 | 51.75 | 60.64 | 71.08 | 51.64 |
| GPT-4V-seed 2 [OpenAI, ] | 35.48 | 83.92 | 68.44 | 73.96 | 58.04 | 60.64 | 75.90 | 52.79 |
| GPT-4V-seed 3 [OpenAI, ] | 36.13 | 81.12 | 67.11 | 72.92 | 56.64 | 62.77 | 73.49 | 52.47 |
| Gemini Pro-seed 1 [Team et al., ] | 40.90 | 79.72 | 60.44 | 78.12 | 43.36 | 71.28 | 65.06 | 52.92 |
| Gemini Pro-seed 2 [Team et al., ] | 35.10 | 75.52 | 63.11 | 75.00 | 44.06 | 71.28 | 69.88 | 50.16 |
| Gemini Pro-seed 3 [Team et al., ] | 35.35 | 74.83 | 64.89 | 77.08 | 43.36 | 67.02 | 63.86 | 49.97 |
| Video-LLaVA-seed 1 [Lin et al., ] | 34.58 | 51.05 | 57.33 | 32.29 | 61.54 | 57.45 | 50.60 | 43.94 |
| Video-LLaVA-seed 2 [Lin et al., ] | 36.77 | 52.45 | 56.00 | 32.29 | 65.03 | 57.45 | 51.81 | 45.35 |
| Video-LLaVA-seed 3 [Lin et al., ] | 36.39 | 50.35 | 55.56 | 33.33 | 62.94 | 59.57 | 44.58 | 44.52 |
| Video-Chat-seed 1 [Li et al., ] | 39.48 | 51.05 | 30.67 | 46.88 | 39.86 | 39.36 | 44.58 | 40.03 |
| Video-Chat-seed 2 [Li et al., ] | 39.48 | 51.05 | 30.67 | 45.83 | 41.26 | 39.36 | 45.78 | 40.15 |
| Video-Chat-seed 3 [Li et al., ] | 39.61 | 51.05 | 31.11 | 45.83 | 40.56 | 39.36 | 44.58 | 40.15 |
| mPLUG-Owl-seed 1  [Ye et al., ] | 31.35 | 65.73 | 45.78 | 61.46 | 28.67 | 48.94 | 65.06 | 41.05 |
| mPLUG-Owl-seed 2 [Ye et al., ] | 28.65 | 65.03 | 44.44 | 58.33 | 21.68 | 37.23 | 57.83 | 37.52 |
| mPLUG-Owl-seed 3  [Ye et al., ] | 27.48 | 61.54 | 52.00 | 60.42 | 20.98 | 39.36 | 63.86 | 38.23 |
| ChatUnivi-seed 1 [Jin et al., ] | 24.13 | 60.14 | 52.00 | 62.50 | 48.95 | 56.38 | 56.63 | 39.77 |
| ChatUnivi-seed 2 [Jin et al., ] | 25.16 | 62.94 | 51.11 | 62.50 | 44.06 | 58.51 | 50.60 | 39.77 |
| ChatUnivi-seed 3 [Jin et al., ] | 24.13 | 59.44 | 52.89 | 58.33 | 45.45 | 55.32 | 50.60 | 38.87 |
| PandaGPT-seed 1 [Su et al., ] | 26.06 | 44.06 | 38.22 | 41.67 | 35.66 | 39.36 | 42.17 | 32.97 |
| PandaGPT-seed 2 [Su et al., ] | 24.77 | 45.45 | 36.89 | 34.38 | 34.27 | 40.43 | 44.58 | 31.88 |
| PandaGPT-seed 3 [Su et al., ] | 25.16 | 38.46 | 43.11 | 39.58 | 36.36 | 45.74 | 33.73 | 32.58 |
| ImageBind-LLM-seed 1 [Han et al., ] | 24.77 | 41.96 | 30.67 | 31.25 | 46.85 | 43.62 | 40.96 | 31.62 |
| ImageBind-LLM-seed 2 [Han et al., ] | 25.03 | 41.96 | 32.44 | 31.25 | 45.45 | 40.43 | 40.96 | 31.69 |
| ImageBind-LLM-seed 3 [Han et al., ] | 24.65 | 44.06 | 33.33 | 28.12 | 48.25 | 40.43 | 42.17 | 31.94 |
| X-Instruct-BLIP-seed 1 [Panagopoulou et al., ] | 21.42 | 14.69 | 22.22 | 29.17 | 16.78 | 21.28 | 26.51 | 21.23 |
| X-Instruct-BLIP-seed 2 [Panagopoulou et al., ] | 20.77 | 16.78 | 24.00 | 28.12 | 20.28 | 22.34 | 25.30 | 21.62 |
| X-Instruct-BLIP-seed 3 [Panagopoulou et al., ] | 21.03 | 16.08 | 21.33 | 28.12 | 18.18 | 23.40 | 26.51 | 21.23 |
| LWM-seed 1 [Liu et al., ] | 11.35 | 18.18 | 16.44 | 19.79 | 24.48 | 24.47 | 10.84 | 15.20 |
| LWM-seed 2 [Liu et al., ] | 12.13 | 17.48 | 15.56 | 19.79 | 24.48 | 22.34 | 8.43 | 15.14 |
| LWM-seed 3 [Liu et al., ] | 12.65 | 16.78 | 14.22 | 21.88 | 28.67 | 19.15 | 15.66 | 15.84 |
| Otter-seed 1 [Li et al., ] | 18.45 | 19.58 | 8.89 | 8.33 | 14.69 | 15.96 | 14.46 | 15.84 |
| Otter-seed 2 [Li et al., ] | 17.29 | 17.48 | 9.33 | 6.25 | 13.99 | 18.09 | 15.66 | 15.14 |
| Otter-seed 3 [Li et al., ] | 15.61 | 18.88 | 9.78 | 6.25 | 11.19 | 13.83 | 15.66 | 13.98 |
| Video-LLaMA-seed 1 [Zhang et al., ] | 5.55 | 21.68 | 24.00 | 29.17 | 15.38 | 21.28 | 18.07 | 13.66 |
| Video-LLaMA-seed 2 [Zhang et al., ] | 6.58 | 20.28 | 20.44 | 31.25 | 13.99 | 17.02 | 32.53 | 14.05 |
| Video-LLaMA-seed 3 [Zhang et al., ] | 6.32 | 21.68 | 22.22 | 33.33 | 16.78 | 19.15 | 24.10 | 14.37 |

### B.2 Results from Amazon Turkers

Table 6 presents the evaluation results from three sets of Amazon Turkers across various disciplines. The results indicate that there is slightly variability in performance across different human evaluators.

**Table 6: Performance of different set of turkers**
| Model | Art& | Business | Science | Health& | Embodied | Tech& | Game& | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sports | Medicine | Tasks | Engineering |  |  |  |  |  |
| Turker Set 1 | 25.224 | 39.860 | 32.444 | 40.625 | 51.049 | 50.000 | 40.964 | 33.227 |
| Turker Set 2 | 30.452 | 46.154 | 35.556 | 42.708 | 53.846 | 51.064 | 46.988 | 37.652 |
| Turker Set 3 | 26.710 | 41.958 | 36.889 | 46.875 | 53.147 | 42.553 | 38.554 | 34.830 |

### B.3 Results for the Two Different Evaluation Strategies

**Table 7: Performance of different MLLMs across different disciplines.**
| Model | Art& | Business | Science | Health& | Embodied | Tech& | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Sports | Medicine | Tasks | Engineering |  |  |  |  |
| Video-Chat (Open-ended) [Li et al., ] | 27.484 | 9.091 | 18.137 | 10.417 | 29.371 | 19.149 | 22.887 |
| Video-Chat  [Li et al., ] | 39.355 | 48.951 | 31.863 | 45.833 | 39.161 | 38.298 | 39.588 |
| Video-LLaMA (Open-ended) [Zhang et al., ] | 5.419 | 27.972 | 24.020 | 31.250 | 11.816 | 15.957 | 16.096 |
| Video-LLaMA [Zhang et al., ] | 27.355 | 31.469 | 31.373 | 48.958 | 16.084 | 28.723 | 28.729 |
| ChatUnivi (Open-ended)  [Jin et al., ] | 21.161 | 61.538 | 42.157 | 61.458 | 30.070 | 37.234 | 32.646 |
| ChatUnivi  [Jin et al., ] | 12.387 | 58.042 | 50.000 | 60.417 | 30.070 | 43.617 | 29.072 |
| Otter (Open-ended) [Li et al., ] | 37.677 | 32.867 | 37.255 | 32.292 | 22.378 | 27.660 | 34.639 |
| Otter [Li et al., ] | 17.677 | 16.783 | 12.255 | 5.208 | 17.483 | 15.957 | 15.876 |
| ImageBind-LLM (Open-ended) [Han et al., ] | 3.355 | 3.497 | 14.706 | 10.417 | 21.678 | 18.085 | 8.179 |
| ImageBind-LLM [Han et al., ] | 23.742 | 34.965 | 51.471 | 33.333 | 48.951 | 56.383 | 33.952 |
| PandaGPT (Open-ended) [Su et al., ] | 22.581 | 16.084 | 24.020 | 21.875 | 19.580 | 21.277 | 21.718 |
| PandaGPT [Su et al., ] | 27.613 | 44.056 | 39.706 | 25.000 | 40.559 | 21.277 | 31.615 |
| LWM (Open-ended) [Liu et al., ] | 16.000 | 20.979 | 14.706 | 16.667 | 19.580 | 20.213 | 16.976 |
| LWM [Liu et al., ] | 16.387 | 18.182 | 18.137 | 19.792 | 22.378 | 21.277 | 17.938 |
| X-Instruct-BLIP (Open-ended) [Panagopoulou et al., ] | 3.613 | 11.888 | 14.706 | 25.000 | 17.483 | 13.830 | 9.416 |
| X-Instruct-BLIP [Panagopoulou et al., ] | 19.355 | 13.287 | 22.549 | 29.167 | 18.881 | 14.894 | 19.519 |

In Table 7, we give additional evaluation results for different MLLMs evaluated in this paper. For closed-source models, the evaluation pipeline is the one used in the main paper, which involves utilizing GPT-4V as a judger. The process consists of presenting GPT-4V with the question, a corresponding answer generated by the baseline model, and the set of possible options. GPT-4V then assesses whether the model-generated answer is accurate within the given context; Another is open-ended generation where we employ a two-step methodology. We first prompt each model to do open-ended generation. Subsequently, we prompt the model to align its generative response with one of the predefined options: ‘a’, ‘b’, ‘c’, or ‘d’.

### B.4 Detailed Results on Multi-faceted Reasoning

**Table 8: Detailed results of different MLLMs on multi-faceted reasoning.**
| Model | Explanation | Counterfactual | Future | Domain | Attribution | Temporal |
| --- | --- | --- | --- | --- | --- | --- |
| Thinking | Prediction | Expertise | Understanding | Understanding |  |  |
| Proprietary Models |  |  |  |  |  |  |
| GPT-4V | 44.90 | 64.90 | 78.59 | 61.07 | 59.61 | 27.17 |
| Gemini Pro | 48.58 | 65.49 | 65.45 | 53.87 | 43.92 | 24.65 |
| Open-source Models |  |  |  |  |  |  |
| Video-LLaVA-7B | 42.46 | 42.55 | 64.96 | 47.86 | 36.86 | 34.45 |
| VideoChat-7B | 41.66 | 43.73 | 45.74 | 40.95 | 30.59 | 25.77 |
| ImageBind-LLM-7B | 29.51 | 26.86 | 50.61 | 33.93 | 34.90 | 19.89 |
| PandaGPT-7B | 29.55 | 37.45 | 46.47 | 33.93 | 26.27 | 28.01 |
| ChatUnivi-7B | 33.91 | 48.82 | 61.80 | 45.95 | 33.33 | 22.97 |
| VideoLLaMA-2-13B | 10.55 | 23.92 | 25.30 | 16.31 | 8.63 | 6.16 |
| X-Instruct-BLIP-7B | 23.05 | 15.29 | 27.25 | 21.07 | 24.31 | 11.20 |
| LWM-1M-JAX | 11.62 | 18.82 | 30.66 | 17.98 | 21.57 | 7.00 |
| Otter-7B | 16.91 | 10.98 | 15.82 | 13.10 | 17.65 | 9.52 |
| mPLUG-Owl-7B | 35.20 | 49.61 | 55.47 | 47.74 | 24.71 | 20.17 |

In Table 8, we give detailed performance numbers of different MLLMs on multi-faceted reasoning corresponding to Figure 4 in the main paper.

## Appendix C Implementation Details

We use the optimum number of video frames and report the performance in the main paper. The numbers of the sampled frames are 10 for GPT-4V/o and Gemini Pro, 8 for Video-LLaVA, 32 for ChatUniVi. For closed-source models, for both Gemini Pro and GPT-4V, we use the default settings provided by their official APIs. We use Katna (^3^33 https://github.com/keplerlab/katna) to extract key video frames as input to these two models. The Gemini Pro is set to process visual input and configured with safety settings to filter a range of harmful content. The configuration thresholds are set to ‘BLOCK_NONE’. For PandaGPT, we set ‘top_p’ to 0.7, and ‘temperature’ to 0.5. For VideoChat, we set ‘max_frames’ to 100. For LWM, we use the LWM-Chat-1M variant. For X-Instruct-BLIP, the model is implemented using four image frames. For Otter, we use the video variant. We use GPT-4-32K as the judge for judging whether the model answer is correct when it can not mapped to the option letter using the rule-based method. The prompt provided to GPT-4-32K is structured as follows: "I will present a response from a question-answering model alongside several answer options. Your task is to evaluate the response and determine which of the following options it most closely aligns with, denoting the most similar option by its corresponding letter (a, b, c, or d).".

#### Query Generation in Synthetic Data Generation Pipeline

For the discipline of Science, queries are generated for subdisciplines such as Geography, Chemistry, Wildlife Restoration, Mycology, Nature, Physics, Weather, Zoology, Math, Botany, Biology, and Geology. In the Tech & Engineering discipline, our queries span across Electronics, Animal Behavior, Mechanical Engineering, Energy & Power, Architecture, Agriculture, Nature, Physics, Robotics, Woodworking, and Gardening. The Sports & Arts discipline encompasses a broad range of cultural and physical activities, including Music, Drawing and Painting, Football, Volleyball, Aerobic Gymnastics, Basketball, Instrument, Baking, Dance, Woodworking, Graffiti, Anatomy, and additional Music-related topics. Embodied Tasks are represented through queries for Assembly, Ego-motion, and Single Object Manipulation, focusing on the interaction between agents and their physical environment. The Health & Medicine discipline is segmented into Pharmacy, Public Health, Clinical Medicine, and Basic Medical Science, reflecting the multifaceted nature of healthcare and medical studies.
The Business discipline is stratified into fundamental areas such as accounting, finance, management, marketing, and economics, each representing key facets of the commercial and economic world. Lastly, the Game discipline consists of Role Playing Game, First Person Shooting game, Racing Game, Adventure Game, Real-Time Strategy Game, Tower Defense game, and Fighting Game.

Each generated query retrieves relevant video content, which is then filtered and processed to align with the specific needs of our research objectives. Videos that meet our criteria in terms of content, length, and quality are downloaded and incorporated into our dataset, forming the basis for subsequent analysis and model training.

Figure: Figure 7: The interface of using Amazon Mechanical Turk to do human evaluation.
Refer to caption: extracted/5762879/figures/amt_img.png

Figure: Figure 8: Human evaluation interface for GPT judger.
Refer to caption: extracted/5762879/figures/human_eval.png

**Table 9: Category-wise and overall error rates**
| Category | Incorrect/Total | Error Rate (%) |
| --- | --- | --- |
| Sports & Arts | 5/62 | 8.06 |
| Health & Medicine | 2/7 | 28.57 |
| Science | 1/52 | 1.92 |
| Robotics | 0/12 | 0.00 |
| Business | 0/10 | 0.00 |
| Tech & Engineering | 1/46 | 2.17 |
| Overall | 9/189 | 4.76 |

Figure: Figure 9: Examples from MMWorld in the Embodied Tasks discipline.
Refer to caption: x8.png

Figure: Figure 10: Examples from MMWorld in the Tech & Engineering discipline.
Refer to caption: x9.png

Figure: Figure 11: Examples from MMWorld in the Science discipline.
Refer to caption: x10.png

Figure: Figure 12: Examples from MMWorld in the Business discipline.
Refer to caption: x11.png

Figure: Figure 13: Examples from MMWorld in the Arts & Sports discipline.
Refer to caption: x12.png

Figure: Figure 14: Examples from MMWorld of explicit temporal understanding and implicit temporal understanding (e.g., in explanation).
Refer to caption: x13.png

## Appendix D Human Evaluation

### D.1 Quality of Data

We hired Amazon Mechanical Turk to do human evaluation on the data with the results shown in Table 6. Workers were required to have completed more than 1000 Human Intelligence Tasks (HITs) and have an HIT approval rate greater than 95% to qualify for our tasks. We show in Figure 7 the human evaluation interface on the generated data. Each worker was compensated $0.20$ for completing an assignment. This amount was determined based on the estimated time and effort required to complete each task. We set the number of unique workers per task to 3 to collect diverse perspectives while avoiding redundancy. Workers were given 1 hour to complete each assignment. This time frame was chosen to enable thoughtful responses from workers.

**Table 10: Comparison of Human Evaluation on subset of 75 videos.**
| Model | Art& | Business | Science | Health& | Embodied | Tech& | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Sports | Medicine | Tasks | Engineering |  |  |  |  |
| Human Evaluation | 31.183 | 59.782 | 42.103 | 48.858 | 56.429 | 50.134 | 43.758 |
| \hdashlineGPT-4V [OpenAI, ] | 30.399 | 89.203 | 68.731 | 80.059 | 38.432 | 69.108 | 48.793 |
| Gemini-Pro [Team et al., ] | 28.745 | 80.909 | 69.425 | 80.023 | 50.987 | 80.479 | 48.083 |

We also hired students from campus to do human evaluation on subset of the data. The results are shown in Table 10. The performance of the human evaluators did not surpass that of GPT-4V and Gemini-Pro. This outcome underscores the challenging nature of the dataset, which often necessitates specialized domain knowledge that our evaluators—primarily non-experts—found demanding. These results highlight the complexity of the questions and the potential necessity for discipline-specific understanding to achieve high accuracy

### D.2 Quality of Using GPT as the Judger

For a comprehensive assessment of GPT-4V’s accuracy when using it as the judger, we devised a human evaluation protocol also resort to Amazon
Mechanical Turk, as visualized in Figure 8. The evaluators present a series of statements derived from the video, and GPT-4V is tasked with selecting the most accurate answer from a set of multiple-choice questions. Through this interface, human evaluators can efficiently gauge GPT-4V’s performance across different types of questions—when using it as the judger.

The results obtained from this human evaluation process are shown in Table 9, across 189 examples, there are only 9 incorrect ones with the error rate of 4.76%, validating the effectiveness of using GPT-4V as the judger.

Figure: Figure 15: Error pattern of lack of domain knowledge and reasoning error. In the first case, the model does not give the correct answer because of lacking the domain knowledge. In the second case, the model makes the wrong reasoning.
Refer to caption: x14.png

Figure: Figure 16: Error pattern of question understanding, visual perceptron error, and hallucinations. In the first case, the model does not understand the question correctly where the model fails to accurately discern the query regarding the pendulum’s period. In the second scenario, the model erroneously identifies objects within the visual input, leading to the hallucination of non-existent elements, such as a red box.
Refer to caption: x15.png

Figure: Figure 17: Error pattern of reject to answer and audio understanding error. In the first case, the model will reject to answer because of the dangerous action showing in the video. In the second case, the model does not understand the audio correctly as the answer to the question is given in the audio of the video.
Refer to caption: x16.png

Figure: Figure 18: Error pattern due to visual perception inaccuracies and insufficient domain knowledge. The first case demonstrates a visual perception error where the model incorrectly identifies the number of capacitors present. The second case showcases a compound error where the model not only fails to discern the colors indicative of different gases but also lacks the domain knowledge necessary to infer their identity correctly.
Refer to caption: x17.png

Figure: Figure 19: The number of videos per subdiscipline in MMWorld. Each horizontal bar indicates the quantity of videos corresponding to a subdiscipline, showcasing the dataset’s diversity and coverage across various domains of knowledge. Synthetic Subset I is collected with audio-only data and Synthetic Subset II is collected with visual-only data.
Refer to caption: x18.png

Figure: Figure 20: The distribution statistics of questions in the MMWorld benchmark by annotations.
Refer to caption: x19.png

## Appendix E Error Analysis

In this section, we delve into the analysis of errors from evaluated MLLMs. We summarized error types as follows:

Question Understanding Error (QUE): Models misinterpret the question’s intent, such as misunderstanding how a pendulum’s period would change if a condition in the scenario is altered.

Audio Understanding Error (AUE): Models fail to interpret audio cues correctly, shown by their failure to recognize blue and red lines on a stock chart.

Visual Perception Error (VPE): There is a misinterpretation of visual content, leading to incorrect assumptions about the visual data presented in the video.

Hallucinations (HE): Models generate content or details that are not present in the actual data, essentially ‘hallucinating’ information.

Reasoning Error (RE): Models demonstrate a lack of logical reasoning, leading to incorrect conclusions based on the given data.

Lack of Domain Knowledge (LDK): Models show an inability to answer questions that require specific domain expertise, indicating a gap in their knowledge.

Reject to Answer (RA): An example of this error was observed when the model was asked to select an answer regarding the outcome of an experiment involving liquid nitrogen. Instead of choosing an option, the model provided an unrelated response concerning a light bulb, indicating either a misunderstanding or a cautious approach due to the potential for the question to be interpreted as pertaining to a sensitive topic, which can trigger content filters focused on safety and compliance policies.

We show in Figure 15, 16, 17, 18 some error cases of Question Understanding Error, Audio Understanding Error, Visual Perception Error, Hallucinations, Reasoning Error, Lack of Domain Knowledge, and Reject to Answer respectively from MLLMs evaluated on MMWorld.

## Appendix F Data Examples

We show in Figure 9, 10, 11, 12, 13, 14 some additional examples from MMWorld.

## Appendix G Additional Data Statistics

For human annotated dataset, the length of each video was capped at approximately two minutes. The statistical distribution of the disciplines within the dataset for this part is as follows:

- •
*Sports & Arts*: The subset that consists of 77 videos, showcasing a vibrant collection that covers a wide range of topics from athletic endeavors to various forms of artistic expression.
- •
*Science*: A subset of 75 videos, which delves into the empirical world of scientific inquiry, spanning a multitude of specializations from fundamental physics to advanced biological studies.
- •
*Tech & Engineering*: Encompassing 54 videos, this segment captures the cutting-edge advancements and foundational concepts that drive innovation and infrastructure in the modern world.
- •
*Embodied Tasks*: With 50 videos, the dataset provides a focused insight into the dynamic field of Embodied Tasks, highlighting the intersection of AI, mechanics, and automation.
- •
*Health & Medicine*: This essential discipline is well-represented with 50 videos, offering perspectives on medical breakthroughs, healthcare practices, and life sciences.
- •
*Business*: This discipline includes 50 videos, reflecting on the multifaceted nature of commerce, from economics to management sciences.
- •
*Game*: This discipline includes 51 videos, reflecting various aspects of gaming.

Altogether, the MMWorld Benchmark’s diversity is visually encapsulated in Figure 19, which delineates the distribution of videos across 61 subdisciplines. The horizontal bar chart provides a quantified representation of the dataset’s range, reflecting the careful curation process that has gone into ensuring breadth across various knowledge areas.

The world we live in is rich with both audio and visual information, and effective world modeling requires an understanding of how these modalities interact and convey meaning. To achieve this, we annotated additional attributes such as "Requires Audio," "Requires Video," and "Question Only." These annotations help determine whether correctly answering a question necessitates audio information, visual cues from the video, or can be addressed based solely on the question itself. By doing so, we ensure that our benchmark tests the full spectrum of multimodal comprehension, reflecting the complex, sensory-rich environment in which real-world understanding takes place. The statistics of these annotations are shown in Figure 20.

## Appendix H Datasheets

### H.1 Motivation

For what purpose was the dataset created?

To introduce a multi-discipline multi-faceted multimodal video understanding benchmark to comprehensively evaluate MLLMs’ abilities in reasoning and interpreting real-world dynamics.

Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?

The dataset is created by authors from UCSC, UCSB, and Microsoft.

Who funded the creation of the dataset?

UCSC, UCSB, and Microsoft Azure.

### H.2 Composition

What do the instances that comprise the dataset represent? (e.g., documents, photos, people, countries)

Videos along with captions and question/answer pairs.

How many instances are there in total (of each type, if appropriate)?

6,627 instances. The data distribution over different types can be found in Figure 2 of the main paper.

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?

Yes.

Is there a label or target associated with each instance?

Yes.

Is any information missing from individual instances?

No.

Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?

N/A.

Are there recommended data splits (e.g., training, development/validation, testing)?

The MMWorld is used for evaluation purpose only.

Are there any errors, sources of noise, or redundancies in the dataset?

No.

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?

Yes.

Does the dataset contain data that might be considered confidential?

No.

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?

No.

### H.3 Collection Process

The data collection process is described in Section 3 of the main paper.

### H.4 Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values

We extract video frames from collected videos in automatically generated.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?

Yes. The raw video urls are given.

Is the software that was used to preprocess/clean/label the data
available?

Yes. The source code can be found in [https://github.com/eric-ai-lab/MMWorld](https://github.com/eric-ai-lab/MMWorld).

### H.5 Uses

Has the dataset been used for any tasks already?

Yes. We have used the dataset to evaluate video question answering.

Is there a repository that links to any or all papers or systems that use the dataset?

Yes. The GitHub repository [https://github.com/eric-ai-lab/MMWorld](https://github.com/eric-ai-lab/MMWorld) here.

What (other) tasks could the dataset be used for?

Video captioning and evaluating faithfulness of evaluation metrics.

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?

No.

Are there tasks for which the dataset should not be used?

The videos in this dataset are from different sources and are unique. The dataset should not be used for tasks such as video editing.

### H.6 Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?

Yes. The benchmark is publicly available.

How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?

We host it on the webpage, GitHub, and Huggingface.

When will the dataset be distributed?

It’s availale and open to the public now.

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?

CC-By 4.0.

Have any third parties imposed IP-based or other restrictions on the data associated with the instances?

No.

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?

No.

### H.7 Maintenance

Who will be supporting/hosting/maintaining the dataset?

The authors will be supporting/hosting/maintaining the dataset.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

The email address is xhe89@ucsc.edu.

Is there an erratum?

No. We will make it if there is any erratum.

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?

Yes. We will make announcements on GitHub if there is any update.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)?

N/A.

Will older versions of the dataset continue to be supported/hosted/maintained?

Yes. Old versions can still be accessed from Huggingface.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?

Yes. Contributors can post issues or submit pull requests on GitHub. We will review and verify contributions, and update the dataset if the contribution is useful.

## Appendix I Author Statement, Hosting, Licensing, and Maintenance Plan

#### Author Statement

We bear all responsibility in case of violation of rights and confirmation of the data license.

#### Hosting

MMWorld is hosted on [https://mmworld-bench.github.io/](https://mmworld-bench.github.io/). The dataset is provided in the JSON file format. The metadata can be found at [https://huggingface.co/datasets/Xuehai/MMWorld](https://huggingface.co/datasets/Xuehai/MMWorld).

#### License

MMWorld is licensed under the CC-BY 4.0 license.

#### Maintenance Plan

We will keep maintaining and updating the dataset and benchmark, including the leaderboard.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: As comments in Weaknesses
soundness: 2
strengths: 1: The benchmark makes a novel contribution by filling an important gap in evaluating MLLMs' world modeling capabilities through video understanding across diverse disciplines.

2: The evaluation framework is comprehensive, covering seven broad disciplines and 69 subdisciplines while testing multiple reasoning types including explanation and counterfactual thinking.

3: The methodology demonstrates rigorous attention to detail, with well-documented data collection processes and multiple evaluation strategies.

4: The research provides valuable insights by identifying significant performance gaps in current MLLMs and revealing interesting differences between human and MLLM capabilities through detailed error analysis.
summary: This paper introduces MMWorld, a new benchmark for evaluating multimodal large language models' (MLLMs) ability to understand and reason about video content across multiple disciplines. The benchmark consists of 1,910 videos spanning 7 broad disciplines and 69 subdisciplines, with 6,627 question-answer pairs. The key innovation is its focus on multi-faceted reasoning (explanation, counterfactual thinking, future prediction, domain expertise) and multi-discipline coverage. The authors evaluate 15 MLLMs (both proprietary and open-source) on this benchmark and provide detailed analysis of model performance and error patterns.
weaknesses: 1. The dataset, while diverse, remains relatively similar compared to other modern benchmarks. Please include the recent works and discuss the difference between MMWorld and recent works, such as VideoMME, MMBench-Video.

2. The evaluation methodology shows potential bias through its heavy reliance on GPT-4V as a judge and in synthetic dataset generation.

3. The temporal reasoning aspect, though claimed as a key feature, could be more thoroughly explored and evaluated in the benchmark.

4. How to get the final results with the three subsets (Main Subset, Synthetic I, Synthetic II) in Table 3?

5. As the evaluated models cannot handle the audio modality, is the model's performance on the audio-related questions underestimated?

6. As the current VLMs are not the world model and the current questions in this benchmark are also limited to basic QA, limiting complex reasoning tasks. I think the author may reconsider the name of the benchmark to avoid over-claiming.

7. More in-depth analysis of the errors from the perspective of current VLM's architecture, training data, or training techniques would improve the work.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Based on the results in Table 3, it appears that MLLMs perform significantly better in the Business discipline compared to other disciplines. Does this imply that the QA pairs in the Business discipline are relatively simpler, or does it indicate that MLLMs are stronger in the Business discipline than in other disciplines?
2. The authors emphasize that they have introduced a new dataset. I did not see an anonymous dataset link in the paper. Where can I find it?
3. Typos:
 • “an characteristic” in Figure 1.
 • “%%” at line 215.
soundness: 3
strengths: * This benchmark encompasses 7 major disciplines and 69 subdisciplines, offering comprehensive coverage and highlighting its advantages over existing benchmarks.
* The authors introduced a new dataset, ensuring its quality through human annotation.
* The experiments are conducted on a large scale, covering mainstream MLLMs and providing analysis across different difficulty levels.
summary: This paper proposed a comprehensive benchmark for MLLMs, covering 7 major disciplines, 69 subdisciplines, and multi-faceted reasoning tasks. The authors conducted experiments on 15 models (including both open-source and closed-source models) and analyzed the results.
weaknesses: * Although the authors conduct an error analysis with six different categories, the error sample size is relatively small. 
* The analysis of experimental results across different models lacks depth. For example, what specific differences exist between model and human reasoning? It would be helpful to compare with human performance, as in business discipline, the model achieves 90% accuracy, showing a strong understanding ability. It is not surprising that MLLMs perform better than average humans. It would be meaningful to see whether the model has reached a level close to or beyond human expert performance, especially in business discipline.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - The hexagon figure (Fig.4) is really difficult to interpret: I think this kind of figures makes sense if one compares 3 models at most. The score normalization makes things even worse: So we cannot see how well the best model performs in this way. I would definitely remove it from the paper, and put the Table 8 into the main text. Also note that the figure is embedded as "PNG/JPEG" instead of "PDF" (but removing it from the paper would make things much more better).
- Fig. 6 is also hard to interpret. I would rather group some selected models instead of error categories, so one can observe which model produces what kind of errors more or less.
- Is there any example for auditory questions in the appendix? Can you please point? Can we see some synthethic questions/answer pairs generated using vision and audio side-by-side?
- How much diverse is the language, i.e. questions and answers?
- One suggestion: Please embed all figures as PDFs.
- It would be good to see many more examples in the appendix in a more compact format by avoiding large text with conversation bubbles.
soundness: 3
strengths: I think the most valuable aspect of this paper, the proposed dataset contains videos from a diverse set of subjects, and questions have a variety focusing on commonsense knowledge and reasoning. This unique combination could position the benchmark as an important resource for the near future, as recent models fails to thrive in this video question answering benchmark.
summary: This work proposes a novel benchmark to evaluate commonsense knowledge and reasoning capabilities of the existing multimodal LLMs that are capable of processing videos or sequences of images. The proposed benchmark is classified as *multi-discipline* since it includes videos from a diverse set of subjects, and it is also classified as *multi-faceted* because it also includes different types of tasks where some tasks are focusing on commonsense reasoning abilities. The authors also collected a synthethic dataset to evaluate either auditory or vision processing capabilities of these models. The experiments reveal that (i) most of the existing models fail to excel in the proposed benchmark, (ii) even the proprietary models lack deficiency in embodied tasks, (iii) and they fail to process auditory information sufficiently.
weaknesses: - Presentation could be improved (see the questions field).
- The paper actually has a main contribution, which is human annotated dataset. There are also two synthetic datasets to evaluate the performance of individual modality processing performances of the models that are capable of processing both vision and audio. This aspect complicates this work a little bit, where I think it would better to have a separate work which carefully examines how well the models process each modality.
- The related work could be improved there are a lot of recent related work which are missing (e.g. [TempCompass](https://arxiv.org/abs/2403.00476), [VILMA](https://openreview.net/forum?id=liuqDwmbQJ), [VITATECS](https://arxiv.org/abs/2311.17404v1), also older but related since nearly half of the questions focus on explainability [Next-QA](https://arxiv.org/abs/2105.08276))
- The imbalanced nature of the subtasks: 48% of the questions belong to the explanation category/task. Also, the main task figure gives the impression of that these tasks could be fluid. For instance, the *explanation* example also requires *spatiotemporal reasoning*, and *future prediction* requires also some *domain-expertise*. Maybe, rather than assigning just single category, they could have multiple categories.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Regarding multi-discipline evaluation, if a model underperforms in specific domains, is it due to a lack of general domain knowledge, specific domain ability, or both? Designing experiments to analyze this would be valuable. For instance, comparative testing with models pre-trained on general knowledge versus those fine-tuned with domain-specific datasets could help isolate the impact of each knowledge type on performance.
soundness: 3
strengths: - MMWorld is the first video benchmark to cover videos beyond the general visual domain.
- The paper includes several analyses to assess the strengths and limitations of current video large language models (LLMs).
summary: This paper introduces MMWorld, a new benchmark designed for multi-disciplinary, multi-faceted multimodal video understanding. MMWorld comprises 1,910 videos spanning seven broad disciplines and 69 sub-disciplines, along with 6,627 question-answer pairs and corresponding captions.
weaknesses: - Regarding multi-faceted abilities, almost all are already evaluated in previous datasets, such as CVRR [1] and Tempcompass [2]. This raises the question of the necessity for another benchmark to assess these capabilities, especially as the data in MMWorld also comes from YouTube, which may reduce its novelty and significance.

- For multi-discipline evaluation, the use of GPT-4V to generate questions raises concerns about ensuring expertise-level quality, as seen in MMMU, which uses college exam questions to maintain a rigorous standard. If the generated questions don’t meet this level, it blurs the distinction between domain-specific and general-domain questions. If no substantial difference exists, a model that performs well in general-domain benchmarks (e.g., CVRR or Tempcompass) may also perform well in MMWorld, potentially limiting the unique insights this new dataset could offer.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: For the first weakness. I hope the authors can provide the followings:

1. The instructions given to human annotators for reviewing and modifying generated QAs.
2. The criteria used for accepting or rejecting generated QAs.
3. Any measures taken to ensure diversity of knowledge sources beyond GPT-4.
4. The percentage of questions that were significantly modified or entirely rewritten by human annotators.

For the second weakness. I suggest that the authors include a comparison that explicitly compares MMWorld to NExT-QA and Causal-VidQA, especially on the types of reasoning tasks.

Finally, I find that the average scores (the right most column) in Table 3 are much below than the average of the 7 disciplines. Any explanations on how to obtain the average scores?
soundness: 3
strengths: 1.To my best knowledge, MMWorld is the first video understanding dataset that contains diverse question types necessitating knowledge from multiple disciplines.

2.The baselines and analyses are comprehensive, covering 15 advanced MLLMs.
summary: The paper introduces MMWorld, a new benchmark designed to evaluate multimodal large language models (MLLMs) across multiple disciplines in their understanding of real-world dynamics. MMWorld includes three datasets: a manually annotated dataset for multimodal reasoning and two automatically generated datasets focused on audio and visual understanding, respectively. Comprehensive baseline results and analyses demonstrate the challenges presented by MMWorld, with a detailed examination of primary errors and valuable insights.
weaknesses: 1.	According to the results in Table 3, GPT-4o already achieves high performance (80%~90%) on some disciplines, significantly surpassing other competitors. However, the QAs are generated and the model predictions are evaluated by GPT-4 families. This raises a concern that the dataset is likely biased to the knowledge of specific models (GPT-4). It would be better to show how the human annotators are instructed to amend the generated QAs.

2.	The paper seems forget to discuss and compare with two related benchmarks: NExT-QA and Causal-VidQA. The two benchmarks also emphasize explanation, temporal dynamics, counterfactual and future predictions, respectively.

Meta-review: This paper proposes a world model evaluation in videos. All the reviews are positive in the final review. The raised concerns have been addressed in the authors' responses and acknowledged by the reviewers. Overall, the AC has checked all the files and stands on the reviewers' side. The proposed evaluation brings metrics to the current foundation model community. The authors shall incorporate all the revisions suggested by reviewers in the camera-ready version.

Decision: Accept (Poster)