MMMU-Pro: A More Robust  Multi-discipline Multimodal Understanding Benchmark

Authors: Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham Neubig
Primary area: datasets and benchmarks
Keywords: Evaluation, Multimodal Understanding, Multimodal LLMs

Abstract:
This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing \textit{a core human cognitive skill of seamlessly integrating visual and textual information}. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8\% to 26.9\% across models. 
We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future multimodal research.

Decision: reject
Reviewer scores: 6, 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729340913225

thread_type:
official_review

<thread_object>
creator: reviewer_9c1c

<content>
summary:
This paper MMMU-Pro presents an upgraded version of MMMU. MMMU-Pro improves MMMU via (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. All these are reasonable upgrades upon MMMU, and the paper also presents some nice insights based on existing models' performances. But the upgrades seem to be incremental and not sure whether the efforts are enough for one standalone work.

strengths:
1/ The paper is easy to follow

2/ The three upgrades upon MMMU are reasonable and can better examine the MLLM's capability in making sense and reasoning about vision

3/ The paper evaluates a wide range of existing MLLMs and share some insights.

weaknesses:
1/ Fig 1 compares almost 20 models, is it necessary to compare so many models? We can probably derive similar conclusions & insights based on comparing just some representative models. In this era, new MLLMs come out every a few days. I understand from the view of maintaining a benchmark/competition, it is good to be inclusive while seems not helpful to have giant table and figures like Table 1 and Fig 1.

2. Despite the 3 upgrades are reasonable, they seem to be incremental given the wonderful work of MMMU. I am not sure whether the efforts in this pro work are enough for one standalone paper. It could be just a nice technical report; while I guess if people want to pass it, I am also fine.

3. The last upgrade of vision-only input is interesting and the analysis of OCR/COT are good. While it feels to be only just scratching the surface, and if the authors would like to create a more solid work, I would expect some deeper contributions e.g. design new model/algorithm that can better make sense of such purely vision input question, create a training dataset that can power existing MLLM to do much better job on this task.

questions:
Please see the above weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651541594

thread_type:
official_review

<thread_object>
creator: reviewer_bstp

<content>
summary:
This paper introduces a new benchmark MMMU-Pro for more accurately and rigorously assessment of a model’s true multimodal understanding and reasoning capabilities by filtering the question that can be answered by LLM dicrectly, adding more options and vision-only input. Experimental results using MMMU-Pro show a significant performance drop in existing model. This paper provides a  more rigorous evaluation tool,

strengths:
1. The proposed benchmark assesses multimodal models’ understanding and reasoning capabilities in a more rigorous method.
2. The data collection pipeline are confidential due to the engagement of human.
3. Experiments are comprehensive, assessing current model's performance more accurate.

weaknesses:
1. As the main contribution of this paper is a benchmark. The authors should provide more data analysis on the collected MMMU-Pro and give more insightful conclusion.
2. Apart from benchmark, the novelty is limited. This paper just tests many models on the proposed benchmark ( I don't think the exploration
 of OCR prompts and COT reasoning can be accounted as a novelty). On the other hand, the dataset collection pipeline is something more like engineering. That's the reason why I think this paper's novelty is limited. Of course, this is not the only criterion for determining whether this paper can be accepted. The proposed benchmark does make a contribution to MLLMs' understanding and reasoning capabilities. My main concern is that the workload and contribution may not be sufficient to be accepted by such a top-tier conference. I may change my rating based on the rebuttal.

questions:
1. Can the author privode more insight anaysis on the proposed benchmark?
2. It's good to see the author's effort about how to improve the open-source model's performance on MMMU-Pro.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726177924

thread_type:
official_review

<thread_object>
creator: reviewer_jvww

<content>
summary:
This paper is an extension of MMMU. The evaluation consists of three steps: 1) filtering out questions that can be answered by text-only models, 2) augmenting candidate options to reduce the chances of guessing correctly, and 3) introducing a "vision-only input" setting to challenge models to comprehend both visual and textual content simultaneously. 
Experimental results demonstrate a significant drop in model performance on MMMU-Pro

strengths:
Rigorous Evaluation:Using multiple LLMs to vote and filter out questions that can be solved by text-only models does enhance the benchmark’s ability to reflect more accurate visual capabilities. Overall, this benchmark is indeed more complex and better demonstrates the model’s ability to integrate text and visual information.

The findings and detailed analysis suggest avenues for future improvements, such as better vision-text integration strategies.

weaknesses:
1. Expanding the number of options is an obvious way to reduce model performance. The emphasis should be on demonstrating the quality of these expanded options. However, the authors only mention this briefly with a single example.
2. The work appears straightforward, with most of the effort concentrated on non-technical human labor, making the overall contribution less innovative.

questions:
Regarding the quality of options: Expanding the number of options does indeed reduce model performance. How do you ensure the quality and diversity of these additional options? If there is a method, could you elaborate further on the validation process?
﻿
Given the high construction cost of the benchmark, is it possible to reduce human effort through automated data generation or other technical means? For instance, could models be used to create new visual inputs or options?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730942382728

thread_type:
official_review

<thread_object>
creator: reviewer_3jhg

<content>
summary:
The paper introduces MMMU-Pro, an enhanced multimodal benchmark to rigorously test AI models’ understanding and reasoning by addressing limitations in the original MMMU benchmark. MMMU-Pro utilizes a three-step process to improve robustness: filtering questions answerable by text-only models, increasing candidate options to prevent guesswork, and implementing a vision-only input mode that embeds questions within images, thus requiring integrated visual and textual processing. Experimental results show a significant drop in model performance compared to MMMU, highlighting multimodal challenges. The study further investigates Chain of Thought prompting and OCR effectiveness, identifying areas where current models struggle, and setting directions for future multimodal research​.

strengths:
- This paper addresses key limitations in existing benchmarks like MMMU. By introducing a vision-only input mode, MMMU-Pro uniquely challenges models to process visual and textual information in a more realistic, integrated manner. This work also enhances question difficulty and mitigates model reliance on shortcuts, providing an essential tool for testing and advancing multimodal AI.

- The clarity of the paper is strong, with well-organized sections detailing the benchmark's construction and evaluation.

- Additionally, the paper examines the impact of Chain of Thought prompting and OCR on performance within the proposed benchmark, further investigating the limitations present in current MLLMs.

weaknesses:
- One limitation is that in the vision-only setting, images are manually captured photos and screenshots over a simulated display environment, but only differences in backgrounds, font styles, and font sizes are considered. However, the diversity of real images should also account for factors such as varied lighting conditions and different camera angles (e.g., rotated text in photos).

- While the paper discusses the Chain of Thought (CoT) prompting and OCR’s impact, these evaluations could be expanded to clarify where CoT specifically improves performance. For example, breaking down CoT's impact across different question types or modalities could reveal deeper insights, guiding future model improvements.

- Moreover, the analysis would benefit from more nuanced evaluation metrics that go beyond accuracy, such as tracking misinterpretation rates or identifying where models are most prone to visual-textual integration issues. This additional layer of analysis could provide more actionable insights for researchers looking to address specific multimodal weaknesses.

questions:
Please refer to weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731072521229

thread_type:
official_review

<thread_object>
creator: reviewer_av7u

<content>
summary:
This paper introduces MMMU-Pro, a more robust version of the MMMU benchmark. MMMU-Pro aims to more accurately assess multimodal models' true understanding and reasoning capabilities across diverse academic domains by addressing limitations found in the original MMMU. The authors achieve this through three main enhancements: (1) filtering out questions that can be answered using only text, ensuring models rely on multimodal input; (2) expanding the number of answer options to reduce reliance on guessing; and (3) introducing a vision-only input setting where questions are embedded in images, challenging models to integrate visual and textual information. These modifications result in a more rigorous benchmark that better approximates real-world scenarios. Experimental results demonstrate that MMMU-Pro challenges existing models more, revealing performance drops across multiple models and encouraging further exploration in multimodal understanding and reasoning.

strengths:
- Significance: MMMU-Pro addresses critical gaps in existing benchmarks, promoting deeper multimodal understanding over shallow pattern recognition. It sets a higher evaluation standard, likely to drive future research and model development.
- Quality: The paper rigorously evaluates MMMU-Pro across multiple state-of-the-art models, showing significant performance drops that underscore the benchmark’s challenge. 
- Insights: Experiments with methods like Chain of Thought reasoning and OCR prompts enrich the analysis, verifying the benchmark’s effectiveness in highlighting model limitations.

weaknesses:
Unclear Justification for OCR Requirement: One of MMMU-Pro's main contributions is embedding text within images to increase difficulty by requiring OCR. However, this addition may detract from the benchmark’s core goal of evaluating multimodal understanding, as it primarily tests the model’s OCR capabilities rather than its deeper multimodal comprehension. Although it is true that embedding text within images is more realistic, whether the extra difficulty from OCR is significant for LMMs needs more justification, as the extra focus on OCR could potentially obscure the true reasoning ability of models that struggle with OCR but perform well in multimodal integration tasks.

Limited Impact on Model Performance Ranking: While it’s acceptable for a benchmark to yield similar performance scores, MMMU-Pro does not alter the ranking of current models, nor does it reveal new insights into their strengths and weaknesses. This lack of differentiation reduces the benchmark’s ability to provide fresh perspectives on model capabilities, potentially weakening its contribution as an evaluation tool.

questions:
A more thorough justification for the OCR requirement and a clearer explanation of the new benchmark's significance could enhance the paper’s impact.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732381716039

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 4

comment:
## **Re “Technical Contribution of MMMU-Pro”**

### **1. Introducing “Reasoning on Integrated Modalities”**
We propose the concept of "reasoning on integrated modalities," which is likely to become a key focus for future foundation models. MMMU-Pro serves as a robust testbed for evaluating these reasoning capabilities. **(See more in *Re Motivation*)**.

### **2. Insights from MMMU-Pro Analysis**
Through analyzing over 20 models, we provide new insights from our overall experimental results, ablation studies, error analysis and model training. **(See more in *Re New Insights*)**.

### **3. Guidance for Future Modeling**
Based on the new insights, we offer recommendations for improving future models. **(See more in *Re Guide for Future Modeling*)**.

### **4. Automatic Data Generator**
We developed an **[$\\textrm{\\color{blue}automatic data generation tool}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/mmmu-pro/tool/README.md)** based on our data curation pipeline, capable of augmenting options and generating screenshot-based questions automatically. This tool facilitates automatic training data generation and supports further analysis, such as studying the impact of different perturbation methods on accuracy.


### **We summarize the new content we revised and added in the rebuttal:**

| **Type**        | **Content**                                                                  | **Position**                                                                | **Reviewers**          |
|:---------------|:----------------------------------------------------------------------------|:----------------------------------------------------------------------------|:----------------------|
| New Result      | Influence of Vision Encoders                                                 | Future Guide (Line 497-509); [$\\textrm{\\color{blue}Table 4}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/table4.pdf)                                                          | bStp                   |
| New Result      | Error Analysis                                                               | Experiments (Line 469-476); [$\\textrm{\\color{blue}Figure 8}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/error_type.pdf)                                        | 3jHG, bStp             |
| New Result      | CoT’s impact on 30 subjects                                                  | Experiments (Line 337-344); [$\\textrm{\\color{blue}Table 2}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/table2.pdf), [$\\textrm{\\color{blue}Figure 10}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/gpt4o_llava72b.pdf)                                | 3jHG, bStp, 9C1c, jvww |
| New Result      | The influence of OCR and Reasoning capability on MMMU-Pro Vision             | Experiments (Line 411-421); [$\\textrm{\\color{blue}Figure 6}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/vis_ocr.pdf)                                         | AV7u, bStp, 9C1c, jvww |
| New Tool        | Automatic screenshot generation of MMMU-Pro style questions| [$\\textrm{\\color{blue}Tool}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/mmmu-pro/tool/README.md) | 9C1c, jvww |
| New Discussions | Guide for future modeling | Section 4  | bStp |
| Revised Result  | Vision Input as the primary setting of MMMU-Pro. Rank significantly changes. | [$\\textrm{\\color{blue}Table 1}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/table1.pdf)                                                                      | AV7u, 9C1c             |
| Revised Figure  | Replace examples to show diverse display environment                         | [$\\textrm{\\color{blue}Figure 2}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/example.pdf)                                                                     | 3jHG                   |



We sincerely appreciate the insightful comments, which have prompted us to reevaluate the positioning of this work and significantly improve its clarity, soundness, and technical contributions. We believe that MMMU-Pro holds great potential to guide the development and training of next-generation multimodal models. We firmly advocate that a **truly capable foundation model of the future must seamlessly transition between different modalities**. MMMU-Pro is explicitly designed to rigorously evaluate this capability. Unlike standalone OCR or reasoning benchmarks that fail to capture the essence of multimodal integration, MMMU-Pro challenges models to perform seamless reasoning across integrated modalities.

### **If our rebuttal addresses your concerns and proves useful, we kindly ask you to consider adjusting your review and scores. We remain open and eager to address any further concerns or questions you may have.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732381767697

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 3

comment:
## **Re: "Guide for Future Model Training"**
We added a new **section 4 “Guide for Future Model Training”** in the revised manuscript.

### **1. Scaling of LLM Backbones**
- Scaling LLMs improves both perception and reasoning capabilities, as shown in performance trends:
  - Examples: GPT-4o > GPT-4o mini, Llava-OneVision-72B > Llava-OneVision-7B, InternVL2-78B > InternVL2-8B.

### **2. Better Vision Encoders that Highlight Visual Representation Learning**
- Vision encoder choice impacts the performance of MMMU-Pro. Self-supervised encoders like DINOv2 show better performance than language-supervised encoders like SigLip.
- Future encoders may combine strengths of language-supervised and self-supervised approaches.

### **3. Better Vision-Text Integration**
- **Cross-modal integration remains challenging** for tasks requiring deep understanding.
- Improved cross-modal feature fusion techniques are promising for bridging this gap.

### **4. CoT Data Generation**
- **CoT prompting significantly enhances reasoning-heavy subjects** like **Tech and Engineering** but has limited or even negative effects in subjective areas like **Art and Design**.
- Generating diverse reasoning-intensive CoT datasets can improve model performance, especially in reasoning-heavy domains.

### **5. Text-Rich Image Generation**
- **Strong OCR and reasoning performance do not guarantee success on MMMU-Pro Vision** due to a lack of training data with text-rich images in reasoning-intensive scenarios.
- A new [$\\textrm{\\color{blue}tool}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/mmmu-pro/tool/README.md) based on the MMMU-Pro Vision annotation process was developed to **generate screenshots embedding questions and images**.
- This tool can scale the generation of reasoning-specific datasets, improving models' ability to handle integrated visual-text tasks.

By addressing these areas, future models can overcome limitations identified by MMMU-Pro and advance multimodal understanding and reasoning capabilities.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732381967312

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 2

comment:
## **Re “New insights from MMMU-Pro”**

### **1. Model Rankings**
- Model **rankings change significantly from MMMU to MMMU-Pro Vision** as noted in revised [$\\textrm{\\color{blue}Table 1}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/table1.pdf), particularly for open-source models. This illustrates reasoning in an integrated multimodal context is more challenging compared with separate multimodal inputs.

### **2. OCR Influence**
- Good OCR ability is **necessary but insufficient** condition for high accuracy on MMMU-Pro Vision.
- High OCR accuracy does not correlate strongly with high performance on MMMU-Pro Vision (e.g., Llava OV-72B, Pixtral-12B), as shown in the newly-added [$\\textrm{\\color{blue}Figure 6}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/vis_ocr.pdf).

### **3. Error Analysis**
- As shown in the new [$\\textrm{\\color{blue}Figure 8}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/error_type.pdf), **Reasoning errors increase significantly in MMMU-Pro Vision compared to MMMU** (46% vs. 26%), highlighting that while MMMU-Pro primarily aims to benchmark the model's reasoning capabilities, **reasoning across integrated modalities remains a greater challenge**.  
- OCR Errors: **OCR does not pose a bottleneck for highly capable models like GPT-4o** (no errors due to incorrect text recognition).
- These findings emphasize that multimodal reasoning requires capabilities beyond OCR in integrated modality environments.

### **4. CoT Impact Across Subjects**
- As shown in the new [$\\textrm{\\color{blue}Figure 10}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/gpt4o_llava72b.pdf) and [$\\textrm{\\color{blue}Table 2}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/table2.pdf), CoT prompting significantly improves performance in reasoning-heavy subjects like Science, Business, and Engineering but shows limited or negative benefits in subjects like Art, Humanities and Health.

### **5. Weaker CoT Ability in Integrated Multimodal Reasoning**
An intriguing phenomenon was observed in [$\\textrm{\\color{blue}Figure 9}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/figure9.pdf): **GPT-4o typically generates fewer CoT tokens in the Vision input setting** compared to the Standard setting. Additionally, the model tends to **allocate more tokens to description rather than analysis**. This highlights that reasoning within integrated multimodal contexts is more challenging, and the CoT ability in such scenarios is comparatively weaker.

### **6. Influence of Vision Encoders**
- In [$\\textrm{\\color{blue}Table 4}$](https://anonymous.4open.science/r/MMMU-Pro-ICLR-7174/supplementary%20material/table4.pdf), we train two MLLMs on 1M samples, utilizing different types of vision encoders: a language-supervised encoder (Siglip) and a self-supervised encoder (DINOv2).
- The self-supervised vision encoder, DINOv2, which emphasizes visual feature learning, achieves better performance on MMMU-Pro Vision. In contrast, the language-supervised vision encoder, Siglip, performs better on MMMU (val). Future work may focus on further enhancing visual feature learning while exploring the integration of language-based training objectives with self-supervised training objectives.


| **Method**               | **MMMU (Val)** | **MMMU-Pro Vision** |
|----------------------|------------|-----------------|
| DINOv2 ViT-G-14     | 37.1       | 17.4            |
| Siglip ViT-SO400M-14 | 37.9       | 16.7            |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732381996293

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 1

comment:
We appreciate all the constructive comments by the reviewers. Most of the major concerns lie in the position of MMMU-Pro compared with MMM, technical contributions, and what kind of new insights we bring by building MMMU-Pro. We respond to these major concerns in the general response and other comments in each individual response.


## **Re “Motivation of building MMMU-Pro”**

Beyond creating a more robust dataset—by filtering for text-only answerable questions and incorporating more diverse options—the primary motivation for introducing MMMU-Pro is to **benchmark reasoning capabilities in an integrated multimodal context**. While the community has yet to reach a consensus, we anticipate that future foundational models will increasingly adopt an integrated modalities paradigm: models will no longer distinguish between input modalities, whether they are images, text, or videos.

This prediction is inspired by how humans perceive and reason about the world. When processing visual signals, **humans do not explicitly segregate them by modality; instead, we seamlessly integrate and transition between textual and visual inputs**. These inputs are perceived holistically through a unified sensory interface—our eyes. Similarly, we believe a foundational model—or any intelligent system—should exhibit this capability. Such models should not be constrained to processing separate multimodal inputs but should handle integrated inputs (e.g., a single screenshot combining text and images) with equal proficiency, seamlessly reasoning across them.
MMMU-Pro represents one of the early steps toward realizing this paradigm, paving the way for future advancements in multimodal reasoning.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces MMMU-Pro, an enhanced multimodal benchmark that addresses limitations of the original MMMU by ensuring reliance on multimodal inputs, expanding answer options, and introducing vision-only settings, providing a more rigorous evaluation of multimodal models' reasoning capabilities.

The paper is generally well-written. The original MMMU has already brought significant impacts to the community. This work is an extension of that, which will be useful for the community as well. 

The paper has generally received scores of 6, with the exception of Reviewer jvww, who did not follow up after the rebuttal.

Upon reviewing all responses, AC noticed that several reviewers were not very excited or enthusiastic about the work, even while supporting its acceptance. They expressed concerns regarding the paper's limited novelty. Specifically, the concerns include:

Similarities between the benchmark dataset and the existing MMMU benchmark. Limited insights derived from extensive model comparisons. The annotations and dataset curation involve significant human effort, with minimal technical innovation introduced.

After discussion among AC and reviewers, due to the limited novelty, the AC decided to reject the paper. However, the authors are highly encouraged to submit an extended version of MMMU to a journal, which often accommodates expansions of original work.