MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

## Abstract

Abstract This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models’ true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly “see” and “read” simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information . Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.

## 1 Introduction

Recent advances in multimodal large language models (MLLMs) have led to remarkable progress in tackling complex reasoning tasks that combine textual and visual information (Yin et al., ; Jin et al., ). Models like GPT-4o (OpenAI, ) have achieved impressive results, e.g., on the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark (Yue et al., ), reaching an accuracy of 69.1% on college-level questions that integrate text and images.

While these achievements are significant, they raise a critical question: Do the current benchmark results truly reflect a deep, multifaceted understanding of diverse subjects, or are these models exploiting subtle shortcuts and statistical patterns to arrive at correct answers without genuine comprehension and reasoning?

This question has profound implications for the development and deployment of AI systems in real-world applications. If models rely on superficial cues rather than true multimodal understanding (Du et al., ; Yuksekgonul et al., ), we risk overestimating their capabilities and potentially deploying systems that fail in unpredictable ways when faced with novel scenarios (Wu and Xie, ; Tong et al., ).

To address this concern and push the boundaries of multimodal AI evaluation, we introduce MMMU-Pro, a more robust and challenging version of the MMMU benchmark. MMMU-Pro is designed to more accurately and rigorously assess a model’s true multimodal understanding and reasoning capabilities across a wide range of academic disciplines. The development of MMMU-Pro is motivated by several key observations, including the text-only solvability of some existing benchmark questions, the limited option space in multiple-choice formats (Wang et al., ), and the need to challenge models’ ability to jointly understand different modalities in a more integrated way.

MMMU-Pro employs a rigorous three-step construction process (as shown in Figure 2) that builds upon MMMU (Yue et al., ): (1) filtering out questions answerable by text-only language models, (2) augmenting candidate options to reduce the effectiveness of guessing based on the options, and (3) introducing a vision-only input setting (as shown in Figure 3) where models are presented with questions embedded in a screenshot or photo.

Figure: Figure 2: An overview of the construction process of MMMU-Pro.
Refer to caption: x2.png

The introduction of the vision-only input setting is particularly crucial, as it tests a fundamental human cognitive ability: the seamless integration and switching between visual and textual information. This setting challenges models to develop the capability to truly “see” and “read” simultaneously, mirroring how humans effortlessly process complex scenes where text and images are intertwined. This ability is crucial for tasks ranging from interpreting scientific diagrams (Li et al., ) to navigating graphical user interfaces (Liu et al., ; Zheng et al., ; Koh et al., ). Moreover, this approach aligns with how users naturally interact with AI systems, often sharing screenshots or photos rather than meticulously separating text and images.

Our experimental results demonstrate the effectiveness of MMMU-Pro in providing a more rigorous evaluation of multimodal models. We observe significant performance drops across all tested models when compared to the original MMMU benchmark, with decreases ranging from 16.8% to 26.9%. These results highlight the limitations of current state-of-the-art models in true multimodal understanding and reasoning. Furthermore, our analysis reveals that while Chain of Thought (CoT) (Wei et al., ) prompting generally improves performance, the benefits vary across models and settings.

Interestingly, we find that explicit OCR prompts do not significantly impact performance for most models, suggesting that advanced multimodal models have already developed robust text extraction capabilities from images. However, this result also underscores that simple OCR is insufficient for the challenges presented by MMMU-Pro’s vision-only input setting. Our further qualitative analysis indicates that when text is embedded within images, it significantly increases the overall complexity of the visual input, requiring models to not only recognize text but also understand its context, relationship to visual elements, and relevance to the question. These findings not only provide a more accurate assessment of current multimodal AI capabilities but also highlight the need for more sophisticated multimodal reasoning abilities.

Figure: Figure 3: Sample questions from MMMU-Pro. In the Vision Input setting, the model is required to answer a multiple-choice question with up to 10 options, each embedded within a screenshot or photo. These images were manually captured by human annotators in diverse display environments to reflect real-world scenarios.

## 2 MMMU-Pro: A More Robust Version of MMMU

### 2.1 Revisiting the MMMU Benchmark

The Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark (Yue et al., ) is a comprehensive dataset designed to evaluate multimodal AI models on college-level tasks requiring subject-specific knowledge and deliberate reasoning. MMMU consists of 11.5K carefully curated multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines across 30 subjects and 183 subfields. Each question in MMMU is a multimodal image-text pair with 4 multiple-choice options, featuring 30 diverse image types such as charts, diagrams, maps, and chemical structures.
MMMU has quickly become a standard evaluation tool in the field, used to assess the capabilities of many prominent multimodal models upon their release (OpenAI, , ; Anthropic, ; Reid et al., ; Li et al., ).

Meanwhile, we received some feedback from the community that text-only LLMs could accurately answer some questions without any need for visual input. We took a close look at these questions and identified two main issues: 1) Text-Only Dependency: Certain questions are relatively independent or irrelevant to the corresponding images. 2) Shortcut Exploitation: Even when questions require images for humans to answer correctly, models could often find shortcuts or correlations within the candidate options, leveraging their pre-existing knowledge (from pre-training) to arrive at the correct answer. Two examples that are answered correctly by Llama-3-70B Instruct (Dubey et al., ) are shown in Figure 4.

Figure: Figure 4: Two MMMU questions that are answered correctly by a text-only LLM Llama-3-70B Instruct. The model finds shortcuts or correlations in the text question and the candidate options.
Refer to caption: x4.png

### 2.2 Methods

To mitigate the issues and construct a more robust benchmark, we implemented a three-step approach:

#### Filtering Questions:

We began by filtering out questions that could be answered by text-only LLMs. We selected four strong open-source LLMs: Llama3-70B-Instruct (Dubey et al., ), Qwen2-72B-Instruct (Yang et al., ), Yi-1.5-34B-Chat (Young et al., ), and Mixtral-8$\times$22B-Instruct ()—and tasked them with answering the MMMU questions without access to images. The models were required to provide answers even when they indicated that visual input was necessary. We repeated this process ten times for each model, considering a question as "answerable" if a model correctly answered it more than five times. We then excluded any question where at least three out of the four models answered correctly across the majority of trials. We then randomly sampled 1800 questions from the remaining pool, evenly distributed across 30 subjects (60 questions per subject).

Figure: Figure 5: Accuracy of text-only LLMs in different sets of MMMU questions.
Refer to caption: x5.png

#### Augmenting Candidate Options:

Despite the filtering, some questions could still be answered by text-only LLMs, often by exploiting subtle hints within the candidate options. To counteract this, we increased the number of candidate options from four to ten, making it more challenging for models to rely on guessing. This augmentation was done by human experts with the assistance of GPT-4o. During this process, experts also reviewed the original annotated questions to ensure their relevance to the images and to eliminate any questions that lacked a clear connection or coherence. This step filtered out 70 questions and we obtained 1730 questions in total.

As illustrated in Figure 5, these two steps significantly reduced the accuracy of text-only models attempting to guess the answers.

#### Enhancing Evaluation with a Vision-Only Input Setting:

To further challenge the multimodal understanding of models, we introduced a vision-only input setting in MMMU-Pro. In this setting, the model is presented with a question embedded within a screenshot or photo, without any text explicitly fed into the model. To implement this setting, we asked the human annotators to manually capture photos and screenshots over a simulated display environment. This process involved varying the backgrounds, font styles, and font sizes to replicate the diversity of real-world conditions. By using different combinations of these elements, we created a broad range of visual contexts, ensuring that the models are not only challenged by the integration of text and images but also by the variability in how this content is presented. Examples of the vision-only input setting are shown in Figure 3.

The motivation for introducing this setting stems from real-world usage patterns and the fundamental cognitive abilities of humans. In everyday scenarios, users often take shortcuts by simply capturing screenshots of questions that include both text and images, rather than inputting text separately. This habit of feeding integrated visual-textual content reflects a natural human tendency to process information holistically. Humans excel at interpreting and understanding information when both text and images are presented together, and our goal with this setting is to push models toward achieving a similar level of comprehension. By mimicking this real-world behavior, the vision-only input setting not only adds realism to the benchmark but also ensures that models are better equipped to handle the complexities of multimodal tasks as they would appear in practical applications.

After these three steps, we finally obtained 3460 questions in total (1730 samples are in the standard format and the other 1730 are in the screenshot or photo form).

**Table 1: Overall results of different models on different subsets. $\Delta_{1}$: Standard (10 options) - Standard (4 options); $\Delta_{2}$: Vision - Standard (10 options); $\Delta_{3}$: MMMU-Pro - MMMU (Val). MMMU-Pro overall score refers to the average of Standard (10 Options) and Vision Input settings.**
|  | MMMU-Pro | MMMU<br>(Val) | $\Delta_{1}$ | $\Delta_{2}$ | $\Delta_{3}$ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Standard<br>(4 Opts) | Standard<br>(10 Opts) | Vision | Overall |  |  |  |  |
| Random Choice | 24.9 | 12.8 | 12.4 | 12.6 | 22.1 | -12.1 | -0.4 | -9.8 |
| Frequent Choice | 27.8 | 12.1 | 12.1 | 12.1 | 26.8 | -15.7 | 0 | -14.7 |
| GPT-4o (0513) (OpenAI, ) | 64.7 | 54.0 | 49.7 | 51.9 | 69.1 | -10.7 | -4.3 | -17.2 |
| Claude 3.5 Sonnet (Anthropic, ) | 63.7 | 55.0 | 48.0 | 51.5 | 68.3 | -8.7 | -6.9 | -16.8 |
| Gemini 1.5 Pro (0801) (Reid et al., ) | 60.6 | 49.4 | 44.4 | 46.9 | 65.8 | -11.2 | -5.0 | -18.9 |
| Qwen2-VL-72B  (Qwen, ) | 59.3 | 49.2 | 43.3 | 46.2 | 64.5 | -10.1 | -5.9 | -18.3 |
| Gemini 1.5 Pro (0523) (Reid et al., ) | 57.6 | 46.5 | 40.5 | 43.5 | 62.2 | -11.1 | -6.0 | -18.7 |
| GPT-4o mini  (OpenAI, ) | 55.3 | 39.9 | 35.2 | 37.6 | 59.4 | -15.4 | -4.7 | -21.9 |
| InternVL2-Llama3-76B  (Chen et al., ) | 55.0 | 41.9 | 38.0 | 40.0 | 58.3 | -13.1 | -3.9 | -18.4 |
| InternVL2-40B  (Chen et al., ) | 47.4 | 36.3 | 32.1 | 34.2 | 55.2 | -11.1 | -4.2 | -21.0 |
| LLaVA-OneVision-72B  (Li et al., ) | 52.3 | 38.0 | 24.0 | 31.0 | 56.8 | -14.3 | -14.0 | -25.8 |
| Qwen2-VL-7B  (Qwen, ) | 46.6 | 34.1 | 27.0 | 30.5 | 54.1 | -12.5 | -7.1 | -23.6 |
| InternVL2-8B  (Chen et al., ) | 42.6 | 32.5 | 25.4 | 29.0 | 51.2 | -10.1 | -7.1 | -22.3 |
| MiniCPM-V2.6  (Yao et al., ) | 40.6 | 30.2 | 24.2 | 27.2 | 49.8 | -10.4 | -6.0 | -22.6 |
| VILA-1.5-40B  (Lin et al., ) | 46.8 | 35.9 | 14.1 | 25.0 | 51.9 | -10.9 | -21.8 | -26.9 |
| LLaVA-NEXT-72B  (Liu et al., ) | 43.0 | 31.0 | 19.2 | 25.1 | 49.9 | -12.0 | -11.8 | -24.8 |
| LLaVA-OneVision-7B  (Li et al., ) | 42.8 | 29.5 | 18.7 | 24.1 | 48.8 | -13.3 | -10.8 | -24.7 |
| LLaVA-NeXT-34B  (Liu et al., ) | 44.5 | 30.3 | 17.2 | 23.8 | 48.1 | -14.2 | -13.1 | -24.4 |
| Idefics3-8B-Llama3  (Laurençon et al., ) | 40.8 | 30.1 | 15.6 | 22.9 | 46.6 | -10.7 | -14.5 | -23.8 |
| Qwen2-VL-2B  (Qwen, ) | 34.8 | 25.3 | 17.2 | 21.2 | 41.1 | -9.5 | -8.2 | -19.9 |
| Phi-3.5-Vision  (Abdin et al., ) | 37.8 | 26.3 | 13.1 | 19.7 | 43.0 | -11.5 | -13.2 | -23.3 |
| LLaVA-NeXT-7B  (Liu et al., ) | 33.7 | 19.4 | 14.6 | 17.0 | 35.3 | -14.3 | -4.8 | -18.4 |
| LLaVA-NeXT-13B  (Liu et al., ) | 33.9 | 19.8 | 14.5 | 17.2 | 36.2 | -14.1 | -5.3 | -19.1 |

## 3 Experiments

### 3.1 Experimental Setups

Baselines. To establish a comprehensive understanding of MMMU-Pro’s difficulty and to provide reference points for future research, we evaluated a diverse set of state-of-the-art multimodal models as baselines. These models represent a range of training approaches and capabilities in the field of multimodal AI.
Our baseline models include:

Proprietary Models: GPT-4o (0513) and GPT-4o mini, Claude 3.5 Sonnet, and Gemini 1.5 Pro (0801 and 0523 versions). These models represent the cutting edge of multimodal AI capabilities.

Open-source models: We evaluated a range of open-source models, including InternVL2 (8B, 40B, and Llama3-76B versions), LLaVA (OneVision-7B, OneVision-72B, and various NeXT versions), VILA-1.5-40B, MiniCPM-V2.6, Phi-3.5-Vision, and Idefics3-8B-Llama3. These models showcase the current state of publicly available multimodal AI systems.
We evaluated these models across three different settings: 1) Standard setting without augmented options (usually 4 options); 2) Standard setting with augmented options (usually 10 options); 3)Vision-only input setting.

The overall performance score for MMMU-Pro is calculated as the average of scores from settings (2) and (3). We include setting (1) and report the original MMMU validation set performance solely for comparison purposes, to highlight the increased difficulty of MMMU-Pro.

We evaluate the models with both Direct and CoT prompts (as shown in Appendix A), and report the higher ones in the overall results. We also discuss the influence of the CoT prompt in subsection 3.4.

### 3.2 Overall Results

We presented the overall results of MMMU-Pro of different models in Table 1.

Effect of Increased Candidate Options: The shift from 4 to 10 candidate options ($\Delta_{1}$) reveals a significant drop in performance for all models. GPT-4o (0513) experienced a decrease of 10.7%, from 64.7% to 54.0%. This indicates that increasing the number of options effectively reduces the likelihood of models guessing the correct answer, forcing them to engage more deeply with the multimodal content.

Impact of Vision-Only Setting: The introduction of the vision-only input setting further challenges models, as evidenced by the additional drop in performance when comparing the vision-only results to the 10-option standard ($\Delta_{2}$). For instance, GPT-4o (0513) dropped another 4.3% in accuracy when evaluated in the vision-only setting, and LLaVA-OneVision-72B saw a dramatic 14.0% decrease. This suggests that the vision-only setting successfully tests the models’ ability to integrate visual and textual information, highlighting their limitations when the text is not explicitly provided.

Combined Effects on MMMU-Pro: The overall $\Delta_{3}$, representing the difference between MMMU-Pro and MMMU (Val), shows a significant decrease across the board. For instance, models like Gemini 1.5 Pro (0801) and Claude 3.5 Sonnet exhibited declines of 18.9% and 16.8%, respectively, while more drastic drops were seen in models like VILA-1.5-40B with a 26.9% decrease.

This significant reduction in accuracy across the board suggests that MMMU-Pro successfully mitigates the shortcuts and guessing strategies that models could exploit in the original benchmark.

### 3.3 Does OCR Help in the Vision Setting?

Figure 6 explores whether Optical Character Recognition (OCR) prompts help in improving performance within the Vision Input setting of MMMU-Pro. The OCR prompt explicitly requires the model to write out the question text in the image (as shown in Appendix A). Across the models evaluated, the inclusion of OCR prompts did not significantly alter performance. These minimal differences suggest that strong capable models are already proficient at extracting and understanding textual information from images, even without explicit OCR prompts.

Figure: Figure 6: OCR’s influence in the Vision setting.
Refer to caption: x6.png

Importantly, this result underscores that simple OCR is not sufficient to solve the challenges presented by MMMU-Pro’s vision-only input setting. When text is embedded within images, it significantly increases the overall complexity of the visual input. This integration requires models to not only recognize and extract text but also to understand its context within the image, its relationship to visual elements, and its relevance to the question at hand. This multi-layered challenge of simultaneously processing textual and visual information makes the task substantially more demanding, pushing models towards more sophisticated multimodal reasoning.

### 3.4 Does CoT Help in Answering MMMU-Pro Questions?

Figure 7 examines the effectiveness of Chain of Thought (CoT) prompting in enhancing model performance on the MMMU-Pro benchmark, both in the Standard and Vision Input settings.
Across both settings, the introduction of CoT prompts generally led to improved performance. However, the extent of improvement varied significantly among models. For instance, Claude 3.5 Sonnet demonstrated a substantial increase in the Standard setting, improving from 42.7% to 55.0%. In contrast, models like LLaVA-OneVision-72B showed only minimal improvement.

Interestingly, we observed a significant performance drop for some models, such as VILA1.5-40B. This decline might be attributed to challenges in instruction-following abilities. When a model struggles to follow instructions accurately, generating CoT explanations becomes more difficult. Additionally, these models may face issues with maintaining the correct response format, leading to what is known as "boiled response format" problems.
These findings highlight the potential of CoT to enhance model performance in complex, real-world tasks that require nuanced reasoning and integration of multiple information sources. However, they also underscore the importance of robust instruction-following capabilities as a prerequisite for effective CoT implementation.

Figure: Figure 7: Impact of CoT prompting of different models in the two settings of MMMU-Pro.
Refer to caption: x7.png

Figure: Figure 8: Comparison of GPT-4o’s responses with and without augmented options.
Refer to caption: x9.png

Figure: Figure 9: Comparison of GPT-4o’s responses between Standard and Vision Input settings.
Refer to caption: x10.png

### 3.5 Qualitative Analysis

To gain deeper insights into model performance beyond quantitative metrics, we conducted a thorough qualitative analysis of MMMU-Pro results, focusing on two key scenarios: 1) Correct answers with four options but failure with ten options in the standard setting; 2) Success in the standard ten-option setting but failure in the vision input setting. Our analysis revealed several critical factors affecting model performance:

Challenges with Increased Options. Models often select the closest answer rather than arriving at a definitive choice, leading to increased errors with more options. An example is shown in Figure 8. Additional options that are conceptually close to the correct answer can cause confusion, particularly in nuanced questions. For instance, in conceptual questions, models struggled to distinguish between subtly different options, revealing limitations in their understanding of fine-grained distinctions within a subject area.

Vision-Text Integration Challenges. One of the major possible reasons is that the integration of text and image significantly increases the complexity of information processing and handling. We observed that the model is more likely to hallucinate and have wrong reasoning chains when simultaneous processing of visual and textual information. An example is shown in Figure 9. Additionally, complex visual inputs or unique layouts could also disrupt logical judgment and increase cognitive load. Rapid transitions between text and image can result in processing biases.

These observations highlight MMMU-Pro’s effectiveness in exposing the limitations of current multimodal models, particularly in handling increased complexity and integrating diverse information types. They point to critical areas for improvement in multimodal AI systems, including enhanced precision in multi-option scenarios, better integration of visual and textual information processing, and improved robustness in handling complex, mixed-format inputs.

## 4 Related Work

Multimodal Large Language Models. Recent progress in multimodal AI has been marked by innovative training approaches (Lu et al., ; Tan and Bansal, ; Chen et al., ; Zhou et al., ; Zhang et al., ; Li et al., ; Alayrac et al., ; Awadalla et al., ). Inspired by the success of large language models, researchers have developed various models with improved instruction-following capabilities (Liu et al., , , ; Li et al., ; Dai et al., ; Zhu et al., ; Zhang et al., ; Gao et al., ; Ye et al., , ; Zhao et al., ; Li et al., ; Monajatipoor et al., ; Zhao et al., ; Li et al., ; Lin et al., ; Zhang et al., ). Proprietary models such as GPT-4V (OpenAI, ), GPT-4o  (OpenAI, ), Gemini  (Team et al., ), and Claude-3.5  (Anthropic, ) have demonstrated strong performance across various vision-language tasks. However, a significant challenge remains in accurately evaluating the capabilities of these advanced multimodal models, highlighting the need for more robust and comprehensive benchmarks.

MLLM Benchmarks.
The rise of more advanced multimodal pre-training and instruction tuning has exposed the limitations of earlier benchmarks like VQA (Antol et al., ; Goyal et al., ), OK-VQA (Marino et al., ), and MSCOCO (Lin et al., ), which no longer suffice to evaluate the full spectrum of LMMs capabilities. To address this, recent benchmarks such as LAMM (Yin et al., ), LVLM-eHub (Xu et al., ), SEED (Li et al., ), MMBench (Liu et al., ),CV-Bench (Tong et al., ), MM-Vet (Yu et al., ), Mantis (Jiang et al., ), and BLINK (Fu et al., ) have emerged, covering aspects from basic perception to hallucination detection (Cui et al., ; Liu et al., ).
However, existing benchmarks often fall short in evaluating expert-level domain knowledge and complex reasoning (Lu et al., ; Zhang et al., ). While MMMU (Yue et al., ) made strides by incorporating multimodal, college-level questions, it still permits text-only models to find shortcuts (Lu et al., ; Zhang et al., ). To overcome these limitations, we introduce MMMU-Pro, a more robust version specifically designed to better evaluate multimodal reasoning by eliminating text-only answerable questions, expanding candidate options, and introducing a vision-only input setting that mirrors real-world scenarios where text and images are naturally intertwined.

## 5 Conclusion

MMMU-Pro presents a more robust multimodal understanding and reasoning benchmark compared with its predecessor MMMU. Our results demonstrate MMMU-Pro’s effectiveness in exposing the limitations of current state-of-the-art multimodal models, with significant performance drops across all tested systems. MMMU-Pro opens up several important avenues for future research: 1) Developing models with consistent performance across all MMMU-Pro settings, particularly in bridging the gap between standard and vision-only inputs. 2) Improving vision-text integration capabilities to handle complex, mixed-format inputs more effectively. 3) Exploring advanced reasoning techniques to address the increased complexity of MMMU-Pro questions.

## Acknowledgment

The authors would thank earlier testing and feedback from multiple VL model development teams including OpenAI, Google Gemini, LLaVA, Qwen-VL, VILA, InternVL, and MiniCPM. The authors would thank Google Gemini and Claude Anthropic for their API support.

## Appendix A Evaluation Prompts

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: A more thorough justification for the OCR requirement and a clearer explanation of the new benchmark's significance could enhance the paper’s impact.
soundness: 2
strengths: - Significance: MMMU-Pro addresses critical gaps in existing benchmarks, promoting deeper multimodal understanding over shallow pattern recognition. It sets a higher evaluation standard, likely to drive future research and model development.
- Quality: The paper rigorously evaluates MMMU-Pro across multiple state-of-the-art models, showing significant performance drops that underscore the benchmark’s challenge. 
- Insights: Experiments with methods like Chain of Thought reasoning and OCR prompts enrich the analysis, verifying the benchmark’s effectiveness in highlighting model limitations.
summary: This paper introduces MMMU-Pro, a more robust version of the MMMU benchmark. MMMU-Pro aims to more accurately assess multimodal models' true understanding and reasoning capabilities across diverse academic domains by addressing limitations found in the original MMMU. The authors achieve this through three main enhancements: (1) filtering out questions that can be answered using only text, ensuring models rely on multimodal input; (2) expanding the number of answer options to reduce reliance on guessing; and (3) introducing a vision-only input setting where questions are embedded in images, challenging models to integrate visual and textual information. These modifications result in a more rigorous benchmark that better approximates real-world scenarios. Experimental results demonstrate that MMMU-Pro challenges existing models more, revealing performance drops across multiple models and encouraging further exploration in multimodal understanding and reasoning.
weaknesses: Unclear Justification for OCR Requirement: One of MMMU-Pro's main contributions is embedding text within images to increase difficulty by requiring OCR. However, this addition may detract from the benchmark’s core goal of evaluating multimodal understanding, as it primarily tests the model’s OCR capabilities rather than its deeper multimodal comprehension. Although it is true that embedding text within images is more realistic, whether the extra difficulty from OCR is significant for LMMs needs more justification, as the extra focus on OCR could potentially obscure the true reasoning ability of models that struggle with OCR but perform well in multimodal integration tasks.

Limited Impact on Model Performance Ranking: While it’s acceptable for a benchmark to yield similar performance scores, MMMU-Pro does not alter the ranking of current models, nor does it reveal new insights into their strengths and weaknesses. This lack of differentiation reduces the benchmark’s ability to provide fresh perspectives on model capabilities, potentially weakening its contribution as an evaluation tool.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to weakness section.
soundness: 3
strengths: - This paper addresses key limitations in existing benchmarks like MMMU. By introducing a vision-only input mode, MMMU-Pro uniquely challenges models to process visual and textual information in a more realistic, integrated manner. This work also enhances question difficulty and mitigates model reliance on shortcuts, providing an essential tool for testing and advancing multimodal AI.

- The clarity of the paper is strong, with well-organized sections detailing the benchmark's construction and evaluation.

- Additionally, the paper examines the impact of Chain of Thought prompting and OCR on performance within the proposed benchmark, further investigating the limitations present in current MLLMs.
summary: The paper introduces MMMU-Pro, an enhanced multimodal benchmark to rigorously test AI models’ understanding and reasoning by addressing limitations in the original MMMU benchmark. MMMU-Pro utilizes a three-step process to improve robustness: filtering questions answerable by text-only models, increasing candidate options to prevent guesswork, and implementing a vision-only input mode that embeds questions within images, thus requiring integrated visual and textual processing. Experimental results show a significant drop in model performance compared to MMMU, highlighting multimodal challenges. The study further investigates Chain of Thought prompting and OCR effectiveness, identifying areas where current models struggle, and setting directions for future multimodal research​.
weaknesses: - One limitation is that in the vision-only setting, images are manually captured photos and screenshots over a simulated display environment, but only differences in backgrounds, font styles, and font sizes are considered. However, the diversity of real images should also account for factors such as varied lighting conditions and different camera angles (e.g., rotated text in photos).

- While the paper discusses the Chain of Thought (CoT) prompting and OCR’s impact, these evaluations could be expanded to clarify where CoT specifically improves performance. For example, breaking down CoT's impact across different question types or modalities could reveal deeper insights, guiding future model improvements.

- Moreover, the analysis would benefit from more nuanced evaluation metrics that go beyond accuracy, such as tracking misinterpretation rates or identifying where models are most prone to visual-textual integration issues. This additional layer of analysis could provide more actionable insights for researchers looking to address specific multimodal weaknesses.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Regarding the quality of options: Expanding the number of options does indeed reduce model performance. How do you ensure the quality and diversity of these additional options? If there is a method, could you elaborate further on the validation process?
﻿
Given the high construction cost of the benchmark, is it possible to reduce human effort through automated data generation or other technical means? For instance, could models be used to create new visual inputs or options?
soundness: 3
strengths: Rigorous Evaluation:Using multiple LLMs to vote and filter out questions that can be solved by text-only models does enhance the benchmark’s ability to reflect more accurate visual capabilities. Overall, this benchmark is indeed more complex and better demonstrates the model’s ability to integrate text and visual information.

The findings and detailed analysis suggest avenues for future improvements, such as better vision-text integration strategies.
summary: This paper is an extension of MMMU. The evaluation consists of three steps: 1) filtering out questions that can be answered by text-only models, 2) augmenting candidate options to reduce the chances of guessing correctly, and 3) introducing a "vision-only input" setting to challenge models to comprehend both visual and textual content simultaneously. 
Experimental results demonstrate a significant drop in model performance on MMMU-Pro
weaknesses: 1. Expanding the number of options is an obvious way to reduce model performance. The emphasis should be on demonstrating the quality of these expanded options. However, the authors only mention this briefly with a single example.
2. The work appears straightforward, with most of the effort concentrated on non-technical human labor, making the overall contribution less innovative.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can the author privode more insight anaysis on the proposed benchmark?
2. It's good to see the author's effort about how to improve the open-source model's performance on MMMU-Pro.
soundness: 3
strengths: 1. The proposed benchmark assesses multimodal models’ understanding and reasoning capabilities in a more rigorous method.
2. The data collection pipeline are confidential due to the engagement of human.
3. Experiments are comprehensive, assessing current model's performance more accurate.
summary: This paper introduces a new benchmark MMMU-Pro for more accurately and rigorously assessment of a model’s true multimodal understanding and reasoning capabilities by filtering the question that can be answered by LLM dicrectly, adding more options and vision-only input. Experimental results using MMMU-Pro show a significant performance drop in existing model. This paper provides a  more rigorous evaluation tool,
weaknesses: 1. As the main contribution of this paper is a benchmark. The authors should provide more data analysis on the collected MMMU-Pro and give more insightful conclusion.
2. Apart from benchmark, the novelty is limited. This paper just tests many models on the proposed benchmark ( I don't think the exploration
 of OCR prompts and COT reasoning can be accounted as a novelty). On the other hand, the dataset collection pipeline is something more like engineering. That's the reason why I think this paper's novelty is limited. Of course, this is not the only criterion for determining whether this paper can be accepted. The proposed benchmark does make a contribution to MLLMs' understanding and reasoning capabilities. My main concern is that the workload and contribution may not be sufficient to be accepted by such a top-tier conference. I may change my rating based on the rebuttal.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the above weakness
soundness: 2
strengths: 1/ The paper is easy to follow

2/ The three upgrades upon MMMU are reasonable and can better examine the MLLM's capability in making sense and reasoning about vision

3/ The paper evaluates a wide range of existing MLLMs and share some insights.
summary: This paper MMMU-Pro presents an upgraded version of MMMU. MMMU-Pro improves MMMU via (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. All these are reasonable upgrades upon MMMU, and the paper also presents some nice insights based on existing models' performances. But the upgrades seem to be incremental and not sure whether the efforts are enough for one standalone work.
weaknesses: 1/ Fig 1 compares almost 20 models, is it necessary to compare so many models? We can probably derive similar conclusions & insights based on comparing just some representative models. In this era, new MLLMs come out every a few days. I understand from the view of maintaining a benchmark/competition, it is good to be inclusive while seems not helpful to have giant table and figures like Table 1 and Fig 1.

2. Despite the 3 upgrades are reasonable, they seem to be incremental given the wonderful work of MMMU. I am not sure whether the efforts in this pro work are enough for one standalone paper. It could be just a nice technical report; while I guess if people want to pass it, I am also fine.

3. The last upgrade of vision-only input is interesting and the analysis of OCR/COT are good. While it feels to be only just scratching the surface, and if the authors would like to create a more solid work, I would expect some deeper contributions e.g. design new model/algorithm that can better make sense of such purely vision input question, create a training dataset that can power existing MLLM to do much better job on this task.

Meta-review: This paper introduces MMMU-Pro, an enhanced multimodal benchmark that addresses limitations of the original MMMU by ensuring reliance on multimodal inputs, expanding answer options, and introducing vision-only settings, providing a more rigorous evaluation of multimodal models' reasoning capabilities.

The paper is generally well-written. The original MMMU has already brought significant impacts to the community. This work is an extension of that, which will be useful for the community as well. 

The paper has generally received scores of 6, with the exception of Reviewer jvww, who did not follow up after the rebuttal.

Upon reviewing all responses, AC noticed that several reviewers were not very excited or enthusiastic about the work, even while supporting its acceptance. They expressed concerns regarding the paper's limited novelty. Specifically, the concerns include:

Similarities between the benchmark dataset and the existing MMMU benchmark. Limited insights derived from extensive model comparisons. The annotations and dataset curation involve significant human effort, with minimal technical innovation introduced.

After discussion among AC and reviewers, due to the limited novelty, the AC decided to reject the paper. However, the authors are highly encouraged to submit an extended version of MMMU to a journal, which often accommodates expansions of original work.

Decision: Reject