MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding
  Benchmark
Authors: name: Xiang Yue; name: Tianyu Zheng; name: Yuansheng Ni; name: Yubo Wang; name: Kai Zhang; name: Shengbang Tong; name: Yuxuan Sun; name: Ming Yin; name: Botao Yu; name: Ge Zhang; name: Huan Sun; name: Yu Su; name: Wenhu Chen; name: Graham Neubig
Published: 2024-09-04 15:31:26

Abstract: This paper introduces MMMU-Pro, a robust version of the Massive
Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
MMMU-Pro rigorously assesses multimodal models' true understanding and
reasoning capabilities through a three-step process based on MMMU: (1)
filtering out questions answerable by text-only models, (2) augmenting
candidate options, and (3) introducing a vision-only input setting where
questions are embedded within images. This setting challenges AI to truly "see"
and "read" simultaneously, testing a fundamental human cognitive skill of
seamlessly integrating visual and textual information. Results show that model
performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8%
to 26.9% across models. We explore the impact of OCR prompts and Chain of
Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT
generally improves performance. MMMU-Pro provides a more rigorous evaluation
tool, closely mimicking real-world scenarios and offering valuable directions
for future research in multimodal AI.