MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and
  Efficient Evaluation
Authors: name: Jinsheng Huang, name: Liang Chen, name: Taian Guo, name: Fu Zeng, name: Yusheng Zhao, name: Bohan Wu, name: Ye Yuan, name: Haozhe Zhao, name: Zhihui Guo, name: Yichi Zhang, name: Jingyang Yuan, name: Wei Ju, name: Luchen Liu, name: Tianyu Liu, name: Baobao Chang, name: Ming Zhang

Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding
and reasoning abilities, often assessed through multiple-choice questions
(MCQs) that include an image, a question, and several options. However, many
benchmarks used for such evaluations suffer from systematic biases. Remarkably,
Large Language Models (LLMs) without any visual perception capabilities achieve
non-trivial performance, undermining the credibility of these evaluations. To
address this issue while maintaining the efficiency of MCQ evaluations, we
propose MMEvalPro, a benchmark designed to avoid Type-I errors through a
trilogy evaluation pipeline and more rigorous metrics. For each original
question from existing benchmarks, human annotators augment it by creating one
perception question and one knowledge anchor question through a meticulous
annotation process. MMEvalPro comprises 2,138 question triplets, totaling
6,414 distinct questions. Two-thirds of these questions are manually labeled
by human experts, while the rest are sourced from existing benchmarks (MMMU,
ScienceQA, and MathVista). Compared with the existing benchmarks, our
experiments with the latest LLMs and LMMs demonstrate that MMEvalPro is more
challenging (the best LMM lags behind human performance by 31.73%, compared
to an average gap of 8.03% in previous benchmarks) and more trustworthy (the
best LLM trails the best LMM by 23.09%, whereas the gap for previous
benchmarks is just 14.64%). Our in-depth analysis explains the reason for
the large performance gap and justifies the trustworthiness of evaluation,
underscoring its significant potential for advancing future research.