Venue: NAACL 2025

# MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation

Jinsheng Huang1,2,3 Liang Chen1,2,* Taian Guo1,2,3 Fu Zeng4, Yusheng Zhao1,2,3 Bohan Wu1,2,3 Ye Yuan1,2,3 Haozhe Zhao1, Zhihui Guo5, Yichi Zhang1, Jingyang Yuan1,2,3 Wei Ju1,2,3 Luchen Liu1,2,3 Tianyu Liu6, Baobao Chang1,21, Ming Zhang1,2,3\({}^{\dagger}\)1National Key Laboratory for Multimedia Information Processing, Peking University 2School of Computer Science, Peking University, 3PKU-Anker LLM Lab 4Chinese Academy of Medical Sciences, 5CUHK, 6Alibaba Group hjs@stu.pku.edu.cn leo.liang.chen@outlook.com {chbb,mzhang_cs}@pku.edu.cn [https://nmevalpro.github.io](https://nmevalpro.github.io)

###### Abstract

Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding and reasoning abilities, often assessed through multiple-choice questions (MCQs) that include an image, a question, and several options. However, many benchmarks used for such evaluations suffer from systematic biases. Remarkably, Large Language Models (LLMs) without any visual perception capabilities achieve non-trivial performance, undermining the credibility of these evaluations. To address this issue while maintaining the efficiency of MCQ evaluations, we propose MMEvalPro, a benchmark designed to avoid Type-I errors through a trilogy evaluation pipeline and more rigorous metrics. For each original question from existing benchmarks, human annotators augment it by creating one perception question and one knowledge anchor question through a meticulous annotation process. MMEvalPro comprises \(2,138\) question triplets, totaling \(6,414\) distinct questions. Two-thirds of these questions are manually labeled by human experts, while the rest are sourced from existing benchmarks (MMMU, ScienceQA, and MathVista). Compared with the existing benchmarks, our experiments with the latest LLMs and LMMs demonstrate that MMEvalPro is **more challenging** (the best LMM lags behind human performance by \(31.73\%\), compared to an average gap of \(8.03\%\) in previous benchmarks) and **more trustworthy** (the best LLM trails the best LMM by \(23.09\%\), whereas the gap for previous benchmarks is just \(14.64\%\)). Our indepth analysis explains the reason for the large performance gap and justifies the trustworthiness of evaluation, underscoring its significant potential for advancing future research.

## 1 Introduction

Ever since the birth of standardized testing, the credibility of its conclusions has been a significant concern. The same problem goes for the evaluation of recently popular Large Multimodal Models (LMMs) such as GPT4-o (OpenAI, 2024), Gemini-1.5 Team et al. (2024), Qwen-VL Bai et al. (2023b) and LAVA Liu et al. (2023b). One classic composition of such an evaluation is the multiple-choice question (MCQ), which includes an image, a question, possible choices, and an answer. This form of evaluation has higher usability

Figure 1: Examples of the probing experiments.

Figure 2: Topic distribution of MMEvalPro’s data.



compared to other evaluation methods, such as text-based evaluation and human evaluation Chen et al. (2024). Many benchmarks Fu et al. (2023); Liu et al. (2023); Lu et al. (2024); Xu et al. (2022); Yue et al. (2023); Zhang et al. (2024); Chen et al. (2023); Kembhavi et al. (2016); Shen et al. (2024) designed for multimodal foundation models include a large portion of MCQs and are widely adopted in testing from basic to most advanced multimodal models OpenAI (2024); Bai et al. (2023); Liu et al. (2023), with some models reaching or even surpassing human scores on certain benchmarks Lu et al. (2022); Kembhavi et al. (2016). However, it is important to consider whether these evaluation results truly reflect the absolute capabilities of the models, especially when we are comparing them with human beings in the pursuit of AGI. In fact, several work such as PCA-Bench Chen et al. (2024), MMStar Chen et al. (2024) and MathVerse Zhang et al. (2024) have pointed out that the multimodal MCQ evaluation has intrinsic bias, which provides Large Language Models LLMs with shortcut to hack the question. FastV Chen et al. (2024) finds that LMM could even achieve better performance on some benchmarks with only partial visual tokens. In this paper, we mainly study three popular multimodal benchmarks MMMU Yue et al. (2023), ScienceQA Lu et al. (2022) and MathVista Lu et al. (2024).

In our preliminary Seeing-or-Not Comparison experiment, we found that LLMs could achieve high scores without processing the visual data, attributed to possible data leakage, visual information problems being not related to answering the question or simply guessing the answer. Notably, the average performance gap between best LLM and best LMM is just 14.64%, which is even smaller than the gap within LMM themselves, revealing the unreliable problem of such evaluations.

Our further Answer Consistency Test into multiple choice questions (MCQ) reveals a prevalent Type-I Error in such evaluation's conclusion, where models could output correct answers without actual comprehension. For example, the model could calculate the degree for a particular angle, but could not recognize the correct angle's name in the figure, which is a prerequisite to compute the degree.

To this end, we propose MMEvalPro to truthfully reflect the true multimodal capabilities of tested models and keep the simplicity of MCQ evaluation. We achieve this by augmenting the original MCQ with prerequisite perception and knowledge questions. We propose Genuine Accuracy as the main metric, which depends on whether the model answers the triplet questions concurrently. Overall, in MMEvalPro we annotate \(2,138\) question

Figure 3: LLMs and LMMs’ performance comparison between original multimodal benchmarks and MMEvalPro. Performance gap between LLM and LMM is much clearer in MMEvalPro.

 

triplets, originating from MMMU, ScienceQA, and MathVista, resulting in \(6,414\) individual questions.

We carry out experiments and analyses involving \(17\) different models and human experts. The findings indicate that MMEvalPro offers a more precise reflection of the tested LMMs' capabilities and poses a more demanding challenge. MMEvalPro is more trustworthy than base benchmarks as the best LLM trails the best LMM by \(23.09\%\) whereas the gap for previous benchmarks is just \(14.64\%\). Significantly, even the most advanced models, including GPT-4o and Qwen-VL-Max, lag considerably behind human performance, with a notable gap of over \(30\%\) in Genuine Accuracy (only \(8.03\%\) for the base benchmarks). Our investigation into the factors contributing to the consistency gap illuminates the existing disparities and supports the evaluation credibility of MMEvalPro, supplying insights for future research endeavors.

## 2 Probing the Credibility of Multimodal Benchmarks

The fundamental assumption of any benchmark is that models achieving higher scores possess superior capabilities. In this section, we question the credibility of such an assumption for existing multimodal benchmarks. We find that the existing benchmarks are not trustworthy enough in either relative or absolute perspectives, which is concluded from two probing experiments: Seeing-or-Not Comparison and Answer Consistency Test. The processes are illustrated in Figure 1. We test the MCQ evaluation in three multimodal benchmarks across various domains including MMMU, ScienceQA-Image and MathVista. We provide the detail of dataset statistics in section 3.1, model inference hyper-parameters and prompts in Appendix-B.

### Seeing-or-Not Comparison

As shown in Figure 1-(a), we prepare two data versions for each benchmark: "Seeing" (with image, question, options, and answer) and "Not-Seeing" (without image). We test leading LMMs on "Seeing" data and non-vision LLMs on "Not-Seeing" data, then compare the results. The outcomes are shown in the first row of Figure 3.

The figure indicates that the performance gap between LMMs and LLMs is significantly narrower than anticipated. Intuitively, one might assume that LLMs, which is unable to process visual information, would perform considerably worse on multimodal benchmarks. In fact, if we compare the scores between the best-performing LLM and LMM, we observe that for MMMU, the best LMM's performance is only 1.1 times that of the best LLM. This performance gap is even smaller than the variability observed within LMMs. Similar results go for the other tested benchmarks. It's more surprising that LLMs sometimes outperform their vision-enabled counterparts (GPT4-o without vision ability outperforms the LLaVA-1.5 series according to Figure 3) and there is not an apparent performance boundary between the two kinds of models. These results suggest that those benchmarks do not accurately reflect the true multimodal understanding capabilities of the tested models. The reason for this phenomenon is three-fold:

**1. Image is not needed:** Some benchmark questions can be answered solely through textual information, making visual input unnecessary. This diminishes the advantage of vision-enabled models (LMMs). For instance, as shown in the ScienceQA question in Figure 1, the knowledge that a falcon uses its feet to capture prey is common and does not require an image for verification.

**2. Data leakage:** During training, LLMs may indarvetently encounter similar questions or datasets, leading to unfair advantages. The existing benchmarks often derive questions from textbooks Lu et al. (2022); Yue et al. (2023), online education resources, and research papers Lu et al. (2024), which are also sources for training datasets Touvron et al. (2023); Clement et al. (2019).

**3. Educated guessing:** LLMs are trained on extensive text datasets, enabling them to make educated guesses even without visual information, which narrows the performance gap with LMMs.

Quantitative AnalysisWe analysed the proportion of samples that can be answered correctly due to "Image is not needed" in the original datasets and the results are shown in the Table 1. As the results shown, the 43.08% proportion in ScienceQA is the highest, which is also reflected in the "Not-Seeing" experiments. From analysis, we can concluded that "Image is not needed" samples significantly

\begin{table}
\begin{tabular}{l c} \hline \hline
**Dataset** & **Proportion** \\ \hline MMMU & 2.97\% \\ ScienceQA & 43.08\% \\ MathVista & 5.37\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Proportions of “Image is not needed” samples in the original datasets 

affected the credibility of previous benchmarks.

These factors collectively result in the unexpectedly narrow performance gap between LLMs and LMMs on multimodal benchmarks. To create a fair multimodal evaluation benchmark: (1) Ensure questions are intrinsically tied to the image details, making visual information essential for deriving the answer. (2) Prevent contamination of training data to ensure models are reasoning through problems instead of recalling memorized answers. (3) Design questions and answers to minimize the likelihood of the model making accurate guesses.

### Answer Consistency Test

To determine whether a model truly understands a question or is simply "hacking" the answer, we simulate the human problem-solving process by creating "anchor questions" that must be answered before the main question. When humans tackle multimodal reasoning questions, they typically follow two key steps: (1) identifying relevant visual clues in the image, and (2) applying their knowledge to reason through the problem before arriving at the final answer. Omitting either step usually leads to an incorrect answer. Similarly, we create a perception anchor question and a knowledge anchor question related to the original question. If a model can answer both anchor questions and the final question, it demonstrates genuine comprehension and reasoning, rather than mere guessing.

An example is shown in Figure 1-(b), we set a perception question and a knowledge question to the MCQ from MathVista. For human examinees, the perception question and knowledge question are easier to answer than the original since they are the prerequisites for the original one in the solution path. We found that even the most advanced LMM such as GPT4o struggles at answering all related questions even if it answers the original question correctly, which is easy for human experts. A detailed case analysis is in Figure 6.

The phenomenon raises another concern for multiple-choice question (MCQ)-based multimodal evaluation benchmarks: correctly answering a question does not necessarily indicate that the model genuinely knows how to derive the final answer. A direct solution to accurately diagnose whether the model truly has the capability to solve the question is to let humans evaluate the model's reasoning process, as done in previous works like MathVista Lu et al. (2024). However, human evaluation is labor-intensive and not reproducible, which complicates the broader application of the method.

Several related works use advanced LLMs like GPT-4 to replace humans in evaluating the reasoning process of LMMs Chen et al. (2024); Zhang et al. (2024). These methods show a strong correlation with human judgments in corresponding tests, but they result in unstable evaluation due to updates of proprietary models and inevitable API costs. Research in using LLMs as evaluators Wang et al. (2023); Chen et al. (2024) also finds systematic bias and a significant gap between open-source and proprietary models in terms of evaluation agreement with human experts. These drawbacks highlight the need for an economical, easy-to-use, and calibrated method for multimodal models.

## 3 MMEvalPro: Calibrating Multimodal Evaluation

In this section, we delve into the detailed process of constructing the MMEvalPro benchmark dataset and elucidate our methodologies for ensuring high-quality evaluation standards. Through these efforts, MMEvalPro sets a new paradigm in the assessment of multimodal models, aiming to foster both accuracy and efficiency in multimodal evaluation.

### Data Source

To enhance the diversity of our benchmark data, MMEvalPro integrates content from three prominent multimodal benchmarks: MMMU, ScienceQA, and MathVista. These benchmarks span educational levels from junior high to undergraduate and cover various subjects. The details of the source datasets are in Appendix A.1. Considering annotator expertise and budget, we selected \(328\) questions from MMMU(dev), \(1,200\) from ScienceQA-Image, and all \(540\) from MathVista, totaling \(2,138\) distinct multimodal MCQs.

### Annotation Pipeline

As illustrated in Figure 4, we design the following pipeline for MMEvalPro to generate question triplets. The triplet consists of an original question, a perception question, and a knowledge question.

**1) Data Preparation:** Annotators begin by thoroughly reviewing the original question to ensure a deep understanding of concepts and solutions.

**2) View and Analyse:** Annotators are tasked with extracting crucial visual information and the logical framework implicit in the original problem, paving the way for the creation of nuanced perception and knowledge questions 



**3) Question Annotation:** Building on the insights gathered, annotators then proceed to enrich the original question by formulating corresponding perception and knowledge questions, thereby expanding the scope of the evaluation.

**4) Double Check:** To maintain the integrity of the MMEvalPro dataset, each annotated question triplet undergoes a rigorous verification process. Two independent checkers, who are not part of the annotation team, review each triplet for any errors or logical inconsistencies. Any issues identified prompt a re-annotation of the affected questions.

We provide the annotator guide in the supplement material. The final distribution of MMEvalPro is shown in Figure 2. We list the key statistics of the benchmark in Table 4 from Appendix 3.1, the annotation guidelines in Appendix C and D.

### Evaluation Metrics

We propose **Genuine Accuracy (GA)** as the primary metric for MMEvalPro. GA equals \(1\) only if the model correctly answers the original question and the corresponding perception and knowledge prerequisite questions simultaneously. The second metric is the **Average Accuracy (AA)**, which computes the average accuracy of all questions.

MMEvalPro can also be viewed as a multi-view evaluation process, where we naturally derive the **Perception Accuracy (PA)** score and the **Knowledge Accuracy (KA)** score by computing the average accuracy for the perception and knowledge anchor questions, respectively.

The **Consistency Gap (CG)** is measured by subtracting the Genuine Accuracy from the accuracy of the original question. This metric reflects the proportion of instances where the model correctly answers the original question but fails on more indepth perception and knowledge questions, leading to inconsistency in its answers. To gain deeper insights into the answer consistency and generalization capabilities of the tested models, we define **Perception Consistency (PC)** and **Knowledge Consistency (KC)** as the conditional probabilities \(P( Perception=1~{}|~{}Origin=1)\) and \(P(Knowledge=1~{}|~{}Origin=1)\), where ( \(=1\) ) indicates correctly answering the corresponding question in a question triplet. PC and KC together reveal the answer consistency of the tested model.

### Quality Evaluation

To ensure the precision and consistency of our annotations, all question triplets are annotated by graduate students. For specialized college-level subjects in the MMMU sub-set, we hire students in the corresponding major to ensure annotation accuracy. All annotators are required to thoroughly familiarize themselves with the guide before commencing their annotation tasks. During the annotation process, we assign a minimum of three annotators for one question triplet. These foundations ensure the quality of the new annotated \(4,276\) questions and there are only \(47\) questions that go through more than two double-check circles. Consensus was reached on all examples in the end.

## 4 Experiments

In this section, we evaluate a wide range of models on our MMEvalPro benchmark. We first introduce the evaluation setup, and then present the quantitative results for both open-source and closed-source models. Finally we investigate the answer consistency gap among LLMs, LMMs and human experts with fine-grained analysis.

### Setup

We test multiple LLMs and LMMs on the original benchmarks and MMEvalPro. To streamline the evaluation process, all questions are converted

Figure 4: Annotation pipeline for MMEvalPro.



into a multiple-choice format, prompting the models to directly provide the answers. This approach simplifies answer matching process and eliminates the need for external models like ChatGPT, which is commonly used in previous studies such as Lu et al. (2024); Chen et al. (2024); Zhang et al. (2024). In line with prior research, we incorporate an intercontext demonstration for LLMs to standardize the output format. This is important as we have observed that different LLMs tend to produce varying response formats when not provided with an example. The detailed prompt and demonstration template are listed in Appendix B. For the original benchmarks, we report the average accuracy on the MCQ questions the same as we sampled for creating MMEvalPro for fair comparison. For MMEvalPro, we report both the Genuine Accuracy and Average Accuracy.

### Evaluated Models

In our evaluation, we evaluate a variety of both LLMs and LMMs. For LLMs, we implement a 1-Shot setting, where a single demonstration is utilized to guide the output format. Among the LLMs assessed are some of the most advanced open-source models, including four versions of Qwen-1.5-Chat Bai et al. (2023) with sizes spanning from 7B to 72B, as well as the LLaMA3-Instruct Touvron et al. (2023) series (8B and 7OB). We also tested API-only models, such as GPT-3.5-Turbo OpenAI (2022) and GPT4-o OpenAI (2024), which rely solely on language. On the other hand, LMMs are evaluated in a zero-shot manner due to their ability to follow instructions to produce valid answer choices. In the open-source category, we tested the LLaVA-1.5 Liu et al. (2023) and LLaVA-Next Liu et al. (2024) series, which include models of 7B, 13B, and 34B in size. We also evaluated two of the most cutting-edge LMMs, MiniCPM-V2.5-LLaMA3-8B Hu et al. (2023) and InternVL-1.5-Chat-20B Chen et al. (2023), both of which have recently been made available to the public. For proprietary models, we tested GPT-4V-1106-preview OpenAI (2023), the latest GPT-4o OpenAI (2024), and Qwen-VL-Max Bai et al. (2023). We provide the prompts for LLMs and the hyperparameters used for LMMs in Appendix B.

### Experiment Results

As shown in Table 2, we compare the performance of different LLMs and LMMs on MMEvalPro and the original benchmarks. We evaluated the performance of graduate students on the benchmarks as a strong baseline. The human evaluation guideline is shown in Appendix E. We also include random guess as a weak baseline performance.

We first evaluate LLMs and LMMs separately. For LLMs, all models perform poorly under the Genuine Accuracy metric in MMEvalPro. For example, the advanced GPT-4o achieves \(67.62\%\) accuracy on the original ScienceQA-Image benchmark but drops to \(10.30\%\) on its calibrated version in MMEvalPro, a decrease of \(57.32\) percentage points. Similar declines are observed in MMMU (down \(37.38\%\)) and MathVista (down \(40\%\)). These low scores more accurately reflect LLMs' general multimodal capabilities due to their lack of visual perception abilities. Open-source LLMs also show drastic declines in Genuine Accuracy. The Qwen-1.5 series suggests that larger LLMs perform better on both benchmarks, supporting the idea that stronger LLMs are better at guessing. However, this trend does not hold for the LLaMA-3 series, indicating a need for our further investigation in the future. Generally, most LLMs score below \(10\%\) in Genuine Accuracy, highlighting the difficulty of the proposed benchmark for LLMs.

On the LMMs side, we also witness a large performance gap between the original benchmarks and MMEvalPro. For proprietary models, GPT-4o and Qwen-VL-Max perform the best. If we compare the performance gap of best LLM and best LMM on the original benchmark and MMEvalProb, we could find that the performance difference in scales is more clear on MMEvalProb. For example, in original MMMU benchmark, GPT-4o with vision (\(54.85\%\)) is only 1.1 times its non-vision LLM version (\(46.09\%\)). While in MMEvalPro, the LMM (\(25.08\%\)) version's performance is \(2.9\) times the LLM (\(8.71\%\)) version of GPT-4o. A similar result also goes for other tasks. The enlarged performance discrepancy is more intuitive given the fundamental functionality difference between LLMs and LMMs. MMEvalPro could better reflect the true abilities of examinees.

### Fine-grained Analysis

To better explain models' performance on MMEvalPro, we conducted a fine-grained analysis on the experiments' result according to the metrics proposed in section 3.3. We select the best performing open-source and proprietary LLMs and LMMs to analyze. The results are shown in Table 3. A detailed case analysis is shown in Figure 6 

More comparisons of different models are listed in Figure 16 from the Appendix.

Why MMEvalPro is challenging and more trustworthy?When comparing human performance with that of LLMs and LMMs, we observe a more significant gap in MMEvalPro evaluations than in original benchmarks. This indicates that MMEvalPro is inherently a more challenging task. The primary difficulty stems from the issue of answer consistency, which demonstrates whether the model genuinely understands how to leverage perceptual abilities and knowledge to solve a given problem. To illustrate this, we compare the Consistency Gap (CG) scores among humans, the best-performing LLMs and LMMs. The results suggest that LLMs generally exhibit a larger Consistency Gap than LMMs, while human experts display a considerably smaller CG. This trend is consistent across both open-source and proprietary models. A large Consistency Gap indicates that a model's robustness and generalization abilities are limited: it may be able to answer the original question but fails to respond accurately to related prerequisite questions based on the same image.

This weakness is difficult to capture for the original benchmarks due to the single MCQ format. In fact, if we only consider the Average Accuracy of MMEvalPro, as shown in Table 2, we ob

\begin{table}
\begin{tabular}{l c|c c c c|c c c c|c c c c} \hline  & & & \multicolumn{4}{c}{**Original Source**} & \multicolumn{5}{c}{**MMEvalPro**} \\ Model & Open* & MMU & ScienceQA & MathVista & Average & MDMU & ScienceQA & MathVista & Average & \multirow{3}{*}{
\begin{tabular}{} \end{tabular} } & \multirow{3}{*}{Average } \\  & & & _(average Accuracy)_ & & & & & & & & & & & & & & \\ \hline \multicolumn{9}{c}{_Large Language Models (L-Shou)_} \\ \hline Owen-15-7B-Chat & Yes & 34.48% & 58.32% & 39.07% & 43.96% & 1.95% / 30.30% & 6.21% / 41.08% & 2.78% / 29.19% & 3.65% / 33.52% \\ Owen-15-14B-Chat & Yes & 32.68% & 61.75% & 43.45% & 45.86% & 5.48% / 35.05% & 8.09% / 37.44% & 4.65% / 35.74% & 6.05% / 38.18% \\ Owen-15-25B-Chat & Yes & 99.21% & 64.66% & 42.96% & 48.49% & 5.81% / 39.56% & 7.40% / 45.31% & 5.56% / 37.53% & 6.26% / 40.80% \\ Owen-15-72B-Chat & Yes & 93.33% & 66.18% & 43.52% & 49.68% & 5.80% / 38.06% & 1.30% / 41.09% & 5.19% / 36.98% & 7.10% / 38.71% \\ LLAMA-38-Instruct & Yes & 38.36% & 68.57% & 34.51% & 50.15% & 4.19% / 34.09% & 6.22% / 45.24% & 4.65% / 34.32% & 5.01% / 37.88% \\ LLAMA-30-Insortect & Yes & 35.56% & 63.47% & 37.95% & 45.48% & 48.34% & 49.85% & 4.02% / 40.26% & 2.59% / 31.85% & 1.56% / 36.95% \\ GPF-3-Turbo & No & 34.21% & 61.85% & 40.18% & 45.41% & 6.13% / 38.28% & 9.98% / 45.87% & 4.44% / 33.35% & 6.85% / 39.24% \\ GPT-4o (non-vision) & No & 46.09% & 67.62% & 45.19% & 52.79% & 8.11% / 39.09% & 10.30% / 48.18% & 5.19% / 37.78% & 8.07% / 41.95% \\ \hline \multicolumn{9}{c}{_Large Multimodal Models (Zero-Shot)_} \\ \hline LiAxVA-1-5-Viuma-7B & Yes & 40.86% & 62.61% & 36.11% & 46.53% & 11.15% / 43.06% & 25.64% / 58.33% & 4.81% / 37.10% & 13.87% / 46.16% \\ LiAxVA-1-Si-Vuma-1B & Yes & 39.92% & 66.76% & 37.95% & 48.09% & 10.82% / 43.28% & 29.94% / 63.20% & 5.19% / 36.98% & 15.32% / 47.83% \\ Alexa-Next-View-13B & Yes & 39.75% & 67.57% & 4.149% & 60.96% & 12.13% / 47.86% & 13.65% / 46.88% & 8.07% / 39.26% & 16.72% / 56.76% \\ LiaVA-Net-Hermes-Y34B & Yes & 47.89% & 74.77% & 53.70% & 58.79% & 23.60% / 59.34% & 43.67% / 73.35% & 18.15% / 54.75% & 28.47% / 62.48% \\ MinConv-25-ALLMA-38-Harr & Yes & 48.86% & 77.68% & 51.32% & 57.95% & 18.36% / 54.86% & 43.65% / 37.96% & 13.60% / 52.28% & 26.07% / 60.37% \\ Intent-11-SCAL-20B & Yes & 46.12% & **87.27%** & **60.44%** & 67.13% & 39.34% / 56.53% & 4.499% / 28.22% & 20.74% / 57.82% & 8.25% / 63.22% \\ GPT-4v110-preview & No & 41.42% & 77.54% & 51.85% & 57.90% & 22.30% / 57.99% & 39.81% / 71.34% & 10.46% / 48.48% & 20.55% / 59.26% \\ GPT-4o & Nune & 48.48% & 84.13% & 55.66% & 61.72% & **50.82%** & **49.49%** & **48.76%** & **15.43%** / 52.63% & **0.066%** / 62.63% \\ Oven-VL-Max & No & 48.75% & 76.22% & 59.07% & 61.36% & 24.38% / 59.45% & 47.61% / 75.02% & **21.48%** / **59.19%** & **31.16%** / **44.55%** \\ Random Guess & - & 26.04% & 35.53% & 32.41% & 31.33% & 1.94% / 28.00% & 2.36% / 29.26% & 3.324/ 9.14% & 2.54% / 29.01% \\ Human (Graduate Student) & - & 49.21% & 76.92% & 92.08% & 72.76% & 38.10% / 66.67% & 64.42% / 81.09% & 86.14% / 93.07% & 62.89% / 80.28% \\ \hline \end{tabular}
\end{table}
Table 2: Main experiments result. For average, we report the macro average scores (the mean of different domains). Highest score is marked **bold** and the second highest is underlined. Best LMM still lags behind human with a substantial gap (\(-31.73\%\) average GA) on MMEvalPro.

Figure 5: Heatmaps of conditional accuracy of MMEvalPro.

 

serve that the performance difference compared to the original benchmarks is much smaller than the difference in Genuine Accuracy (GA). This suggests that LLMs can often guess correct answers for questions even without image input. MMEVAL-Pro addresses this issue effectively through the GA metric, making it a more reliable evaluation method compared to previous benchmarks.

Why the Consistency Gap is large?The gap between Genuine Accuracy and Average Accuracy on the original benchmarks reveals the answer inconsistency problem. We are further interested in what causes the problem. There are two possible reasons for a large Consistency Gap, that the model correctly answer the original question however fails on the perception or knowledge one. We compare the **Perception Consistency (PC)** and **Knowledge Consistency (KC)** of the evaluated models and humans. We find that there is a clear performance border between humans and LMMs, LMMs and LLMs. Humans could reach at least \(90\%\)PC and \(80\%\)KC in various sub-tasks, showing strong answer consistency. While the numbers for LMMs are \(50\%\)PC and \(55\%\)KC, for LLMs are \(23\%\)PC and \(41\%\)KC according to Table 3.

PC and KC intuitively reflect the model's likelihood of correctly answering perception and knowledge questions if it has already solved the original question. Low PC and KC scores lead to a significant consistency gap. Beyond PC and KC, we visualize all conditional probabilities for the best

Figure 6: Case study on the answer in-consistency problem of LMMs. We could tell from the figure that GPT-4o could answer the original question however failed on the perception and knowledge question. The wrong reasoning process is marked red. The multiply mentioned items are marked with the same background color. In the reasoning process of original question, GPT-4o directly points out that \(\angle\)BOC = 90\({}^{\circ}\) + (1/2) \(\angle\)BAC. However in the knowledge question while the ”/BOC = 90\({}^{\circ}\) + (1/2) \(\angle\)BAC” is just one of the options, the model failed to figure it out, while human could easily achieve it. It shows the answer in-consistency problem of the LMM, which can not be pinpointed by single MCQ evaluation. If we further look at the reasoning process of the perception question, we could find that the model could not even recognize the correct number of triangles in the figure. For example it recognized AOC as a trangle, which is actually not. This also causes its problematic reasoning for the knowledge question.

 

open-source LLM and LMM in Figure 5. Humans generally exhibit high probabilities of correctly answering one question given a correct answer to another, indicating consistent thinking. In contrast, LLMs show the lowest probabilities compared to LMMs and humans. This is expected, as LLMs lack consistent multimodal problem-solving paths due to their absence of visual perception, thus supporting the credibility of the benchmarks.

Examining the Perception Accuracy (PA) and Knowledge Accuracy (KA) in Table 3, we find that LMMs demonstrate a greater advantage in PA compared to KA when contrasted with LLMs. This is because PA depends directly on visual capabilities, which LLMs lack. The above conclusion explains why tested models have larger CG compared to humans, which is a potential and promising direction for future LMMs to improve on.

## 5 Related Work

There have been several benchmarks built for evaluating LMMs Feng et al. (2024); Yang et al. (2024), such as MMBench, MME, Seed-Bench Liu et al. (2023c); Fu et al. (2023); Li et al. (2023a) that assess LMMs performance from multiple fine-grained dimensions. LVLM-eHub, M3IT Xu et al. (2023); Li et al. (2023b) focus on the general instruction following ability. MMMU, MathVista, ScienceQA Yue et al. (2023); Lu et al. (2024, 2022) require perception from the vision part and knowledge in the language part.

Nonetheless, critiques have been raised regarding the limitations of these existing benchmarks in effectively evaluating LMMs. PCA-Bench, MatHverse Chen et al. (2024b); Zhang et al. (2024b) adopt strong LLMs such as GPT4 and GPT4-Vision to score the reasoning process of LMMs in embodied-AI and math diagram questions, in order to pinpoint cases where the LMM gets the correct answer by a fluke. Yet, using a proprietary model to conduct evaluation hinders the broader usage of the method, moreover, the evaluation result has bias itself due to the proxy model and would change over time. MMStar Chen et al. (2024d) filters out the questions that do not rely on visual information in existing multimodal benchmarks. However, it does not address the issue inherent in MCQ, where models can potentially get the correct answer without truly understanding the content. Compared with those benchmarks, MMEvalPro is more economical, easy-to-use, and calibrated for evaluating multimodal models.

## 6 Conclusion

We propose MMEvalPro, a multimodal benchmark designed to address issues identified in previous evaluations and built upon MMMU, ScienceQA-Image, and MathVista. MMEvalPro introduces twin perception and knowledge anchors to the original framework and defines Genuine Accuracy as its primary metric, thereby reducing the likelihood of LLMs manipulating the questions. Our extensive experiments and analyses on a wide array of models and human experts demonstrate that MMEvalPro more accurately reflects the true capabilities of the tested LMMs and presents a more challenging task. Notably, even the most advanced models, such as GPT-4o and Qwen-VLMax, trail behind human performance by a substantial gap of more than \(30\%\) in Genuine Accuracy. Our analysis into the reasons behind the consistency gap problem elucidates the disparity and provides valuable insights for future research.

### Limitations

In order to ensure the high quality and accuracy of MMEvalPro, we employed manual annotation with human experts to construct the dataset. A certain level of human effort and expertise are necessary. To some extent, this requirement may limit the expansion efficiency of MMEvalPro.

#### Acknowledgments

This paper is partially supported by the National Key Research and Development Program of China with Grant No.2023YFC3341203 as well as the National Natural Science Foundation of China with Grant Numbers 61876004 and 62306014.

## References

* Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Bingyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jiawei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023a. Qwen technical report. _Preprint_, arXiv:2309.16609 

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A frontier large vision-language model with versatile abilities. _ArXiv preprint_, abs/2308.12966.
* Chen et al. (2024) Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, et al. 2024a. Next token prediction towards multimodal intelligence: A comprehensive survey. _arXiv preprint arXiv:2412.18619_.
* Chen et al. (2023a) Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang. 2023a. Towards end-to-end embodied decision making via multi-modal large language model: Explorations with gpt4-vision and beyond. _ArXiv_.
* Chen et al. (2024b) Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baohao Chang. 2024b. Pcabench: Evaluating multimodal large language models in perception-cognition-action chain. _Preprint_, arXiv:2402.15527.
* Chen et al. (2024c) Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baohao Chang. 2024c. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. _Preprint_, arXiv:2403.06764.
* Chen et al. (2024d) Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024d. Are we on the right way for evaluating large vision-language models? _arXiv preprint arXiv:2403.20330_.
* Chen et al. (2023b) Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. _arXiv preprint arXiv:2312.14238_.
* Clement et al. (2019) Colin B. Clement, Matthew Bierbaum, Kevin P. O'Keeffe, and Alexander A. Alemi. 2019. On the use of arxiv as a dataset. _Preprint_, arXiv:1905.00075.
* Feng et al. (2024) Bin Feng, Zequn Liu, Nanlan Huang, Zhiping Xiao, Haomiao Zhang, Srbuhi Mirzoyan, Hanwen Xu, Jiaran Hao, Yinghui Xu, Ming Zhang, et al. 2024. A bioactivity foundation model using pairwise meta-learning. _Nature Machine Intelligence_, 6(8):962-974.
* Fu et al. (2023) Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. _Preprint_, arXiv:2306.13394.
* Hu et al. (2020) Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Large multilingual models pivot zero-shot multimodal learning across languages. _arXiv preprint arXiv:2308.12038_.
* Kembhavi et al. (2016) Aniruddhha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishrizi, and Ali Farhadi. 2016. A diagram is worth a dozen images. _ArXiv_, abs/1603.07396.
* Li et al. (2023a) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023a. Seed-bench: Benchmarking multimodal lllms with generative comprehension. _Preprint_, arXiv:2307.16125.
* Li et al. (2023b) Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. 2023b. M\({}^{3}\)IT: A large-scale dataset towards multi-modal multilingual instruction tuning. _ArXiv preprint_, abs/2306.04387.
* Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning.
* Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llavanext: Improved reasoning, ocr, and world knowledge.
* Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. _ArXiv preprint_, abs/2304.08485.
* Liu et al. (2023c) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023c. Mmbench: Is your multi-modal model an all-around player? _Preprint_, arXiv:2307.06281.
* Lu et al. (2024) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishrizi, Hao Cheng, Kaik-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. _Preprint_, arXiv:2310.02255.
* Lu et al. (2022) Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kaik-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_.
* OpenAI (2022) OpenAI. 2022. Introducing chatgpt. _Web_.
* OpenAI (2023) OpenAI. 2023. Gpt-4v(sion) system card. _Web_.
* OpenAI (2024) OpenAI. 2024. hello-gpt-4o.
* Shen et al. (2024) Jianhao Shen, Ye Yuan, Srbuhi Mirzoyan, Ming Zhang, and Chenguang Wang. 2024. Measuring vision-language stem skills of neural models. _Preprint_, arXiv:2402.17205 

* [34] M. C. Bershadsky, A. K. D. Lane, and J. E. Gunn, "The Origins of the Universe: The Role of Dark Matter in the Early Universe," _Astrophys. J. Suppl._**106**(2), 579-680, doi:10.1142/S021771X1060 

shen Wang, Dave Lacey, Anaststaja Ilic, Yao Zhao, Adam Ivanicki, Alejandro Lince, Alexander Chen, Christina Lyu, Carl Lebsack, Jordan Griffith, Meenu Gaba, Paramjit Sandhu, Phil Chen, Anna Koop, Ravi Rajwar, Soheil Hassas Yeganeh, Solomon Chang, Rui Zhu, Soroush Radpour, Elnaz Davoodi, Ying Ian Lei, Yang Xu, Daniel Toyama, Constant Segal, Martin Wicke, Hanzhao Lin, Anna Bulanova, Adria Puigdomenech Badia, Nemanja Rakicevic, Pablo Sprechmann, Angelos Filios, Shaobo Hou, Victor Campos, Nora Kassner, Devendra Sachan, Meire Fortunato, Chimeze Iwuanyanwu, Vitaly Nikolaev, Balaji Lakshminarayanan, Sadogh Jazayeri, Mani Varadarajan, Chetan Tekur, Doug Fritz, Misha Khalman, David Reitter, Kingschuk Dasgupta, Shourya Sarcar, Tina Ornduff, Javier Snaider, Fantine Huot, Johnson Jia, Rupert Kemp, Nejer Trdin, Anitha Vijayakumar, Lucy Kim, Christof Angermuller, Li Lao, Tianqi Liu, Haibin Zhang, David Engel, Somor Greene, Anais White, Jessica Austin, Lilly Taylor, Shereen Ashraf, Dangyi Liu, Maria Georgaki, Irene Cai, Yana Kulizhskaya, Sonnam Goenka, Brennan Saeta, Ying Xu, Christian Frank, Davio de Cesare, Brona Robenek, Harry Richardson, Mahmoud Alhanhai, Christopher Yew, Priya Ponnamalli, Marco Tagliascchi, Alex Korchemnyi, Yelin Kim, Dinghua Li, Bill Rosgen, Kyle Levin, Jeremy Wiesner, Praseem Banzal, Praveen Srinivasan, Hongkun Yu, Caglar Unla, David Reid, Zora Tung, Daniel Finchelstein, Ravin Kumar, Andre Elisewef, Jin Huang, Ming Zhang, Ricardo Aguilar, Mai Gimenez, Jiawei Xia, Olivier Dousse, Willi Giereke, Damion Yates, Komal Jalan, Lu Li, Eri Latorre-Chimoto, Duc Dung Nguyen, Ken Durden, Praveen Kallakuri, Yaxin Liu, Matthew Johnson, Tomy Tsai, Alice Talbert, Jaymine Liu, Alexander Neitz, Chen Elkind, Marco Selvi, Jimi Jasarevic, Livio Baldini Soares, Albert Cui, Pidong Wang, Alek Wenjiao Wang, Xinyu Ye, Krystal Kalintackal, Lucia Loher, Hoi Lam, Josef Broder, Dan Holtmann-Rice, Nina Martin, Bramadia Ramadhana, Mrinal Shuklai, Sujoy Basu, Abhi Mohan, Nick Fernando, Noah Fieldel, Kim Paterson, Hui Li, Ankush Garg, Jane Park, DongHyun Choi, Diane Wu, Sankalg Singh, Zhishua Zhang, Amir Globerson, Lily Yu, John Carpenter, Felix de Chaumont Quitry, Carey Rabdebapathi, Chu-Cheng Lin, Alex Tudor, Pakash Shroff, Drew Garmon, Dayou Du, Neera Vats, Han Lu, Shariq Iqbal, Alex Yakubovich, Nilesh Tripuraneni, James Manyika, Haroon Qureshi, Nan Hua, Christel Ngani, Maria Abi Raad, Hannah Forbes, Jeff Stanway, Mukund Sundararajan, Victor Ungreuneau, Colton Bishop, Yunje Li, Balaji Venkatraman, Bo Li, Chao Thornton, Salvatore Scellato, Nishesh Gupta, Yicheng Wang, lan Tenney, Xihui Wu, Ashish Shenoy, Gabriel Carvajal, Diana Gage Wright, Ben Bariach, Zhuyu Xiao, Peter Hawkins, Sid Dalmia, Clement Farabet, Pedro Valenzuela, Quan Yuan, Ananth Agarwal, Mia Chen, Wooyool Kim, Brice Hulse, Nandita Dukkipati, Adam Paszke, Andrew Bolt, Kiam Choo, Jennifer Beattie, Jennifer Prendki, Harsha Vashishth, Rebecca Santamaria-Fernandez, Luis C. Cobo, Jarek Wilkiex, David Madras, Ali Elapursh, Grant Uy, Kevin Ramirez, Matt Harvey, Tyler Liecthy, Heiga Zen, Jeff Seibert, Clara Huiyi Hu, Andrey Khorlin, Maigo Le, Asaf Aharoni, Megan Li, Lily Wang, Sandeep Kumar, Norman Casagrande, Jaay Hoover, Dalia El Badavy, David Soegerlei, Denis Vukov, Matt Sirenikowski, Jiri Simsa, Praveen Kumar, Thibault Skallam, Daniel Vlasic, Samira Daruki, Nir Shabat, John Zhang, Guolong Su, Jiageng Zhang, Jeremiah Liu, Yi Sun, Evan Palmer, Alireza Ghaffarkhah, Xi Xiong, Victor Cotrata, Michael Fink, Lucas Dixon, Ashwin Stewartesavatas, Adrian Goedeckemeyer, Aleks Dimmitriev, MohesneJafari, Remmi Crocker, Nicholas FitzGerald, Aviral Kumar, Sanjay Ghemawat, Ivan Philips, Frederick Liu, Yannie Liang, Rachel Sterneck, Alena Repina, Marcus Wu, Laura Knight, Marin Georgeiv, Hyo Lee, Harry Askham, Abhishek Chaklader, Annie Louis, Carl Crous, Hardie Cate, Dessein Petrova, Michael Quinn, Demese Owusu-Afrique, Achindya Singhal, Nan Wei, Solomon Kim, Damien Vincent, Milad Nasr, Christopher A. Choquette-Choo, Reiko Tojo, Shawn Lu, Diego de Las Casas, Yuchung Cheng, Tolga Bolukhasi, Katherine Lee, Saaber Fatehi, Rajagopal Ananthanarajanan, Miteyan Patel, Charbel Kaed, Jing Li, Shereyas Rammohan Belle, Zhe Chen, Jaclyn Konzelmann, Siim Poder, Roopal Garg, Vinod Kovachtuh, Adam Brown, Chris Dyer, Rosanne Liu, Azade Nova, Jun Xu, Alanna Walton, Alicia Parrish, Mark Epstein, Sara McCarthy, Slav Petrov, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _Preprint_, arXiv:2403.05530.
* [69] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLama: Open and efficient foundation language models. _ArXiv preprint_, abs/2302.13971.
* [70] Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. _ArXiv_, abs/2305.17926.
* [71] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. _Preprint_, arXiv:2306.09265.
* [72]Junwei Yang, Hanwen Xu, Srbuhi Mirzoyan, Tong Chen, Zixuan Liu, Zequn Liu, Wei Ju, Luchen Liu, Zhiping Xiao, Ming Zhang, et al. 2024. Poisoning medical knowledge using large language models. _Nature Machine Intelligence_, 6(10):1156-1168.
* [73] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yukuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Reniliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Weinu Chen. 2023. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. _Preprint_, arXiv:2311.16502 

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yadong Liu, Yu-Hsuan Tsai, Fengji Zhang, Chenghua Lin, Wenhao Huang, Wenhu Chen, and Jie Fu. 2024a. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. _Preprint_, arXiv:2401.11944.
* Zhang et al. (2024b) Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. 2024b. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? _Preprint_, arXiv:2403.14624.

## Appendix A Details on MMEvalPro

### Data Source

**MMMU Yue et al. (2023):** The MMMU is a benchmark designed to evaluate multimodal models on massive multi-discipline tasks. The benchmark sources its questions from college examinations, quizes, and textbooks, encompassing \(30\) subjects and \(183\) subfields across six core disciplines, i.e. Art & Design, Business, Science, Health & Medicine, Humanities & Social Science and Tech & Engineering. This benchmark is meticulously designed to assess models' capabilities in handling multi-disciplinary tasks, drawing upon college-level subject knowledge.

**ScienceQA Lu et al. (2022):** The Science Question Answering (ScienceQA) is another pivotal resource, which consists of 21,208 multimodal multiple choice questions with diverse science topics. There are only \(48.7\%\) questions of ScienceQA that have an image context. It is renowned for its application in multimodal tasks and features a domain diversity spanning three primary science subjects, i.e., natural, language, and social. The dataset comprises multimodal science questions that are collated from elementary and high school science curricula, ensuring a breadth of scientific inquiry and comprehension. The subjects of the questions can be categorized by Biology, Physics, Chemistry, and others.

**MathVista Lu et al. (2024):** The MathVista benchmark is developed to evaluate the reasoning ability of the multimodal models, which consists of \(6,141\) examples from \(31\) datasets (\(28\) mathematics and IQTest, FunctionQA, PaperQA). The dataset offers exclusively mathematical and visual tasks. This source enriches our dataset with rigorous computational and analytical problems, providing a robust framework for evaluating quantitative reasoning in multimodal contexts.

Finally, we select the validation set of MMMU (\(722\) questions), the questions with images in ScienceQA (\(2,097\) questions), and the testmini set of MathVista (\(540\) questions) to construct the MMEvalPro. From the \(722\) questions in the validation set of MMMU, we chose the questions with topics suited for the annotator's major and other questions in easy-level to annotate, which resulted in \(339\) final questions. And we annotated all questions selected from the testmini set of MathVista 

and \(1,259\) of ScienceQA-Image.

The distribution of MMEvalPro is shown in the Figure 2. There are \(58.89\%\) questions originating from ScienceQA, \(25.26\%\)from MathVista, and \(15.86\%\) from MMMU. We further categorize the problems by task categories, subjects, and domains, computing their relative percentages to the total questions. In the annotated questions originated from ScienceQA, there are \(16.68\%\) of them subject to Geography, \(12.61\%\) subject to Physics, \(12.02\%\) with the Biology topic, and the remaining questions distributed in History, Chemistry, Economics, etc. In MathVista, the annotated questions contain 5 tasks, including GPS(geometry problem solving), VQA(visual question answering), FQA(figure question answering), TQA(textbook question answering), and MWP(math word problem), the last one only accounted for a very small proportion and is not annotated in the figure. According to the subjects of questions from MMMU, there are \(4.91\%\) questions in Medicine, \(3.27\%\) in Art, \(3.65\%\) in Business, \(2.71\%\) in Science, and \(1.32\%\) from other subjects with easy-level.

### License

We check all the datasets' licenses and they all permit customization and redistribution for noncommercial use.MMMU is under Apache 2.0 License, ScienceQA is under MIT License and MathVista is covered by CC BY-SA 4.0.

MMEvalPro can be used commercially as a test set, but using it as a training set is prohibited. By accessing or using this dataset, users acknowledge and agree to abide by these terms in conjunction with the CC BY-SA 4.0 license. We make sure there is no offensive content found during the whole annotation process.

### MmEvalPro Examples

We list examples of different splits of MMEvalPro as shown in Figure 7, 8 and 9.

### Statistics of MmEvalPro

The key statistics of MMEvalPro is shown in the table 4. Figure 10 display the distribution of answers and the number of options in the MMEvalPro.

## Appendix B Experiment Setup

Prompt Format for Different Models For LLMsu given a question you need to choose the best answer from the given options. I will first

\begin{table}
\begin{tabular}{l c} \hline \hline
**Statistic** & **Number** \\ \hline Source datasets & 3 \\ Number of question triplets & 2,138 \\ Number of unique questions & 6,414 \\ Triplets from MMMU & 339 \\ Triplets from ScienceQA & 1259 \\ Triplets from MathVista & 540 \\ \hline Maximum question length & 165 \\ Maximum choice number & 12 \\ Average question length & 9.60 \\ Average choice number & 3.94 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Key statistics of MMEValPro.

Figure 8: Examples of the ScienceQA subset of MMEvalPro.

Figure 7: Examples of the MathVista subset of MMEvalPro.

 

give you an example, you need to follow the output format of the answer. Example: Question: {example question} Options: {example options} Output: {example answer}. Just answer the following question with only the letter of the correct option, or you will get no credit. Question:{question} Options: {options}"

We use a fixed demonstration for all inferences to format the output. The example is:"**Question:** Baxter Company has a relevant range of production between 15,000 and 30,000 units. The following cost data represents the average variable costs per unit for 25,000 units of production. If 30,000 units are produced, what are the per unit manufacturing overhead costs incurred? **Options:** (A) SS6 (B) S7 (C) $8 (D) $9 **Output:** A".

For LMM\({}^{*}\)Analyse the image and choose the best answer for the following question:{question} Options: {options} Just output the letter of the correct answer."

### Model Hyper-parameters

For open-source models, we use the default inference script provided in corresponding papers and githubs. For proprietary models, we follow the official guide to call the API. In particular, we do not use sampling techniques during generation to ensure our results are reproducible. All experiments are done on a local server with 4 NVIDIA-A100 GPUs.

Figure 10: Distribution of Answer Choices in MMEvalPro.

 



## Appendix C MMEvalPro Triplet Annotation Guideline

We developed an annotation Web UI to enable expert annotators to construct question triplets of MMEvalPro. The Web UI is shown in the figure 11. The annotators of MMEvalPro were trained with the guideline shown in figure 12 before formal work.

## Appendix D MMEvalPro Triplet Checking Guideline

In our study, we employed double checking to maintain the quality of MMEvalPro. We also developed a web page shown in figure 13 for checking. The checkers were trained with checking instructions shown in figure 14.

Figure 11: UI for annotations of MMEvalPro.

Figure 14: Checking Guideline of MMEvalPro.

Figure 12: Annotation Guideline of MMEvalPro.

Figure 13: UI for checking of MMEvalPro.





## Appendix E Human Evaluation Guide

To ensure the comprehensiveness of our study, we employed five graduates with special knowledge to do human evaluation in MMEvalPro. The designed web ui of human evaluation is shown in figure 15.

### Comparison of Different Models

We list three cases comparing the output of GPT-4o (OpenAI, 2024), GPT-4o (non-vision) (OpenAI, 2024), InternVL-Chat (Chen et al., 2023b) and LLaVA-1.5 (Liu et al., 2023b). We observed that, while both GPT-4o (non-vision) and LLaVA-1.5 could answer all the origin questions correctly, they struggled with most perception and knowledge questions. This highlights a Type-I error in current evaluation benchmarks: correctly answering a question does not necessarily indicate genuine understanding by the model. On the other hand, more advanced models like GPT-4o and InternVL-Chat demonstrate higher consistency in answering different types of questions.

Figure 15: UI for human evaluation of MMEvalPro.



* [14] M. C. B. Gret, A. J. van der Bijnenfeld, and R. D. K. Wilson, "The role of the measurement of the mass of a single particle in an event with a large number of particles in events with high energy muons in proton-proton collisions at RHIC energies", _Phys. Rev. Lett._**83** (2001) 156902, doi:10.1103/PhysRevLett.83.156902, arXiv:hep-ex/0107015.
* [15] S. Frixione et al., "A study of the production cross section for the production of heavy quarks in pp collisions at \(\sqrt{s}=7\) TeV", _JHEP_**01** (2001) 043, doi:10.1007/JHEP01(2001)043, arXiv:hep-ph/0106146.
* [16] T. Sjostrand, S. Mrenna, and P. Skands, "PYTHIA 6.4 physics and manual", _Eur. Phys. J. C_**73** (2006) 159, doi:10.1140/epjc/s10052-006-x, arXiv:hep-ph/0503175.
* [17] CMS Collaboration, "Event generator tunes obtained from underlying event and multiparton scattering measurements", _Eur. Phys. J. C_**74** (2016) 155, doi:10.1140/epjc/s10052-016-3988-y, arXiv:1512.0021.
* [18] GEANT4 Collaboration, "GEANT4--a simulation toolkit", _Nucl. Instrum. Meth. A_**506** (2003) 250, doi:10.1016/S0168-9002(03)01368-8.
* [19] N. Bahr et al., "Herwig++: The POWHEG method", _Comput. Phys. Commun._**181** (2008) 1, doi:10.1016/j.cpc.2008.01.021, arXiv:0707.0018.
* [20] NNPDF Collaboration, "Parton distributions for LHC Run II", _JHEP_**07** (2014) 040, doi:10.1007/JHEP07(2014)040, arXiv:1407.0020.

[MISSING_PAGE_POST]