M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models

## Abstract

Abstract Multilingual multimodal reasoning is a core component in achieving human-level intelligence. However, most existing benchmarks for multilingual multimodal reasoning struggle to differentiate between models of varying performance; even language models without visual capabilities can easily achieve high scores. This leaves a comprehensive evaluation of leading multilingual multimodal models largely unexplored. In this work, we introduce M4U, a novel and challenging benchmark for assessing the capability of multi-discipline multilingual multimodal understanding and reasoning. M4U contains 8,931 samples covering 64 disciplines across 16 subfields in Science, Engineering, and Healthcare in Chinese, English, and German. Using M4U, we conduct extensive evaluations of 21 leading Large Multimodal Models (LMMs) and Large Language Models (LLMs) with external tools. The evaluation results show that the state-of-the-art model, GPT-4o, achieves only 47.6% average accuracy on M4U. Additionally, we observe that the leading LMMs exhibit significant language preferences. Our in-depth analysis indicates that leading LMMs, including GPT-4o, suffer performance degradation when prompted with cross-lingual multimodal questions, such as images with key textual information in Chinese while the question is in German. We believe that M4U can serve as a crucial tool for systematically evaluating LMMs based on their multilingual multimodal reasoning capabilities and monitoring their development. The homepage , codes and data are public available.

## 1 Introduction

Multilingual multimodal reasoning is an essential aspect of human intelligence. AI systems with strong multimodal reasoning capabilities have extensive applications, including automatic scientific discovery, autonomous driving, and healthcare. The rapid advancements in Large Language Models (LLMs)[, , ] have led to the development of Large Multimodal Models (LMMs)[, , , ], which demonstrate remarkable performance across a broad range of tasks, such as image captioning and visual question answering. Numerous benchmarks have been established to comprehensively evaluate these leading LMMs in real-world scenarios [, , , , ]. Unlike perceptual tasks [, ], multimodal reasoning tasks, including mathematical reasoning [] and scientific question answering [, ], present significant challenges for neural models. These tasks necessitate an understanding of domain-specific knowledge and the ability to perform complex logical reasoning alongside visual content. Additionally, multilingual capability is crucial for real-world applications, as these models are typically deployed across various countries and languages.

Many datasets are curated to evaluate the capability of multilingual multimodal reasoning. However, the multimodal component of existing benchmarks [, , ] is limited in scale. More importantly, we observe that the current data on multilingual multimodal reasoning suffers from significant data contamination and language disparities in task complexity. For instance, the multimodal section of M3Exam [] contains 61% high-difficulty questions in English, but only 23% high-difficulty questions in Chinese. Furthermore, as shown in Table 2, without any visual information, the multilingual LLM, Qwen-1.5-14B Chat, easily achieves high scores: 66.4% and 56.0% accuracy on the Chinese and English sections of M3Exam, respectively. This suggests that M3Exam struggles to differentiate between models of varying multimodal capabilities. Consequently, a systematic evaluation of multilingual multimodal understanding and reasoning for leading models remains largely unexplored.

To advance the development of multilingual LMMs, we introduce M4U, a novel and challenging benchmark for evaluating foundational models on expert-level multilingual multimodal understanding and reasoning. Specifically, we assembled a team of over 10 college and graduate students to collect a high-quality dataset and assess its difficulty and correctness. As shown in Figure 3, M4U consists of 8,931 multiple-choice questions covering 64 disciplines across 16 subfields in Science, Engineering, and Healthcare. To minimize the risk of data contamination, samples are collected from college exams and quizzes from online video lectures. Additionally, a significant portion (35%) of the questions in M4U are written by our team based on textbooks. Figure 2 illustrates an example from the Chemistry-Inorganic part of M4U, demonstrating that our dataset requires expert-level multimodal reasoning and multilingual capability.

With M4U, we conduct a comprehensive evaluation, both quantitative and qualitative, on the zero-shot performance of 17 leading LMMs and 4 LLMs. Furthermore, we assess the performance of the LMMs with chain-of-thought prompting [, ] and the LLMs with external tools, such as a powerful captioning model. As shown in Table 3 in §3.3, the most advanced model, GPT-4o [], achieves only 47.6% average accuracy with zero-shot prompting on the M4U dataset, demonstrating the significant challenge M4U poses for existing models. Additionally, we observe significant language preferences among the leading LMMs: InstructBLIP Vicuna-7B achieves 29.8% accuracy on the English section, but only 13.7% and 19.7% accuracy on the Chinese and German sections, respectively. Further results (§3.4) indicate that leading LMMs suffer performance degradation when prompted with cross-lingual multimodal questions, such as images with key textual information in Chinese while the question is in German. Our in-depth analysis (§4) reveals that the errors of GPT-4V(ision) are mainly due to limited perception ability, domain-specific knowledge, and reasoning. These findings demonstrate that LMMs still have significant room for improvement, particularly in multilingual multimodal reasoning.

Figure: Figure 3: Key statistics of M4U dataset. M4U covers a wide scope of tasks from Science, Engineering and Health in Chinese, English and German, and supports the interleaved vision-language documents.
Refer to caption: x3.png

## 2 The M4U Benchmark

### 2.1 Overview

In this section, we introduce M4U, a novel and challenging benchmark for assessing the multilingual multimodal understanding and reasoning of foundational models. To investigate whether differences exist in the multimodal reasoning capabilities of LMMs across different languages, we first construct the Chinese part of M4U and then translate it into English and German. This approach ensures that the domain-specific knowledge and reasoning abilities tested in different languages remain consistent. Specifically, we assemble a team of over 10 college students to collect questions from the Internet, textbooks, online video lectures, and college exams. Subsequently, a team of graduate students from related majors assessed the quality of the curated questions. Following this, we utilize GPT-4 Turbo (gpt-4-turbo-preview) to translate the questions into other languages, and then manually checked the quality of the translated questions.

The key statistics of M4U are detailed in Figure 3. M4U contains 8,931 multiple-choices questions, covering 64 subjects of 16 fields from Science (36.4%), Engineering (28.7%) and Health (34.9%) in Chinese, English and German. Different from the prior work [], M4U includes interleaved image-text documents: 8.2% of the questions have multiple images, while the images of 14.8% and 7.4% of the questions are placed in the middle of question and the options, respectively. The average length of questions and options is 33.2 words and 6.1 words. The image sources of M4U cover 13 categories in different scenarios, e.g., diagrams, technical blueprints and medical images.

**Table 1: Comparison between M4U and the existing benchmarks. ^∗We report the size of test set of the multimodal part for the benchmarks.**
| Benchmark | Multilingual | Multimodal | Size^∗ | Difficulty | Fields |
| --- | --- | --- | --- | --- | --- |
| CMMLU [] | ✗ | ✗ | - | ★ ★ ★ | STEM, Humanities, etc. |
| C-Eval [] | ✗ | ✗ | - | ★ ★ ★ | STEM, Humanities, etc. |
| MMLU [] | ✗ | ✗ | - | ★ ★ ★ | STEM, Humanities, etc. |
| MathVista [] | ✗ | ✓ | 6,141 | ★ ★ ★ | Mathematics |
| CMMMU [] | ✗ | ✓ | 12,012 | ★ ★ ★ | Art, STEM, Humanities, etc. |
| MMMU [] | ✗ | ✓ | 11,550 | ★ ★ ★ | Art, STEM, Humanities, etc. |
| MGSM [] | ✓ | ✗ | - | ★ ★ ★ | Mathematics |
| EXAMS-V [] | ✓ | ✓ | 1,221 | ★ ★ | STEM, Humanities, etc. |
| M3Exam [] | ✓ | ✓ | 2,816 | ★ | STEM, Humanities, etc. |
| M4U (ours) | ✓ | ✓ | 8,931 | ★ ★ ★ ★ | STEM, Healthcare |

**Table 2: The zero-shot accuracy of the multilingual LLM, Qwen-1.5-14B-Chat [], on the multimodal part of M3Exam and M4U dataset. We provide the scores of random choices in blue as the reference baseline. The higher scores of LLM indicate that the multimodal benchmark requires less visual efforts or suffers from more serious data contamination during the training of LLMs.**
| Benchmark | Chinese$\downarrow$ | English$\downarrow$ | German$\downarrow$ | Average$\downarrow$ |
| --- | --- | --- | --- | --- |
| M3Exam [] | 66.4 (25.9) | 56.0 (25.0) | - | 61.2 (25.5) |
| M4U (ours) | 28.0 (25.9) | 19.7 (25.9) | 27.6 (25.9) | 25.1 (25.9) |

We present a comparison of M4U with existing benchmarks in Table 1. Unlike MMMU [] and CMMMU [], our dataset focuses on the evaluation of multilingual multimodal reasoning. Furthermore, M4U is larger and has a more balanced distribution of difficulty across different languages compared to M3Exam []. This ensures a fair comparison of models’ capabilities in multimodal reasoning within multilingual scenarios. More importantly, we implement strict collection guidelines and quality control measures to minimize the risk of data contamination. To quantitatively measure the level of data contamination in these benchmarks, we use the scores of LLMs without any visual information as a reference. Higher scores indicate that the benchmark is less effective at differentiating between multimodal models of varying performance. As shown in Table 2, without any visual information, Qwen-1.5-14B Chat easily achieves 66.4% and 56.0% accuracy on the Chinese and English parts of M3Exam, respectively, while it only achieves 25.1% average accuracy on our dataset. This suggests that M4U is more challenging and less exposed to the training corpus of LLMs.

### 2.2 Data Collection

#### Data sources.

Following MMMU [], we go through the educational programs of top universities, then select 64 subjects of 16 subfields from Science, Engineering and Healthcare whose applications highly rely on visual information. We recruit a team of over 10 college students to collect multiple-choices questions from public available sources. To minimize the risk of data contamination for foundation models, different from M3Exam [], we do not include the samples from the official exam papers, e.g., National Postgraduate Entrance Examination and national professional exams. Although these resources usually have higher quality and are well organized, they are also easy to be curated for the training of LLMs. Therefore, we carefully select the data sources for M4U: most questions of our dataset are collected from the quizzes of online video lectures and college exams in PDF documents. Further 35% of questions are written by our team according to the textbooks. As shown in Table 3, the state-of-the-art open-source multilingual LLM, Qwen-1.5-14B Chat [], only has 25.1% average accuracy on M4U dataset without any visual information. This proves that our data is less exposed in the training of language models.

#### Data processing.

The primary sources of M4U include college exams, the quizzes of online video lectures and the written questions. Most of college exams are uploaded by their students as images or scanned PDF documents, while the quizzes of online video lectures can be taken as the screenshot. We first adopt the OCR tools to convert these images into plain texts, then manually correct the potential errors of OCR results. Besides, we also write a large portion (35%) of questions according to the textbooks. For the mathematical formulas and the chemical structures, we require the annotators to convert them into LaTeX format. Since the samples of M4U may include multiple images in the questions or options, the annotators also annotate the location and type of each image (e.g. tables, blueprints and medical images).

After collecting the data, we design a two-stage post-processing pipeline to further improve the quality of M4U. We first design the guidelines to allow each annotator to score the collected samples from three dimensions: image quality, question description quality and the difficulty of visual understanding, and filter out the questions with average scores lower than 2.0. The detailed guidelines are summarized in the Appendix A.3. Then we recruit a team of graduate students of related major to assess the difficulty and quality of the curated questions. We further filter out the questions with the minimum visual efforts and the wrong answer. After that, we use GPT-4 Turbo to translate the Chinese part of M4U to English and German. Then the annotators will check and correct the potential errors introduced by machine translation.

### 2.3 Evaluation

We evaluate the zero-shot performance of 17 leading LMMs of different scales on M4U. The models are required to follow the instruction to directly generate the predicted option for each question. To minimize the format discrepancy between training and evaluation, we handle models that support interleaved image-text documents by inserting the visual tokens of each image into the corresponding position as in training. For models that only support image-text pairs as input, we place all visual tokens at the beginning of the sentence and use annotated positions (e.g., <image_1>, <image_2>) to refer to each image. Furthermore, we also evaluate the performance of various LMMs with chain-of-thought prompting [, ] and LLMs equipped with detailed visual captions.

As shown in Table 3, the existing models still lack the capability for expert-level multilingual multimodal understanding and reasoning: the most advanced GPT-4o achieves only 47.6% average accuracy with zero-shot prompting on the M4U dataset. Additionally, we observe that these models exhibit strong language preferences. For instance, InstructBLIP Vicuna-7B achieves 29.8% accuracy on the English part, but only 13.7% and 19.7% accuracy on the Chinese and German parts, respectively. These results indicate that there remains significant room for improvement in LMMs, particularly regarding multilingual capability and complex multimodal reasoning.

## 3 Experiments

### 3.1 Setup

We evaluate the performance of zero-shot learning for various LMMs and LLMs of different scales across different languages on M4U dataset. The models are prompted to directly generate the option’s letter. Further we also evaluate the performance of the LMMs with chain-of-thought prompting [, ]: the models should first generate the rationale for the question and the options, then give the predicted option. For reference, we add the baseline of Random choices: we randomly select an option, and use the average accuracy of 30 runs with different seeds. We provide more details about the instruction prompt in Appendix A.4.

### 3.2 Models

#### LMMs.

For the open-source models, we select VisualGLM [], Ying-VLM [], InstructBLIP series [], InternLM-XComposer [], CogVLM-Chat [], Qwen-VL-Chat [], Yi-VL-series [], DeepSeek-VL [] and LLaVA-NeXT series []. For closed source models, we choose Gemini 1.0 Pro [], GPT-4V(ision) [] and GPT-4o [] using the provided API, gemini-pro-vision, gpt-4-vision-preview and gpt-4o, respectively. As for the augmented LMMs, we evaluate the performance of Gemini 1.0 Pro and GPT-4V with the chain-of-thought prompting [, ].

#### LLMs.

We select Mistral-Instruct-v0.2-7B [], Qwen-1.5-7B-Chat, Qwen-1.5-14B-Chat [] and Gemini 1.0 Pro (gemini-pro) for the open and closed source LLMs. We use Gemini 1.0 Pro (gemini-pro-vision) to generate the detailed caption in Chinese, English and German for each image. The visual captions are placed at the beginning of the prompt. The annotated image positions are used to refer each image.

### 3.3 Main results

**Table 3: The zero-shot accuracy of various LLMs, augmented LLMs and LMMs on M4U dataset. CoT is short for chain-of-thought prompting.**
| Models | Size | Chinese$\uparrow$ | English$\uparrow$ | German$\uparrow$ | Average$\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| Random choices | - | 25.9 |  |  |  |
| *Large Language Models* |  |  |  |  |  |
| Qwen-1.5-7B-Chat [] | 7B | 29.5 | 15.0 | 28.5 | 24.3 |
| Qwen-1.5-14B-Chat [] | 14B | 28.0 | 19.7 | 27.6 | 25.1 |
| *Augmented Large Language Models (+ Visual Caption)* |  |  |  |  |  |
| Mistral-Instruct-v0.2-7B [] | 7B | 24.9 | 24.9 | 26.9 | 25.6 |
| Gemini 1.0 Pro [] | - | 31.6 | 31.1 | 30.9 | 31.2 |
| Qwen-1.5-7B-Chat [] | 7B | 34.2 | 27.7 | 31.7 | 31.2 |
| Qwen-1.5-14B-Chat [] | 14B | 32.7 | 32.0 | 33.8 | 32.8 |
| *Large Multimodal Models* |  |  |  |  |  |
| VisualGLM [] | 6B | 8.7 | 22.4 | 13.5 | 14.9 |
| Ying-VLM [] | 13B | 22.3 | 11.2 | 15.6 | 16.4 |
| InstructBLIP-Vicuna-13B [] | 13B | 10.5 | 23.4 | 18.6 | 17.5 |
| InstructBLIP-Vicuna-7B [] | 7B | 13.7 | 28.1 | 19.7 | 20.5 |
| LLaVA-NeXT-Vicuna-7B [] | 7B | 11.8 | 29.8 | 28.2 | 23.3 |
| LLaVA-NeXT-Vicuna-13B [] | 13B | 21.9 | 30.9 | 29.3 | 27.4 |
| Qwen-VL-Chat [] | 7B | 29.7 | 29.9 | 27.1 | 28.9 |
| CogVLM-Chat [] | 7B | 28.9 | 30.2 | 28.5 | 29.2 |
| LLaVA-NeXT-Mistral-7B [] | 7B | 28.2 | 30.6 | 29.4 | 29.4 |
| InternLM-XComposer [] | 7B | 31.8 | 31.6 | 29.1 | 30.8 |
| DeepSeek-VL [] | 7B | 30.4 | 32.8 | 30.8 | 31.3 |
| Yi-VL-6B [] | 6B | 33.4 | 31.4 | 29.7 | 31.5 |
| Yi-VL-34B [] | 34B | 33.5 | 33.3 | 30.5 | 32.4 |
| Gemini 1.0 Pro [] | - | 34.9 | 32.7 | 30.8 | 32.8 |
| LLaVA-NeXT-34B [] | 34B | 38.5 | 36.2 | 35.2 | 36.6 |
| GPT-4V(ision) [] | - | 39.7 | 39.4 | 37.3 | 38.8 |
| GPT-4o [] | - | 49.4 | 47.8 | 45.6 | 47.6 |
| * Augmented Large Multimodal Models* |  |  |  |  |  |
| Gemini 1.0 Pro [] + CoT | - | 34.4 | 34.2 | 33.9 | 34.2 |
| GPT-4V(ision) [] + CoT | - | 43.9 | 43.6 | 40.3 | 42.6 |

In this section, we present the comprehensive evaluation results of 17 leading LMMs and 4 LLMs with different prompt strategies. Table 3 demonstrates the performance of various LMMs and LLMs across Chinese, English and German on M4U dataset.

For the text-only LLMs, we first only use the text part of question to prompt these models. As shown in Table 3, Qwen-1.5-14B Chat has only 25.1% average accuracy on M4U dataset, which is lower than 25.9% of random choices. It proves that M4U requires significant visual efforts to answer these questions, and does not suffer from the data contamination during the training of LLMs. Further, we equip these LLMs with the detailed visual captions generated by Gemini 1.0 Pro. Qwen-1.5-14B Chat with additional captions outperforms itself without any visual information by a gain of 7.7%, and achieves 32.8% average accuracy, the highest scores among the baselines. Mistral-Instruct-v0.2-7B has 25.6% average accuracy, since it does not follow the instruction to generate the valid option. We observe that Mistral-Instruct-v0.2-7B tends to reject to give an answer when not being provided with enough visual information.

For LMMs, most of them do not have the satisfactory results on M4U dataset. The state-of-the-art model, GPT-4o, achieves only 47.6% average accuracy with zero-shot prompting. It indicates that M4U is quite challenging for the existing models, and the reasoning capability of the multimodal models still has much room for future improvement. With the powerful LLM, Nous-Hermes Yi-34B (^1^11[https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)), LLaVA-NeXT-34B scores highest among the open-source LMMs, even significantly outperforms Gemini 1.0 Pro by a gain of 3.8% on average accuracy. As for augmented LMMs, chain-of-thought prompting further boosts the performance. GPT-4V with chain-of-thought prompting outperforms itself with zero-shot prompting by a gain of 3.8% on average accuracy. It demonstrates that explicitly generating the reasoning steps is also beneficial for complex multimodal reasoning.

Furthermore, we observe that the existing models has strong language preferences on multilingual multimodal reasoning tasks. InstructBLIP Vicuna-7B achieves 28.1% accuracy on the English part of M4U, while only has 13.7% and 19.7% accuracy on the Chinese and German part, respectively. For GPT-4V, the average accuracy on the Chinese and English is both 3% higher than it on the German. Besides, we observe that the effect of chain-of-thought prompting also differs across different languages. For instance, chain-of-thought improves the performance of Gemini 1.0 Pro on English and German part by a gain of 1.5% and 3.1% accuracy, while leads to a degradation of 0.5% on Chinese part. We argue that this results from the lack of the multilingual vision-language corpus used for multimodal training, and the LLMs of these LMMs (e.g., Vicuna-7B, Vicuna-13B) do not well support the multilingual capability.

**Table 4: The zero-shot accuracy of various LMMs, augmented LMMs on the cross-lingual set of M4U dataset. CoT is short for chain-of-thought prompting.**
| Models | Size | Chinese$\uparrow$ | English$\uparrow$ | German$\uparrow$ | Average$\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| DeepSeek-VL [] | 7B | 32.8 | 34.0 | 33.3 | 33.4 |
| Yi-VL-6B [] | 6B | 39.2 | 34.3 | 30.1 | 34.5 |
| Gemini 1.0 Pro [] | - | 38.0 | 36.3 | 32.9 | 35.7 |
| Yi-VL-34B [] | 34B | 41.6 | 38.7 | 34.2 | 38.2 |
| LLaVA-NeXT-34B [] | 34B | 44.6 | 40.9 | 36.1 | 40.5 |
| GPT-4V(ision) [] | - | 45.3 | 41.2 | 38.2 | 41.6 |
| GPT-4o [] | - | 52.0 | 47.5 | 45.2 | 48.2 |
| Gemini 1.0 Pro [] + CoT | - | 38.1 | 35.7 | 37.8 | 37.2 |
| GPT-4V(ision) [] + CoT | - | 46.7 | 48.0 | 42.6 | 45.8 |

**Table 5: The detailed results of different LMMs on Health, Science and Engineering of M4U dataset. Sci. and Eng. are short for Science and Engineering, respectively.**
| Models | Chinese | English | German |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Health | Sci. | Eng. | Health | Sci. | Eng. | Health | Sci. | Eng. |  |
| Yi-VL-6B | 31.2 | 34.1 | 34.9 | 32.1 | 32.2 | 30.0 | 29.0 | 29.2 | 30.8 |
| DeepSeek-VL | 40.1 | 22.6 | 28.5 | 38.0 | 31.9 | 28.6 | 35.2 | 29.3 | 27.8 |
| Yi-34B | 32.9 | 34.1 | 33.6 | 34.0 | 33.2 | 32.6 | 29.4 | 30.2 | 32.0 |
| LLaVA-NeXT-34B | 38.1 | 40.4 | 37.0 | 37.2 | 36.8 | 34.7 | 36.9 | 34.2 | 34.5 |
| Gemini 1.0 Pro | 38.8 | 34.5 | 31.4 | 34.9 | 33.4 | 29.8 | 33.1 | 30.5 | 28.8 |
| + Chain-of-thought prompting | 37.8 | 33.3 | 32.6 | 38.8 | 33.3 | 30.6 | 37.8 | 32.2 | 31.8 |
| GPT-4V(ision) | 41.9 | 39.3 | 37.9 | 43.9 | 37.8 | 36.6 | 41.1 | 36.0 | 34.6 |
| + Chain-of-thought prompting | 43.9 | 46.2 | 41.7 | 45.8 | 43.3 | 41.9 | 42.5 | 39.1 | 39.3 |
| GPT-4o | 56.0 | 47.3 | 45.0 | 56.2 | 44.3 | 42.8 | 52.9 | 40.9 | 43.0 |

Figure: (a) Image types
Refer to caption: x4.png

### 3.4 Cross-lingual Multimodal Evaluation

To measure the cross-lingual multimodal capability of the LMMs, we select a subset from M4U: the image of each sample in this subset contains the text that labels or explains the key concepts or objects in the picture, while the textual description of the question is multilingual. For example, as illustrated in Figure 2, the visual content contains the key text in Chinese that labels bond length between atoms and explains the single-layer structure of the material, and the question part is multilingual. The models are required to perform complex reasoning given the multilingual both textual and visual contents. The cross-lingual set contains 1,065, 417 and 531 samples from Science, Engineering and Healthcare, respectively, resulting in up to 2,013 samples in total.

We evaluate the performance of different LMMs on the cross-lingual set of M4U. As shown in Table 4, almost all models suffer from a degradation of performance when the image contains the key textual information in Chinese but the question is English or German. It shows that these models are short for following multilingual instructions to understand the visual contents with the textual information of another language. Furthermore, as for the augmented LMMs, we observe that the chain-of-thought prompting significantly improves the performance of GPT-4V on English and German. This is aligned with our previous evaluations on the full set of M4U in Table 3.

### 3.5 Fine-grained results

#### Different Disciplines and Languages.

We present the detailed results of various LMMs on different fields of Chinese, English and German in Table 5. GPT-4o outperforms the other models by large improvements on all fields of all languages. For the open-source models, we observe that LLaVA-NeXT-34B shows impressive results on scientific reasoning, and DeepSeek-VL demonstrates good performance on Health. Further, we observe that on Science, the chain-of-thought prompting significantly improves the performance of GPT-4V by a gain of over 6% accuracy in Chinese and English, while only boosts the performance by an improvement of 3.1% accuracy on German. The similar phenomenon also exists for Gemini 1.0 Pro. On Health part, Gemini 1.0 Pro with the chain-of-thought prompting outperforms it with zero-shot prompting by a gain of 4.9% and 4.7% on English and German, but it leads to a degradation of 1.0% accuracy on Chinese. These results show that the effect of the chain-of-thought prompting also differ from different languages.

#### Different Image Types and Positions.

We demonstrate the visualization of the detailed results of various LMMs on different image types and positions in Figure 4. We reclassified 13 image types into 7 categories based on the style and application of the image. As shown in Figure 4(a), GPT-4o shows impressive performance on the image type of "Plots & Charts" and "Medical" compared with the other models, but has unsatisfactory results on Blueprints. We argue that this is because the Blueprints contain many engineering sketches that require the capability of the fine-grained perception and domain-specific knowledge about engineering standards. M4U not only supports the image-text pairs as the input, but includes interleaved image-text documents. Thus, we conduct the analysis about the performance of the selected LMMs on different positions of the images. We divide these questions into four groups according to the image position: image at the beginning, end, middle of the question and in the options. As shown in Figure 4(b), on the questions with images in the options, GPT-4o and GPT-4V outperform the other models by a large gain, and LLaVA-NeXT-34B performs poorly on this types of the questions. We argue that this is because the LLaVA-NeXT series are only trained with a high-quality corpus of image-text pairs. Instead DeepSeek-VL is pre-trained with a large mixture of image-text pairs and interleaved documents, and it does not suffer from a significant degradation of performance on the questions with images in the options.

## 4 Qualitative Analysis

Figure: (a) Chinese
Refer to caption: x6.png

In this section, we conduct qualitative analysis for the results of GPT-4V with the chain-of-thought prompting. We randomly sample 75 questions (2.5%) from different disciplines of each language. In these instances, GPT-4V has errors in responses and analysis in at least one language. We analyze the cause of these wrong cases, and divided them into 6 categories: perceptual error, lack of knowledge, reasoning error, textual understanding, annotation error and answer extraction error. Furthermore, we illustrate the distribution of the selected samples across different categories in Figure 5. Perceptual error, lack of knowledge, and reasoning error account for the major causes of failed cases (96% in Chinese, 95% in English, and 92% in German). GPT-4V tends to exhibit lack of knowledge on the Chinese part of M4U, while reasoning errors are more likely to occur in German and English. We present the summary of the top-3 types of failed cases in the following. More results can be found in the Appendix B.

#### Perceptual Error.

Perceptual error is the most frequent error made by GPT-4V. It corresponds to the illusion phenomenon that occurs when extracting visual information from images provided by the questions. These kinds of hallucination could be divided in two main categories: visual information deficiency and misinterpretation. As presented in Figure 21, visual information deficiency occurs when GPT-4V overlooked crucial conditions and information provided in the image associated with the question, such as dimensions and scales annotated in engineering blueprints and numerical values provided in physics experiments thereby failing to complete the reasoning chain. Figure 15 shows a typical case for the visual information misinterpretation: the extracted information is complete but contains mistakes. A portion of these mistakes are common perceptual errors in OCR and visual localization.

#### Lack of Knowledge.

We define the lack of knowledge as the model has factual misunderstanding about the key concepts in questions and provides erroneous premise to the reasoning process. Figure 23 illustrates that GPT-4V ignores that the amplitude needs to be judged on an equivalent basis to the same period, and as shown in Figure 24, the model equates the average kinetic energy of a molecule to the kinetic energy of a single molecule, overlooking key preconditions of physical laws.

#### Reasoning Error.

The reasoning error is categoried as the mathematical miscalculations and logical errors in the analysis procedure, which often occur in subjects need numerical computations and logical inference, such as math, physics, and electronics. As demonstrated in Figure 25, GPT-4V only considers the power supply on the left and does not consider the power supply on the right.

## 5 Conclusion and Future Work

In this work, we introduce M4U, a novel and challenging benchmark for evaluating the capability of multilingual multimodal understanding and reasoning. M4U contains 8,931 multiple-choice questions, covering 64 disciplines across 16 subfields in Science, Engineering, and Healthcare in Chinese, English, and German. Table 2 demonstrates that M4U requires significant visual efforts and has less exposure to the training corpus of LLMs compared with M3Exam. As shown in Table 3, the state-of-the-art model, GPT-4o, achieves only 47.6% average accuracy with zero-shot prompting, indicating that M4U is quite challenging for existing models. Furthermore, we observe that the leading LMMs exhibit significant language preferences. These results demonstrate that there is still significant room for improvement in LMMs, especially in expert-level multilingual multimodal reasoning. In the future, we aim to extend M4U to support more languages (e.g., Japanese and French) and investigate the performance of multilingual LMMs on questions associated with cultural backgrounds (e.g., history and politics). Additionally, we plan to include multilingual rationales for M4U to construct a fine-grained metric that considers the correctness of both reasoning steps and final predictions.

## 6 Acknowledgments

We would like to acknowledge Jiayu Gao, Jialu Ma, Wenbo Nie, Tianze Xia, Jiawei Zhang, MingYang Zhang and Weihao Kong for their contribution in data collection and quality control.

## Appendix A Dataset Documentation

### A.1 Usage

- •
M4U dataset is public available at [https://huggingface.co/datasets/M4U-Benchmark/M4U](https://huggingface.co/datasets/M4U-Benchmark/M4U).
- •
The dataset is saved in Parquet format. We present the data format and the examples in the README.md file. Besides, we also provide the example codes to show how to evaluate the multimodal models on the dataset.
- •
License: M4U is under CC BY-NC-SA Liences.

### A.2 Data sources

M4U consists of 8,931 multiple-choices questions, covering 64 disciplines of 16 subfields from Science, Engineering and Healthcare. To minimize the risk of data contamination, the samples are collected from college exams, the quizzes of online video lectures. Further a large portion (35%) of the questions are written by our team according to the textbooks.

### A.3 First-stage Processing Guidelines

We summarize the detailed first-stage processing guidelines for the annotators. For each dimension, we require the annotator to score the sample following the below guidelines. The question with the higher scores indicates higher quality. We filter out the questions with average scores lower than 2.0. As for the image quality, the standard is:

- •
0 score: The image is extremely blurry, difficult to recognize, or most of it is cropped, resulting in severe information loss.
- •
1 score: The image is relatively blurry, details are hard to discern, or parts of the image are cropped, leading to some information loss.
- •
2 score: The image is slightly blurry; most content is recognizable but details are unclear, or the image is slightly cropped, but most information is complete.
- •
3 score: The image is mostly clear; all major content is recognizable, though some details may not be clear.
- •
4 score: The image is clear; all content and details are easily recognizable with no apparent defects.
- •
5 score: The image is very clear; details are excellently represented, complete without any cropping or obstructions, meeting or exceeding the expected quality standards.

The guideline for measuring the question description quality is:

- •
0 score: The question is vague and completely unintelligible, with no clear intent.
- •
1 score: The question statement is ambiguous, difficult to fully understand its intent, with multiple possible interpretations.
- •
2 score: The question statement is basically clear, but there are some ambiguities or lack of rigor that need further clarification.
- •
3 score: The question statement is clear, though there are some details that are not rigorous or there is slight ambiguity.
- •
4 score: The question statement is both clear and rigorous, with details well handled, and only very minor issues present.
- •
5 score: The question statement is extremely clear and rigorous, logical, without any ambiguity, fully meeting high standards.

The standard for measuring the difficulty of visual understanding is:

- •
0 score: The question almost does not rely on visual ability, can be fully understood without any visual information.
- •
1 score: The question does not completely rely on visual ability, both visual and non-visual information are balanced.
- •
2 score: Although the question relies on visual ability, a considerable proportion of non-visual information assists understanding.
- •
3 score: The question largely depends on visual ability, but some non-visual information is provided.
- •
4 score: The question greatly depends on visual ability, with very little content provided by non-visual information.
- •
5 score: The question completely depends on visual ability, without it, the content is incomprehensible.

Furthermore, we recruit a team of graduate student of related majors to access the difficulty and correctness for the questions. The team will filter out the questions with wrong answer or minor visual efforts.

### A.4 Evaluation prompt

Figure 6 shows the prompt template used in the zero-shot evaluation of M4U. As for the chain-of-thought prompting, we first prompt the model to generate the rationale for the question and the options using the template shown in Figure 7, then append the generated rationale after the options. After that, the model are prompted to generate the predicted option.

Figure: Figure 6: The template of different languages used for the evaluation of the LLMs and LMMs.
Refer to caption: x10.png

Figure: Figure 7: The template of different languages used for the evaluation of the LMMs with the chain-of-thought prompting.
Refer to caption: x11.png

## Appendix B Detailed Results of Qualitative Analysis

### B.1 Correct Cases

Figure: Figure 8: A sample correct case (subject: agriculture, language: Chinese).
Refer to caption: x12.png

Figure: Figure 9: A sample correct case (subject: agriculture, language: English).
Refer to caption: x13.png

Figure: Figure 10: A sample correct case (subject: agriculture, language: German).
Refer to caption: x14.png

Figure: Figure 11: A sample correct case (subject: physics, language: German).
Refer to caption: x15.png

### B.2 Perceptual Error Cases

Figure: Figure 12: A sample perceptual error case (subject: geography, language: Chinese).
Refer to caption: x16.png

Figure: Figure 13: A sample perceptual error case (subject: architecture, language: English).
Refer to caption: x17.png

Figure: Figure 14: A sample perceptual error case (subject: math, language: Chinese).
Refer to caption: x18.png

Figure: Figure 15: A sample perceptual error case (subject: math, language: English).
Refer to caption: x19.png

Figure: Figure 16: A sample perceptual error case (subject: math, language: German).
Refer to caption: x20.png

Figure: Figure 17: A sample perceptual error case (subject: laboratory medicine, language: Chinese).
Refer to caption: x21.png

Figure: Figure 18: A sample perceptual error case (subject: laboratory medicine, language: English).
Refer to caption: x22.png

Figure: Figure 19: A sample perceptual error case (subject: laboratory medicine, language: German).
Refer to caption: x23.png

Figure: Figure 20: A sample perceptual error case (subject: architecture, language: Chinese).
Refer to caption: x24.png

Figure: Figure 21: A sample perceptual error case (subject: architecture, language: English).
Refer to caption: x25.png

Figure: Figure 22: A sample perceptual error case (subject: architecture, language: German).
Refer to caption: x26.png

### B.3 Lack of Knowledge Cases

Figure: Figure 23: A sample lack of knowledge case (subject: physics, language: Chinese).
Refer to caption: x27.png

Figure: Figure 24: A sample lack of knowledge case (subject: energy, language: Chinese).
Refer to caption: x28.png

### B.4 Reasoning Error Cases

Figure: Figure 25: A typical case of reasoning error (subject: electronics, language: English).
Refer to caption: x29.png

Figure: Figure 26: A sample reasoning error case (subject: physics, language: English).
Refer to caption: x30.png

Figure: Figure 27: A sample reasoning error case (subject: geography, language: English).
Refer to caption: x31.png

Figure: Figure 28: A sample reasoning error case (subject: geography, language: German).
Refer to caption: x32.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Is there any language-specific samples covered in the current dataset?

* Is it possible to evaluate only the OCR ability of different models to probe if these models fail because of the multilingual OCR ability?
soundness: 3
strengths: * **Writing**: The paper is well-written and easy to follow.
* **Well-designed Curation Process**: A large portion of the dataset is curated to avoid data contamination.
* **Contribution**: It would be a useful resource for the community to assess the multilingual performance of LMMs.
summary: This paper curates an evaluation benchmark (called M4U) for multilingual multimodal models. It consists of 8,931 samples covering 64 disciplines across 16 subfields in Science, Engineering, and Healthcare in Chinese, English, and German. The evaluation shows that it's a challenging benchmark for most LMMs, where GPT-4o achieves only 47.6% average accuracy on M4U.
weaknesses: * **Language Coverage**: Only three languages (Chinese, English, and German) are covered in this benchmark.

* **Language-specific characteristics**: The multilingual samples are directly translated from one language therefore the language-specific characteristics are not considered in the curation process. Especially for STEM, it's very much evaluating the translation ability or the multilingual ability of the model. It would be interesting to cover samples where the interesting characteristics (e.g., culture) behind multilingualism really matter. For example, for healthcare, the guideline in different countries may differ so having those knowledge is important and will make the benchmark more interesting.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Q1: How is the accuracy of the multilingual data translation ensured? Were native speakers involved in verifying the accuracy?

Q2: Why was the translation of text in the images not considered?

Q3: The paper mentions that the dataset questions come from online videos, PDFs, and textbooks. What are the respective proportions of these sources?

Q4: How is the accuracy of the questions generated ensured? Was there a cross-checking process, and what is the consistency rate of the statistics?
soundness: 2
strengths: Through extensive labor in collection and processing, the authors have created a relatively large multimodal reasoning assessment dataset that reflects university-level knowledge. This effort addresses the issues of high data leakage risk in existing datasets and the inability to differentiate performance among various models.
summary: The paper begins by collecting examination questions with relatively low leakage risk from Chinese universities, resulting in a Chinese multimodal assessment dataset comprising 8,931 items. The authors then constructed English and German translation assessment datasets by translating the text only. They claim that this dataset is a particularly challenging one for multimodal large models and explore the cross-linguistic performance of these models.
weaknesses: > 1. Exaggeration of Contributions:

The article's scientific contributions focus on "challenging visual reasoning tasks" and "multilinguality." However, it lacks sufficient experimentation and effort to substantiate and showcase these contributions.

- Regarding the contribution of "challenging visual reasoning tasks," the authors attempt to demonstrate this by collecting university-level questions with low leakage risk and arguing that these tasks are "difficult" from the perspectives of "answering based solely on text" and "low model accuracy." However, they do not provide data on human expert performance for these questions, which undermines their definition of "difficulty" due to the lack of reference to human accuracy rates.

- Concerning the "multilingual" contribution, merely translating text into two other languages does not convincingly support the claim of "multilinguality," especially in light of the absence of experimental data to ensure the accuracy of the translation, casting doubt on the quality of the multilingual aspect of the dataset.

- Most importantly, as described in the paper, if the images contain Chinese text, retaining the Chinese in the assessments of other languages is clearly unreasonable. Furthermore, considering that most multimodal models' visual encoders are not trained on Chinese text-image pairs, their ability to encode Chinese text is limited, significantly constraining the models' reasoning capabilities and potentially leading to inaccurate assessments of the dataset.

> 2. Lack of Rigor and Missing Information in Data Processing:

For the critical data quality verification aspect of the dataset, the authors use vague language. For instance, they repeatedly refer to "over 10 college and graduate students" when discussing the number of annotators, which I believe is an inadequate expression for a scientific paper. Additionally, there are other missing data statistics, as detailed in the Questions section below.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Were more evaluation methods attempted, such as translating questions into English or incorporating OCR results of images into the questions, to compare the models' understanding and reasoning abilities?
soundness: 2
strengths: -  The motivation behind this benchmark is well-founded, the benchmark provides support for subsequent evaluations of the capabilities of LMMs/LLMs in multidisciplinary, multilingual, and multimodal understanding and reasoning.
- The construction of the benchmark is thorough, the paper offers a complete assessment and analysis of LMMs and LLMs.
- The paper is well-written and easy to understand.
summary: This paper proposes a new challenging benchmark for assessing the capabilities of LMMs in multi-discipline, multilingual, and multimodal understanding and reasoning. The benchmark covers various disciplines and languages, and the authors have meticulously collected and constructed this benchmark. The authors conducted extensive evaluations of leading LMMs and LLMs, demonstrating that there is still significant room for improvement in existing models' capabilities in multilingual multimodal reasoning.
weaknesses: - The data sources for the benchmark are mainly focused on science exam questions, and there is still room for improvement in terms of comprehensiveness.
- Compared to other benchmarks, there does not seem to be a significant difference.
- The evaluation method is relatively simple and may not fully reflect the model's multilingual understanding and reasoning abilities.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. The benchmark content is quite challenging. Given the difficulty level, it’s hard to envision which specific models or fields would absolutely need this benchmark.

2. While M4U offers a more refined, discipline-specific benchmark, it may not serve as a true innovation in assessing multimodal understanding and reasoning, as it primarily tests subject-specific knowledge. The need for such specialized knowledge might be limited outside of subject-focused educational contexts. Narrowing the scope to specific application areas—such as healthcare, education, or scientific research—would help make the benchmark’s significance and applicability more compelling.

Therefore, I suggest you provide evidence or examples demonstrating how M4U tests multimodal understanding and reasoning capabilities beyond just subject knowledge. These examples would help to clarify the benchmark's value and innovation.

3. As shown in the analysis in Figure 4, the top performance barriers are “Perceptual Error”, “Lack of Knowledge” and “Reasoning Error” with the first two accounting for over 69%. This suggests that the main performance limitations stem from the MLLM’s image encoder and LLM lacking subject-relevant knowledge, rather than reasoning capability. Therefore, describing this dataset as a measure of reasoning and understanding might be somewhat misleading.

It is suggested to provide additional analyses or examples that demonstrate how the dataset tests reasoning capabilities despite the prevalence of perceptual and knowledge-based errors. You can show examples of those failures due to “Reasoning Error”. These examples are the characteristics of your benchmark.

4. In Line 112-113, "Unlike MMMU and CMMMU, our dataset focuses on the evaluation of multilingual multimodal reasoning". As far as I know, MMMU and CMMMU are also multi-science and multi-modal benchmarks. The difference is that your benchmark can evaluate problems in multiple languages, and the problems are harder. 

I wonder if the multilingual aspect truly necessary? While it is essential in certain contexts, this point isn’t clearly presented. Additionally, Chinese and German are not particularly low-resource languages. It’s unclear why multilingualism is a primary focus in this work. 

5. It is suggested to detail the process of the data collection and annotation processes. A high-quality dataset requires not only accurate labels but also well-defined annotation standards and quality control procedures. This information is essential for assessing the dataset’s reliability and usefulness, as well as aiding other researchers in replicating and leveraging the data. The paper should include details about annotator selection, annotation processes, quality control measures, and how label consistency and accuracy were ensured. For example, the following could be expanded: “To minimize the risk of data contamination, samples are collected from college exams and quizzes from online video lectures. Additionally, a significant portion (35%) of the questions in M4U are written by our team based on textbooks” (line 82-84). For the first part, what type of crowdsourcing platform was used? For the latter, how did you extract questions from textbooks, formulate questions, and choose answer options? Highlighting the technical aspects of crowdsourcing and data engineering might make this work more engaging.
soundness: 2
strengths: 1. This paper introduces a carefully annotated multimodal, multilingual reasoning dataset, consisting of 8,931 multiple-choice questions covering 64 disciplines across 16 subfields in Science, Engineering, and Healthcare.
2. The paper tests various MLMMs on this dataset to evaluate their performance on subject-specific multimodal reasoning and analyzes the causes of their limitations.
summary: This paper introduces M4U, a novel and challenging benchmark for evaluating multilingual multimodal understanding and reasoning capabilities across 16 subfields in Science, Engineering, and Healthcare in three languages. The experiments show that even the state-of-the-art model, GPT-4, still has significant room for improvement, especially in expert-level multilingual multimodal reasoning.
weaknesses: 1. Necessity of the benchmark. It is suggested to provide specific examples of real-world applications or use cases where this benchmark would be valuable for evaluating MLLMs. Emphasizing its importance in areas like educational robots or medical scenarios might help clarify its necessity.
2. General applicability of the benchmark. This benchmark requires extensive subject-specific knowledge, making reasoning skills secondary. The emphasis on subject knowledge could limit its general applicability. It is suggested to reorganize the title and the Introduction section to highlight specific scenarios and necessary requirements for the benchmark. For example, in MMMU(MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI), the title emphasizes the scenario is "expert AGI". 
3. Limited Contribution. The main focus of the paper is on introducing the benchmark and evaluating various MLLMs on it. However, there is no novel approach in terms of the design, construction, or evaluation methods of the benchmark, nor is there an attempt to improve upon methods that performed poorly. This makes the contribution seem limited.