Venue: NAACL 2025

JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation

 Shota Onohara\({}^{1*}\)  Atsuyuki Miyai\({}^{1*}\) Yuki Imajuku\({}^{1*}\) Kazuki Egashira\({}^{1*}\) Jeonghun Baek\({}^{1*}\) Xiang Yue\({}^{2}\) Graham Neubig\({}^{2}\) Kiyoharu Aizawa\({}^{1}\)

Equal contribution.

###### Abstract

Accelerating research on Large Multimodal Models (LMMs) in non-English languages is crucial for enhancing user experiences across broader populations. In this paper, we introduce **JMMMU** (_Japanese MMMU_), the first large-scale Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the Japanese cultural context. To facilitate comprehensive culture-aware evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA) subset, where the clutter-independent subjects (e.g., Math) are selected and translated into Japanese, enabling one-to-one comparison with its English counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly crafted subjects that reflect Japanese cultural context. Using the CA subset, we observe performance drop in many LMMs when evaluated in Japanese, which is purely attributable to language variation. Using the CS subset, we reveal their inadequate Japanese cultural understanding. Further, by combining both subsets, we identify that some LMMs perform well on the CA subset but not on the CS subset, exposing a _shallow_ understanding of the Japanese language that lacks depth in cultural understanding. We hope this work will not only help advance LMM performance in Japanese but also serve as a guideline to create high-standard, culturally diverse benchmarks for multilingual LMM development.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 932–950

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

In recent years, large language models (LLMs) have revolutionized the field of language processing Chen et al. (2023); vic (2023); Touvron et al. (2023); Wei et al. (2023). Building on the success of LLMs, large multimodal models (LMMs) have demonstrated remarkable performance across tasks ranging from common sense reasoning to domain-specific, expert-level challenges Antol et al. (2015); Liu et al. (2023); Ma et al. (2024); Yue et al. (2024). As their capabilities grow, the need for robust criteria to evaluate LMMs has become increasingly important, highlighting the role of comprehensive benchmarks in assessing the full scope of their abilities.

However, current benchmarks focus primarily on performance in English Liu et al. (2024); Yue et al. (2024); Li et al. (2024); Liu et al. (2023); Yu et al. (2024); Fu et al. (2024), with less emphasis on evaluation in other languages. Given that LMMs are widely used across diverse languages, it is imperative to evaluate their performance beyond English. Additionally, such multilingual evaluations should actively involve contributions from diverse communities, ensuring that the associated cultural contexts are appropriately considered.

In this paper, we introduce **JMMMU** (_Japanese MMMU_), the first benchmark designed to evaluate LMMs on extensive, multi-disciplinary tasks in Japanese that require college-level subject knowledge, deliberate reasoning, and cultural understanding. The overview of JMMMU is shown in Figure 1. JMMMMU draws inspiration from the well-established MMMU Yue et al. (2024) and expands existing culture-aware Japanese benchmarks Inoue et al. (2024); SakanaAI (2024c) by over 10 times, with 1,320 questions using 1,118 images, covering a diverse range of subjects.

JMMMMU offers two key subsets: (i) **Culture-Agnostic (CA) Subset**: We extracted and translated the culture-agnostic components from MMMU. This subset allows for a direct comparison of the performance gaps between English and Japanese that are purely attributable to language variations. (ii) **Culture-Specific (CS) Subset**: We carefully crafted brand-new questions that align with the Japanese cultural context. With CS subset, developers can assess capabilities specifically tailored to Japanese culture. Together, JMMMU 

serves as a diagnostic tool for model developers, providing valuable feedback for future improvements.

Evaluating 16 open-source LMMs and three advanced proprietary LMMs on JMMMU, our key findings are summarized as follows:

* Overall performance is up to 58.6%, leaving great room for improvement in the utility of the Japanese context.
* The CA subset reveals that most models perform worse when asked in Japanese than in English (up to 8.6%), even when the question asks exactly the same content. This apple-to-apple comparison clearly indicates that the utility in non-English languages is falling behind in current LMMs.
* The CS subset reveals that models trained on Japanese datasets perform the best among open-source models, suggesting that such fine-tuning effectively contributes to incorporating Japanese cultural knowledge into the models.
* Combining both subsets, we reveal a significant discrepancy among the state-of-the-art proprietary models. While they perform similarly on English benchmarks and even on culture-agnostic questions in Japanese, their performances are significantly different on CS subset. This finding is particularly alarming, as it indicates that evaluation exclusively on a translation-based benchmark could risk overestimation of an LMM's multilingual capability without truly understanding the context of the individual cultures.

Our findings indicate that English-centered performance evaluation may lead to biased development, neglecting non-English languages. We hope our findings not only spark interest in Japanese performance but also motivate the community to craft a variety of high-standard benchmarks that encompass diverse cultures and their associated languages, thereby promoting more inclusive LMM development.

## 2 Related Work

Large Multimodal Models (LMMs)Following the success of large language models (LLMs), many LMMs have been developed with improved knowledge and instruction-following capabilities Liu et al. (2023b, 2024a, b); Li et al. (2024a); Ye et al. (2024); Zhao et al. (2023); Li et al. (2023); Monajatipoor et al. (2023); Zhao et al. (2024). However, the progress of these models is typically evaluated on English benchmarks Yue et al. (2024a); Liu et al. (2024c). For example, there is a study measuring LMM performance in Japanese--linguistically and culturally distinct from English--by generating recipe texts from food photos. Interestingly, these experiments found that existing proprietary LMMs sometimes perform worse than smaller open-source models fine-tuned on Japanese recipe data, suggesting that these proprietary models may lack sufficient knowledge of Japanese foodsImajuku et al. (2025). Therefore, a significant challenge remains in accurately evaluating the capabilities of other

Figure 1: **Overview of the JMMMU dataset. JMMMU includes 720 culture-agnostic (translation-based) questions and 600 culture-specific (newly created) questions, totaling 1,320 questions, thus expanding the existing culture-aware Japanese benchmark Inoue et al. (2024b) by over 10 times. JMMMU serves as a diagnostic tool for assessing both Japanese cultural understanding and culture-agnostic language understanding capability.**



languages, highlighting the need for non-English benchmarks.

LMM BenchmarksAmong various recent benchmarks [14; 236; 50; 78; 96; 10], MMMU [79] is the most widely used to measure the advancements of cutting-edge LMMs. MMMU requires advanced university-level knowledge and reasoning across a broader range of subjects, enabling a more comprehensive and expert-level evaluation. Subsequently, COMMMU [11] has been proposed as its Chinese counterpart. While COMMUM comprises entirely new culture-specific questions, our JMMMU has not only culture-specific subjects but also translation-based culture-agnostic subjects, facilitating one-to-one comparisons between English and Japanese using the exact same questions. In line with multilingual ability evaluation, several VQA benchmarks have been proposed [12, 13, 14, 15, 16, 17, 18, 19]. However, unlike the MMMU series, their primary focus is on daily knowledge, (e.g., _Pop Culture, Sports_ in CVQA [12]), still leaving the multilingual _expert-level_ reasoning skills as an important direction for future work.

Japanese LMM BenchmarksThe development of Japanese LMM benchmarks remains behind that of English benchmarks. While efforts have been made to create Japanese benchmarks as shown in Table 1, they still exhibit the following critical limitations: (i) Existing benchmarks [13, 14, 15, 16, 17, 18] focus primarily on common sense knowledge but do not adequately address expert-level knowledge, despite the advancement in LMMs and the importance of evaluating such capabilities. (ii) Many do not account for cultural differences. They are often created by directly translating existing English benchmarks [13, 14, 15], resulting in questions that may feel unfamiliar to Japanese people due to cultural context. (iii) Although recent benchmarks attempt to consider cultural differences [16, 17, 18], they are limited in size (up to 102 questions), raising concerns about the reliability of quantitative evaluation. Our proposed JMMMU addresses all three of the aforementioned challenges, significantly advancing the benchmark in the realm of Japanese evaluation.

## 3 JMMMU Benchmark

### Overview of JMMMU

As illustrated in Figure 1, JMMMU contains a total of 1,320 questions and 1,118 images, covering 28 different subjects. This benchmark is strategically divided into two distinct categories: culture-agnostic and culture-specific subjects.

Culture-agnostic subset consists of 24 subjects with 720 questions across five disciplines: (1) Art & Psychology, (2) Business, (3) Health & Medicine, (4) Science, and (5) Tech & Engineering. Culturespecific subset consists of 600 questions across four subjects: (1) Japanese Art, (2) Japanese Heritage, (3) Japanese History, and (4) World History. We provide sample questions in Appendix E

### Data Curation Process

JMMMU is derived from the widely-used validation set of MMMU, consisting of 900 questions across 30 subjects. To construct JMMMU, we first examined the cultural dependencies in the original MMMM subjects. For culture-agnostic subjects, we translated the questions into Japanese. We further replaced culture-dependent subjects with new subjects that are conceptually similar, but better aligned with the Japanese context. All the process has been conducted with the help of 19 university students, including the authors, who have expert knowledge in the respective fields and native fluency in Japanese. Here, we describe the dataset creation process in detail.

**Examining Cultural Dependencies in MMMU** Among the 30 subjects in MMMU, we identified that questions in six subjects are particularly unfamiliar to Japanese people and thus we categorized them as culture-specific subjects; _Art, Art Theory, Geography, History, Literature, and Sociology_. The remaining subjects (e.g., _Biology, Chemistry, Computer Science, Electronics_) exist

\begin{table}
\begin{tabular}{l|c c c c} \hline Benchmark & Culture & Level & Questions Images \\ \cline{2-4} I-VQYA-XQQ [18] & ✗ & Common sense & 500 & 500 \\ IA-VA-Bench+in-the-wild [19] & ✗ & Common sense & 60 & 24 \\ JA-Math-Image-VQA [18] & ✓ & Common sense & 55 & 39 \\ JA-VLML-bench+wide [18] & ✓ & Common sense & 50 & 42 \\ Heorehosh [18] & ✓ & Common sense & 102 & 21 \\ MMMU [19] & ✓ & Expert & 1,320 & 1,118 \\ \hline \end{tabular}
\end{table}
Table 1: **Overview of Japanese LMM benchmarks. JMMMU is the first benchmark that evaluates expert-level skills and is the largest among culture-aware benchmarks.** 

in Japan with similar contents, and thus we categorized them as culture-agnostic subjects. As a result, we excluded the six culture-specific subjects while keeping the remaining 24 culture-agnostic subjects in JMMMU.

Translating Culture-Agnostic SubjectsThe experts were provided with the original English texts, GPT-4o-translated question texts in Japanese, and corresponding images. For texts, their task involved: (i) refining the auto-translated Japanese text to ensure naturalness and fluency; (ii) confirming that technical terms and academic expressions adhere to conventional Japanese usage; and (iii) adjusting the currency to reflect typical digit lengths in Japanese yen (Y). For currency conversion, a simplified conversion ($1 \(\rightarrow\) $100) was employed to avoid making the calculation unnecessarily complicated. For images, we asked the experts to overwrite the English text with Japanese text by using an image editing tool. An example of the image translation process is presented in Figure 2.

Consequently, we obtained 720 questions covering 24 culture-agnostic subjects fully translated and adapted for Japanese usage.

Creating Culture-Specific SubjectsRecognizing that most of the removed subjects are related to art or social studies, we created the following new subjects to test similar knowledge in the Japanese context:

* _Japanese Art_: Questions about traditional Japanese art, such as Ukiyo-e and Noh.
* _Japanese Heritage_: Questions about traditional, culturally significant locations and buildings in Japan such as temples and shrines.
* _Japanese History_: Questions about historical incidents in Japan.
* _World History_: Questions about global historical incidents, but based on the content typically covered in Japanese textbooks to better reflect the Japanese educational context than _History_ in MMMU.

We carefully designed the subject selection process to include only standard fields of study at the Japanese university level. The images are primarily sourced from Wikimedia Commons1, ensuring that all selected images are available under licenses suitable for public release. In crafting questions, we aimed to keep the text as simple as possible and ensure that no options stand out, making it hard to guess the correct choice without referring to the image.

Footnote 1: [https://commons.wikimedia.org/](https://commons.wikimedia.org/)

### Comparison with Other Japanese Multimodal Benchmarks

Here, we compare JMMMU with other Japanese multimodal benchmarks, provided in Table 1, to demonstrate its uniqueness. First and foremost, JMMMU is the only benchmark that includes expert-level questions, while the rest of the benchmarks Shimizu et al. (2018); Turing (2024a); SakanaAI (2024a, c); Inoue et al. (2024b) are focused on common knowledge. Further, JMMMU is carefully designed to take the Japanese cultural context into account. While some existing benchmarks consider Japanese culture, they are all limited in size (only up to 102 questions in Inoue et al. (2024b)), raising concerns about whether reliable quantitative evaluations can be conducted. In contrast, JMMMU contains more than 10 times larger than any of the existing culture-aware benchmarks.

## 4 Experiments

### Setup

LMMsWe evaluate a diverse set of LMMs.

* **Proprietary LMMs**: GPT-4o (OpenAI, 2024) Gemini 1.5 Pro (DeepMind, 2024; Reid et al., 2024) and Claude 3.5 Sonnet (anthropic, 2024).
* **Japanese LMMs**: LLaVA CALM2 (Inagaki, 2024) and EvoVLM JP v2 (Inoue et al.,

Figure 2: **Example of the image translation process.** English words in the image are manually overwritten with Japanese.

 

2024a), which are trained on both English and Japanese datasets.
* **Open-source LMMs**: LLaVA-OneVision 0.5B & 7B Li et al. (2024), LLaVA1.6-13B & 34B Liu et al. (2024b), Phi-3 & 3.5 Vision Abdin et al. (2024), InternVL2-2B & 8B Chen et al. (2023b), xGen-MM Xue et al. (2024), Idefics2-8B Laurencon et al. (2024b), Idefics3-8B Laurencn et al. (2024a), CogVLM2-19B Hong et al. (2024), Mantiss-8B Jiang et al. (2024), and Pangeea-7B Yue et al. (2024b).

We run all experiments with LMMs-Eval Zhang et al. (2024b). In Appendix A, we provide further details of these models, with a particular focus on Japanese language support.

Text-only LLMAs a reference, we present the accuracy of GPT-4o when provided only with the question text and choices, without images.

EvaluationThe evaluation method is based on the setup in MMMU Yue et al. (2024a). Prompts are translated as follows: for multiple-choice questions,  []\(\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xright{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{} \xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\xleftarrow{}\x{}\xleftarrow{}\xleftarrow{}\x{}\xleftarrow{}\x{}\xleftarrow{}\x{}\xleftarrow{}\x{}\xleftarrow{}\x{}\xleftarrow{}\x{}\x{}\}}

 

scores across all subjects, culture-agnostic (CA) subjects, and culture-specific (CS) subjects, as well as scores on individual subjects. For comparison, we also provide the performance on CA suset in English CA (EN) . Note that CA (EN) is often smaller than the overall average of MMMU given by Yue et al. (2024a) because subjects selected as CA are relatively difficult among all subjects in MMMU as it often requires stronger reasoning capabilities _(e.g., Math)_.

Here, we summarize our key observations.

Challenging NatureIn our experiment, the performance is up to 40.5% for open-source, and 58.6% for proprietary models, leaving great room for improvement. This also highlights a significant gap between open-source and proprietary models, presenting a more difficult challenge for open-source models.

The Effect of Translation in CA SubsetFirst, as a general trend, the score on the CA subset is significantly lower than its English counterpart (CA (EN) in Table 2) with an average drop of 4.3%. This indicates that, even for the same questions, many models perform worse when asked in Japanese. Second, despite such a general trend, Japanese-made LMMs (i.e., LLaVA CALM2 and EvoVLM JP v2) face a minimal drop (up to 1.7 %), which implies that incorporating the Japanese dataset successfully mitigates the performance gap between English and Japanese.

The Performance of Japanese LMMsFigure 3 demonstrates the correlation between the scores on the CA and CS subjects. The Japanese LMMs (LLaVA CALM2 and EvoVLM JP v2) and Pangea (a culture-aware multilingual LMM) show higher scores on CS subjects compared to other models that perform similarly on CA subjects. This strongly indicates their proficiency in CS subjects. As shown in appendix A, Japanese LMMs uses not only translated corpora but also texts originally written in Japanese. This may contribute to their performance in CS subjects. On the other hand, however, compared to stronger models such as InternVL2-8b, LLaVA1.6-34b, and LLaVA-OV-7b, the Japanese LMMs show lower scores on CA subjects, suggesting room for improvement in their general reasoning and problem-solving capabilities in culture-agnostic context.

Scores on Japanese HeritageAmong CS subjects, the performance of open-source models is particularly low in Japanese Heritage (Table 2). Even the best-performing open-source model (EvoVLM JP v2) scores 30.7% lower than GPT-4o in Japanese Heritage, while in other CS subjects, there is at least one open-source model whose gap from GPT-4o is \(\leq\) 16.7%, indicating the particular inadequacy of the open-source model in Heritage domain.

GPT-4o vs. Claude 3.5 SonnetWe reveal a significant performance gap between the two leading models; GPT-4o and Claude 3.5. They are state-of-the-art models and their performance is known to be similar with only 0.8% difference on the MMMU benchmark in English (Anthropic, 2024). Further, their performance is similar even on CA split in Japanese (1.2% difference in Table 2). However, on the CS split, GPT-4o outperforms Claude 3.5 Sonnet by a substantial 15.7%.

This strongly indicates that a model's Japanese language skill and its understanding of Japanese culture should be separately discussed. Our research is pioneering in revealing this, a discrepancy that would have remained obscured without combining translation-based CA subjects and brand-new CS subjects. Our finding underscores the limitations of relying exclusively on auto-translated benchmarks for a thorough evaluation of model capabilities in non-English languages, highlighting the importance of evaluating models on culture-specific questions.

Figure 3: **Score correlation between subsets. While proprietary models (\(\blacksquare\)) perform the best on both subsets, Japanese LMMs (\(\bigstar\)) and Pangea (\(\blacklozenge\), a culture-aware multilingual LMM) perform remarkably high on CS subset compared to models that perform similarly on CA subset.**





## 5 Analysis

### Ablation on Image Translation

Here, we investigate how translating text and images affects the model performances. Using 360 questions from the culture-agnostic subset which involved translation of both texts and images, we compare the scores in English (\(I_{en}T_{en}\)), when only text is translated (\(I_{en}T_{jp}\)), and when both text and images are translated (\(I_{jp}T_{jp}\)). We provide scores for selected models in Table 3 and the full set in Appendix B.2. Many models experience a drop in scores by text translation, with further degradation observed when images are also translated (i.e., \(0>\Delta_{1}>\Delta_{2}\)). However, some models exhibit different performance trends, showing a drop by text translation but an improvement by translating both (i.e., \(\Delta_{1}<0<\Delta_{2}\)), or vice versa. Overall, while the trends are complex, our result indicates that text-only translation, as is done in many non-English benchmarks, could result in a biased performance evaluation. Rigorous investigation on this point is left for future work.

### Errors in Culture-agnostic Subjects

JMMMU shares 600 culture-agnostic questions with MMMU, which allows us to compare the output one by one. Using these questions, we evaluate how translation affects model performance. Taking GPT-4o as an example, we classify the responses into four categories based on whether they are correct or incorrect in each language. Figure 4 presents the results before and after translation. The results on the other models are provided in Appendix B.1

While GPT-4o performs similarly in both languages on culture-agnostic split (only 0.3% difference in Table 2), we have found that there are a significant amount (28.6 %) of questions to which it answered correctly only in either one of the languages. We now investigate this phenomenon. For questions answered correctly only in English (orange in Figure 4(a)), we observe simple performance degradation after translation. In contrast, we have found some distinctive examples in the opposite case (yellow). In an example of Figure 4(b), GPT-4o outputs only the direct answer in English, whereas in Japanese, the model includes the reasoning process in its response although the model is instructed to generate the choice directly by using the prompt in Section 4.1. Approximately 70% of the cases marked in yellow exhibit this tendency to include the reasoning process only in Japanese. For a fair comparison with MMMU Yue et al. (2024), we count a response to be correct as far as the model's response is accurate and can be parsed by a rule-based algorithm, regardless of its instruction-following ability. As a result, the scores can sometimes be counterintuitively overestimated due to the lack of instruction following skills in Japanese. While the primary focus of JMMMU is on evaluating expert knowledge and supporting the improvement of such capabilities, our findings highlight a crucial direction for future work: mea

suring and enhancing instruction-following ability in non-English languages.

### Errors in Culture-specific Subjects

This section presents an analysis of the tendency of GPT-4o's errors in the culture-specific subjects. To investigate the causes of these errors, we manually review GPT-4o's responses and classify the errors into four categories: (i) **Lack of Knowledge**, where the model successfully extracts the necessary information from the image but lacks the culture-specific knowledge required to produce a correct answer, (ii) **Image Recognition Errors**, where it fails to correctly interpret the image during the visual understanding stage, (iii) **Answer Rejection**, where it declines to provide an answer, and (iv) **Textual Misunderstanding**, where the response is not aligned with the question. The overall distribution of these error types is shown in Figure 5. **Lack of Knowledge** is the overwhelming majority at over 50%, indicating that culture-specific knowledge is the most critical requirement to achieve high performance in JMMMU. In this section, we discuss notable examples for each error category.

 



### Limitations

Throughout our experiment and extensive analysis, we have shown a number of critical directions of improvement in multilingual benchmarks and model developments. While they are outside of the scope of this paper, they are left as important directions for future work, and thus we summarize them here:

Subject Set ExpansionWhile JMMMU can assess the latest LMMs' expert-level skills, it cannot evaluate model performance on subjects outside of those currently covered. As models gain more knowledge and improve their reasoning abilities, it will be necessary to expand the range of subjects and include more challenging questions.

Benchmarks in Other CulturesSince JMMMU only covers the Japanese, evaluating model performance in other languages and cultural contexts remains an important area for future work. We hope these efforts will help mitigate the underrepresentation of diverse cultures and languages.

Instruction Following Ability in JapaneseIn Section 5.2, we have shown a gap in instruction-following ability between languages and that models go against the instruction and generate their reasoning more often in Japanese. While the primary focus of our benchmark is on evaluating expert knowledge and thereby helping improve such skills, it is left as an important future work to improve the instruction-following ability in Japanese. Further, it is also important to design an evaluation protocol to measure instruction-following ability to enhance the development of such skills. While there are some methods to evaluate the model's instructions following ability Zhou et al. (2023); Qian et al. (2024), these should be appropriately incorporated in the context of multilingual performance evaluation.

## Acknowledgments

This work was partially supported by JST BOOST, Japan Grant Number JPMJBS2418, JSPS KAKENHI Grant Number 24K23882, and JST JPMICR22U4.

## References

* A. Abdin, J. Aneja, H. Awadalla, A. Awadallah, et al. (2024)Phi-3 technical report: a highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. Cited by: SS6.
* T. Akiba, M. Shing, Y. Tang, Q. Sun, and D. Ha (2024)Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187. Cited by: SS6.
* A. Amtoglu, A. Agrawal, I. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh (2015)Vqa: visual question answering. In ICCV, External Links: Link Cited by: SS6.
* S. Changpinyo, L. Xue, M. Yarom, A. V. Thapliyal, I. Szpektor, J. Amelot,

Xi Chen, and Radu Soricut. 2022. Maxm: Towards multilingual visual question answering. _arXiv preprint arXiv:2209.05401_.
* Chen et al. (2023a) Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2023a. Alpagasus: Training a better alpaca with fewer data. _arXiv preprint arXiv:2307.08701_.
* Chen et al. (2024) Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2024. Sharegpt4v: Improving large multi-modal models with better captions. In _ECCV_.
* Chen et al. (2023b) Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. _arXiv preprint arXiv:2312.14238_.
* DeepMind (2024) Google DeepMind. 2024. Gemini 1.5 pro. [https://deepmind.google/technologies/gemini/pro/](https://deepmind.google/technologies/gemini/pro/).
* Fu et al. (2024) Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. In _ECCV_.
* Gao et al. (2015) Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question. _Advances in neural information processing systems_, 28.
* Gupta et al. (2020) Deepak Gupta, Pabitra Lenka, Asif Ekbal, and Pushpak Bhattacharyya. 2020. A unified framework for multilingual and code-mixed visual question answering. In _Proceedings of the 1st conference of the Asia-Pacific chapter of the association for computational linguistics and the 10th international joint conference on natural language processing_, pages 900-913.
* Hong et al. (2024) Wenyi Hong, Weinhan Wang, Ming Ding, Wenmeng Yu, Qingssong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. 2024. Cogylvm2: Visual language models for image and video understanding. _arXiv preprint arXiv:2408.16500_.
* Hu et al. (2024) Jinyi Hu, Yuan Yao, Chongyi Wang, SHAN WANG, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. Large multilingual models pivot zero-shot multimodal learning across languages. In _ICLR_.
* Imajkuku et al. (2025) Yuki Imajkuku, Yoko Yamakata, and Kiyoharu Aizawa. 2025. Foodmllm-jp: Leveraging multimodal large language models for japanese recipe generation. In _International Conference on Multimedia Modeling_, pages 401-414.
* Inagaki (2024) Aozora Inagaki. 2024. Ilava-calm2-siglip. [https://huggingface.co/cyberagent/llava-calm2-siglip](https://huggingface.co/cyberagent/llava-calm2-siglip).
* Inoue et al. (2024a) Yuichi Inoue, Takuya Akiba, and Shing Makoto. 2024a. Llama-3-evovlrm-jp-v2.
* Inoue et al. (2024b)Yuichi Inoue, Kento Sasaki, Yuma Ochi, Kazuki Fujii, Kotaro Tanahashi, and Yu Yamaguchi. 2024b. Heronbench: A benchmark for evaluating vision language models in japanese. In _CVPR workshop_.
* Jiang et al. (2024) Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. 2024. Mantis: Interleaved multi-image instruction tuning. _arXiv preprint arXiv:2405.01483_.
* Laurencon et al. (2024a) Hugo Laurencon, Andres Marafioti, Victor Sanh, and Leo Tronchon. 2024a. Building and better understanding vision-language models: insights and future directions. _arXiv preprint arXiv:2408.12637_.
* Laurencon et al. (2023) Hugo Laurencon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kela, Matthieu Cord, and Victor Sanh. 2023. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Laurencon et al. (2024b) Hugo Laurencon, Leon Tronchon, Matthieu Cord, and Victor Sanh. 2024b. What matters when building vision-language models? _arXiv preprint arXiv:2405.02246_.
* Li et al. (2023) Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023. Otter: A multi-modal model with in-context instruction tuning. _arXiv preprint arXiv:2305.03726_.
* Li et al. (2024a) Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. _arXiv preprint arXiv:2408.03326_.
* Li et al. (2024b) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2024b. Seed-bench: Benchmarking multimodal llms with generative comprehension. In _CVPR_.
* Liu et al. (2021) Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. _arXiv preprint arXiv:2109.13238_.
* Liu et al. (2023a) Fuxiao Liu, Hao Tan, and Chris Tensmeyer. 2023a. Documentclip: Linking figures and main body text in reflowed documents. _arXiv preprint arXiv:2306.06306_.
* Liu et al. (2024a) Haotiian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In _CVPR_ 

* Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llava-next: Improved reasoning, ocr, and world knowledge.
* Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In _NeurIPS_.
* Liu et al. (2024c) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2024c. Mmbench: Is your multi-modal model an all-around player? In _ECCV_.
* Lu et al. (2024d) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024d. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In _ICLR_.
* Miyai et al. (2024) Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. 2024. Unsolvable problem detection: Evaluating trustworthiness of vision language models. _arXiv preprint arXiv:2403.20331_.
* Monajatipoor et al. (2023) Masoud Monajatipoor, Liumian Harold Li, Mozhdeh Rouhsedaghat, Lin F Yang, and Kai-Wei Chang. 2023. Metavl: Transferring in-context learning ability from language models to vision-language models. _arXiv preprint arXiv:2306.01311_.
* OpenAI (2024) OpenAI. 2024. Gpt-4o.
* Pfeiffer et al. (2021) Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan Martin O Steitz, Stefan Roth, Ivan Vulic, and Iryna Gurevych. 2021. xgqa: Cross-lingual visual question answering. _arXiv preprint arXiv:2109.06082_.
* Qian et al. (2024) Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. 2024. Mia-bench: Towards better instruction following evaluation of multimodal llmns. _arXiv preprint arXiv:2407.01509_.
* Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-bantigtse Alayrac, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _arXiv preprint arXiv:2403.05530_.
* Romero et al. (2024) David Romero, Chenyang Lyu, Haryo Akbarianto Wibowe, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnfau Lambebo Tonja, et al. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. In _NeurIPS Datasets and Benchmarks Track_.
* SakanaAI (2024a) SakanaAI. 2024a. Ja-multi-image-vqa. [https://huggingface.co/datasets/SakanaAI/JA-Multi-Image-VQA](https://huggingface.co/datasets/SakanaAI/JA-Multi-Image-VQA).
* SakanaAI (2024b) SakanaAI. 2024b. Ja-vg-vqa-500. [https://huggingface.co/datasets/SakanaAI/JA-VG-VQA-500](https://huggingface.co/datasets/SakanaAI/JA-VG-VQA-500).
* SakanaAI (2024c) SakanaAI. 2024c. Ja-vlm-bench-in-the-wild. [https://huggingface.co/datasets/SakanaAI/JA-VLM-Bench-In-the-Wild](https://huggingface.co/datasets/SakanaAI/JA-VLM-Bench-In-the-Wild).
* Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivatso, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2023. Language models are multilingual chain-of-thought reasoners. In _The Eleventh International Conference on Learning Representations_.
* Shimizu et al. (2018) Nobuyuki Shimizu, Na Rong, and Takashi Miyazaki. 2018. Visual question answering dataset for bilingual image understanding: A study of cross-lingual transfer using attention maps. In _COLING_.
* Tang et al. (2024) Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitzi FairE Bin Mahmood, Hao Feng, Zhen Zhao, et al. 2024. Mtvaq: Benchmarking multilingual text-centric visual question answering. _arXiv preprint arXiv:2405.11985_.
* Tuvron et al. (2023) Hugo Tuvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Turing (2024a) Turing. 2024a. Llava-bench-in-the-wild. [https://huggingface.co/datasets/liuhaoatin/llava-bench-in-the-wild/tree/main](https://huggingface.co/datasets/liuhaoatin/llava-bench-in-the-wild/tree/main).
* Turing (2024b) Turing. 2024b. Llava-bench-in-the-wild (journales). [https://github.com/turingmotors/heron/tree/main/playground/data/llvma-bench-in-the-wild](https://github.com/turingmotors/heron/tree/main/playground/data/llvma-bench-in-the-wild).
* Turing (2024c) Turing. 2024c. Llava-bench-ja. [https://github.com/turingmotors/heron/tree/main/playground/data/llava-bench-ja](https://github.com/turingmotors/heron/tree/main/playground/data/llava-bench-ja).
* Wang et al. (2024) Yukia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024. Do-not-answer: Evaluating safegu guards in LLMs. In _Findings of the Association for Computational Linguistics: EACL 2024_, pages 896-911, St. Julian's, Malta. Association for Computational Linguistics.
* Wei et al. (2023) Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. _arXiv preprint arXiv:2303.03846_.
* Xue et al. (2024) Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, et al. 2024. xgen-mm (blip-3): A family of open large multimodal models. _arXiv preprint arXiv:2408.08872_.
* Ye et al. (2024) Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In _CVPR_ 

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. Mm-vet: Evaluating large multimodal models for integrated capabilities. In _ICML_.
* Yue et al. (2024a) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoiqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxun Sun, et al. 2024a. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In _CVPR_.
* Yue et al. (2024b) Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanija, Anjalil Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. 2024b. Pangea: A fully open multilingual multimodal llm for 39 languages. _arXiv preprint arXiv:2410.16153_.
* Zhang et al. (2024c) Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyau Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, et al. 2024a. Cmmnu: A chinese massive multi-discipline multimodal understanding benchmark. _arXiv preprint arXiv:2401.11944_.
* Zhang et al. (2024d) Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Chayono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. 2024b. L'mms-eval: Reality check on the evaluation of large multimodal models. _arXiv preprint arXiv:2407.12772_.
* Zhao et al. (2023) Bo Zhao, Boya Wu, and Tiejun Huang. 2023. Svit: Scaling up visual instruction tuning. _arXiv preprint arXiv:2307.04087_.
* Zhao et al. (2024) Haozhee Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2024. Mmtl: Empowering vision-language model with multi-modal in-context learning. In _ICLR_.
* Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. _arXiv preprint arXiv:2311.07911_.

 



## Appendix A LMMs' Japanese Support

To discuss the multilingual capabilities of LMMs, we summarize whether each model officially supports Japanese. Table 1 presents the Japanese language support status for each model. "\(\mathbf{\check{\mathsf{\mathcal{V}}}}\)" indicates official support for Japanese, while "\(\mathbf{\check{\mathsf{X}}}\)" indicates the absence of such support. Also, we denote "?" for models of which we could not find the information.

Even if a model is marked as "\(\mathbf{\check{\mathsf{\mathcal{K}}}}\)", it may still demonstrate some Japanese language capability due to the presence of Japanese data in publicly available datasets like ShareGPT-4V [30] and ShareGPT-4o2, or data crawled from the web.

Footnote 2: [https://huggingface.co/datasets/OpenGVLab/ShareGPT-4o](https://huggingface.co/datasets/OpenGVLab/ShareGPT-4o)

Proprietary commercial models, such as GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet, do not publicly disclose detailed information about their training data. However, based on their release blog posts, it can be inferred that these models support many languages, including Japanese.

LLaVA CALM2 is based on the Japanese LLM CALM23, and it has been trained using Japanese multimodal datasets, officially supporting Japanese. EvoVLM JP v2, a merged model [7], also incorporates Japanese data for optimization and is officially released as a Japanese LAM.

Footnote 3: [https://huggingface.co/cyberagent/calm2-7b-chat](https://huggingface.co/cyberagent/calm2-7b-chat)

Phi-3.5 Vision does not officially support Japanese, despite its base model, Phi-3.5, having official support for multiple languages, including Japanese. Phi-3 Vision, likewise, does not support non-English languages.

In the LLaVA series, LLaVA-OneVision explicitly mentions support for Chinese in its training but does not extend this to other non-English languages. However, Qwen2, the base LLM for the LLaVA-OneVision models, officially supports Japanese. LLaVA-1.6 models are trained from different base LLMs, such as Vicuna v1.5 and Nous Hermes 2 Yi, neither of which officially support Japanese. Thus, Japanese language capabilities are not guaranteed in their visual instruction training.

InternVL and its base model, InternLM2, officially support only English and Chinese. Similarly, CogVLM2 claims proficiency in both English and Chinese, with no explicit mention of Japanese support.

Idefics2, Idefics3, xGen-MM, and Mantis use large-scale datasets for multimodal training. However, there is no clear evidence of Japanese data inclusion, and in some datasets, such as OBELICS [9], non-English data is explicitly filtered out. While Llama 3, the base model for some of these LMMs, mentions multilingual training, it does not explicitly confirm support for Japanese. Mistral v0.1 also does not disclose its training data.

Pangea is designed as a multilingual LMM. This model is trained on 6 million instruction datasets across 39 languages, including Japanese.

The performance of these models depends on a complex interplay of factors, including the quantity and quality of the training data and the size and capabilities of the base language model. Official support for Japanese is not the only consideration; there are reports of models trained on English-only multimodal data generalizing to other languages [8], including Japanese. Moreover, since many models are designed with Chinese support, the cultural and linguistic proximity between Japanese and Chinese-speaking regions may result in a high performance in Japanese.



More Result

### Error Analysis in Culture-Agnostic subjects

In Section 5.2, we present the error analysis for GPT-4o on the CA subjects. In this section, we provide the error analysis for all models. While we have shown in Table 2 that most models perform worse in Japanese, there are some amount of questions where the model answers correctly only in Japanese for every model. The number of such questions is particularly high for LLaVA-OV 0.5B and InternVL2 2B. This occurrence, however, appears to be a random phenomenon, likely attributable to the overall weaker performance of these models.

### Ablation on Image Translation

The full set of Table 3 is presented in Table B. As discussed in Section 5.1, each model reacts differently as the translation proceeds, and the tendency is difficult to summarize. Notably, here, GPT-4o shows a 7.2% improvement in score after text translation. This partly stems from its weak instruction-following skills in Japanese, as discussed in Section 5.2, which allows it to infer answers more easily. Note that our experiment here has been conducted by using questions that involved translation of both texts and images. Many of them consist of table data, which requires stronger reasoning based on data processing, so the result may vary when investigating different data types that do not exist in the CA subset of JMMMU.

### Score Correlation between languages

Using the culture-agnostic subset, we have demonstrated in Section 4.2 that (i) models perform worse in Japanese and (ii) Japanese LMMs show robustness to translation. To illustrate these points, we provide Figure B.

### Chain of Thought Performance

We investigate the effect of Chain of Thought (CoT) prompting on performance enhancement. The prompts are derived from the CoT prompts utilized in MMMU-Pro Yue et al. (2024), translated into Japanese for this study. The prompts are translated as follows: for multiple-choice questions, \(\uparrow\downarrow\downarrow\rangle\)\(\rightarrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\uparrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\leftarrow 

English compared to English (Shi et al., 2023).

To investigate reasoning abilities across different languages in JMMMU, we evaluate the models with English direct prompts. English Direct Prompts refer to cases where the question remains in Japanese, but the instruction is given in English. Direct Prompts are defined as follows: for multiple-choice questions, "Answer with the option's letter from the given choices directly."; and for open-ended questions, "Answer the question using a single word or phrase."

We show the results in fig. D. Notable improvements in scores are observed in Pangea's CA subjects and GPT-4o's CS subjects.

A detailed analysis of the model outputs reveals that Pangea exhibits significant deficiencies in instruction-following ability when prompted in Japanese, frequently failing to provide answers in correct formats. In contrast, the underlying cause of the substantial score increase in GPT-4o's CS subjects remains unclear and requires further investigation.

## Appendix C Further Experimental Details

### Experimental Setup

Computing InfrastructuresWe conduct all our evaluations of open-source models on a single NVIDIA A100 (80GB) GPU.

Parameters for LMM InferenceA maximum output length is set to 1,024 and a temperature is set to 0 for all models during inference.

### Evaluation Protocol

**Answer Extraction in Multiple Choice Question** While the models are instructed to answer their 

choice directly, they often generate some contextual information or unnecessary symbols. To tackle this point, following MMMU Yue et al. (2024), we extract an answer from the model response with a rule-based method. For multiple-choice questions, this parser can extract the model's choice even when the choice is surrounded by some symbol (e.g., '(A)', 'A.', 'A ') or by text.

For example, these answers, which are all some variants of _"The answer is A."_ in Japanese, can be parsed as "A":

* []
* []
* A 15.3

While this allows an evaluation robust against some variety of answer generation styles, we have shown in Section 5.2 that this can sometimes overestimate the performance in Japanese because models' instruction-following abilities are relatively low in Japanese.

## Appendix D Annotation Instruction

Recruitment and PaymentAnnotators were paid at least the minimum wage set in Japan, according to the time spent on the task.

Data ConsentThey were informed that translated data would be used for evaluation purposes.

Instructions Given to ParticipantsThe document containing the instructions presented to the annotators is shown in Figure E.

## Appendix E Examples

We provide sample questions from culture-agnostic subset in Figure F, and questions from culture-specific subset in Figure G 

* [15] M. J. C. Collins, and R. S. Thorne, _Theoretical Computer Science Series_**2006**, 379-487 (2006)
* [16] A. D. Krizhevsky, I. Sutskever, and V. E. Hinton, _Deep Learning: An Introduction to Machine Learning_. Springer, New York, NY, USA, 2006.
* [17] P. G. LeCun, L. Bottou, Y. Bengio, and T. Darrell, _Coucet classification with deep convolutional neural networks_, arXiv preprint arXiv:1412.6098v1 (2014)
* [18] B. Li, X. Liu, Z. Wang, and H. Zhang, _Deep learning for image recognition_, arXiv preprint arXiv:1412.6098v1 (2014)
* [19] W. Lin, _A Practical Guide of Deep Learning in Image Processing_. Springer, Berlin Heidelberg, 2014.
* [20] F. Liu, Y. Bengio, and T. Darrell, _Deep Learning: A Survey on the Foundations of Computer Vision_. Springer, Berlin Heidelberg, 2014.

[MISSING_PAGE_POST]

 

* [125] M. C. Bettler, J. A. K. Wilson, and R. D. Wiese, _Theoretical Computer Science of the Universe: The New York_, (2006).
* [126] G. F. Hinton, _Statistical Inference in Machine Learning_, volume 349 of _Lecture Notes in Statistics_, pages 78-86. Springer Berlin Heidelberg, 2006.
* [127] S. Kullback, P. H. Baer, and T. L. Cover, _Elements of Information Theory_. Cambridge University Press, Cambridge, 1994.
* [128] E. Candes, _Bayesian Analysis with Applications to Data Mining_. Chapman and Hall/CRC, 2006.
* [129] V. N. Kolmogorov and I. A. Rubin, _Random Processes_. Princeton University Press, Princeton, 1984.
* [130] V. N. Kolmogorov and I. A. Rubin, _Probability and Randomness_. Academic Press, Boston, MA, 1984.

[MISSING_PAGE_POST]

 

* [150] M. C. Bader, J. A. Kwiecinski, and D. R. Sorensen. The role of the \({}^{3}\)-image 1-\({}^{>}\) in image 1-\({}^{>}\).
* [151] P. G. De Silva, L. F. de Oliveira, and T. E. V. Lebedev. Image classification using a deep neural network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'2014)_, pages 769-782, June 2014.
* [152] H. Elsasser, N. G. Hinton, and I. Goodfellow. Imagenet classification with deep convolutional networks. In _Advances in Neural Information Processing Systems (NIPS'09)_, pages 1398-1403, 2009.

[MISSING_PAGE_POST]