Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

## Abstract

Abstract Despite recent advances in multimodal large language models (MLLMs), their development has predominantly focused on English- and western-centric datasets and tasks, leaving most of the world’s languages and diverse cultural contexts underrepresented.
This paper introduces Pangea , a multilingual multimodal LLM trained on PangeaIns , a diverse 6M instruction dataset spanning 39 languages. PangeaIns features: 1) high-quality English instructions, 2) carefully machine-translated instructions, and 3) culturally relevant multimodal tasks to ensure cross-cultural coverage.
To rigorously assess models’ capabilities, we introduce PangeaBench , a holistic evaluation suite encompassing 14 datasets covering 47 languages.
Results show that Pangea significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts. Ablation studies further reveal the importance of English data proportions, language popularity, and the number of multimodal training samples on overall performance. We fully open-source our data, code, and trained checkpoints, to facilitate the development of inclusive and robust multilingual MLLMs, promoting equity and accessibility across a broader linguistic and cultural spectrum.

## 1 Introduction

Multimodal large language models (MLLMs) (Liu et al., ; Deitke et al., ; OpenAI, ; Team et al., ) have demonstrated impressive capabilities in tasks such as image captioning, visual question answering, and complex reasoning (Yue et al., ; ).
Despite this rapid progress in their reasoning capabilities, a critical flaw persists: *the overwhelming focus on English- and western-centric training datasets and evaluation benchmarks* (Liu et al., ; Song et al., ).

This homogeneity results in a lack of representation for the vast majority of the world’s languages and diverse cultural contexts (Yu et al., ).
Consequently, models predominately trained on such data exhibit: (a) diminished performance in multilingual settings (Blasi et al., ) with poor tokenization further leading to higher inference costs (Ahia et al., ); (b) generate outputs misaligned with the socio-cultural norms of underrepresented languages (AlKhamissi et al., ); and (c) lack the ability to recognize objects from geographically diverse regions (Ramaswamy et al., ) or rare objects belonging to the long-tail (Gupta et al., ).
With the increased adoption of these models into real-world applications across the globe, there’s an urgent need to develop multilingual MLLMs that equitably serve a diverse set of users. Few efforts have been made to develop multilingual MLLMs (Geigle et al., ; Rasheed et al., ), however, their performance still exhibits inequalities across languages and lacks evaluation of cultural understanding.

In this paper, we address how to train and evaluate multilingual MLLMs that are culturally inclusive, using limited open-source resources, tackling four major challenges (Yu et al., ): 1) Data scarcity: high-quality multilingual multimodal data is scarce, especially in low-resource languages, which makes it difficult to create large-scale training data; 2) Cultural nuances: visual interpretations are context-dependent and vary across cultures (Ramaswamy et al., ; Khanuja et al., ); 3) Catastrophic forgetting: training on many languages or modalities often results in suboptimal performance on some subsets and require careful balancing; 4) Evaluation complexity: developing an evaluation suite that accurately measures performance across languages and cultures requires substantial resources and expertise.

To tackle these challenges, we introduce Pangea, an open-sourced multilingual MLLM designed to bridge linguistic and cultural gaps in visual understanding tasks. Pangea is trained on PangeaIns (Figure 2), a high-quality multilingual multimodal instruction tuning dataset comprising 6 million samples in 39 typologically diverse languages.
PangeaIns combines existing open-source resources with newly created instructions focused on multicultural understanding. We curate high-quality English instructions, carefully translate them, and adapt them for multilingual contexts. To address Western-centric biases in visual representations, we source images from LAION-Multi (Schuhmann et al., ), which includes images from various countries and captions in multiple languages. However, LAION-Multi contains many images that are not culturally representative of the country’s speaking population. Additionally, the associated alt text is often short, noisy, and lacks sufficient detail. To combat these issues, we develop a multi-cultural multilingual multimodal instruction generation pipeline. This pipeline leverages an LLM (Dubey et al., ) to score and filter images based on cultural informativeness. We then enhance the remaining data by generating detailed descriptions and creating complex instructions that combine culturally relevant tasks with general multilingual scenarios. This approach improves the model’s cultural understanding while maintaining robust multilingual performance.

To evaluate Pangea’s capabilities, we present PangeaBench, a comprehensive multilingual and multimodal evaluation suite comprising five multimodal and three text-based tasks across 14 datasets in 47 languages. PangeaBench assesses MLLMs’ performance on open-domain multimodal chat, image captioning, cultural understanding, multimodal reasoning, and text-only tasks including question answering and complex math reasoning.
A key highlight of PangeaBench is the introduction of xChat, a human-crafted benchmark designed to evaluate open-ended, information-seeking multimodal conversations. xChat employs a fine-grained evaluation pipeline where human annotators annotate both reference answers and scoring rubrics for each query. An LLM then uses these rubrics to score the model’s predictions on a 1-5 scale. This approach offers a more precise assessment of MLLM performance, addressing limitations of coarse LLM-as-Judge methods (Zheng et al., ). Additionally, we introduce xMMMU, a translated version of MMMU (Yue et al., ), testing college-level multimodal reasoning across seven languages. Together, these components provide a robust, nuanced evaluation of Pangea’s cross-lingual and cross-cultural capabilities.

Our results demonstrate Pangea abilities in both English and multilingual scenarios, significantly outperforming existing open-source MLLMs across PangeaBench datasets, surpassing the best open MLLMs by 7.3 points on English tasks and 10.8 points on multilingual tasks on average. Notably, Pangea excels in multilingual and multi-cultural understanding, as evidenced by its performance on xChat and CVQA benchmarks. Pangea also matches or outperforms state-of-the art proprietary LLMs, namely Gemini-1.5-Pro and GPT4o, on several tasks. However, some performance gaps remain in multimodal chat and complex reasoning, shedding light on the need for further improvements in open multimodal LLMs. We discuss key insights from Pangea’s training, including the scaling effect of instructions, the role of English data, and the impact of language-specific training proportions on performance. Additionally, we explore preliminary methods to improve multilingual OCR. We fully open-source PangeaIns, PangeaBench, Pangea models, and code, enabling advances in culturally inclusive MLLMs across diverse languages.

## 2 PangeaIns : Multilingual Multimodal Instruction Tuning

Creating a truly multilingual, multicultural MLLM presents unique challenges.
We developed PangeaIns, a diverse and high-quality dataset specifically designed for instruction tuning. Comprising 6 million samples in 39 languages, PangeaIns was curated with a focus on linguistic and cultural diversity.
We implemented three key strategies to ensure comprehensive coverage, each addressing the specific hurdles encountered in multilingual multimodal learning. Figure 2 shows the details of our PangeaIns’s distribution.

### 2.1 Machine Translated Instructions

The scarcity of human-annotated multilingual multimodal training datasets presents a significant challenge. To address this, we primarily adopt machine translation as a practical and scalable solution to extend our dataset beyond English. While human annotation is ideal, it is resource-intensive and impractical for covering a wide range of languages.

Constructing a High-quality Pool of English Instructions from Existing Sources.
We first collect a high-quality set of English multimodal instructions, which serve as the foundation for translation into other languages. These instructions span a wide range of visual understanding tasks, including general visual instructions and conversations (Tong et al., ; Liu et al., ), visual reasoning, captioning, and chart question answering (Masry et al., ). Besides, we also added text-only high-quality English instructions, covering general instructions (Teknium, ), code (Zheng et al., ), and math (Li et al., ). Figure 2 shows the statistics of our translated datasets. By leveraging existing English instructions, we ensured comprehensive coverage of visual interpretation and text instruction following tasks in English, preparing a pool of high-quality data for translation.

Translation Model Selection.
To expand the English instructions to other languages, we initially experimented with strong open-source machine translation models, such as NLLB-3B (NLLB Team, ). However, we found that these models struggled with complex instruction-following scenarios and context-switching tasks, particularly in specialized domains like code generation and mathematical reasoning. For example, in code-related tasks, the model failed to recognize and correctly translate programming language keywords, significantly reducing the quality of the instructions. Based on these limitations, we shifted to using the proprietary Gemini 1.5 Pro model, which shows slightly better performance in small-scale human evaluations compared with GPT4o.

Post-Processing Translated Data. Even with high-quality translations, inconsistencies arose. To resolve issues such as mismatched conversation turns or missing candidates in multiple-choice questions, we developed a post-processing pipeline. This pipeline automatically corrected these errors or directly dropped the examples, ensuring that all translated instructions remained consistent.

Overall, we found the Gemini 1.5 Pro translation to be satisfactory—providing a fast, cost-effective alternative to human annotation, especially for scaling across multiple languages. However, we acknowledge that machine translation still has some limitations, particularly in handling nuanced contexts and cultural subtleties.

Figure: Figure 3: Overview of multicultural understanding instructions data generation pipeline.
Refer to caption: x4.png

### 2.2 Multicultural Understanding Instructions

While machine translation enables us to scale across multiple languages, data translated from English is still Anglo-centric in its coverage of cultural concepts (Yu et al., ). To address this, we developed a pipeline focused on creating instructions for multicultural understanding.
Both visual and textual elements can convey deep cultural significance, and our goal is to design a dataset that allows models to not only recognize these nuances but also respond appropriately across various cultural contexts. The pipeline of creating multicultural understanding instructions is shown in Figure 3.

Curation of Culturally Diverse Images. To ensure that our dataset captures a wide array of cultural contexts, we began by sampling 10 million images from the LAION-Multi dataset (Schuhmann et al., ), which includes images and short alt texts from diverse languages and regions. A filtering process was proposed to guarantee both the quality and cultural relevance of the images.

- •
Heuristic Filtering: We implemented automatic filtering based on several key criteria: Image Size, Aspect Ratio, Text Length, NSFW content, Offensive Text, Deduplication, and CLIP Score (used to assess the alignment between the image and its textual description). This helped remove low-quality or inappropriate images and ensured the remaining dataset adhered to quality standards.
- •
LLM Scoring: To further refine the dataset, we employed the Llama-3.1-8B-Instruct model (Dubey et al., ) to evaluate the quality, subjects, and cultural relevance of the accompanying text descriptions (alt text) for each image. The model was instructed to perform the following tasks: 1) Evaluate Text Quality: The alt text was rated on a scale from 1 to 5 based on how well it described the corresponding image, assuming the model could not access the image itself. Alt text scoring below 4 was removed. 2) Subject Classification: The model assigned a subject or category to the alt text based on its content. 3) Country/Region Classification: The model determined whether the alt text was closely related to a specific country’s culture. Images classified as “no specific country” (approximately 60% of the dataset) were excluded to ensure we focused on culturally identifiable content. The full LLM scoring prompt is included in Appendix B.
- •
Avoiding Overrepresentation: To maintain a balanced representation, we downsampled images from frequently occurring subjects, such as objects, materials, and clothing, to avoid skewing the dataset toward specific topics or regions. Additionally, an accessibility check was conducted, which removed around 30% of the remaining samples due to image download or other issues. Ultimately, we curated a final set of 1 million high-quality, culturally specific images that form the foundation of our dataset.

Captioning of Multicultural Images with Different Languages. To provide context and enhance the models’ ability to interpret the images accurately, we regenerated a more detailed caption using Gemini 1.5 Pro based on high-quality original text. Each image was accompanied by a caption written in the language corresponding to its cultural origin. This step was crucial as writing captions in the native language of the image’s cultural context adds a layer of authenticity and helps the model learn language-specific nuances.

Generating Multilingual and Cross-Cultural Instructions. For each image, we generated captions in the corresponding native language using Gemini 1.5 Pro. However, our approach was not just about using a capable model. The filtered high-quality alt text played a critical role in enriching the data, as it often contained culturally specific and contextually important information that would otherwise be absent from the images alone. For example, in the Appendix Figure 9, with high-quality alt text, a model can incorporate details such as “President and CEO of The Walt Disney Company standing in front of a model of Shanghai Disneyland,” which adds significant context that may not be immediately evident from the image. This additional layer of information helps the model generate captions that better capture the cultural and contextual nuances.

After recaptioning, we generated multilingual instructions based on the detailed captions with Gemini 1.5 Pro. Instead of only prompting the model to generate random instructions, we did a careful prompt engineering where we first came up with 13 task types (e.g., Information Seeking, Coding & Debugging, Critical Reasoning, Cultural Interpretation, etc.). Then for each image, up to two QA pairs were created, representing different instruction types to ensure a diverse set of interactions. This approach ensures that the model not only recognizes these visual elements but also responds appropriately across varied linguistic and different instruction contexts. The captioning and instruction generation prompts are included in Appendix B.

### 2.3 Curating Existing Multilingual Instructions

To further enrich PangeaIns, we conducted an extensive survey of the available multilingual multimodal literature and datasets, including those hosted on HuggingFace. As a result, we incorporated several high-quality, open-source datasets into our PangeaIns mixture. These include Chinese ALLaVA-4V (Chen et al., ), Viet Document and OCR QA (Doan et al., ), Llava Chinese (LinkSoul-AI, ), Llava Medical Chinese Instruction (BUAA, ), LLaVA-Japanese-Instruct (Toshi456, ), MTVQA (Tang et al., ), Japanese STAIR Captions (Yoshikawa et al., ), Russian GQA (Belopolskih & Spirin, ), French Doc-VQA (Sonagu & Sola, ), and French Table-VQA (Agonnoude & Delestre, ). Each of these datasets brings unique linguistic and cultural perspectives to the mix, covering a wide range of languages and task types.

### 2.4 Dataset Statistics

By combining these three methods, we created PangeaIns as a comprehensive dataset that addresses the major challenges in building multilingual multimodal models: data scarcity, linguistic diversity, and cultural nuance. The dataset’s balanced distribution across languages and tasks supports the development of more sophisticated LLMs that can handle complex visual and textual content in a multilingual, multicultural context.

Language and Task Distribution: PangeaIns features an extensive and balanced distribution of languages, tasks, and cultural contexts (as shown in Figure 2). We empirically keep the final language ratio of English to Multilingual as 40%:60% as we found a significant portion of English data plays an important role in cross-lingual transfer. See more discussions about the ratio in section 5 and Figure 6. The inclusion of diverse multimodal instructions ensures that the model develops a deeper understanding of varied linguistic and cultural environments. Examples of training samples from different languages and categories are provided in Appendix G.

The comprehensive nature of PangeaIns lays a solid foundation for training Pangea, enabling it to become a truly multilingual, multicultural multimodal LLM, capable of understanding and interacting effectively with users from diverse linguistic and cultural backgrounds.

Figure: Figure 4: Overview of PangeaBench, which contains 5 multimodal and 3 text tasks covering 14 datasets (including two newly curated xChatBench and xMMMU datasets). The table provides details about the datasets, while the figure shows evaluation examples from five different multimodal eval tasks in our PangeaBench.
Refer to caption: x5.png

## 3 PangeaBench : Evaluation of Multilingual Multimodal Models

### 3.1 Overview of PangeaBench

To assess the capabilities of Pangea across a variety of languages, cultures, and task types, we have developed PangeaBench, a comprehensive multilingual and multimodal evaluation suite. PangeaBench integrates diverse benchmarks that encompass both multimodal and text-only tasks, enabling a holistic evaluation of Pangea’s performance in cross-lingual, cross-cultural, and multimodal contexts. Each task within PangeaBench is designed to probe specific aspects of Pangea’s proficiency, ensuring robust testing across a wide range of scenarios.

### 3.2 Multimodal Tasks

The multimodal tasks in PangeaBench are categorized as follows: Multimodal Chat, Captioning, Cultural Understanding, Multilingual Visual Question Answering (VQA), and Multi-Subject Reasoning. These tasks incorporate datasets from PangeaBench to ensure comprehensive testing of Pangea’s multimodal capabilities. The overview and examples of PangeaBench from each task are shown in Figure 4.

Multimodal Chat. This task tests the model’s ability to engage in natural, dynamic real-world conversations involving both text and images. In this category, Multilingual LlavaBench (Rasheed et al., ) (M-LlavaBench for short) stands as the only benchmark for evaluating multilingual long-form generation capabilities from MLLMs. Moreover, following the evaluation pipeline from Zheng et al. () and Liu et al. (), M-LlavaBench uses a coarse-grained evaluation criteria (i.e., “Please rate the helpfulness, relevance, accuracy, level of details of their responses.”). Prior works have shown that employing such coarse-grained evaluation criteria might lead to automatic evaluation results that diverge from how humans would evaluate them (Ye et al., ; Kim et al., ; Lee et al., ; Kim et al., ; ). To assess our baselines based on a more accurate evaluation pipeline with fine-grained evaluation criteria on diverse scenarios, we additionally annotate a new multilingual multimodal generation benchmark called the xChatBench. An additional explanation of the annotation process of xChatBench is included in Appendix E. For the multimodal chat scenarios, we found that many English-centric models tend to respond in English regardless of the query language. This behavior is problematic, as it undermines the fundamental capability of a multilingual model, which should ideally respond in the language of the query. To address this, we implemented a strict evaluation criterion where such responses were penalized and assigned a score of 0. We believe this is crucial, as users may not understand English, and failing to respond in the appropriate language can hinder effective communication and user experience.

Captioning. The XM3600 (Thapliyal et al., ) dataset was developed to evaluate models’ capability in multilingual image captioning. This dataset contains images paired with captions in 36 different languages. The original datasets often contain similar images and captions. To address this, we clustered the images based on their captions and manually selected 100 representative images from these clusters. This approach enhances the diversity of the dataset while also accelerating the evaluation process. We call the dataset XM100.

Cultural Understanding. To assess the model’s ability to reason about and understand culturally diverse visual content, we use the CVQA (Romero et al., ) and MaRVL (Liu et al., ) datasets. These datasets are designed to test the model’s performance in reasoning tasks involving culturally relevant imagery and concepts across multiple languages.

Multilingual VQA. This task measures the model’s proficiency in answering questions about images across multiple languages. The xGQA (Pfeiffer et al., ) and MaXM (Changpinyo et al., ) datasets provide a diverse range of visual question-answering challenges in several languages and scripts, addressing cross-lingual visual understanding.

Multi-Subject Reasoning. The xMMMU and M3Exam (Zhang et al., ) datasets are used to evaluate the model’s reasoning abilities across different academic subjects. xMMMU is a machine-translated version of MMMU validation questions, which focuses on multimodal reasoning in multiple subjects. We randomly sample 300 questions from MMMU (Yue et al., ) validation set and employ GPT-4o for the six languages translation. M3Exam challenges the model with complex, real-world educational questions requiring both textual and visual comprehension. Further details on how we ensure the quality of translations for xMMMU, as well as a detailed description of other datasets, can be found in Appendix D.

### 3.3 Text-Only Multilingual Datasets

While multimodal tasks are critical for evaluating the holistic capabilities of models like Pangea, text-only multilingual tasks provide an equally essential dimension to assess. Most existing multimodal evaluations tend to overlook the importance of text-only evaluation, especially across diverse languages. Including text-only tasks in PangeaBench allows us to examine whether the model can perform well in scenarios that require deep linguistic understanding without the aid of visual context, highlighting its performance as a foundation model. We include three tasks QA, Translation, and Reasoning covering five datasets for the text-only evaluations in PangeaBench.

Specifically, we include TydiQA (Clark et al., ) to test the model’s ability to answer questions across 11 typologically diverse languages. We adopt the FLORES (NLLB Team, ) dataset to assess machine translation performance across multiple languages. We sample 11 languages (denoted as FLORES-Sub). We use MMMLU (OpenAI, ), a human-translated version of MMLU to test the general language understanding of models. We use XStoryCloze (Lin et al., ) and MGSM (Shi et al., ) to test the model’s commonsense and mathematical reasoning ability in multilingual contexts respectively.

**Table 1: Overall performance on the multilingual multimodal benchmarks in PangeaBench. The best-performing open model on each dataset is in bold and the second best is \ulunderlined.**
| Models | AVG (all) | Multimodal Chat | Cultural Understanding |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| xChatBench | M-LlavaBench | CVQA | MaRVL |  |  |  |  |  |  |  |
| en | mul | en | mul | en | mul | en | mul | en | mul |  |
| Gemini-1.5-Pro | 67.1 | 62.5 | 67.0 | 54.4 | 103.4 | 106.6 | 75.9 | 75.7 | 76.4 | 72.0 |
| GPT4o | 68.6 | 64.6 | 71.0 | 64.4 | 104.6 | 100.4 | 79.1 | 79.4 | 81.4 | 82.1 |
| Llava-1.5-7B | 45.4 | 28.4 | 28.5 | 11.8 | 66.1 | 40.8 | 48.9 | 36.5 | 56.2 | 53.7 |
| Llava-Next-7B | 51.1 | 32.7 | 40.5 | 18.9 | 78.9 | 50.7 | 55.7 | 42.6 | 62.8 | 50.9 |
| Phi-3.5-Vision | 54.0 | 35.0 | 38.5 | 13.2 | 70.8 | 58.0 | 56.3 | 42.3 | 72.1 | 56.5 |
| Cambrian-8B | 50.9 | 36.4 | 27.5 | 11.3 | 78.4 | 61.8 | 59.7 | 47.5 | \ul75.4 | 61.8 |
| Llava-OV-7B | \ul59.5 | 41.3 | 51.0 | \ul28.5 | 89.7 | 55.3 | \ul65.2 | 53.7 | 72.7 | 57.5 |
| Molmo-7B-D | 55.4 | 34.1 | 49.5 | 21.1 | 95.9 | 13.8 | 59.4 | 48.3 | 65.3 | 54.9 |
| Llama3.2-11B | 57.2 | 41.9 | \ul49.0 | 27.8 | \ul93.9 | 58.2 | 70.2 | 61.4 | 64.5 | 58.1 |
| PaliGemma-3B | 37.3 | 25.8 | 6.0 | 3.5 | 32.1 | 31.9 | 52.9 | 42.9 | 56.5 | 52.2 |
| PALO-7B | 46.3 | 32.2 | 27.0 | 11.8 | 68.9 | 71.2 | 50.9 | 39.2 | 63.3 | 54.2 |
| mBLIP mT0-XL | 35.1 | 29.8 | 2.5 | 0.5 | 32.7 | 28.2 | 40.5 | 37.5 | 67.3 | \ul66.7 |
| mBLIP BLOOMZ | 36.1 | 30.0 | 4.0 | 1.6 | 43.5 | 41.0 | 44.9 | 36.9 | 62.3 | 58.6 |
| Pangea-7B (Ours) | 59.9 | 52.7 | 46.0 | 35.6 | 84.2 | 89.5 | 64.4 | \ul57.2 | 87.0 | 79.0 |
| $\Delta$ over SoTA Open | +0.4 | +10.8 | -3.5 | +7.1 | -11.7 | +18.3 | -5.8 | -4.2 | +11.6 | +12.3 |
| Models | Captioning | Short VQA | Multi-subject Reasoning |  |  |  |  |  |  |  |
| XM100 | xGQA | MaXM | xMMMU | M3Exam |  |  |  |  |  |  |
| en | mul | en | mul | en | mul | en | mul | en | mul |  |
| Gemini-1.5-Pro | 27.6 | 19.1 | 54.2 | 48.7 | 56.4 | 63.5 | 65.8 | 57.7 | 77.4 | 64.7 |
| GPT4o | 27.7 | 19.1 | 55.8 | 51.0 | 60.7 | 65.4 | 69.1 | 58.3 | 68.0 | 61.0 |
| Llava-1.5-7B | 28.6 | 1.1 | 62.0 | 30.6 | 49.8 | 20.4 | 36.2 | 31.5 | 32.3 | 29 |
| Llava-Next-7B | 29.3 | 9.4 | 64.8 | 37.8 | \ul54.9 | 21.4 | 36.7 | 34.3 | 36.5 | 28.4 |
| Phi-3.5-Vision | 30.2 | 5.2 | \ul64.7 | 38.4 | 55.3 | 25.0 | 42.6 | 38.8 | 55.8 | 37.2 |
| Cambrian-8B | 20.6 | 9.9 | 64.6 | 39.8 | 55.3 | 28.7 | 41.8 | 33.2 | 34.7 | 33.4 |
| Llava-OV-7B | 30.6 | 7.0 | 64.4 | \ul48.2 | 54.9 | 34.8 | \ul46.3 | 41.0 | \ul60.4 | 45.8 |
| Molmo-7B-D | 22.1 | 9.1 | 51.5 | 43.0 | 52.9 | 37.5 | 44.5 | 40.4 | 57.1 | 39.1 |
| Llama3.2-11B | 27.6 | 4.5 | 55.6 | 45.4 | 55.3 | \ul43.9 | 46.5 | 41.4 | 51.8 | 36.6 |
| PaliGemma-3B | 18.7 | 0.8 | 59.7 | 30.5 | 47.9 | 19.9 | 26.3 | 25.2 | 36.0 | 25.6 |
| PALO-7B | \ul30.4 | 0.8 | 60.5 | 37.8 | 51.4 | 16.3 | 33.1 | 30.5 | 30.8 | 27.8 |
| mBLIP mT0-XL | 31.9 | 3.1 | 44.2 | 39.9 | 44.7 | 36.8 | 29.3 | 30.4 | 22.8 | 25 |
| mBLIP BLOOMZ | 22.5 | \ul10.3 | 43.3 | 36.9 | 44.7 | 24.8 | 29.2 | 30.8 | 30.3 | 29.5 |
| Pangea-7B (Ours) | \ul30.4 | 14.2 | \ul64.7 | 60.2 | 55.3 | 53.2 | 45.7 | 43.7 | 61.4 | \ul42.1 |
| $\Delta$ over Best Open Model | -0.2 | +3.9 | -0.1 | +12.0 | 0.0 | +9.3 | -0.8 | +2.3 | +1.0 | -3.7 |

## 4 Experiments

### 4.1 Experimental Setup

We train Pangea on PangeaIns, our multilingual multimodal dataset comprising 6 million samples across 39 languages. The model is based on LLaVA-Next architecture (Liu et al., ) with Qwen2-7B-Instruct (Yang et al., ) as the language model backbone. We employ a learning rate of 2e-5, a batch size of 512, coupled with a cosine decay schedule with 0.03 warmup steps. We train the model for 1 epoch.

For evaluation, we compare Pangea against several state-of-the-art open source baselines, including English-centric models Llava-1.5-7B (Liu et al., ), Llava-Next-7B (Liu et al., ), Phi-3.5-Vision (Abdin et al., ), Cambrian-8B (Tong et al., ), Llava-OV-7B (Li et al., ), Molmo-7B-D (Deitke et al., ) Llama3.2-11B (Dubey et al., ) and multilingual models PaliGemma-3B (Beyer et al., ), PALO-7B (Rasheed et al., ), mBLIP mT0-XL and mBLIP BLOOMZ (Geigle et al., ). We also consider two text-only LLMs baselines Vicuna-1.5-7B (Zheng et al., ) and Qwen2-7B-Instruct (Yang et al., ), which are the backbones of Llava-Next and our Pangea respectively. We integrate our multimodal tasks in PangeaBench into lmms-eval (Li et al., ),(^1^11[https://github.com/EvolvingLMMs-Lab/lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval)) a multimodal evaluation package that supports many English multimodal benchmarks. We use lm-evaluation-harness (Biderman et al., )(^2^22[https://github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)) to evaluate text-only tasks. We follow the original paper for their best models’ prompts in different tasks, and mostly reproduce their original numbers on datasets reported in the original papers.

### 4.2 Multilingual Multimodal Results

The results in Table 1 provide clear insights into the strengths and remaining challenges of Pangea-7B in multilingual and multimodal tasks. Key observations from the evaluation include:

Superior English and Multilingual Performance: Pangea-7B outperforms existing open-source models across both English and multilingual tasks. While concurrent multimodal models such as Molmo (Deitke et al., ) or Llama 3.2 show strong performance on English datasets, they struggle in multilingual evaluation settings.
Particularly in multilingual subsets like xChatBench, M-LlavaBench, and MaRVL, it has achieved substantial gains, highlighting its effectiveness in both cross-lingual and cross-cultural contexts.

Balanced Cross-Language Capabilities: Unlike many models that exhibit a significant drop in performance when moving from English to multilingual tasks, Pangea-7B is relatively consistent. For instance, in Multimodal Chat tasks, the performance gap between English and multilingual remains relatively small, indicating its ability to handle multiple languages effectively.

Challenges Compared to Proprietary Models: While Pangea-7B leads in open-source models, some gaps remain when compared to closed-source models like GPT4o. Additionally, though Pangea-7B narrows the gap between English and multilingual performance, there is still room for improvement in fully closing this divide across all tasks.

**Table 2: Overall performance on text-only multilingual benchmarks in PangeaBench.**
| Models | AVG (all) | FLORES-Sub | TyDiQA | XStoryCloze | MGSM | MMMLU |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| en | mul | x→en | en→x | en | mul | en | mul | en | mul | en | mul |  |
| Vicuna-1.5-7B | 52.1 | 38.7 | 55.6 | 42.4 | 59.7 | 52.7 | 78.1 | 57.4 | 17.6 | 6.4 | 49.5 | 34.7 |
| Qwen2-7B-Instruct | \ul66.6 | 54.5 | 61.8 | 46.0 | 72.2 | 71.2 | 80.3 | 61.9 | 48.8 | \ul40.4 | 70.1 | 53.1 |
| Llava-1.5-7B | 53.1 | 39.0 | 54.7 | 41.5 | 66.8 | 52.8 | 79.1 | 57.6 | 14.8 | 7.6 | 50.2 | 35.7 |
| Llava-Next-7B | 54.0 | 38.9 | 54.8 | 41.4 | 68.3 | 52.1 | 79.1 | 57.1 | 15.6 | 7.5 | 52.1 | 36.5 |
| Phi-3.5-Vision | 60.7 | 41.7 | 28.5 | 32.5 | 75.9 | 51.3 | 77.9 | 54.8 | 59.2 | 33.1 | 62.0 | 36.7 |
| PALO-7B | 52.0 | 37.5 | 52.9 | 40.4 | 69.4 | 50.8 | 77.4 | 57.2 | 13.6 | 5.8 | 46.7 | 33.4 |
| Pangea-7B (Ours) | 72.8 | \ul54.3 | \ul60.7 | \ul44.9 | \ul73.7 | \ul66.0 | \ul79.1 | \ul61.2 | 82.0 | 47.4 | \ul68.4 | \ul52.2 |

### 4.3 Multilingual Text-only Results

We further evaluate our model in text-only scenarios in Table 2. Interesting findings include:

Best Text Performance Among Multimodal LLMs: Pangea-7B demonstrates the strongest performance among all multimodal LLMs in the text-only tasks consistently outperforming baselines like Llava-Next-7B. This highlights that, despite being trained as a multimodal model, Pangea-7B maintains superior text understanding and reasoning capabilities compared to other MLLMs.

Maintained Performance from its Text Backbone. Pangea-7B generally maintains or sees slight drops in performance on most text-only benchmarks compared with its text backbone Qwen2-7B-Instruct. Notably, the model shows a significant improvement in MGSM. This improvement is directly attributable to the inclusion of math-related instructions in PangeaIns, which enhances the model’s capability to handle complex multilingual reasoning and mathematical tasks.

## 5 Discussion

Finally, we explore the implications of our findings and their potential impact on future developments in the field. We examine the scaling effects of instruction quantity, the persistent role of English data, the relationship between training sample proportions and performance, qualitative examples of model behavior, and challenges in multilingual OCR. Through this discussion, we aim to provide a comprehensive understanding of our model and chart a course for future advancements.

Scaling Effect of Number of Instructions. Understanding how the quantity of instructions affects model performance is crucial for optimizing training strategies and resource allocation. Figure 5 reveals a clear scaling effect related to the number of instructions used during training. Performance improvements were consistent as we increased the number of multilingual instructions in PangeaIns, for both English and multilingual performance. This demonstrates the necessity of scaling multilingual multimodal instruction tuning.

Figure: Figure 5: Scaling effect of training samples on English and multilingual scores across datasets.
Refer to caption: x6.png

Figure: Figure 6: Impact of English training data proportion on English vs. multilingual performance.
Refer to caption: x7.png

Role of English Data. In multilingual scenarios, English data plays a pivotal role in cross-lingual transfer. To investigate this, we sampled 500K examples from the translated data described in subsection 2.1, ensuring a consistent data distribution. We varied the ratio of English data while keeping the total number of training samples fixed at 500K. For the 17 multilingual languages in the translated subset, we evenly distributed the number of samples across languages. As shown in Figure 6
, English performance generally improves as the percentage of English data increases. More surprisingly, using no English data (full multilingual data) results in relatively lower multilingual performance. As we introduce more English data, multilingual performance improves, peaking at 38.7% with 40% English. However, performance drops sharply when English data reaches 100%. This suggests that English data aids cross-lingual transfer, however, over-reliance on it harms multilingual performance.

Figure: Figure 7: The relationship between training sample size (relative to English) and performance (relative to English) of different languages across four datasets.
Refer to caption: x8.png

How does the proportion of training samples in a language affect downstream performance? An interesting question to ask is whether the downstream task performance is correlated with the number of training samples.
Our analysis in Figure 7 revealed a nuanced relationship between training sample proportion and downstream performance. While there is a general positive correlation, the impact varies significantly across languages and tasks. For widely spoken languages with rich linguistic resources, we observed a near-linear relationship. However, for low-resource languages, even a small increase in proportion yielded disproportionately large performance gains. Interestingly, we also noted instances of positive transfer between typologically similar languages. These findings suggest that strategic allocation of training samples, considering both language prevalence and linguistic similarities, can optimize overall model performance.

Multimodal Chat in the Wild. One important application of MLLMs is to answer users’ queries in the wild. Here, we show the outputs of Pangea for the multimodal chat queries from our xChatBench. As shown in Appendix Figure 10, 11, 12, 13, 14, 15, Pangea successfully interprets the figures in different tasks and generates fluent and readable in certain languages. These qualitative examples further demonstrate the remarkable visual understanding ability of Pangea in multilingual contexts. On the other hand, we also identified a few bad cases shown in Figure 16, 17. Despite generating relevant responses to the queries, the model does not capture the key details of the images due to the lack of knowledge, which points out potential improvement directions in the future.

Figure: Figure 8: A preliminary exploration of multilingual OCR.
Refer to caption: x9.png

Preliminary Explorations of Multilingual OCR. Multilingual OCR emerged as a particularly challenging aspect of Pangea’s functionality. We made efforts to enhance its multilingual OCR capabilities. Specifically, we constructed a dataset of 500K multilingual OCR instructions spanning 10 languages, with 50K examples per language, sourced from web user interfaces. Webpages naturally serve as image-rich environments containing abundant text, and by capturing screenshots of websites from various countries in different languages, we were able to gather a substantial number of OCR images. Further details on the construction of the multilingual OCR training dataset are available in Appendix I.

We employed the same model architecture as Pangea but trained it exclusively on these OCR images, reserving a portion of the data as a test set. As shown in Figure 8
, the results indicate that improving multilingual OCR performance is feasible with an increase in training samples. However, the OCR accuracy for non-Latin scripts (e.g., Chinese, Japanese, and Korean) remains lower than for Latin-based languages. Looking ahead, we aim to further expand the multilingual OCR training dataset to include more languages and integrate this data into PangeaIns.

## 6 Conclusion

In this paper, we introduced Pangea, a novel multilingual multimodal large language model designed to bridge linguistic and cultural gaps in visual understanding tasks. By leveraging PangeaIns, our newly curated 6M multilingual multimodal instruction data samples, we demonstrated significant improvements in cross-lingual and cross-cultural understanding across 39 typologically diverse languages. Our comprehensive evaluation using PangeaBench revealed Pangea’s superior performance compared to existing open-source models, particularly in tasks requiring nuanced cultural interpretation. We also highlight ongoing challenges in areas such as low-resource language support and multilingual OCR. We fully open-source Pangea, PangeaIns, and PangeaBench to facilitate future research to build open and inclusive MLLMs.

## Acknowledgments

This work was supported in part by the Carnegie Bosch Institute Fellowship and a grant from DSTA Singapore. The training is supported by the CMU FLAME Center. The authors would like to thank CMU NeuLab colleagues for their constructive comments. The authors would like to thank Google Gemini credits for data construction and evaluation.

## Appendix A Related Work

#### Visual Instruction Tuning.

Visual instruction tuning is a key technique for enhancing multimodal large language models by aligning visual inputs with textual instructions to improve understanding and generation tasks Liu et al. (). Traditionally, these instructions are built using English-language data from visual question answering and other datasets Liu et al. (; ; ); Tong et al. (); Beyer et al. (). Researchers often supplement this with synthetic instruction tuning data, generating large volumes of instructional pairs to possibly cover multiple languages too Geigle et al. (). However, these instruction-tuning datasets have mostly been task-focused and lack conversational capabilities. Further, while translation gives lends to multilingual capabilities, the data remains to be culturally homogeneous. By curating multilingual and multicultural instruction tuning data across various task types, our model is designed to intuitively understand and engage with users from diverse demographics.

#### Multilingual Multimodal LLMs.

Multilingual MLLMs have evolved from dual-encoder-based models, only capable of understanding and reasoning Ni et al. (); Zeng et al. (); Jain et al. (), to encoder-decoder models capable of multilingual text generation as well Shan et al. (); Chen et al. (); Geigle et al. (). Despite their advancements, these models have remained focused on conventional tasks such as VQA and image captioning. Moreover, most efforts have centered around training with multilingual text, while little attention has been given to curating culturally diverse image datasets. Even for text, despite the focus on multilinguality, few attempts have been made to reflect cultural diversity in instructions and captions. As a result, these models tend to reflect a Western-centric bias. By selecting culturally diverse images from LAION and intentionally integrating this diversity into our instructions and captions, our model aims to serve a wide range of users in an inclusive and equitable manner.

## Appendix B Prompts used in the data construction

In this appendix, we will list the detailed prompts we used when constructing cultural understanding instruction tuning data described in subsection 2.2.

## Appendix C Recaptioning Example from LAION-Cultural

Figure: Figure 9: An example from LAION-Cultural illustrating why the filtered informative alt text helps generate a more informative caption. With the high-quality alt text, the model incorporates important details like “President and CEO of The Walt Disney Company standing in front of a model of Shanghai Disneyland” into the generated caption.
Refer to caption: x10.png

## Appendix D Datasets used in PangeaBench

To comprehensively assess the capabilities of Pangea across diverse languages, cultures, and task types, we developed PangeaBench. We list the details of each dataset included in the PangeaBench.

### D.1 Multimodal Datasets

- •
xGQA (Pfeiffer et al., ): A cross-lingual visual question-answering dataset featuring 9,666 questions in eight languages covering five scripts. The dataset includes 300 unique images from Visual Genome (Krishna et al., ). xGQA tests the model’s ability to understand and reason about visual content across multiple languages.
- •
MaXM (Changpinyo et al., ): A VQA dataset in seven languages and five scripts, with questions and answers in the same language. Images are culturally matched to the target language regions. MaXM specifically addresses the challenge of cultural diversity in multimodal understanding.
- •
MaRVL (Liu et al., ): A Multicultural Reasoning over Vision and Language dataset in five languages and three scripts, featuring 4,914 culturally diverse images matched to respective languages. MaRVL focuses on evaluating models’ ability to reason about culturally diverse visual concepts.
- •
XM100 (Thapliyal et al., ): We create a subset of 3600 instances (100 instances per language) from the original XM100 dataset, a large multilingual image captioning dataset comprising 36 languages, with 261,375 captions for 100 unique images per language, culturally matched to each language. XM100 evaluates a model’s ability to generate culturally appropriate captions across a wide range of languages. For sampling, we select 100 instances per language, ensuring that all languages share the same set of images for their respective 100 instances. To ensure diversity within our sample, we use Sentence-BERT (Reimers & Gurevych, ) to cluster the 3600 English instances from the original dataset into 100 groups, and then select one instance from each group. This method ensures that the sampled instances are as diverse as possible. We evaluate models on this new sample of 3600 instances, which allows for a more time-efficient evaluation while still accurately reflecting the multilingual capabilities of models in diverse contexts.
- •
M3Exam (Zhang et al., ): A novel benchmark sourced from real and official human exam questions, featuring 12,317 questions in 9 languages across three educational levels. Approximately 23% of the questions require image processing. M3Exam tests the model’s ability to handle complex, multi-step reasoning tasks in an educational context.
- •
xMMMU: MMMU contains multimodal questions from college-level materials across six disciplines and 30 subjects. The dataset features 183 subfields and 30 diverse image types, including charts, diagrams, and chemical structures. We sample 300 questions from the original MMMU validation set and translate them using GPT-4o into xx languages. To ensure the quality, we translated each sampled question multiple times and then back-translated it to English. We select the translation with the highest BLEU score. xMMMU evaluates the model’s capacity to understand and reason about specialized academic content across languages and modalities.

### D.2 Text-Only Multilingual Datasets

- •
TyDiQA (Clark et al., ): A question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. Questions are written by native speakers without seeing the answers, ensuring a realistic information-seeking task. TyDiQA is designed to test linguistic diversity and avoid translation artifacts.
- •
FLORES (NLLB Team, ): A machine translation benchmark for 200 languages, including many low-resource languages. It consists of 3,001 sentences from 842 web articles, divided into dev, devtest, and test splits. FLORES-200 includes translations from multiple pivot languages and provides script alternatives for some languages, making it a comprehensive test of translation capabilities.
- •
MMMLU (OpenAI, ): A human-translated version of MMLU (Hendrycks et al., ), covering 57 subjects across STEM, humanities, social sciences, and more. It ranges in difficulty from elementary to advanced professional levels, testing both world knowledge and problem-solving ability in a zero-shot and few-shot setting across multiple languages.
- •
MGSM (Shi et al., ): Multilingual Grade School Math Benchmark, featuring 250 grade-school math problems translated into 10 languages. Based on GSM8K, it requires multi-step reasoning and tests the model’s ability to solve complex mathematical word problems across languages.

This diverse set of datasets in PangeaBench allows for a comprehensive evaluation of Pangea’s capabilities across various languages, cultures, modalities, and task types, providing a holistic assessment of its performance in multilingual and multimodal contexts.

## Appendix E Explanation of xChatBench

#### Task Category

We first divide into 10 task categories, namely art_explanation, bar_chart_interpretation, defeasible_reasoning, figurative_speech_explanation, iq_test, ocr, graph_interpretation, image_humor_understanding, science_figure_explanation, unusual_images. The task categories are inspired by existing papers that do not use a free-form generation format (Lu et al., ; Yue et al., ; Han et al., ; Hessel et al., ; Kim et al., ).

#### Construction Procedure

To annotate the instances, we mainly follow the procedure of Kim et al. (). Two human annotators first hand-crafted the instances by searching through appropriate images for the task and then hand-crafting each component of the instance. As our motivation for fine-grained evaluation, each instance consists of not only an instruction, reference answer, but also a unique evaluation criteria tailored to each instance (e.g., Does the response effectively explain the humor in the image based on the juxtaposition of a character’s portrayal in different scenarios?) and a description for each score between 1 and 5 (e.g., score4_description: The response understands the juxtaposition and relates it to the humor involving machine learning models, but may miss some nuances or the related aspect of the humor). During the annotation process, we asked the annotators to not copy-and-paste results from LLM services like ChatGPT or directly from the web. Then, we hire four additional annotators to assess the quality of the instances. Each participant to asked to grade if each instance (1) fits into the devised task category, (2) if the quality of the reference answer is good enough, and (3) if the score rubric is suitable to assess the response. We iteratively ask the annotators who made the instances to revise them if the instance does not satisfy all three criteria. The resulting dataset consists of 50 instructions, reference answers, and evaluation criteria with a corresponding score rubric.

#### Translation Procedure

To assess the multilingual generation capabilities of MLLMs, we translate the hand-crafted 50 instances into 6 different languages, namely Chinese, Hindi, Indonesian, Japanese, Korean, and Spanish. We first use GPT-4o-2024-08-06 to translate the instruction and reference answer of each instance with a naive prompt, “Translate the following sentences into {target_language}. Sentences: {sentences}”. Then, the coauthors who are native speakers of each language reviewed the instances and made adjustments if the translated results were unnatural.

#### Evaluation Pipeline

Similar to prior works employing LLM-as-a-Judge, we use GPT-4o-2024-08-06 as the judge model and prompt it in a direct assessment manner. As input, the judge model is given the instruction, the model’s response, the reference answer, the evaluation criteria, and the descriptions for each score. As output, the judge generates verbal feedback and an integer score between 1 and 5. For this procedure, we use the prometheus-eval library (Kim et al., ) and employ their default hyperparameter setting for evaluation. Lastly, the final score is acquired by averaging the results across the 50 instances for each language. Note that in the main result and breakdown result tables, we normalize the score from 1-5 to 0-100 by $(\text{score}-1)\times 25$. Also, for postprocessing, we use langdetect (^3^33[https://pypi.org/project/langdetect/](https://pypi.org/project/langdetect/)) to identify whether the response is written in the given language and change the score to 1 when it is written in a different language, a phenomenon called language hallucination (Xue et al., ; Pfeiffer et al., ).

## Appendix F Qualitative Examples from xChatBench

We show some interesting qualitative examples from xChatBench, including good cases in Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, as well as bad cases in Figure 16, Figure 17. The examples included the scoring rubric, query, response from our Pangea, reference answer, and LLM-as-Judge feedback.

Figure: Figure 10: An xChat example from Japanese subset and our model Pangea’s response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by Pangea.
Refer to caption: x11.png

Figure: Figure 11: An xChat example from Hindi subset and our model Pangea’s response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by Pangea.
Refer to caption: x12.png

Figure: Figure 12: An xChat example from the Korean subset and our model Pangea’s response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by Pangea.
Refer to caption: x13.png

Figure: Figure 13: An xChat example from the Indonesian subset and our model Pangea’s response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by Pangea.
Refer to caption: x14.png

Figure: Figure 14: An xChat example from Spanish subset and our model Pangea’s response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by Pangea.
Refer to caption: x15.png

Figure: Figure 15: An xChat example from the Chinese subset and our model Pangea’s response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by Pangea.
Refer to caption: x16.png

Figure: Figure 16: An xChat example from the Chinese subset and our model Pangea’s response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by Pangea.
Refer to caption: x17.png

Figure: Figure 17: An xChat example from the Korean subset and our model Pangea’s response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by Pangea.
Refer to caption: x18.png

## Appendix G Training Examples

### G.1 Machine Translated Instructions

We include a few machine-translated training examples from PangeaIns in Figure 18, Figure 19, Figure 20,
Figure 21,
Figure 22, Figure 23.

### G.2 Multicultural Understanding Instructions

We include a few multicultural understanding instructions from PangeaIns in Figure 24, Figure 25, Figure 26, Figure 27.

Figure: Figure 18: Translated Task: An example from the Cambrian dataset where it discusses the concept of hourly wages based on a given prompt in Russian.
Refer to caption: x19.png

Figure: Figure 19: Multimodal Translated Task: An example from the ALLaVa-LAION dataset where the GPT model answers a prompt in Turkish regarding the maximum time displayed on a digital timer.
Refer to caption: x20.png

Figure: Figure 20: Multimodal Translated Task: An example from the ShareGPT-4v dataset where the model describes an image of a wine rack in Thai, detailing its structure and the arrangement of wine bottles.
Refer to caption: x21.png

Figure: Figure 21: Text-only Translated Task: An example from the OpenHermes2.5 dataset translated into Japanese.
Refer to caption: x22.png

Figure: Figure 22: Text-only Translated Task: An example from the NumininaMath dataset translated into Spanish.
Refer to caption: x23.png

Figure: Figure 23: Text-only Translated Task: An example from the Code-Feedback dataset translated into German.
Refer to caption: x24.png

Figure: Figure 24: Multicultural Understanding: An example from the LAION-Culture dataset where the GPT model describes and analyzes the Petrobras logo in Portuguese.
Refer to caption: x25.png

Figure: Figure 25: Multicultural Understanding: Example from the LAION-Culture dataset where the model interprets the meaning of a Japanese character and its representation in calligraphy, described in Bulgarian.
Refer to caption: x26.png

Figure: Figure 26: Multicultural Understanding: LAION-Culture sample in Hebrew.
Refer to caption: x27.png

Figure: Figure 27: Multicultural Understanding: LAION-Culture sample in Tamil.
Refer to caption: x28.png

## Appendix H Breakdown Results of Different Languages on PangeaBench

### H.1 xChat

We show the performance of different models’ performance on the xChat benchmark in Table 3.

**Table 3: Comparison of models on the xChat dataset across different languages.**
| Models | English | Multi | Spanish | Hindi | Indonesian | Japanese | Korean | Chinese |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-1.5-Pro | 71.0 | 65.6 | 66.0 | 62.0 | 65.5 | 68.0 | 66.5 | 65.5 |
| GPT4o | 67.0 | 65.1 | 66.0 | 64.0 | 65.0 | 66.5 | 67.5 | 61.5 |
| Llava-1.5-7B | 22.5 | 16.7 | 22.5 | 3.5 | 18.0 | 23.0 | 12.0 | 21.0 |
| Llava-Next-7B | 40.5 | 20.4 | 33.0 | 1.5 | 19.0 | 25.0 | 15.0 | 29.0 |
| Phi-3.5-Vision | 38.5 | 21.1 | 37.0 | 11.5 | 10.5 | 31.0 | 12.5 | 24.0 |
| Cambrian-8B | 27.5 | 15.8 | 22.5 | 4.0 | 20.0 | 20.0 | 10.5 | 18.0 |
| Llava-OV-7B | 51.0 | 33.1 | 45.5 | 6.5 | 42.0 | 36.5 | 26.0 | 42.0 |
| Molmo-7B-D | 49.5 | 34.7 | 45.0 | 19.5 | 36.5 | 36.0 | 35.0 | 46.0 |
| Llama3.2-11B | 49.0 | 31.3 | 42.5 | 19.5 | 45.0 | 26.0 | 21.0 | 43.0 |
| PaliGemma-3B | 6.0 | 3.8 | 4.5 | 0.5 | 6.5 | 6.5 | 2.0 | 3.0 |
| PALO-7B | 27.0 | 16.2 | 23.0 | 3.0 | 19.0 | 20.0 | 13.5 | 18.5 |
| mBLIP mT0-XL | 2.5 | 0.5 | 0.0 | 0.0 | 0.5 | 2.0 | 0.5 | 0.0 |
| mBLIP BLOOMZ-7B | 4.0 | 1.7 | 2.0 | 2.5 | 2.5 | 0.0 | 0.0 | 3.0 |
| Pangea-7B (Ours) | 46.0 | 35.8 | 43.5 | 23.5 | 34.5 | 39.0 | 33.5 | 40.5 |

### H.2 Multilingual LLaVABench

We show the performance of different models’ performance on the Multilingual LLaVABench benchmark in Table 4.

**Table 4: Comparison of models on the Multilingual LLaVABench benchmark across different languages.**
| Models | English | Multi | Arabic | Bengali | Chinese | French | Hindi | Japanese | Russian | Spanish | Urdu |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-1.5-Pro | 103.4 | 106.6 | 112.9 | 117.1 | 104.1 | 115.5 | 106.2 | 118.1 | 95.7 | 88.2 | 101.6 |
| GPT4o | 104.6 | 100.4 | 98.3 | 111.9 | 96.5 | 101.1 | 99.7 | 104.0 | 88.5 | 100.9 | 102.5 |
| Llava-1.5-7B | 66.1 | 40.8 | 26.4 | 11.9 | 50.7 | 63.8 | 23.2 | 70.0 | 46.5 | 59.2 | 15.4 |
| Llava-Next-7B | 78.9 | 50.7 | 24.9 | 11.2 | 72.8 | 91.4 | 18.0 | 70.1 | 71.8 | 82.9 | 13.4 |
| Phi-3.5-Vision | 70.8 | 58.0 | 50.1 | 35.1 | 69.2 | 86.0 | 35.9 | 63.0 | 67.6 | 75.6 | 39.3 |
| Cambrian-8B | 78.4 | 61.8 | 54.1 | 35.4 | 80.9 | 87.3 | 44.2 | 64.4 | 76.4 | 90.3 | 23.3 |
| Llava-OV-7B | 89.7 | 55.3 | 45.5 | 33.8 | 90.0 | 89.4 | 35.3 | 70.3 | 44.7 | 75.5 | 13.3 |
| Molmo-7B-D | 95.9 | 13.8 | 10.1 | 4.2 | 0.3 | 59.6 | 5.5 | 6.0 | 8.7 | 29.5 | 0.0 |
| Llama3.2-11B | 93.9 | 58.2 | 39.4 | 48.1 | 47.2 | 85.6 | 67.8 | 53.7 | 68.5 | 77.8 | 35.3 |
| PaliGemma-3B | 32.1 | 31.9 | 37.3 | 38.2 | 29.1 | 30.0 | 35.8 | 33.4 | 26.1 | 32.3 | 25.1 |
| PALO-7B | 68.9 | 71.2 | 79.1 | 54.6 | 71.5 | 83.9 | 61.9 | 66.6 | 80.9 | 74.4 | 68.2 |
| mBLIP mT0-XL | 32.7 | 28.2 | 33.7 | 26.2 | 3.6 | 39.8 | 26.9 | 26.8 | 34.1 | 36.9 | 26.0 |
| mBLIP BLOOMZ-7B | 43.5 | 41.0 | 48.1 | 44.1 | 30.6 | 53.3 | 39.1 | 29.8 | 38.1 | 51.5 | 34.0 |
| Pangea-7B (Ours) | 84.2 | 89.5 | 91.0 | 94.9 | 94.4 | 93.8 | 84.9 | 92.8 | 91.2 | 87.4 | 75.5 |

### H.3 CVQA

We show the performance of different models’ performance on the CVQA benchmark in Table 5 and Table 6.

**Table 5: Comparison of models on CVQA across different country-language pairs (in local languages). Includes Macro-Acc.**
| Models | ar-es | br-pt | bu-bg | ch-es | ch-zh | co-es | ec-es | eg-ar | et-am | et-or |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llava-1.5-7B | 37.8 | 51.1 | 35.6 | 42.4 | 44.4 | 50.6 | 48.6 | 31.5 | 27.8 | 31.8 |
| Llava-Next-7B | 52.5 | 62.3 | 41.5 | 59.0 | 51.1 | 54.8 | 50.8 | 33.5 | 29.5 | 36.9 |
| Phi-3.5-Vision | 54.0 | 57.2 | 36.9 | 57.7 | 51.1 | 52.3 | 50.1 | 38.4 | 27.8 | 32.2 |
| Cambrian-8B | 59.6 | 60.6 | 42.0 | 64.5 | 59.5 | 57.7 | 56.1 | 40.9 | 27.8 | 25.7 |
| Llava-OV-7B | 64.5 | 69.7 | 49.6 | 67.1 | 69.1 | 66.8 | 65.5 | 47.8 | 32.5 | 41.1 |
| Molmo-7B-D | 61.1 | 69.0 | 54.9 | 60.7 | 66.2 | 58.5 | 54.9 | 56.7 | 58.1 | 60.7 |
| Llama3.2-11B | 69.1 | 74.6 | 64.2 | 70.5 | 73.6 | 69.3 | 66.9 | 68.5 | 68.4 | 63.1 |
| PaliGemma-3B | 48.7 | 53.9 | 39.1 | 53.4 | 53.7 | 50.6 | 45.3 | 40.4 | 24.8 | 28.0 |
| PALO-7B | 50.9 | 56.7 | 36.7 | 55.1 | 45.3 | 48.5 | 46.4 | 28.6 | 19.2 | 32.7 |
| mBLIP BLOOMZ-7B | 45.3 | 51.4 | 30.5 | 45.3 | 51.1 | 46.9 | 44.8 | 35.9 | 23.9 | 25.7 |
| mBLIP mT0-XL | 40.8 | 44.4 | 38.0 | 44.9 | 39.9 | 41.9 | 42.5 | 31.0 | 35.9 | 26.6 |
| Pangea-7B (Ours) | 68.3 | 72.9 | 53.9 | 70.5 | 74.0 | 64.7 | 63.5 | 49.3 | 36.3 | 35.5 |
| Models | fr-br | in-bn | in-ta | in-te | ind-id | ind-jv | ind-mi | ind-sv | ir-ir | ja-jp |
| Llava-1.5-7B | 29.4 | 31.1 | 29.8 | 28.0 | 41.7 | 32.0 | 32.7 | 33.5 | 42.6 | 37.4 |
| Llava-Next-7B | 27.4 | 31.1 | 28.8 | 28.0 | 42.2 | 38.7 | 40.2 | 35.5 | 42.6 | 32.5 |
| Phi-3.5-Vision | 29.3 | 39.0 | 40.0 | 36.8 | 45.0 | 38.2 | 38.2 | 30.8 | 39.6 | 39.7 |
| Cambrian-8B | 31.6 | 47.2 | 38.1 | 44.0 | 50.2 | 43.8 | 39.4 | 45.5 | 47.9 | 40.9 |
| Llava-OV-7B | 34.3 | 56.3 | 43.9 | 46.5 | 58.0 | 45.8 | 45.4 | 40.5 | 50.6 | 49.8 |
| Molmo-7B-D | 44.2 | 61.9 | 61.2 | 58.5 | 52.9 | 53.9 | 54.6 | 55.0 | 64.4 | 42.9 |
| Llama3.2-11B | 49.4 | 76.9 | 80.4 | 80.5 | 65.8 | 60.6 | 68.9 | 64.0 | 76.4 | 54.2 |
| PaliGemma-3B | 29.9 | 46.2 | 46.0 | 43.5 | 45.4 | 41.4 | 39.8 | 33.0 | 34.4 | 43.3 |
| PALO-7B | 29.1 | 37.8 | 31.2 | 25.0 | 41.3 | 32.3 | 32.3 | 32.0 | 42.9 | 30.5 |
| mBLIP BLOOMZ-7B | 26.7 | 41.9 | 40.0 | 42.0 | 41.9 | 35.4 | 35.1 | 32.0 | 29.4 | 31.0 |
| mBLIP mT0-XL | 23.5 | 36.4 | 44.2 | 39.0 | 37.4 | 37.4 | 34.7 | 31.0 | 35.3 | 30.0 |
| Pangea-7B (Ours) | 34.6 | 59.1 | 51.9 | 54.5 | 62.1 | 49.5 | 47.8 | 53.0 | 56.4 | 48.3 |
| Models | ke-sw | ma-my | me-es | mo-mg | ni-ig | no-ng | pk-ur | ph-fi | ro-ro | ru-ru |
| Llava-1.5-7B | 34.4 | 42.2 | 42.4 | 26.9 | 34.5 | 47.5 | 26.4 | 43.8 | 47.0 | 51.0 |
| Llava-Next-7B | 46.2 | 45.7 | 51.4 | 33.3 | 35.0 | 56.9 | 36.6 | 46.8 | 52.3 | 53.5 |
| Phi-3.5-Vision | 46.0 | 45.1 | 46.3 | 31.9 | 33.3 | 50.0 | 35.2 | 41.4 | 47.4 | 50.5 |
| Cambrian-8B | 50.5 | 52.1 | 56.7 | 34.6 | 36.0 | 53.5 | 48.6 | 47.3 | 52.0 | 61.5 |
| Llava-OV-7B | 46.5 | 55.6 | 59.4 | 35.9 | 33.5 | 62.5 | 58.3 | 56.2 | 60.3 | 75.5 |
| Molmo-7B-D | 73.3 | 54.6 | 53.6 | 51.9 | 53.0 | 54.8 | 67.1 | 57.6 | 63.6 | 61.5 |
| Llama3.2-11B | 79.1 | 72.1 | 66.6 | 54.5 | 61.5 | 66.9 | 78.7 | 70.0 | 76.8 | 74.5 |
| PaliGemma-3B | 44.0 | 44.1 | 47.4 | 29.2 | 32.0 | 52.2 | 44.9 | 39.9 | 50.3 | 53.5 |
| PALO-7B | 35.9 | 42.5 | 44.3 | 28.8 | 29.5 | 49.2 | 44.4 | 39.4 | 46.0 | 47.0 |
| mBLIP BLOOMZ-7B | 37.0 | 42.5 | 44.8 | 28.8 | 33.0 | 49.2 | 47.7 | 31.5 | 46.0 | 34.0 |
| mBLIP mT0-XL | 45.1 | 40.6 | 44.9 | 29.2 | 30.5 | 42.8 | 40.3 | 32.0 | 43.7 | 42.0 |
| Pangea-7B (Ours) | 64.1 | 59.7 | 62.2 | 42.3 | 46.0 | 64.5 | 66.2 | 58.6 | 64.6 | 74.0 |
| Models | rw-ki | sg-zh | sk-ko | sp-es | sr-si | ur-es | macro |  |  |  |
| Llava-1.5-7B | 31.1 | 44.3 | 44.5 | 56.9 | 24.9 | 37.8 | 38.7 |  |  |  |
| Llava-Next-7B | 34.5 | 44.8 | 43.4 | 63.5 | 29.8 | 41.0 | 42.6 |  |  |  |
| Phi-3.5-Vision | 31.1 | 43.9 | 55.2 | 62.4 | 28.0 | 43.3 | 42.4 |  |  |  |
| Cambrian-8B | 31.9 | 54.7 | 54.5 | 70.4 | 36.4 | 45.7 | 47.5 |  |  |  |
| Llava-OV-7B | 35.3 | 70.3 | 65.2 | 79.9 | 31.6 | 47.3 | 53.8 |  |  |  |
| Molmo-7B-D | 57.4 | 69.3 | 65.2 | 70.1 | 68.0 | 50.8 | 59.4 |  |  |  |
| Llama3.2-11B | 57.9 | 80.7 | 73.8 | 81.4 | 72.4 | 52.4 | 70.1 |  |  |  |
| PaliGemma-3B | 27.2 | 48.6 | 61.0 | 60.1 | 31.6 | 39.4 | 43.0 |  |  |  |
| PALO-7B | 28.9 | 45.8 | 44.5 | 64.8 | 28.0 | 39.4 | 39.3 |  |  |  |
| mBLIP BLOOMZ-7B | 29.4 | 47.6 | 33.1 | 56.6 | 28.0 | 39.4 | 36.9 |  |  |  |
| mBLIP mT0-XL | 33.2 | 36.8 | 38.3 | 53.5 | 31.1 | 39.1 | 37.6 |  |  |  |
| Pangea-7B (Ours) | 35.7 | 65.6 | 70.7 | 72.6 | 39.1 | 49.8 | 57.2 |  |  |  |

**Table 6: Comparison of models on CVQA across different country-language pairs (in English). Includes Macro-Acc.**
| Models | ar-es | br-pt | bu-bg | ch-es | ch-zh | co-es | ec-es | eg-ar | et-am | et-or |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llava-1.5-7B | 56.2 | 61.6 | 52.3 | 60.2 | 54.0 | 55.6 | 55.5 | 50.2 | 51.3 | 53.3 |
| Llava-Next-7B | 53.9 | 61.3 | 50.9 | 59.8 | 58.8 | 60.2 | 52.8 | 54.7 | 52.9 | 58.9 |
| Phi-3.5-Vision | 59.2 | 61.9 | 54.9 | 64.1 | 58.2 | 59.3 | 57.5 | 50.7 | 54.7 | 58.4 |
| Cambrian-8B | 57.7 | 66.5 | 56.1 | 65.4 | 64.3 | 59.3 | 60.2 | 56.7 | 60.3 | 56.5 |
| Llava-OV-7B | 63.0 | 73.9 | 59.3 | 65.8 | 68.8 | 65.1 | 63.3 | 62.1 | 59.8 | 59.3 |
| Molmo-7B-D | 57.7 | 65.8 | 45.6 | 63.7 | 68.5 | 57.3 | 55.0 | 43.8 | 31.6 | 38.8 |
| Llama3.2-11B | 66.8 | 72.9 | 54.4 | 72.6 | 72.0 | 66.4 | 65.2 | 56.7 | 41.9 | 32.2 |
| PaliGemma-3B | 51.7 | 59.5 | 49.3 | 51.7 | 54.9 | 54.8 | 47.2 | 51.2 | 52.6 | 51.4 |
| PALO-7B | 50.2 | 57.0 | 48.8 | 53.4 | 52.1 | 51.9 | 53.0 | 48.3 | 47.0 | 52.3 |
| mBLIP mT0-XL | 38.1 | 45.4 | 39.1 | 42.7 | 43.7 | 41.1 | 40.9 | 42.9 | 34.2 | 42.1 |
| mBLIP BLOOMZ-7B | 46.0 | 51.4 | 41.5 | 44.4 | 48.9 | 49.0 | 45.0 | 45.3 | 38.9 | 46.3 |
| Pangea-7B (Ours) | 67.2 | 72.9 | 60.1 | 68.8 | 67.2 | 64.7 | 61.6 | 59.1 | 60.7 | 56.0 |
| Models | fr-br | in-bn | in-ta | in-te | ind-id | ind-jv | ind-mi | ind-sv | ir-ir | ja-jp |
| Llava-1.5-7B | 37.3 | 52.1 | 61.4 | 63.5 | 47.8 | 50.8 | 49.0 | 44.0 | 61.3 | 41.9 |
| Llava-Next-7B | 37.5 | 60.8 | 61.4 | 60.5 | 48.5 | 48.1 | 51.4 | 49.0 | 66.6 | 40.9 |
| Phi-3.5-Vision | 41.7 | 58.7 | 60.5 | 60.0 | 51.7 | 45.5 | 51.4 | 47.5 | 62.6 | 41.4 |
| Cambrian-8B | 40.7 | 68.5 | 65.6 | 63.0 | 55.1 | 50.2 | 58.2 | 56.0 | 66.6 | 42.4 |
| Llava-OV-7B | 44.2 | 69.6 | 72.0 | 70.5 | 59.0 | 55.9 | 59.4 | 58.5 | 76.4 | 47.3 |
| Molmo-7B-D | 29.6 | 47.9 | 36.4 | 41.5 | 50.5 | 45.1 | 43.4 | 39.5 | 43.6 | 44.8 |
| Llama3.2-11B | 36.3 | 62.9 | 66.4 | 66.5 | 63.6 | 48.8 | 58.2 | 54.0 | 57.4 | 58.1 |
| PaliGemma-3B | 37.3 | 59.1 | 66.0 | 62.5 | 49.3 | 48.1 | 43.4 | 46.0 | 58.3 | 44.8 |
| PALO-7B | 36.8 | 52.4 | 53.5 | 56.5 | 45.1 | 45.8 | 44.2 | 42.0 | 55.6 | 37.4 |
| mBLIP mT0-XL | 30.4 | 43.0 | 46.0 | 41.0 | 38.1 | 39.1 | 38.6 | 32.5 | 37.4 | 34.0 |
| mBLIP BLOOMZ-7B | 34.6 | 43.4 | 52.6 | 49.5 | 41.0 | 44.8 | 38.2 | 30.5 | 42.3 | 36.5 |
| Pangea-7B (Ours) | 45.2 | 67.1 | 71.0 | 68.0 | 60.4 | 57.2 | 56.9 | 56.0 | 72.7 | 45.8 |
| Models | ke-sw | ma-my | me-es | mo-mg | ni-ig | no-ng | pk-ur | ph-fi | ro-ro | ru-ru |
| Llava-1.5-7B | 68.9 | 52.1 | 47.9 | 45.8 | 51.0 | 58.5 | 63.9 | 52.7 | 55.6 | 59.0 |
| Llava-Next-7B | 71.1 | 54.9 | 51.1 | 44.2 | 53.0 | 57.2 | 67.1 | 56.7 | 62.6 | 58.5 |
| Phi-3.5-Vision | 72.9 | 57.1 | 46.3 | 50.7 | 53.0 | 56.2 | 60.6 | 57.6 | 61.9 | 58.5 |
| Cambrian-8B | 74.4 | 61.9 | 56.7 | 48.7 | 56.5 | 60.5 | 73.1 | 60.1 | 66.6 | 61.5 |
| Llava-OV-7B | 79.1 | 65.1 | 63.2 | 52.6 | 57.5 | 64.2 | 75.0 | 64.0 | 72.5 | 72.5 |
| Molmo-7B-D | 47.6 | 51.7 | 55.1 | 35.9 | 36.0 | 49.2 | 46.8 | 43.3 | 52.0 | 63.5 |
| Llama3.2-11B | 61.5 | 69.2 | 64.7 | 41.0 | 39.5 | 65.9 | 65.7 | 66.0 | 75.5 | 74.5 |
| PaliGemma-3B | 59.7 | 54.9 | 51.7 | 43.4 | 46.0 | 55.2 | 67.6 | 48.8 | 60.9 | 56.0 |
| PALO-7B | 65.9 | 49.2 | 53.4 | 42.9 | 49.0 | 54.5 | 60.6 | 52.7 | 55.0 | 53.5 |
| mBLIP mT0-XL | 50.2 | 41.6 | 34.7 | 33.9 | 39.5 | 43.1 | 45.4 | 36.9 | 43.7 | 41.0 |
| mBLIP BLOOMZ-7B | 54.6 | 45.7 | 39.3 | 38.1 | 45.0 | 47.2 | 60.6 | 36.9 | 50.3 | 44.0 |
| Pangea-7B (Ours) | 77.2 | 62.5 | 61.6 | 52.9 | 59.5 | 64.9 | 72.2 | 64.0 | 71.9 | 68.5 |
| Models | rw-ki | sg-zh | sk-ko | sp-es | sr-si | ur-es | macro |  |  |  |
| Llava-1.5-7B | 51.1 | 60.8 | 56.9 | 66.0 | 58.7 | 42.5 | 54.2 |  |  |  |
| Llava-Next-7B | 52.8 | 62.3 | 60.0 | 67.6 | 59.1 | 38.7 | 55.7 |  |  |  |
| Phi-3.5-Vision | 52.3 | 59.4 | 66.5 | 66.7 | 61.3 | 46.3 | 56.3 |  |  |  |
| Cambrian-8B | 56.2 | 66.0 | 63.1 | 71.7 | 63.1 | 47.0 | 59.7 |  |  |  |
| Llava-OV-7B | 55.7 | 73.6 | 67.9 | 80.2 | 72.9 | 48.9 | 65.2 |  |  |  |
| Molmo-7B-D | 34.9 | 66.0 | 56.9 | 66.7 | 31.6 | 44.8 | 48.3 |  |  |  |
| Llama3.2-11B | 40.4 | 73.6 | 73.1 | 83.3 | 51.1 | 56.2 | 61.2 |  |  |  |
| PaliGemma-3B | 44.7 | 59.4 | 58.3 | 61.0 | 62.2 | 40.6 | 52.9 |  |  |  |
| PALO-7B | 51.9 | 56.1 | 55.9 | 62.9 | 54.2 | 42.2 | 50.9 |  |  |  |
| mBLIP mT0-XL | 38.3 | 43.9 | 41.4 | 51.9 | 48.0 | 34.9 | 40.5 |  |  |  |
| mBLIP BLOOMZ-7B | 45.1 | 53.8 | 46.9 | 58.5 | 46.7 | 34.0 | 44.9 |  |  |  |
| Pangea-7B (Ours) | 56.6 | 71.7 | 66.6 | 75.2 | 70.6 | 52.7 | 64.4 |  |  |  |

### H.4 MaRVL

We show the performance of different models’ performance on the MaRVL benchmark in Table 7.

**Table 7: Comparison of models on the MaRVL dataset across different languages.**
| Models | English | Multi | Indonesian | Swahili | Tamil | Turkish | Chinese |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GPT4o | 81.8 | 82.3 | 81.9 | 80.8 | 80.2 | 86.4 | 82.1 |
| Gemini-1.5-Pro | 76.4 | 72.0 | 71.2 | 67.8 | 70.0 | 75.4 | 75.8 |
| Llava-1.5-7B | 56.2 | 53.7 | 56.1 | 49.8 | 49.7 | 55.4 | 57.5 |
| Llava-Next-7B | 62.8 | 50.9 | 52.2 | 50.6 | 50.5 | 50.4 | 50.6 |
| Phi-3.5-Vision | 72.1 | 56.5 | 58.6 | 51.4 | 52.0 | 58.6 | 61.7 |
| Cambrian-8B | 75.4 | 61.8 | 64.7 | 53.6 | 56.7 | 65.2 | 68.9 |
| Llava-OV-7B | 72.7 | 57.5 | 60.9 | 51.2 | 51.9 | 63.5 | 60.0 |
| Molmo-7B-D | 65.3 | 54.9 | 61.1 | 49.6 | 49.6 | 52.2 | 62.2 |
| Llama3.2-11B | 64.5 | 58.1 | 62.7 | 52.4 | 54.0 | 61.6 | 59.5 |
| PaliGemma-3b | 56.5 | 52.2 | 53.4 | 49.6 | 50.5 | 56.3 | 51.3 |
| PALO-7B | 63.3 | 54.2 | 58.3 | 50.6 | 51.9 | 54.9 | 55.3 |
| mBLIP mT0-XL | 67.3 | 66.7 | 64.9 | 64.8 | 69.7 | 68.1 | 65.9 |
| mBLIP BLOOMZ-7B | 62.3 | 58.6 | 59.1 | 56.2 | 60.3 | 57.7 | 59.7 |
| Pangea-7B | 87.0 | 79.0 | 81.3 | 75.1 | 69.4 | 84.8 | 84.3 |

### H.5 XM100

We show the performance of different models’ performance on the XM100 benchmark in Table 8.

**Table 8: Comparison of models on the XM100 dataset across different languages.**
| Models | English | Multi | Arabic | Bengali | Czech | Danish | German | Greek |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-1.5-Pro | 27.6 | 19.1 | 1.7 | 7.5 | 25.9 | 32.8 | 27.6 | 5.0 |
| GPT4o | 27.7 | 19.1 | 15.8 | 13.5 | 21.1 | 25.3 | 19.3 | 21.1 |
| Llava-1.5-7B | 28.6 | 1.1 | 0.0 | 0.0 | 2.1 | 1.0 | 3.1 | 0.0 |
| Llava-Next-7B | 29.3 | 9.4 | 5.6 | 0.1 | 12.1 | 15.7 | 14.4 | 4.2 |
| Phi-3.5-Vision | 30.2 | 5.2 | 0.4 | 2.4 | 16.6 | 16.2 | 0.0 | 20.7 |
| Cambrian-8B | 20.6 | 9.9 | 1.4 | 6.6 | 7.4 | 15.1 | 15.5 | 4.4 |
| Llava-OV-7B | 30.6 | 7.0 | 0.2 | 0.6 | 5.2 | 16.8 | 14.0 | 0.4 |
| Molmo-7B-D | 22.1 | 9.1 | 5.4 | 7.9 | 5.7 | 13.8 | 12.2 | 4.2 |
| Llama3.2-11B | 27.6 | 4.5 | 0.0 | 0.0 | 1.5 | 11.8 | 4.6 | 1.2 |
| PaliGemma-3B | 18.7 | 0.8 | 0.0 | 0.0 | 1.1 | 3.1 | 2.7 | 0.0 |
| PALO-7B | 30.4 | 0.8 | 0.0 | 0.0 | 2.0 | 1.0 | 2.7 | 0.0 |
| mBLIP mT0-XL | 31.9 | 3.1 | 3.2 | 1.6 | 3.7 | 2.1 | 2.9 | 3.1 |
| mBLIP BLOOMZ | 22.5 | 10.3 | 9.5 | 6.4 | 11.5 | 15.9 | 14.5 | 10.9 |
| Pangea-7B (Ours) | 30.8 | 14.2 | 18.1 | 16.4 | 16.2 | 20.7 | 20.6 | 11.2 |
| Models | Spanish | Persian | Finnish | Filipino | French | Hebrew | Hindi | Croatian |
| Gemini-1.5-Pro | 39.5 | 4.2 | 29.0 | 28.7 | 42.4 | 4.3 | 2.2 | 33.8 |
| GPT4o | 28.3 | 26.6 | 13.1 | 26.4 | 23.1 | 20.4 | 17.0 | 19.4 |
| Llava-1.5-7B | 3.7 | 0.0 | 0.4 | 1.1 | 2.0 | 0.1 | 0.0 | 0.3 |
| Llava-Next-7B | 23.6 | 9.4 | 5.5 | 9.3 | 23.0 | 2.7 | 10.2 | 7.5 |
| Phi-3.5-Vision | 20.7 | 0.0 | 1.0 | 1.7 | 21.2 | 0.3 | 0.0 | 0.5 |
| Cambrian-8B | 18.6 | 9.6 | 5.1 | 19.6 | 18.3 | 5.8 | 6.8 | 7.2 |
| Llava-OV-7B | 24.9 | 3.8 | 1.5 | 4.2 | 22.0 | 0.0 | 4.4 | 7.2 |
| Molmo-7B-D | 19.8 | 11.3 | 3.1 | 13.0 | 19.8 | 8.3 | 9.4 | 6.9 |
| Llama3.2-11B | 10.2 | 0.0 | 2.4 | 8.4 | 12.0 | 0.0 | 0.2 | 0.7 |
| PaliGemma-3B | 0.7 | 0.0 | 0.1 | 0.1 | 0.6 | 0.0 | 0.0 | 1.3 |
| PALO-7B | 1.5 | 0.0 | 0.4 | 0.9 | 2.1 | 0.0 | 0.0 | 0.2 |
| mBLIP mT0-XL | 8.3 | 5.5 | 1.7 | 2.8 | 6.4 | 4.0 | 1.8 | 0.9 |
| mBLIP BLOOMZ | 18.9 | 13.8 | 4.8 | 7.7 | 19.1 | 7.5 | 10.1 | 3.2 |
| Pangea-7B (Ours) | 26.2 | 19.3 | 3.8 | 18.9 | 26.7 | 18.2 | 17.4 | 10.8 |
| Models | Hungarian | Indonesian | Italian | Japanese | Korean | Maori | Dutch | Norwegian |
| Gemini-1.5-Pro | 37.2 | 55.4 | 27.6 | 1.2 | 8.2 | 3.8 | 27.7 | 36.7 |
| GPT4o | 21.8 | 28.4 | 21.0 | 0.0 | 11.1 | 26.8 | 26.4 | 24.7 |
| Llava-1.5-7B | 3.3 | 0.9 | 4.3 | 0.0 | 0.0 | 0.2 | 2.9 | 3.7 |
| Llava-Next-7B | 9.3 | 14.7 | 17.6 | 4.2 | 5.2 | 9.2 | 23.8 | 16.3 |
| Phi-3.5-Vision | 3.4 | 3.2 | 17.5 | 1.6 | 0.3 | 0.2 | 17.2 | 14.1 |
| Cambrian-8B | 6.6 | 15.7 | 15.5 | 7.2 | 2.0 | 3.2 | 20.3 | 16.0 |
| Llava-OV-7B | 3.6 | 16.4 | 12.8 | 0.6 | 0.0 | 1.7 | 24.7 | 13.9 |
| Molmo-7B-D | 3.5 | 17.2 | 17.8 | 5.2 | 2.4 | 7.5 | 15.7 | 13.8 |
| Llama3.2-11B | 12.7 | 1.2 | 16.0 | 0.0 | 0.0 | 9.3 | 22.0 | 1.1 |
| PaliGemma-3B | 2.0 | 0.2 | 1.8 | 0.0 | 0.0 | 4.0 | 2.6 | 2.3 |
| PALO-7B | 3.4 | 1.1 | 3.2 | 0.0 | 0.0 | 0.1 | 3.5 | 0.7 |
| mBLIP mT0-XL | 2.8 | 6.0 | 2.8 | 0.3 | 2.1 | 1.5 | 3.4 | 3.1 |
| mBLIP BLOOMZ | 11.8 | 16.0 | 16.5 | 0.0 | 4.5 | 0.1 | 18.2 | 14.5 |
| Pangea-7B (Ours) | 7.7 | 27.9 | 22.9 | 2.1 | 8.1 | 0.7 | 26.6 | 24.9 |
| Models | Polish | Portuguese | Quechua | Romanian | Russian | Swedish | Swahili | Telugu |
| Gemini-1.5-Pro | 35.5 | 35.7 | 0.7 | 31.2 | 32.4 | 37.8 | 10.7 | 0.0 |
| GPT4o | 22.2 | 28.0 | 4.4 | 19.1 | 20.7 | 26.0 | 20.0 | 12.5 |
| Llava-1.5-7B | 0.8 | 2.5 | 0.0 | 1.6 | 0.5 | 2.0 | 0.1 | 0.0 |
| Llava-Next-7B | 13.5 | 21.3 | 0.0 | 11.5 | 13.5 | 16.0 | 3.2 | 0.0 |
| Phi-3.5-Vision | 1.0 | 21.0 | 0.4 | 3.2 | 0.7 | 12.5 | 0.4 | 0.0 |
| Cambrian-8B | 9.3 | 17.5 | 0.0 | 13.4 | 11.3 | 17.9 | 3.7 | 2.3 |
| Llava-OV-7B | 7.4 | 24.6 | 0.0 | 6.8 | 5.5 | 15.0 | 2.0 | 0.0 |
| Molmo-7B-D | 8.2 | 16.2 | 0.6 | 11.6 | 12.3 | 14.1 | 3.8 | 0.4 |
| Llama3.2-11B | 1.0 | 18.6 | 0.0 | 10.1 | 0.6 | 7.4 | 5.8 | 0.0 |
| PaliGemma-3B | 0.9 | 1.3 | 0.1 | 0.8 | 0.0 | 2.0 | 0.0 | 0.0 |
| PALO-7B | 0.8 | 1.7 | 0.0 | 1.1 | 0.5 | 0.9 | 0.2 | 0.0 |
| mBLIP mT0-XL | 3.5 | 5.8 | 0.2 | 2.3 | 3.1 | 3.7 | 3.8 | 2.7 |
| mBLIP BLOOMZ | 11.8 | 16.5 | 0.1 | 13.7 | 14.5 | 14.5 | 8.4 | 3.0 |
| Pangea-7B (Ours) | 16.2 | 28.1 | 0.0 | 21.4 | 20.9 | 19.4 | 18.7 | 0.1 |
| Models | Thai | Turkish | Ukrainian | Vietnamese | Chinese |  |  |  |
| Gemini-1.5-Pro | 0.0 | 0.9 | 0.0 | 0.0 | 0.9 |  |  |  |
| GPT4o | 0.0 | 17.6 | 16.9 | 30.9 | 0.4 |  |  |  |
| Llava-1.5-7B | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |  |  |  |
| Llava-Next-7B | 0.0 | 0.0 | 0.3 | 0.0 | 6.3 |  |  |  |
| Phi-3.5-Vision | 0.5 | 1.9 | 0.0 | 2.2 | 0.0 |  |  |  |
| Cambrian-8B | 0.4 | 9.3 | 5.9 | 17.8 | 11.3 |  |  |  |
| Llava-OV-7B | 0.0 | 0.0 | 0.0 | 0.0 | 2.9 |  |  |  |
| Molmo-7B-D | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |  |  |  |
| Llama3.2-11B | 0.0 | 0.0 | 0.0 | 0.0 | 2.9 |  |  |  |
| PaliGemma-3B | 0.5 | 0.0 | 0.0 | 0.2 | 0.0 |  |  |  |
| PALO-7B | 0.2 | 0.0 | 0.0 | 0.1 | 0.0 |  |  |  |
| mBLIP mT0-XL | 0.0 | 3.9 | 2.0 | 7.1 | 0.0 |  |  |  |
| mBLIP BLOOMZ | 0.5 | 1.9 | 0.0 | 2.2 | 0.0 |  |  |  |
| Pangea-7B (Ours) | 0.0 | 0.0 | 0.3 | 0.0 | 4.9 |  |  |  |

### H.6 xGQA

We show the performance of different models’ performance on the xGQA benchmark in Table 9.

**Table 9: Comparison of models on the xGQA dataset across different languages**
| Models | English | Multi | Bengali | German | Indonesian | Korean | Portuguese | Russian | Chinese |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-1.5-Pro | 54.2 | 48.7 | 49.4 | 50.2 | 48.6 | 46.4 | 51.2 | 44.8 | 50.2 |
| GPT4o | 55.8 | 51.0 | 49.4 | 52.6 | 50.4 | 51.0 | 52.2 | 50.0 | 51.4 |
| Llava-1.5-7B | 62.0 | 30.7 | 15.6 | 28.4 | 33.4 | 38.2 | 27.5 | 33.1 | 38.4 |
| Llava-Next-7B | 64.8 | 37.8 | 11.5 | 41.5 | 37.3 | 42.5 | 39.8 | 43.5 | 48.2 |
| Phi-3.5-Vision | 64.7 | 38.4 | 7.7 | 51.4 | 36.0 | 36.3 | 49.6 | 46.2 | 41.4 |
| Cambrian-8B | 64.6 | 39.8 | 32.3 | 44.6 | 36.0 | 43.6 | 41.6 | 44.2 | 36.2 |
| Llava-OV-7B | 64.4 | 48.2 | 41.8 | 49.2 | 48.8 | 45.3 | 52.4 | 54.0 | 45.9 |
| Molmo-7B-D | 51.5 | 43.0 | 25.6 | 45.9 | 44.9 | 44.2 | 46.5 | 45.6 | 48.1 |
| Llama3.2-11B | 55.6 | 45.4 | 42.9 | 46.7 | 46.2 | 44.5 | 46.5 | 44.7 | 46.1 |
| PaliGemma-3B | 59.7 | 30.5 | 13.3 | 44.5 | 21.3 | 22.8 | 34.7 | 35.8 | 41.2 |
| PALO-7B | 60.5 | 37.8 | 42.2 | 39.1 | 36.8 | 41.7 | 31.7 | 27.0 | 46.5 |
| mBLIP mT0-XL | 44.2 | 39.9 | 39.1 | 41.1 | 39.1 | 39.7 | 40.7 | 40.2 | 39.4 |
| mBLIP BLOOMZ-7B | 43.3 | 36.9 | 37.7 | 36.3 | 39.3 | 28.5 | 40.7 | 36.6 | 39.1 |
| Pangea-7B (Ours) | 64.7 | 60.2 | 58.9 | 61.6 | 60.1 | 58.9 | 61.8 | 60.4 | 59.6 |

### H.7 MAXM

We show the performance of different models’ performance on the MAXM benchmark in Table 10.

**Table 10: Comparison of models on the MAXM dataset across different languages.**
| Models | English | Multi | French | Hindi | Hebrew | Romanian | Thai | Chinese |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-1.5-Pro | 56.4 | 63.5 | 60.2 | 66.5 | 65.7 | 57.4 | 73.9 | 57.4 |
| GPT4o | 60.7 | 65.4 | 59.8 | 68.8 | 70.0 | 61.3 | 76.5 | 56.3 |
| Llava-1.5-7B | 49.8 | 20.4 | 32.2 | 17.3 | 12.9 | 15.1 | 17.2 | 27.8 |
| Llava-Next-7B | 54.9 | 21.4 | 33.7 | 16.2 | 10.7 | 15.5 | 18.3 | 33.9 |
| Phi-3.5-Vision | 55.3 | 25.0 | 38.3 | 31.9 | 17.5 | 10.9 | 24.3 | 27.4 |
| Cambrian-8B | 55.3 | 28.7 | 41.7 | 23.8 | 17.1 | 32.0 | 25.7 | 31.8 |
| Llava-OV-7B | 54.9 | 34.8 | 37.9 | 31.9 | 17.8 | 30.2 | 53.0 | 37.9 |
| Molmo-7B-D | 52.9 | 37.5 | 45.5 | 33.5 | 30.7 | 28.9 | 46.3 | 40.4 |
| Llama3.2-11B | 55.3 | 43.9 | 48.1 | 50.4 | 41.8 | 36.6 | 56.7 | 30.0 |
| PaliGemma-3B | 47.9 | 19.9 | 8.0 | 36.5 | 19.3 | 13.4 | 31.3 | 10.8 |
| PALO-7B | 51.4 | 16.3 | 33.7 | 15.8 | 12.1 | 11.3 | 14.6 | 10.5 |
| mBLIP mT0-XL | 44.7 | 36.8 | 36.0 | 42.7 | 28.9 | 30.3 | 56.3 | 26.4 |
| mBLIP BLOOMZ-7B | 44.7 | 24.8 | 33.0 | 47.3 | 8.9 | 16.9 | 9.7 | 33.2 |
| Pangea-7B (Ours) | 48.6 | 34.3 | 36.4 | 40.4 | 36.4 | 33.1 | 36.2 | 23.1 |

### H.8 xMMMU

We show the performance of different models’ performance on the xMMMU benchmark in Table 11.

**Table 11: Comparison of models on the xMMMU dataset across different languages.**
| Models | English | Multi | Arabic | French | Hindi | Indonesian | Japanese | Portuguese |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-1.5-Pro (0801) | 65.8 | 57.7 | 57.7 | 58.1 | 55.5 | 60.2 | 55.0 | 59.6 |
| GPT4o (0513) | 69.1 | 58.3 | 56.7 | 58.1 | 58.1 | 59.9 | 58.0 | 58.9 |
| Llava-1.5-7B | 36.2 | 31.5 | 29.5 | 34.9 | 27.5 | 31.6 | 32.0 | 33.7 |
| Llava-Next-7B | 36.7 | 34.3 | 30.5 | 35.6 | 30.9 | 37.0 | 34.9 | 37.0 |
| Phi-3.5-Vision | 42.6 | 38.8 | 35.6 | 44.0 | 30.9 | 36.7 | 37.9 | 47.8 |
| Cambrian-8B | 41.8 | 33.2 | 32.6 | 34.6 | 30.9 | 31.3 | 33.5 | 36.0 |
| Llava-OV-7B | 46.3 | 41.0 | 41.6 | 43.0 | 34.7 | 43.4 | 40.1 | 43.4 |
| Molmo-7B-D | 42.9 | 40.4 | 40.6 | 42.6 | 32.6 | 40.7 | 43.9 | 42.1 |
| Llama3.2-11B | 39.2 | 34.0 | 33.6 | 39.6 | 32.3 | 36.7 | 29.0 | 33.0 |
| PaliGemma-3B | 26.3 | 25.2 | 29.2 | 23.8 | 21.6 | 24.2 | 24.5 | 27.6 |
| PALO-7B | 33.1 | 30.5 | 30.5 | 33.2 | 28.9 | 34.0 | 27.1 | 33.3 |
| mBLIP mT0-XL | 29.3 | 30.4 | 30.2 | 33.2 | 28.2 | 26.9 | 31.6 | 32.3 |
| mBLIP BLOOMZ-7B | 29.2 | 30.8 | 28.5 | 33.9 | 27.8 | 33.3 | 31.6 | 29.6 |
| Pangea-7B (Ours) | 45.7 | 43.7 | 42.3 | 45.3 | 41.6 | 46.5 | 40.5 | 46.1 |

### H.9 M3Exam

We show the performance of different models’ performance on the M3Exam benchmark in Table 12.

**Table 12: Comparison of models on the M3Exam dataset across different languages.**
| Models | English | Multi | Afrikaans | Chinese | Italian | Portuguese | Thai | Vietnamese |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-1.5-Pro | 77.4 | 64.7 | 80.4 | 74.1 | 76.3 | 61.8 | 49.9 | 46.0 |
| GPT4o | 68.0 | 61.0 | 73.0 | 68.0 | 67.0 | 58.0 | 52.0 | 48.3 |
| Llava-1.5-7B | 32.3 | 29.0 | 28.2 | 24.3 | 40.1 | 28.2 | 23.7 | 29.3 |
| Llava-Next-7B | 36.5 | 28.4 | 28.2 | 25.4 | 37.8 | 27.0 | 23.7 | 28.4 |
| Phi-3.5-Vision | 55.8 | 37.2 | 44.2 | 40.8 | 51.4 | 40.3 | 25.2 | 21.6 |
| Cambrian-8B | 34.7 | 33.4 | 36.8 | 34.2 | 45.2 | 30.3 | 28.9 | 25.0 |
| Llava-OV-7B | 60.4 | 45.8 | 50.3 | 58.0 | 57.2 | 43.8 | 30.9 | 34.5 |
| Molmo-7B-D | 57.1 | 39.1 | 35.6 | 56.4 | 49.4 | 40.2 | 27.4 | 25.9 |
| Llama3.2-11B | 51.8 | 36.6 | 42.3 | 46.4 | 45.8 | 28.4 | 26.4 | 30.2 |
| PaliGemma-3B | 36.0 | 25.6 | 26.4 | 24.7 | 32.2 | 24.3 | 27.2 | 19.0 |
| PALO-7B | 30.8 | 27.8 | 31.9 | 22.1 | 36.9 | 32.3 | 22.7 | 20.7 |
| mBLIP mT0-XL | 22.8 | 25.0 | 16.0 | 25.6 | 33.7 | 21.2 | 22.4 | 31.0 |
| mBLIP BLOOMZ-7B | 30.3 | 29.5 | 28.2 | 29.8 | 37.3 | 28.3 | 22.9 | 30.2 |
| Pangea-7B (Ours) | 61.4 | 42.1 | 52.1 | 49.2 | 54.9 | 43.3 | 32.9 | 19.8 |

### H.10 TyDiQA

We show the performance of different models’ performance on the TyDiQA benchmark in Table 13.

**Table 13: Comparison of models on the TyDiQA dataset across different languages.**
| Models | English | Multi | Arabic | Bengali | Finnish | Indonesian | Korean | Russian | Swahili | Telugu |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vicuna-1.5-7B | 59.7 | 52.7 | 32.3 | 68.1 | 63.0 | 72.6 | 58.8 | 57.6 | 51.3 | 18.1 |
| Qwen2-7B-Instruct | 72.2 | 71.2 | 67.6 | 75.9 | 67.1 | 78.0 | 64.9 | 67.2 | 75.3 | 73.8 |
| Llava-1.5-7B | 66.8 | 52.8 | 61.8 | 33.4 | 60.2 | 72.8 | 63.3 | 55.0 | 55.0 | 20.6 |
| Llava-Next-7B | 68.3 | 52.1 | 64.5 | 24.9 | 63.0 | 74.3 | 61.9 | 58.4 | 53.1 | 17.0 |
| Phi-3.5-Vision | 75.9 | 51.3 | 63.1 | 24.8 | 57.3 | 70.6 | 60.2 | 57.5 | 48.7 | 28.3 |
| PALO-7B | 69.4 | 50.8 | 60.9 | 46.0 | 61.8 | 70.6 | 56.8 | 56.7 | 42.5 | 10.8 |
| Pangea-7B (Ours) | 73.7 | 66.0 | 55.5 | 65.3 | 66.3 | 74.5 | 69.4 | 60.1 | 76.6 | 60.0 |

### H.11 XStoryCloze

We show the performance of different models’ performance on the XStoryCloze benchmark in Table 14.

**Table 14: Comparison of models on the XStoryCloze dataset across different languages.**
| Models | English | Multi | Arabic | Spanish | Basque | Hindi | Ind. | Burmese | Russian | Swahili | Telugu | Chinese |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vicuna-1.5-7B | 78.1 | 57.4 | 52.7 | 69.4 | 50.8 | 54.5 | 61.0 | 48.4 | 66.5 | 52.1 | 54.5 | 63.5 |
| Qwen2-7B-Instruct | 80.3 | 61.9 | 64.0 | 71.6 | 51.6 | 59.6 | 68.5 | 50.7 | 72.7 | 53.2 | 55.3 | 72.1 |
| Llava-1.5-7B | 79.1 | 57.6 | 52.7 | 69.2 | 50.9 | 54.9 | 62.6 | 49.0 | 65.9 | 51.7 | 55.8 | 63.9 |
| Llava-Next-7B | 79.1 | 57.1 | 51.7 | 68.8 | 50.3 | 54.5 | 62.0 | 46.7 | 65.5 | 52.1 | 55.2 | 63.8 |
| Phi-3.5-Vision | 77.9 | 54.8 | 53.7 | 67.2 | 50.4 | 54.9 | 51.7 | 47.8 | 61.3 | 49.3 | 52.5 | 59.5 |
| PALO-7B | 77.4 | 57.2 | 56.5 | 68.4 | 49.8 | 58.6 | 58.5 | 47.4 | 65.6 | 51.2 | 53.1 | 62.8 |
| Pangea-7B (Ours) | 79.1 | 61.2 | 60.5 | 67.8 | 50.0 | 61.8 | 66.4 | 48.7 | 69.4 | 58.9 | 60.4 | 68.2 |

### H.12 MGSM

We show the performance of different models’ performance on the MGSM benchmark in Table 15.

**Table 15: Comparison of models on the MGSM dataset across different languages.**
| Models | English | Multi | Bengali | German | Spanish | French | Japanese | Russian | Swahili | Telugu | Thai | Chinese |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vicuna-1.5-7B | 17.6 | 6.4 | 0.0 | 14.4 | 9.6 | 14.4 | 2.8 | 10.8 | 3.6 | 0.0 | 2.0 | 14.8 |
| Qwen2-7B-Instruct | 48.8 | 40.4 | 0.0 | 67.2 | 67.6 | 68.8 | 11.2 | 71.2 | 10.8 | 2.4 | 45.6 | 59.2 |
| Llava-1.5-7B | 14.8 | 7.6 | 0.0 | 15.2 | 10.8 | 18.0 | 2.8 | 11.2 | 0.4 | 0.0 | 1.6 | 15.6 |
| Llava-Next-7B | 15.6 | 7.5 | 0.0 | 13.6 | 13.2 | 16.0 | 1.6 | 12.8 | 2.0 | 0.0 | 1.6 | 14.0 |
| Phi-3.5-Vision | 59.2 | 33.1 | 0.0 | 64.0 | 59.6 | 58.0 | 20.0 | 54.0 | 4.0 | 0.0 | 18.8 | 52.4 |
| PALO-7B | 13.6 | 5.8 | 0.0 | 11.6 | 9.6 | 13.2 | 1.6 | 8.8 | 0.4 | 0.0 | 0.0 | 12.4 |
| Pangea-7B (Ours) | 82.0 | 47.3 | 0.0 | 68.4 | 74.8 | 63.2 | 22.0 | 68.0 | 54.0 | 5.6 | 49.6 | 68.0 |

### H.13 MMMLU

We show the performance of different models’ performance on the MMMLU benchmark in Table 16.

**Table 16: Comparison of models on the MMMLU dataset across different languages.**
| Models | English | Multi | Arabic | Bengali | Portuguese | Chinese | French | German |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vicuna-1.5-7B | 49.5 | 34.7 | 30.3 | 28.5 | 39.6 | 36.9 | 40.4 | 39.8 |
| Qwen2-7B-Instruct | 70.1 | 53.1 | 51.0 | 43.4 | 60.7 | 63.8 | 61.5 | 57.7 |
| Llava-1.5-7B | 50.2 | 34.9 | 29.7 | 28.5 | 40.3 | 36.8 | 40.1 | 39.8 |
| Llava-Next-7B | 52.1 | 35.6 | 30.0 | 28.8 | 40.7 | 37.3 | 41.4 | 41.4 |
| Phi-3.5-Vision | 62.0 | 39.1 | 34.9 | 27.9 | 47.6 | 41.5 | 49.2 | 45.8 |
| PALO-7B | 46.7 | 32.6 | 30.3 | 29.5 | 36.0 | 34.2 | 36.9 | 35.8 |
| Pangea-7B (Ours) | 68.4 | 52.2 | 49.3 | 44.4 | 58.9 | 60.5 | 58.9 | 56.7 |
| Models | Hindi | Indonesian | Italian | Japanese | Korean | Spanish | Swahili | Yoruba |
| Vicuna-1.5-7B | 29.8 | 36.5 | 39.5 | 35.9 | 34.1 | 40.3 | 27.9 | 26.8 |
| Qwen2-7B-Instruct | 45.7 | 57.1 | 60.8 | 58.0 | 54.6 | 61.9 | 36.0 | 31.8 |
| Llava-1.5-7B | 29.2 | 37.1 | 41.0 | 35.1 | 34.1 | 41.6 | 28.0 | 27.3 |
| Llava-Next-7B | 29.6 | 37.5 | 41.2 | 36.0 | 34.2 | 42.7 | 28.5 | 28.7 |
| Phi-3.5-Vision | 32.9 | 38.3 | 47.0 | 40.0 | 36.6 | 49.6 | 28.9 | 27.8 |
| PALO-7B | 29.6 | 33.7 | 36.4 | 32.7 | 30.6 | 37.0 | 26.4 | 27.1 |
| Pangea-7B (Ours) | 45.7 | 55.4 | 58.8 | 55.3 | 52.7 | 59.7 | 42.8 | 31.3 |

## Appendix I A Preliminary Exploration of Constructing Multilingual OCR Instructions

Optical Character Recognition (OCR) is a critical capability for multimodal LLMs, enabling them to interpret and process textual information embedded within images. However, most existing OCR training datasets are predominantly English-centric, which limits the models’ performance in non-English contexts. To address this gap, we have curated a comprehensive set of 500K multilingual OCR training samples from web user interfaces.

We utilize URLs from the CC-News-Multilingual(^4^44https://huggingface.co/datasets/intfloat/multilingual_cc_news)dataset (Hamborg et al., ) to obtain a diverse set of multilingual web pages. Using Playwright(^5^55https://github.com/microsoft/playwright), we render each website and automatically capture screenshots under various device settings and resolutions to achieve a wide range of image dimensions and aspect ratios. Each screenshot includes a red bounding box that highlights a specific element targeted for OCR extraction. We focus on ten languages for this dataset: English, Chinese, Japanese, Korean, Indonesian, Hindi, Spanish, French, Portuguese, and Arabic. We totally have 1M samples (50K for each language).

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Would the other open source models used as the baseline in this paper could also benefit from training on the PANGEAINS data?
2. How to validate if there is potential overfitting of training on the PANGEAINS and evaluating on PANGEABENCH?
soundness: 3
strengths: 1. This paper addressed an important research problem on multilingual multimodal learning, by providing a large instruction dataset and evaluation suite that covers multiple languages with the corresponding culture reference annotations. This data could be of interest to a broad audience in the research community. 
2. The paper is well written and easy to follow. The proposed PANGEAINS instruction data and the PANGEABENCH evaluation suite were described with good amount of details. The authors further provided examples of the data which is helpful in understanding the dataset proposed.
3. The authors conducted comprehensive experiments of the proposed model and data, by comparing it to several strong baseline models and evaluating on a diverse set of tasks. The performance of the model seems to be pretty strong from the evaluation results.
summary: This paper describes PANGEA model and data, with a focus on multilingual multimodal settings. 	PANGEAINS is an instruction dataset that spans 39 languages that covers the different languages through translation and the culture references aspects of the data. PANGEABENCH is an evaluation suite consists of 14 datasets covering 47 languages. PANGEA, a model based on the LLaVA-Next architecture and trained on the PANGEAINS data, showed strong performance on several tasks comparing to strong baseline models.
weaknesses: 1. It could be helpful to a section to discuss related multilingual multimodal datasets (e.g. M3IT etc), and show how PANGEA compares to them. This could further illustrate the significance of the proposed data here.
2. Although the authors highlighted the introduction of PANGEA, the discussion on the modeling aspect of PANGEA is light. It seems to be a directly application of a LLaVA-Next based model trained using the new data proposed. It could help to further discuss the training specifics.
3. The proposed method (model and data) is only evaluated on one eval set, which is the proposed PANGEABENCH. It could be helpful to evaluate on additional evaluation suits to see how the improvement from training on PANGEAINS would generalize.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: A. For text-only performance, how is the chat performance compared to the other models? Some evaluation benchmarks such as MTBench, ArenaHardAuto or AlphacaEval can be used for this comparison. 

B. Have you checked if there is any data leakage from the benchmarks that has been tested?
soundness: 4
strengths: A. This paper includes a very significant set of contributions which will be made public together with the code:
1. PangeaIns with 6M instructions from 39 languages that include (1) high-quality English instructions, (2) machine-translated instructions, (3) culturally relevant multimodal tasks
2. PangeaBench, a multimodal evaluation suite consists of 14 diverse datasets covering 47 languages
3. Pangea an MLLM covering 39 languages outperforms existing open-source models by 7.3 points on English tasks and 18.0 points in multilingual tasks.

B. The paper presents a detailed multilingual multi-modal data generation pipeline showcasing the limitation of open-source machine translation models, and the use of LLMs to enrich the multilingual data where the LLM score and filter images based on cultural informativeness. Then remaining data is enriched with re-generating descriptions and creating complex instructions. 

C. PangeaBench covering 47 languages in 14 datasets introduces a comprehensive evaluation for multilingual multimodal LLMs. Notably, the new evaluation dataset xChat is a human-curated benchmark where reference answers and scoring rubrics are annotated for each query. This enables a better use of LLM-as-a-judge for scoring other models' generations using a scale of 1-5 for each query.
summary: This paper introduces Pangea, a multilingual multi-modal LLM together with its multi-modal instruction fine-tuning dataset PangeaIns and a comprehensive benchmark PangeaBench:
- PangeaIns includes 6M examples from 39 languages and consists of (1) high-quality English instructions, (2) machine-translated instructions, (3) culturally relevant multimodal tasks
- PangeaBench, a multimodal evaluation suite consists of 14 diverse datasets covering 47 languages, including a human-curated multimodal chat benchmark xChat 

Pangea outperforms other open-source models by 7.3 points on English tasks and 18.0 points in multilingual tasks. Finally, the paper discusses the impact of English data, the necessity of language-specific datasets, and the limitations in multimodal chat and multilingual OCR.
weaknesses: There are only two minor weaknesses mainly focusing on the need for clarity and the presentation:

A. In Section 2.1, the paper mentions a post-processing pipeline for noisy translations, however, details are missing. This might be quite important for practitioners to build similar models. Therefore, I suggest authors to include more details there.

B. In Section 4.1, it has been mentioned that the model is based on Llava-Next architecture but this needs to be detailed. For example, does the training include multiple stages such as adapter alignment and SFT, or is only SFT has been used? If so what datasets are used in which stage?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Data Representation: What strategies can be implemented to ensure a more balanced representation of low-resource languages within the PANGEAINS dataset?

Quality Assessment: How can we establish rigorous quality assessment protocols to evaluate the accuracy and relevance of data for each language in the dataset?

Cultural Context: What methodologies can be employed to enrich the dataset with culturally relevant examples and ensure that cultural nuances are adequately captured across all languages?

Data Augmentation: What role can data augmentation techniques play in enhancing the dataset, particularly for underrepresented languages, to improve model performance?


As a reviewer, I would like to provide detailed technical advice regarding the concerns particularly focusing on the potential imbalances in data quantity and quality across different languages, especially for low-resource languages. Perhaps I (not a robot) missed the methods described in your manuscript. Please let me know and provide additional information if possible.

1. Addressing Imbalances in Quantity and Quality
- Data Augmentation Techniques:

Synonym Replacement: Identify and replace words with their synonyms to generate paraphrased sentences that maintain the original meaning.
Back Translation: Translate sentences into another language and then back to the original language to create varied sentence structures while preserving semantic content.
Noise Injection: Introduce small amounts of noise (e.g., typos, grammatical variations) to simulate real-world language use and increase dataset diversity.

- Collaboration with Native Speakers:

Engage with native speakers or linguists proficient in low-resource languages to enhance the quality of your dataset. This could involve Crowdsourcing Platforms with native speakers for data validation and enrichment.

- Detailed Data Distribution Analysis:

Provide a comprehensive analysis of the data distribution across the 39 languages included in the dataset. This should include:
Quality Metrics: If applicable, include metrics that assess the quality of the data per language (e.g., linguistic diversity, error rates).

Visual Representations: Use graphs or charts to visually represent the distribution and highlight any significant imbalances.

2. Enhancing Quality of Cultural Representation

When collecting data, consider the cultural context of each language. This can be achieved by:
Ethnographic Research: Integrate ethnographic methods to understand cultural nuances that might not be reflected in traditional data collection.

Focus Groups: Conduct focus groups with speakers of the languages to gather insights on cultural expressions and values that should be represented in the dataset.

Examples of Cultural Representation: Provide specific examples of how the dataset captures cultural differences. Detail how various cultural elements (e.g., idioms, customs, social norms) are represented in the data. Case studies that illustrate the inclusion of culturally relevant examples. An analysis of how the dataset addresses or fails to address complex cultural concepts.



3. insufficient in-depth analysis related to specific tasks and languages, particularly in areas such as multilingual visual question answering (VQA) and multi-subject reasoning. 

Below are some actionable suggestions to enhance your analysis:

- Conducting Ablation Studies
Ablation Studies on Language Groups: Perform ablation studies to assess how different language groups affect model performance. This involves systematically removing specific languages or language features from the dataset to observe changes in performance metrics. Analyze which languages contribute the most to overall model effectiveness and identify any languages that may cause degradation in performance.

Ablation Studies on Task Types: Similarly, conduct ablation studies focusing on different task types (e.g., VQA versus multi-subject reasoning). By isolating the performance impacts of each task type, you can gain insights into which tasks are more challenging for your model and why.

Expert Collaboration: How can collaboration with linguistic and cultural experts be structured to inform the data collection process, ensuring a more comprehensive understanding of low-resource languages?

Evaluation Metrics: What specific evaluation metrics can be developed to assess the model's performance across different languages, particularly focusing on those with limited data?

Feedback Mechanisms: How can feedback mechanisms be incorporated into the dataset's development process to continuously improve data quality and representation based on model performance?

Cross-Linguistic Comparisons: What methods can be utilized to conduct cross-linguistic comparisons that identify specific challenges faced by the model when processing low-resource languages?
soundness: 2
strengths: Originality
The paper presents approaches to multilingual and multimodal learning through the development of the PANGEAINS dataset and the PANGEABENCH evaluation framework. By focusing on both linguistic and cultural diversity, it addresses gaps in existing datasets that often overlook these critical aspects. The integration of machine translation and culturally relevant guidelines for data curation showcases an innovative methodology that enhances the relevance and applicability of the dataset. This originality is further emphasized by the emphasis on cultural understanding in multimodal tasks, which is a relatively underexplored area in the field.

Quality
The quality of the research is evident in the rigorous methodology employed for dataset creation and model evaluation. The paper provides a thorough explanation of the processes involved in curating the PANGEAINS dataset, including the use of high-quality image captioning and the implementation of robust evaluation metrics. The empirical results presented are well-supported, demonstrating PANGEA's superior performance in various tasks. Additionally, the use of established benchmarks and the careful selection of evaluation metrics contribute to the overall robustness of the findings.

Clarity
The paper is well-structured and clearly organized, making it accessible to readers across different backgrounds. Each section is logically connected, and the authors effectively communicate complex ideas through concise language and well-defined terminology. The inclusion of visual aids, such as figures and tables, enhances understanding by illustrating key concepts and results. Overall, the clarity of presentation allows for easy comprehension of the methodologies and findings.

Significance
The significance of the paper lies in its potential impact on the field of multilingual and multimodal learning. By providing a comprehensive dataset and a structured evaluation framework, the authors contribute valuable resources that can facilitate further research and development in this area. The focus on cultural diversity and understanding not only enriches the training of models but also promotes more inclusive and applicable AI systems. The findings underscore the importance of addressing cultural nuances in AI applications, which is essential for creating systems that are effective and relevant in a global context.
summary: PANGEAINS is a multilingual and multicultural dataset comprising 6 million samples across 39 languages, developed for instruction tuning. This dataset emphasizes linguistic and cultural diversity and implements strategies such as machine translation, cultural understanding guidelines, and high-quality image caption generation to address challenges in multilingual multimodal learning. By using machine translation, English instructions are extended into other languages, and culturally biased issues are resolved by curating images from the LAION-Multi dataset to capture cultural contexts.

PANGEABENCH is a comprehensive evaluation tool designed to assess PANGEA's performance across various languages and tasks. It includes diverse benchmarks that evaluate capabilities in cross-lingual and cross-cultural contexts. The multimodal tasks encompass natural conversation, image captioning, cultural understanding, multilingual visual question answering, and multi-subject reasoning, while text-only tasks are also included to examine the model's linguistic understanding.

Experimental results show that PANGEA-7B outperforms existing models in both English and multilingual tasks, particularly demonstrating significant improvements in cultural understanding tasks. The performance gap between English and multilingual tasks is small, indicating strong multilingual processing capabilities, although there remains a performance disparity compared to closed-source models. Additionally, PANGEA-7B exhibits the strongest performance in text-only tasks, with notable enhancements in handling complex multilingual reasoning and mathematical tasks attributed to the inclusion of math-related instructions.

The discussion explores the scaling effect of instruction quantity, the role of English data, the relationship between training sample proportions and performance, real-world applications of multimodal chat, and preliminary explorations of multilingual OCR. These insights aim to provide a comprehensive understanding of the model and outline future advancement directions. The development of PANGEAINS and PANGEABENCH is expected to significantly contribute to enhancing the performance of multilingual and multimodal models.
weaknesses: Limitations of the Dataset
The PANGEAINS dataset comprises samples from 39 languages; however, there exists a potential imbalance in both the quantity and quality of data for specific languages. This issue is particularly pronounced for low-resource languages that the authors do not explicitly address. The scarcity of data for these languages can lead to significant challenges in training models that are capable of performing effectively across diverse linguistic contexts. Insufficient representation may result in suboptimal performance, as the models may lack the necessary exposure to the varied linguistic structures and cultural nuances inherent in these languages.

Moreover, the quality of the data is as crucial as its quantity. If the available data for certain languages is of lower quality—whether due to inaccuracies in translation, lack of contextual relevance, or inadequate cultural representation—this can further exacerbate performance issues. Such disparities in data quality can hinder the model's ability to generalize effectively across languages, potentially leading to biased outputs and diminished reliability in multilingual applications.

Consequently, these imbalances may impede the overall generalization capabilities of multilingual processing, as the model’s performance may be skewed towards languages with richer datasets. Thus, it is imperative for future iterations of the dataset to ensure a more equitable distribution of data quality and quantity across all included languages, particularly for those that are underrepresented, in order to enhance the robustness and applicability of multilingual models.

-Depth of Cultural Context
In evaluating cultural understanding, the discussion regarding whether the PANGEAINS dataset adequately reflects diverse cultural backgrounds is somewhat lacking. Cultural differences are complex and nuanced, extending beyond mere linguistic expressions or visual representations to encompass social contexts, customs, values, and other interrelated elements. These factors shape the meanings within specific cultural contexts and are essential for multilingual multimodal models to accurately understand and process cultural nuances. However, the current dataset primarily focuses on the matching of images and captions, which may not sufficiently capture the depth of cultural context.

For instance, metaphorical expressions or symbolic meanings in one culture may be interpreted entirely differently in another. Capturing these subtle distinctions necessitates a more in-depth analysis that transcends simple dataset curation. It would be beneficial to incorporate expert consultations on cultural backgrounds and conduct comprehensive research on each culture to inform data collection efforts. Such an approach would enhance the model's ability to understand meanings across various cultural contexts and generate more nuanced responses.

- Limitations of Experimental Design
From the perspective of experimental design, there is a notable lack of in-depth analysis concerning specific tasks or languages. For example, in tasks such as multilingual visual question answering (VQA) or multi-subject reasoning, a detailed exploration of the factors influencing model performance is missing. Without such analysis, it becomes challenging to clearly understand the model's strengths and weaknesses, ultimately limiting the ability to establish future improvement directions.

Particularly, without a thorough examination of various determinants of performance—such as the characteristics of specific languages, dataset imbalances, or hyperparameter adjustments during training—understanding why a model excels or falters in certain tasks becomes difficult. Therefore, it is crucial that the experimental design explicitly defines the specific conditions and variables for each task, along with comparative analyses of performance across different environments. This approach would not only enhance model performance but also deepen the overall understanding of multilingual and multimodal systems.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: In Figure 7, the model's poor performance on English OCR is counterintuitive. What might explain this unexpected result?
soundness: 4
strengths: 1. This paper is well-written, offering comprehensive details on the curated instruction-tuning datasets, benchmarks, and evaluation results. The authors present their work with clarity and thoroughness.
2. The meticulously curated datasets for both training and evaluation are of significant practical value. They will undoubtedly make a substantial contribution to the multilingual research community, providing a robust foundation for future studies in this field.
summary: The paper introduces PANGEA, a multilingual multimodal large language model (MLLM) addressing the limitations of English-centric data in existing models. The model was trained on a dataset called PANGEAINS, which includes 6 million instruction samples across 39 languages. PANGEINS combines high-quality English instructions, machine-translated content, and culturally relevant tasks. The authors also developed PANGEABENCH, an evaluation suite to assess the model’s multilingual and multimodal capabilities, including tasks like chat, image captioning, and visual question answering. PANGEA outperforms existing open-source models, especially in tasks requiring cross-cultural understanding, though it still lags behind proprietary models like GPT-4 in areas like complex reasoning. Fully open-sourced along with its dataset and evaluation suite, PANGEA aims to promote further research in inclusive multilingual LLMs, addressing challenges like data scarcity, cross-lingual transfer, and multilingual OCR.
weaknesses: 1. The paper lacks some crucial details about the model and training process. Specifically, it would be beneficial to provide more information about the architecture, including the type of vision encoder used and the final total number of parameters. Additionally, details regarding the training infrastructure and duration are absent. 
2. Some evaluation setup details are missing, such as the number of shots for each dataset, which makes the results difficult to reproduce. For instance, the results for Qwen2-7B-instruct are significantly worse than those reported in the official release (https://qwenlm.github.io/blog/qwen2/#multilingual-capability-of-instruction-tuned-models). What is the reason?
3. Although the authors claim the model supports 39 languages, it's unclear which specific languages have shown improvement and to what extent compared to the backbone model. This is particularly uncertain for languages that comprise only a small portion of the PANGEAINS dataset.

Meta-review: This paper introduces PANGEA, a multilingual multimodal LLM, and two resources: PANGEAINS (a 6M instruction dataset covering 39 languages) and PANGEABENCH (an evaluation suite spanning 47 languages). The authors claim significant improvements over existing open-source models on both English and multilingual tasks.

Strengths: This work emphasizes the importance of culturally aware training data and evaluation, with rigorous evaluation across multiple languages and tasks.

Weaknesses: Some architectural and training details were initially missing (though addressed in rebuttal), these should be added to the final version of the paper. Some reviewers also raise concerns about the potential data imbalances and limited discussion/handling of low-resource languages.

Despite some limitations, the work makes substantial contributions with multilingual and multimodal resources. I recommend accepting this paper.

Decision: Accept (Poster)