Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
Authors: Xiang Yue
Yueqi Song
Akari Asai
Seungone Kim
Jean de Dieu Nyandwi
Simran Khanuja
Anjali Kantharuban
Lintang Sutawika
Sathyanarayanan Ramamoorthy
Graham Neubig
Venue: ICLR 2025 Poster (2025)
Keywords: Multilingual
Multimodal
LLMs
Primary area: foundation or frontier models, including LLMs
TL;DR: We develop a fully open multilingual multimodal LLM for 39 languages.

Abstract: Despite recent advances in multimodal large language models (MLLMs), their development has predominantly focused on English- and western-centric datasets and tasks, leaving most of the world's languages and diverse cultural contexts underrepresented.  
This paper introduces PANGEA, a multilingual multimodal LLM trained on PANGEAINS, a diverse 6M instruction dataset spanning 39 languages. PANGEAINS features: 1) high-quality English instructions, 2) carefully machine-translated instructions, and 3) culturally relevant multimodal tasks to ensure cross-cultural coverage. 
To rigorously assess models' capabilities, we introduce PANGEABENCH, a holistic evaluation suite encompassing 14 datasets covering 47 languages. 
Results show that PANGEA significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts. Ablation studies further reveal the importance of English data proportions, language popularity, and the number of multimodal training samples on overall performance.  We fully open-source our data, code, and trained checkpoints, to facilitate the development of inclusive and robust multilingual MLLMs, promoting equity and accessibility across a broader linguistic and cultural spectrum.

Published as a conference paper at ICLR 2025
PANGEA
PANGEA: A FULLY OPEN MULTILINGUAL MULTI- MODAL LLM FOR 39 LANGUAGES
Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, Graham Neubig [xyue2, yueqis, greubigg00cs cmu.edu
Carnegie Mellon University
https: http://eeuuaa...tubbb.o..
ABSTRACT
Despite recent advances in multimodal large language models (MLLMss, their development has predominantly focused on English- and western-centric datasets and tasks, leaving most of the world's languages and diverse cultural contexts underrepresented. This paper introduces PANGEA, a multilingual multimodal LLM trained on PANGEAINS, a diverse 6M instruction dataset spanning 39 lan- guages. PANGEAINS features: 1) high-quality English instructions, 2) carefully instructions, and 3) culturally relevant multimodal tasks to en- sure cross-cultural coverage. To rigorously assess models' capabilities, we intro- duce VANGEABENCE a holistic evaluation suite encompassing datasets cov- ering 47 languages. Results show that PANGEA significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts. Abla- tion studies further reveal the importance of English data proportions, language popularity, and the number of multimodal training samples on overall perfor- mance. We fully open-source the development of inclusive and robust multilingual MLLMs, promoting equity and accessibility across broader linguistic and cultural spectrum.
data, code, and trained checkpoints, facilitate
Aggregate Performance on Multimodal PangeaBench
70
GPT46
Proprietary
Multilingual Performance
Carnegic Mellon University
Open-weights
Cemini-1,5-Pro
Open-weights and Open-data
Pangea-7B (Ours)
O
Multilingual MLLMs
Llama3.2-11B
Iel ByteDance Llava-OV-7B
40
NYU Cambrian-SH
Ai2
Molmo-7B-D
mBLIP BLOOMZ
PALO-7B
30
English-centric MLLMs
Phi-3.5-Vision
Llava-Next-7B
PaliGemma-3g
30
35
40
45
50
55
60
65
70
75
English Performance
Figure 1: Overview of the aggregate performance of various multimodal LLMs on PANGEABENCH. Our PANGEA-7B demonstrates comparable performance to SoTA open-source models in English settings, while significantly outperforming them multilingual scenarios.
Equal Contributions.

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
Multimodal large language models (MLLMs) Liu et al. 2023b; 2024 Dubey et al. 2024 Deitke
et al., 2024 OpenAl, 2024b; Team et al. 2023) have demonstrated impressive capabilities in tasks such as image captioning, visual question answering, and complex reasoning (Yue et al. 2024a
Despite this rapid progress in their reasoning ability, a critical flaw persists: the overwhelming focus on English- and western-centric training and evaluation datasets (Liu et al. 202 1 Song et al., 2023
This homogeneity results a lack of representation for the vast majority of the world's languages and diverse cultural contexts (Yu et al., (022). Consequently, models predominately trained on such data exhibit: (a) diminished performance in multilingual settings (Blasi et al. (022)) with poor tokenization further leading higher inference costs (Ahia al. 2023); (b) generate outputs mis- aligned with the socio-cultural norms of underrepresented languages (AiKhamissi et al.) (224);; and (c) lack the ability recognize objects from geographically diverse regions (Ramaswamy et al., 2024) or rare objects belonging to the long-tail (Gupta et al. 2019). With the increased adoption of these models into real-world applications across the globe, there's S an urgent need develop multi- lingual MLLMs that equitably serve a diverse set of users. Few efforts have been made to develop multilingual MLLMs (Geigle et al., 2024b; Rasheed et al.] 2025), however, their performance still exhibits inequalities across languages and lacks evaluation of cultural understanding.
In this paper, we address how to train and evaluate culturally inclusive multilingual MILMs, using limited open-source resources, tackling four major challenges (Yu et al. 2022): 1) Data scarcity: high-quality multilingual multimodal data is scarce, especially in low-resource languages, making it difficult to create large-scale training data; 2) Cultural nuances: visual interpretations are context- dependent and vary across cultures (Ramaswamy et al. 2023 Khanuja et al. 2024): 3) Catastrophic forgetting: training on many languages or modalities often results in suboptimal performance some subsets and require careful balancing: 4) Evaluation complexity: substantial resources and expertise are required accurately measure performance across languages and cultures.
To tackle these challenges, we introduce PANGEA, an open-source multilingual MLLM designed to bridge linguistic and cultural gaps in visual understanding tasks. PANGEA is trained on PANGEAINS (Figure 2). a high-quality multilingual multimodal instruction tuning dataset comprising 6 million samples in 39 typologically diverse languages. PANGEAINS combines existing open-source re- sources with newly created instructions focused on multicultural understanding. We curate high- quality English instructions, carefully translate and adapt them for multilingual contexts. To address Western-centric biases in visual representations, we source images from LAION-Multi (Schuhmann al. 2022), which includes images from various countries and captions in multiple languages. However, LAION-Mull contains images that are not culturally representative of the country's 's speak- ing population, and the associated alt text is often short, noisy, and lacks sufficient detail. To com- bat these issues, we develop multicultural multilingual multimodal instruction generation pipeline, leveraging an LLM (Dubey et al., 2024) to score and filter images based on cultural informativeness. We then enhance the remaining data by generating detailed descriptions and creating complex in- structions that combine culturally relevant tasks with general multilingual scenarios. This approach improves the model's cultural understanding while maintaining robust multilingual performance.
To evaluate PANGEA'S 's capabilities, we present PANGEA BENCH, a comprehensive multilingual and multimodal evaluation suite comprising five multimodal and three text-based tasks across 14 datasets in 47 languages. PANGEABENCI assesses MLLMS performance on open-domain multimodal chat, image captioning, cultural understanding, multimodal reasoning, and text-only tasks including ques- tion answering and complex math reasoning. A key highlight of LANGEABENCH, is the introduction of xChat, human- craffted benchmark designed evaluate open-ended, information-sedd seeking mul- timodal conversations. xChat employs fine-grained evaluation pipeline where human annotators annotate both reference answers and scoring rubrics for each query. An L.M then uses these rubrics to score the model's 's predictions 1-5 scale. This approach offers more precise assessment of MLLM performance, addressing limitations of coarse LLM-as-Judge methods (Zheng et al. 2023). Additionally, introduce xMMMU, translated version of MMMU (Yue et al. 2024a), testing college-level multimodal reasoning across seven languages. Together, these components provide a robust, nuanced evaluation of PANGEA'S 's cross-ling lingull and cross-cultural capabilities.
Our results demonstrate PANGEA'S 's abilities in both English and multilingual scenarios, significantly outperforming existing open-source MI.LMs on PANGRA BENCH, surpassing the best open MILMM
2

---

Published as a conference paper at ICLR 2025
Pangealns: 6M Multilingual Multimodal Instructions for 39 Languages
General (3,4))
Doc/Chart (325K)
Cultural (1.5M)
LViS-InstructVV (350K)
ALLaVA-4V (1.3M)
$ ChariQ (45K)
Doc-VQA-Fr
LAION-Culture (15M))
Cultural Samples 1.M (20%))
Cambrian (IM)
Llava-Mulillnggal (250K)
Viet-Doc-VOA (116K)
Toble-VQA-Fr (168)
Domain-spec (60K)
Llava-Med Zh (60K)
English 2.5M (60%)
Multi 3.7M (605)
Translation 12M (199))
PALO (100K)
Viet-OCR-VQA (137K)
ShareGiTT-oe (203K)
Text Only (820K)
Open-scurce (17%5)
CQA-ru (40K)
Caption (100K)
Code-Fredback (50K)
MIVQA (6K)
+ LAION-GITAY (17K)
NuminaMath (150K)
NEV (58K)
Captions (33K)
Openi (620K)
Distribution of Multilingual Tra Samples
500000
Sampless
Latin Cyrillic
30000
Arabic
Logegraphic
230000
Drovidian
Others
Language
Figure 2: Statistics of PANGEAINS, comprising 6M multimodal instructions in 39 languages. The distribution of multilingual training data shows the percent of instances for each language among the multilingual instructions. PANGEAINS includes general instructions, document and chart question answering, captioning, dppain-specific, culturally relevant, and text-only instructions.
by 0.4% on English tasks and 10.9% on multilingual tasks on average. Notably, PANGEA excels in multilingual and multicultural understanding, evidenced by its performance xChat, CVQA, and MaRVL benchmarks. PANGEA also matches or outperforms state-of-the-ar proprietary LLMs, namely Deminni-155-PP and GPT4o, several tasks such as XGQA. However, some performance gaps remain in multimodal chat and complex reasoning, shedding light on the need for further im- provements in open MLLMs. We discuss key insights from trainingPANGEA, including the scaling effect of instructions, the role of English data, the impact of language-specific training proportions, and preliminary methods improve multilingual OCR. We fully open-souree PANGEAINS, PANGE- ABENCH, PANGEA-7B. and code, to advance culturally inclusive MLLMs across diverse languages.
PANGEAINS: MULTILINGUAL MULTIMODAL INSTRUCTION TUNING
Creating truly multilingual, multicultural MLLM presents unique challenges. We developed PANGEAINS, a diverse and high-quality instruction tuning dataset. Comprising 6 million samples in 39 languages, PANGEAINS was curated with focus on linguistic and cultural diversity. We imple- mented three key strategies ensure comprehensive coverage, each addressing the specific hurdles encountered in multilingual multimodal learning. Figure 2 shows the distribution of PANGEAINS.
2.1 MACHINE TRANSLATED INSTRUCTIONS
To address the scarcity of human- annotated multilingual multimodal data, we primarily adopt ma- chine translation as practical and scalable solution extend data beyond English. While human annotation is ideal, it is resource-intensiv and impractical to cover a wide range of languages.
Constructing a High-quality Pool of English Instructions from Existing Sources. We first collect a high qushttuulityy set of English multimodal instructions, which serve as the foundation for translation into other languages. These instructions span wide range of visual understanding tasks, including general visual instructions and conversations (Tong et al. 2024 Liu et al., 2024). visual reasoning, captioning, and chart question answering (Masry et al. 2022). Besides, we also added text-only high-quality English instructions, covering general instructions TTeknium 2023), code (Zheng al., 2024), and math (Li et al.) 2024c). Figure 2) shows the statistics of our translated datasets. By leveraging existing English instructions, we ensured comprehensive coverage of visual interpretation and text instruction following tasks in English, preparing a pool of high-quality data for translation.
Translation Model Selection. To expand the English instructions to other languages, initially experimented with strong open-source machine translation models, such as NLLB-3B (NLLB Team 2024). However, we found that these models struggled with complex ietruction-followweee scenar-
3

---

Published as a conference paper at ICLR 2025
LAION-Multi
Houristic Filtering
LLM Scoring
Data Generation
Instruction Response
Imagee Size (224-4006pz)
Informativeness
Recoption
Pare the Entooingaai turtrr
Plasse descctoe thwurtage datti Himgunge The emage might bo relatee!
Langh (B-5000icher)
soaleffom lu bused on it
Aspect Ratio (3 20-2--33
quadity is decosboii tne truge...
oo the joorress The right de
NSFW Contem (Jkrliklly)
3 Topic Claseity
ralooe to joshagory. preriaso
Image proct
Oftensive lest
cesed onits contast. Chooss
Instruction Ganeration
cadipl
from the fralnsnn categoiee...
CUP Score (+0.3))
a Country Classity
pall desedd un the absse conters o'ac
Language Batonce
Dheaide whthe netatedd
pappectic country's culuue...
smage. Choose moo tast erom the Vit
nelow puitle reoottng process...
Figure 3: Overview of multicultural understanding instructions data generation pipeline.
ios and context-switching tasks, particularly in specialized domains like code generation and math- ematical reasoning. For example, in code-related tasks, the model failed to recognize and correctly translate programming language keywords, significantly reducing the quality of the instructions. Based on these limitations, we shifted to using the proprietary Gemini 1.5 Pro model, which shows slightly better performance in small-scale human evaluations compared with GPT4o.
Post-Prooe post-Processing Translated Data. Even with high-quality translations, inconsistencies arose. To resolve issues such as mismatched conversation turns or missing candidates multiple-choice ques- tions, we developed a post-processing pipeline. This pipeline automatically corrected these errors or directly dropped the examples, ensuring that all translated instructions remained consistent.
Overall, Gemini 1.5 Pro's translation seems satisfactory, providing fast, cost-effective alternative to human annotation, especially for scaling across languages. However, we acknowledge that machine translation still has limitations, particularly in handling nuanced contexts and cultural subtleties.
2.2 MULTICULTURAL UNDERSTANDING INSTRUCTIONS
While machine translation enables scaling across multiple languages, data translated from English is still Anglo-centric in coverage of cultural concepts (Yu al. 2022). To address this, we developed a pipeline focused on creating instructions for multicultural understanding. Both visual and textual elements can convey deep cultural significance, and our goal is to design a dataset that allows models to not only recognize these nuances but also respond appropriately across various cultural contexts. The pipeline of creating multicultural understanding instructions shown in Figure 3
Curation of Culturally Diverse Images. To ensure that our dataset captures wide array of cultural contexts, we began by sampling 10 million images from the LAION-Multi dataset (Schuhmann) et al. 2022), which includes images and short alt texts from diverse languages and regions. A filtering process was proposed to guarantee both the quality and cultural relevance of the images.
Heuristic Filtering: We implemented automatic filtering based on several key criteria: Image Size, Aspect Ratio, Text Length, NSFW content, Offensive Text, Deduplication, and CLIP Score (used to assess the alignment between the image and its textual description). This helped remove low- quality or inappropriate images and ensured the remaining dataset adhered to quality standards.
LLM Scoring: To further refine the dataset, we employed the Llama-3. --8--Instruct model (Dubey jtt al. 2024) to evaluate the quality, subjects, and cultural relevance of the accompanying text de- scriptions (alt text) for each image. The model was instructed to perform the following tasks: 1) Evaluate Text Quality: The alt text was rated on a scale from to 5 based on how well described the corresponding image, assuming the model could not access the image itself. Alt text scoring below 4 was removed. 2) Subject Classification: The model assigned a subject or category to the alt text based on its content. 3) Country/Region Classification: The model deter- mined whether the alt text was closely related to a specific country's culture. Images classified as "no specific country" (approximately 60% of the dataset) were excluded to ensure we focused on culturally identifiable content. The full LLM scoring prompt is included in Appendix B
Avoiding Overrepresentatiin:: To maintain a balanced representation, we downsampled images from frequently occurring subjects, such as objects, materials, and clothing, to avoid skewing the dataset toward specific topics or regions. Then, we conducted an accessibility check, removing 30% of the remaining samples due to image download or other issues. Ultimately, we curated a final set of 1M high-quality, culturally specific images, forming the foundation of PA PANGRAIN INS.
4

---

Published as a conference paper at ICLR 2025
Captioning Images with Different Languages. To provide context and enhance the model's ability to interpret images, we regenerated more detailed captions using Gemini 1.5 Pro based on high- quality alt texts. In this step, each image was accompanied by caption written in the language corresponding to its cultural origin. However, our approach was not just about using a capable model. The alt text played critical role in enriching the data, it often contained culturally specific and contextually important information that would otherwise be absent from the images alone. For example, in Figure 8 with high-quality alt text, models can incorporate details such as "President and CEO of The Walt Disney Company" and "a model of Shanghai Disneyland." " adding significant context that may not be immediately evident from the image. This additional layer of information helps the model generate captions that better capture the cultural and contextual nuances.
Generating Multilingual and Cross-Cultural Instructions.
After recaptioning, we generated multilingual instructions based on the detailed captions with Gem- ini 1.5 Pro. Instead of only prompting the model to generate random instructions, we did a careful prompt engineering where we first came up with 13 task types (e.g.. Information Seeking, Coding & Debugging, Critical Reasoning. Cultural Interpretation, etc.). Then for each image, up to two QA pairs were created, representing different instruction types ensure a diverse set of interactions. This approach ensures that the model not only recognizes these visual elements but also responds appropriately across varied linguistic and different instruction contexts. The captioning and instruc- tion generation prompts are included in Appendix
2.3 CURATING EXISTING MULTILINGUAL INSTRUCTIONS
To further enrich PANGEAINS, we conducted extensive survey of available multilingual multi- modal literature and datasets, including those hosted on HuggingFace. As result, we incorpo- rated several high-quality, open-source datasets into INSS These include Chinese AL 4V (Chen et al., (024)). Viet Document and OCR QA (Doan et al. 2024), Llava Chinese (LinkSoul- 20.3),3) Llava Medical Chinese Instruction (BUAA 2023), LLaVA-Japanese -nntteepeeeee (Tshh4456]
2023), MTVQA (Tang et al. 2024). Japanese STAIR Captions Yoshikawa et al.) (017). Russian
GQA (Belopolskih & Spirin, 2024). French Doe- VQA (Sonagu & Sola (024), and French Table-
VQA Agonnoude & Delestre 2024). Each of these datasets brings unique linguistic and cultural
perspectives the mix, covering wide range of languages and task types.
2.4 DATASET STATISTICS
By combining these three methods, we created PANGEAINS, a comprehensive dataset addressing major challenges in building multilingual MILMs: data scarcity, linguistic diversity, and cultural nuance. Its balanced language and task distribution supports the development of more sophisticated LLMs that can handle complex visual and textual content in multilingual, multicultural context.
PANGEAINS features an extensive and balanced distribution of languages, tasks, and cultural contexts (as shown in Figure 2}. We empirically keep the final lan- guage ratio of English to Multilingual as 40%:60% as we found significant portion of English data plays an important role in cross-lingual transfer. See more discussions about the ratio in sscciinn 5) and FFigure 6] The inclusion of diverse multimodal instructions ensures that the model develops deeper understanding of varied linguistic and cultural environments. Examples of training samples from different languages and categories are provided in Appendix H The comprehensive nature of PANGEA A INS lays a solid foundation for training PANGEA, enabling it to become a truly multilingual, multicultural multimodal LLM, capable of understanding and interacting effectively with users from diverse linguistic and cultural backgrounds.
Language and Task Distribution:
3 PANGEABENCH: EVALUATION OF MULTILINGUAL MULTIMODAL MODELS
3.1 OVERVIEW OF PANGEABENCI
To assess the capabilities of PANGEA across variety of languages, cultures, and task types, have developed PANGEABENCH, a comprehensive multilingual and multimodal evaluation suite. PANGFABENCH integrates diverse benchmarks that encompass both multimodal and text-only
5

---

Published as a conference paper at ICLR 2025
Multimodal Chat
Captioning XM9600!
Cultural Uncerstanding Dataiet Marv))
Shart VQJA
Reasoning
Dhraeet SChaedennn Mellesfanch
Dotuent MarM
arans
3g g22 44
whtt co nauts 10 prahh
proforence??
for the provided image Japanese
Provide on one ttence Jaption
82 Whre is the twre fur it
Singah ambeng It
caleli (Potel capeid
maddle the recely
[Ji whlice are muss srs bustests
is smenty Quralde Dutdoors
Q: Hocvecl kubusse word benody
ne 48-55-) que
ggaic --mm pirt, advdd UC peephet ofherrn gortd
herse une hepplle
SWote contsiver?
1A: A small box with sliver decorations on the lable))
C. (ersssy mand
0. lolung beua beokn
(t: wi atreet toursoo with ite colicc fomall foreo///4 furoad)
(A)120 (nj1n6 (53120 (0j116
Category
Tasks
Datasets
Forms
Size
Languages
Metric
Multinodal Chat
xChatBeuch
Long
400
LLM-as-Jude Ju-Je
M-LlavaBench
Long
600
ar ww. ja..ar..iij.............
LLM-as-Judge
Captioning
XMI00
Long
3.6K
36 langunges
ROUGE-L
Multinedal
Cultural
CVQA
MC
21K
en,zh .0.2h.u. mo. jtm.j.dddjjj....... dd.ijvn
Accuracy
Understanding
MaRVI.
Short
6K
id.sw.ta.tr.hh
Accuracy
Multilingual
xGQA
Short
77K
en.de.pt pt. bn.
Accuracy
VQA
MaXM
MC
2K
wt.th.zh.fr.en.iv....
Accuracy
Reasoning
XMMMU
Short/MC
3K
cn.ar, fir bi.dd..
Accuracy
(Multi-subject)
M3Exam
MC
3K
en.zh..i.tt,
Accuracy
QA
TyDiQA
Short
5.1K
Accuracy
Translation
FLORES Suh Long
18K
ChrF
Text-only
Reasoning (Multi-subject,
MMMLU
MC
197K
un un, de,es, es ft..iiiiiiiiiiii.. wdw.........mmm.u......... wtw..eeees.f..j..ss......
Accuracy
XStoryCloze
MC
21K
Accuracy
Commonsense, Math)
MGSM
Open
3K
Accuracy
Figure 4: Overview of PANGEABENCH, which contains 5 multimodal and 3 text tasks covering 14 datasets (including two newly curated xChatBench and XMMMU datasets). The table provides details about the datasets, while the figure shows evaluation examples from five different multimodal eval tasks in our PANGEABENCH.
tasks, enabling holistic evaluation of PANGEA'S performance in cross-lingual, cross-cultural, and multimodal contexts. Each task within PANGEABENCH is designed to probe specific aspects of PANGEA's 's proficiency, ensuring robust testing across wide range of scenarios. All tasks in PANGE- ABENCH are evaluated under a zero-shot setting.
3.2
MULTIMODAL TASKS
The multimodal tasks in PANGEABENCH are categorized as follows: Multimodal Chat, Caption- ing, Cultural Understanding, Multilingual Visual Question Answering (VQA), and Multi-Subjer Reasoning. We incorporate these in PANGEABENCE to ensure comprehensive testing of PANGEA'S multimodal capabilities. The overview and examples of PANGEABENCE are shown in Figure
Multimodal Chat. This task tests the model's ability to engage in natural and dynamic rorl-world
conversations involving both text and images. Multilingual LlavaBench (Rasheed et al.) 2025) (M-
LlavaBench for short) stands as the only benchmark for evaluating multilingual long-form genera- tion capabilities from MLLMs. Following the evaluation pipeline from Zheng et al. (0023) and Liu et al. (2023b), M-LlavaBench uses a coarse- -ranined evaluation criteria (e.g.. "Please rate the help- fulness, relevance, accuracy, level of details of their responses."). "). Previous works suggest that em- ploying such coarse-grained evaluation criteria may lead to automatic evaluation results that diverge from how humans would evaluate them (Ye et al.. 2023] Kim al. 2023] [ee et al., 2024] Kim al.) b0)4alb). To assess baselines with more accurate evaluation pipeline with fine-graines grained evaluation criteria on diverse scenarios, we additionally annotate a new multilingual multimodal generation benchmark called the xChatBench, included in the multimodal chat category of PANGEABENCH. A more detailed explanation of the annotation process of xChatBench is included in Appendix E
Captioning. The XM3600 (Thapliyal al. 2022) dataset developed to evaluate models' pability in multilingual image captioning. It contains images paired with captions in 36 different languages. However, it includes many similar images and captions. To address this, we clustered

---

Published as a conference paper at ICLR 2025
the images based captions and manually selected 100 representative images (denoted XM100).
This approach enhances the diversity of the samples and accelerates the evaluation process.
Cultural Understanding. To assess the model's ability to reason about and understand culturally diverse visual content, we use the CVQA (Romero et al. 2024) and MaRVL (Liu et al., (02T) datasets. These datasets designed to test the model's performance in reasoning tasks involving culturally relevant imagery and concepts across multiple languages.
Multilingual VQA. This task measures the model's proficiency in answering questions about im- ages across multiple languages. The XGQA (Pfeiffer et al. (2022b) and MaXM (Changpinyo et al.) 2022) datasets provide a diverse range of visual question-answerrngg challenges in several languages and scripts, addressing rross-lingual visual understanding.
Multi-Subject Reasoning. The XMMMU and M3Exam (Zhang et 2023
datasets are used to evaluate the model's 's reasoning abilities across different academic subjects. XMMMU is a machine- translated version of MMMU validation questions, which focuses on multimodal reasoning multi- ple subjects. We randomly sample 300 questions from MMMU (Yue et al. (2024)) validation set and employ GPT-4o for the six languages translation. M3Exam challenges the model with real-world educational questions requiring both textual and visual comprehension. Details on how we ensure the translation quality, as well as detailed descriptions of other datasets, can be found in in/ppendix
3.3 TEXT-ONLY MULTILINGUAL DATASETS
While multimodal tasks are critical for evaluating the holistic capabilities of models like PANGEA, text-only multilingual tasks provide equally essential dimension assess. Most existing multi- modal evaluations tend to overlook the importance of text-only evaluation, especially across diverse languages. Including text-only tasks in PANGEABENCI allows us examine whether the model can perform well in scenarios that require deep linguistic understanding without the aid of visual con- text, highlighting its performance as foundation model. We include three tasks QA, Translation, and Reasoning covering five datasets for the text-only evaluations in PANGEABENCH,
Specifically, we include TydiQA (Clark et al.) 2020) test the model's ability to answer questions across 11 typologically diverse languages. We adopt the FLORES (NLLB Team 2024 dataset to assess machine translation performance. We sample 11 languages (denoted as FLORES-SuL
We use MMMLU (OpenAl 2024a), human-translated version of MMLU to test the general language understanding We use XStoryCloze (Lin et al., (02T) and MGSM (Shi et al., 2022) to test the model's 's commonsense and mathematical reasoning ability in multilingual contexts respectively.
Stages
Pretraining
Finetuning
EXPERIMENTS
Training Data
4.1 EXPERIMENTAL SETUP
Dataset
LLaVA LCS-558K
PANGEAINS 6M
#Samples
558K
We train PANGEA on PANGEAINS, our multilingual multimodal dataset compris- ing 6 million samples across 39 languages. The model uses LLa VA-Nett as archi- tecture (Liu et al. 2024). Qwen2-7B- Instruct (Yang et al. 2024) as the lan- guage model backbone and clip-vit-large- patch14-336 (Radford et al. 2021) the vision encoder. The training consists of two stages. First, we pretrain the vision- language connector that aligns the outputs of vision encoder backbone, with the LLaVA LCS-558K LLu et al. 2023b a). Then, we perform finetuning on PANGEAINS, where we employ learning rate of 2e-5, a batch size of 512, coupled with a cosine decay schedule with 0.03 warmup steps. We pretrain and finetune the model for epoch, where pretraining took 4 hours with 8 H100 (32 GPU hours), and finetuning took 168 hours with H100 (1344 GPU hours).
Model
Trainable
Projector (20M)
Full Model (8B)
Training
Batch Size
128 I x 10-3
128
LR: Rivision
2 x 10-6
LR: (ppaj, (LLM) Epoch
1 x 10-3
10
1
GPU Hours (H100)
32
1344
Table PANGBA's 's training configurations.
https:///uugginnffccc..oo// www.eettttttuuhhttaiiiiia

---

Published as a conference paper at ICLR 2025
Multimodal Chat
Cultural Understanding
AVG (all)
Models
XChatBench
M-LlavaBench
CVQA
MaRVL
en
mul
en
mul
en
mul
en
mul
en
mul
Gemini-1.5-PP
67.1
62.5
67.0
54.4
103.4
106.6
75.9
75.7
76.4
72.0
GPT4o
68.6
64.6
71.0
64.4
104.6
100.4
79.1
79.4
81.4
82.1
Llava-1.5-7B
45.4
28.4
28.5
11.8
66.1
40.8
48.9
36.5
53.7
Llava-Next-7B
51.1
32.7
40.5
18.9
78.9
50.7
55.7
42.6
62.8
50.9
Phi-3.5-Vision
54.0
35.0
38.5
13.2
70.8
58.0
56.3
42.3
72.1
56.5
Cambrian-8B
50.9
36.4
27.5
11.3
78.4
61.8
59.7
47.5
75.4
61.8
Llava-Ov-7B
59.5
41.3
51.0
28.5
89.7
55.3
65.2
53.7
72.7
57.5
Molmo-7Ba
55.4
34.1
21.1
13.8
59.4
48.3
54.9
Llama3 2-11B
57.2
41.9
49.0
27.8
93.9
58.2
70.2
61.4
64.5
58.1
PaliGemma-3B
37.3
25.8
3.5
32.1
31.9
52.9
42.9
56.5
52.2
PALO-7B
46.3
32.2
27.0
11.8
68.9
71.2
50,9
39.2
63.3
54.2
mBLIP mTO-XL
35.1
29.8
2.5
0.5
32.7
28.2
40.5
37.5
67.3
66.7
mBLIP BLOOM7.
36.1
30.0
4.0
1.6
43.5
41.0
44.9
36.9
62.3
58.6
PANGEA-7B (Ours)
59.9
52.8
46.0
35,8
84.2
89.5
64.4
57.2
87.0
79.0
A over SoTA Open
+0.4
+10.9
-3.5
+7.3
-11.7
+18.3
-5.8
-4.2
+11.6
+12.3
Captioning
Short VQA
Multi-subject Reasoning
Models
XM100
XGQA
MaXM
XMMMU
M3Exam
en
mul
mul
mul
mul
mul
Gemini-1.5-Pro
27.6
19.1
54.2
48.7
56.4
63.5
65.8
57.7
77.4
64.7
GPT4o
27.7
19.1
55.8
51.0
60.7
65.4
69.1
58.3
68.0
61.0
Llava-1,5-7B
28.6
1.1
62.0
30.6
49.8
20.4
36.2
31.5
32.3
29
Llava-Next-7B
29.3
9.4
64.8
37.8
54.9
21.4
36.7
34.3
36.5
28.4
Phi-3.5-Vision
30.2
5.2
38.4
55.3
25.0
42.6
38.8
55.8
37.2
Cambrian-8B
20.6
9.9
64.6
39.8
55.3
28.7
41.8
33.2
34.7
33.4
Llava-Ov-7B
30.6
7.0
64.4
48.2
54.9
34.8
46.3
41.0
60.4
45.8
Molmo- 7B-D
22.1
9.1
51.5
43.0
52.9
37.5
44.5
40.4
57.1
39.1
BBB
27.6
4.5
55.6
45.4
55.3
43.9
46.5
41.4
51.8
36.6
PaliGemma-3B
18.7
0.8
59.7
30.5
47.9
19.9
26.3
25.2
36.0
25.6
PALO-7B
30.4
0.8
60.5
37.8
51.4
16.3
33.1
30.5
30.8
27.8
mBLIP mTO-XL
31.9
39.9
44.7
36.8
29.3
30.4
25
mBLIP BLOOM7.
22.5
10.3
43.3
36.9
44.7
24.8
29.2
30.8
30.3
29.5
PANGEA-7B (Ours)
30.4
14.2
60.2
55.3
53.3
45.7
43.7
42.1
over Best Open Model
-0.2
+3.99
-0.1
+12.0
0.0
+9.4
-0.8
+2.3
+1.0
-3.7
Table 2: Overall performance on the multilingual multimodal benchmarks in PANGEABENCH. The best-performing open model on each dataset bold and the second best underlined.
For evaluation, we compare PANGEA against several state-of-the-art open source baselines, includ- ing English-centric models Llava-1.5-7B (Liu et al. 2022a)). Llava-Next (Liu et al., (024)), Phi- 3.5-Vision (Abdin et al., 2024).
Cambrian-88 (oog et al. 2024). Llava-OV-7B (Li al. 2024b),
Molmo-7B-D (Deitke et al.
2024)
Llama3.2- 1B (Dubey et al. 2024) and multilingual models
PaliGemma-3B (Beyer et al. 2024).
PALO-7B (Rasheed et al.] 2025). mBLIP mTO-XL and mBLIP
BLOOMZ (Geigle et al.) (024b).
We also consider two text-only LLMs baselines Vicuna-1.5- 7B (Zheng et al. 2023) and Qwen2-7B-Instruc (Yang et al., 2024). which are the backbones of Llava-Next and our PANGEA respectively. We integrate our multimodal tasks in PANGEABENCI into lmms evall (Li et al. 2024a), multimodal evaluation package that supports many English multimodal benchmarks. We use Im-evaluat www.llatiinnhaaannn on-harness (Biderman et al.) (2024) to evaluate text-only tasks. We follow the original paper for their best models' prompts different tasks, and mostly reproduce their original numbers on datasets reported in the original papers.
4.2 MULTILINGUAL MULTIMODAL RESULTS
The results in Table 2 provide clear insights into the strengths and remaining challenges of PANGEA- 7B in multilingual and multimodal tasks. Key observations from the evaluation include:
8

---

Published as a conference paper at ICLR 2025
AVG (all)
FLORES Sub
TyDiQA
XStoryCloze
MGSM
MMMLU
Models
en
mul
X->en
en Xx
en
mul
en
mul
en
mul
en
mul
Vieuna-1.55 1.5-BB
52.1
38.7
55.6
42.4
59.7
52.7
78.1
57.4
17.6
6,4
49.5
34.7
Owen2-7B-IIstrre
66.6
54.5
61.8
46.0
72.2
71.2
80.3
61.9
48.8
40.4
70.1
53.1
Alava-1.5-78
53.1
39.0
54.7
41.5
66.8
52.8
79.1
57.6
14.8
7.6
50.2
35.7
Llava-Next-7B
54.0
38.9
54.8
41.4
68.3
52.1
79.1
57.1
15.6
7.5
52,1
36.5
Phi-3.5-Vision
60.7
41.7
28.5
32.5
75.9
51.3
77.9
54.8
59.2
33.1
62.0
36.7
PAL.07BB
52.0
37.5
52.9
40.4
69.4
50.8
77.4
57.2
13.6
5.8
46.7
33.4
PANGEA-7B (Ours)
72.8
54.3
60.7
44.9
66.0
79.1
61.2
82.0
47.4
68.4
52.2
Table 3: Overall performance on text-only multilingual benchmarks in PANGEABENGE
Superior English and Multilingual Performance: PANGEA-7B outperforms existing open-source models across both English and multilingual tasks. While concurrent multimodal models such as Molmo (Deitke et al. 2024 Llama 3.2 show strong performance English datasets, they strug- gle in multilingual evaluation settings. Particularly in multilingual subsets like xChatBench, M- LlavaBench, and MaRVI., it has achieved substantial gains, highlighting its effectiveness in both cross-lingual and cross-cultural contexts.
Balanced Cross-Language Capabilities: Unlike many models that exhibit significant drop in performance when moving from English to multilingual tasks, PANGEA-7B is relatively consistent. For instance, in Multimodal Chat tasks, the performance gap between English and multilingual remains relatively small, indicating its ability to handle multiple languages effectively.
Challenges Compared to Proprictary Models: While PANGEA-7B leads in open-source mod- els. some gaps remain when compared to closed-source models like GPT4o. Additionally, though PANGEA-7B narrows the gap between English and multilingual performance, there still room for improvement in fully closing this divide across all tasks.
4.3 MUL.TIINGUAL TEXT-ONLY RESULTTS
We further evaluate our model in text-only scenarios in Table 3 Interesting findings include:
Best Text Performance Among Multimodal LLMs: PANGEA-7B 7B demonstrates the strongest per- formance among all multimodal LLMs the text-only tasks consistently outperforming baselines like Llava-Next-7B. This highlights that, despite being trained multimodal model, PANGEA-7B maintains superior text understanding and reasoning capabilities compared to other MLLMs.
Maintained Performance from its Text Backbone. PANGEA-7B generally maintains sees slight drops in performance on most text-only benchmarks compared with its text backhone Qwen2-7B- Instruct. Notably, the model shows significant improvement in MGSM. This improvement is directly attributable to the inclusion of math-related instructions in PANGEAINS, which enhances the model's capability to handle complex multilingual reasoning and mathematical tasks.
DISCUSSION
Finally, we explore implications of our findings and their potential impact on future developments in the field. We examine the scaling effects of instruction quantity, the persistent role of English data, the relationship between training sample proportions and performance. Through this discussion, we aim to provide a comprehensive understanding of our model and chart a course for future advance- ments. More discussion on qualitative examples of model behavior on multilingual multimodal chat and challenges in multilingual OCR can be found in Appendix and Appendix
Scaling Effect of Number of Instructions. Understanding how the quantity of instructions affects model performance is crucial for optimizing training strategies and resource allocation. Figure 5) reveals a clear scaling effect related to the number of instructions used during training. Perfor- mance improvements
increased the number of multilingual instructions in PANGEAINS, for both English and multilingual performance. This demonstrates the necessity of scaling multilingual multimodal instruction tuning.
consistent

---

Published as a conference paper at ICLR 2025
XGQA
XMMMU
MaXM
60
Accuracy
Accuracy
65
Accurary
-4
Accuracy
52.5
50
63
50.0
English
42
Muitilinuual
47.5
2
Dataset Size (M)
Datasel Size (M)
Datasel Size (M)
Datasel Size (M)
Figure 5: Scaling effect of training samples on English and multilingual scores across datasets.
Role of English Data. In multilingual scenarios, English data plays a pivotal role in cross-ling lingual transfer. To investigate this, we sampled 500K examples from the translated data described in sub- section 2.1] ensuring consistent data distribution. We varied the ratio of English data while keeping the total number of training samples fixed at 500K. For the 17 multilingual languages in the trans- lated subset, we evenly distributed the number of samples across languages.
As shown in Figure 6], English per- formance generally improves the percent of English data increases. Surprisingly, using only multilingual data results in relatively lower mul-
English Performance
Multilingual Performance
Accuracy (AVG)
52.5
37.5
36.7
385
50.0
$1.0
51.6
50.7
51.
35.2
tilingual performance. As we intro-
47.5
duce more English data, multilingual performance improves, peaking at 38.7% with 40% English. However, performance drops sharply when En- glish data reaches 100%. This suggests that English data aids cross-lingual transfer, however, over- reliance on harms multilingual performance.
40
80
100
40
80 100
Percentage of English (16)
Percentage of English (%6
Figure 6: Impact of English training data proportion on En- glish vs. multilingual performance.
How does the proportion of training samples in language affect downstream performance? Is downstream task performance correlated with the number of training samples? Our analysis in Fig- ure reveeled the relationship between training sample proportion and downstream performance. While there is a general positive correlation, the impact varies significantly across languages and tasks. For widely spoken languages with rich resources, we observed near-linear relationship. However, for low-resource languages, even a small increase in proportion yielded disproportion- ately large performance gains. Interestingly, typologically similar languages, These findings suggest that strategic allocation of training samples, considering both language prevalence and linguistic similarities, can optimize model performance.
also noted instances of positive transfer between
CVQA
XGQA
XMMMU
MARVL
Perf. to EN (%)
100
100
94
80
90
SW
60
90
80
20
5
10
15
15
10
Train Size English (%)
Train Size to English (9%)
Train Size to English (%)
Train Size to English (96)
Figure 7: The relationship between training sample size (relative to English) and performance (rel- ative to English) of different languages across four datasets.
CONCLUSION
In this paper, we introduced PANGEA, multilingual MILMM designed to bridge linguistic and cul- tural gaps in visual understanding tasks. By leveraging PANGEAINS, our newly curated 6M mul- tilingual multimodal instruction data samples, we demonstrated significant improvements cross- lingual and cross-cultural understanding across 39 typologically diverse languages. Our compre- hensive evaluation using PANGEABENCE revealed PANGEA'S 's superior performance compared to existing open-source models. We also highlight ongoing challenges in areas such low-resource language support and multilingual OCR. We fully open-source PANGEA, PANGEAINS, and PANGE- BENCH facilitate future research to build open and inclusive MI..Ms.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by a Carnegie Bosch Institute Fellowship to Xiang Yue, as well grants from DSTA Singapore, and the Programs for Bridging the gap between R&D and the IDeal society (society 5.0) and Generating Economic and social value (BRIDGE)Practical Global Research in the AI Roboties Services, implemented by the Cabinet Office, Government of Japan. The training is supported by the CMU FLAME Center. The authors would like to thank Google Gemini credits for data construction and evaluation. The authors would also like thank CMU NeuLab colleagues for their constructive comments.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language language model locally on your phone. ArXiv preprint, abs/2404.14219, 2024. [ttps:ts:://arxxtt www.wvorrgggaae abs 2404.
Tom Agonnoude and Cyrile Delestre. Table vqa dataset, 2024. URL htt.htttt//:hginggteette
co/ httttaastts/maakkkaaaaaae taaaakkeaaatbaevaaaaaa
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization the era of commercial language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proc. of EMNLP, pp. 9904-9923, 2023.
org/ 2023. emnlp-maiiee in. n. 614.611666 pdf
Badr AlKhamissi, Muhammad EINokrashy, Mai Alkhamissi, and Mona Diab. Investigating cul- tural alignment of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proc. ACL, pp. 12404-12422, 2024. URL/tttsst //cc//acllccaa htt.tppss//aallaaaccloo.oooooo www.aanooooo..r///000000
ac oog. ang.671.pd1.
Daniil Belopolskih and Egor Spirin. Gqa-ru, 2024. URL httpss//huggiggaace.co
datasets/deepvk/ ru
Lucas Beyer, Andreas Steiner, André Susamo Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Grit- senko, Neil Houlsby, Manoj Kumar, Keran Rong. Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Bal- azevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3B VLM for transfer. ArXiv preprint, 0242240700726, 2024. URL /ttp:::t://a hbtp://bb/224000 07726
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Al- ham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches reproducible evaluation of language models. ArXiv preprint, 147722, 2024.
httpaa////2222.2 14782] 2240b.1282221
Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. Systematic inequalities in language technology performance across the world's languages. In Proc. of ACL. pp. 5486-5505, Dublin, Ireland, 2022. Association for Computational Linguistics. URI. htt.:tttss///claatttttt
prg/2022.0 ac - 376
BUAA. Chinese 2023. URL https://hogginnacce...//AAAaa Chinese-LLavV-Medd- 7B Accessed: 4024-10-01.
Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish V Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. Maxm: Towards multilingual visual question answering. ArXiv preprint, abs/2209. 005401, 2022.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Hamessing syt4v-sytthesieed data for a lite
model. ArXiv preprint, abs/2402. 2024. URL https:
7/arxiv. //axxiv..org/ab//220.1111
11

---

Published as a conference paper at ICLR 2025
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. ArXiv preprint, has/2305.11565, 2023. URL https:
2305. 10005.18865.. 18565
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: benchmark for ieboommttionseekinng question answering in typologically diverse languages. Transactions of the Association for Compulational Linguistics, 8:454 470, 2020. URL. https: /aap::/aalllatll aatalnnthhooooo wor.tholooy....oo org/ 020 -tacl1l...
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty-seventh Conference on Newral Information Processing Systems, 2023. RRRLttts///ppeeeeeeeet
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Moham- madreza Salchi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. ArXiv preprint, abs/2409.17144 2024. URL httpp: 2ttp://.wwr//aab/24....1222
Khang T Doan, Bao G Huynh, Dung T Hoang. Thue Pham, Nhat Pham, Quan Nguyen. Bang Q Vo, and Suong N Hoang. Vintern lb: An efficient multimodal large language model for viet- namese. ArXiv preprint, 1a//2448.114880 2024. URL http:pp:///aax...ooo/aa/////////
12480
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL httthtttt////xiii org/ abs 27400.1177
Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavaš. mBLIP: Efficient bootstrapping of multilingual vision-I..Ms. In Jing Gu, Tsu-Jui (Ray) Fu, Drew Hudson, Asli Celikyilmaz, and William Wang (eds.), Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), pp. 7-25, Bangkok, Thailand, August 2024a. Association for Computational Linguistics, doi: 10. 18653/v122024.avvr 1.2. URL htLhttps:/// peattps://aclanntclogo. ortptt/oooooooo.o.222..... alvr-1.2
Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavaš. mBLIP: of multilingual vision-LLMs. In Jing Gu, Tsu-Jui (Ray) Fu, Drew Hudson, Asli Celikyilmaz, and William Wang (eds.). Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), pp. 7-25, 2024b. URL https:://aalantoooogy...oo 2.pdf
Efficient bootstrapping
www.20........l11...a.. alvr-1.
Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 20/9, Long Beach, CA, USA, June 16-20, 2019, pp. 5356-5364. Computer Vision Foundation / IEEE, 2019, URL http:::tt////acccccccc thecvf. com www/ccontent__vvP__0I11 A_____AA___ A_A___ataet__eeee__ee__ www.ooooaaaeeeaaalallaaaaaaaaaaa Bertrrrnnnntttneeeeeeeenetnteeee CVPR 2019 papper.htm.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Informa- tion Science, pp. 218-223, 2017. URL https: http:////oc....bbeeeia.a.... www.hhuubbbriii.e... de www.:/eee/eeeeeeeeeeeeeeeeeeeeee core/bitstreamm/ /D0/382333600000 0fb5 2t-.-------0-------31188222228-1
api/
Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Jiwan Chung, Yejin Son, Yejin Choi, and Youngjae Yu. Reading books is great, but not if you are driving! visually grounded reasoning about defeasible commonsense norms. ArXiv preprint. abs/2310.1044 2023. URL https:
//aaaiiv.rg/aab///////// 1041
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proc. of ICLR. OpenRe- view.net, 2021. wtt.htttss://ppneeee
net. fett
12

---

Published as a conference paper at ICLR 2025
Jack Hessel, Ana Marasović, Jena D Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor "understanding" benchmarks from the new yorker caption contest. In Proc. of ACL, pp. 688-714, 2023. :
paclanthoolgge ogy. ott//o/gg..../.oooo ltt.eecoooooo.oor//22...... won...loo...... 41 .1.1.f..f
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. Mural: multimodal, multitask retrieval across languages. ArXiv preprint, abs/2109.05122 125, 2021.
Simran Khanuja, Sathyanarayanan Ramamoorthy, Yueqi Song, and Graham Neubig. An image speaks a thousand words, but can everyone listen? on translating images for cultural relevance. arXiv preprint M4Xv:22004.0244,, 2024.
Geewook Kim, Teakgyu Hong. Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document un- derstanding transformer. In European Conference on Computer Vision (ECCV). 2022. URL
https: /atpps://axxv.../// org/
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine-grained evalua- tion capability in language models. In The Twelfth International Conference on Learning Repre- sentations, 2023. URL http::://///..... / rxx v. org/ /2300 08491
Seungone Kim, Juyoung Suk, Ji Yong Cho. Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Gui- jin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al. The biggen bench: A principled benchmark for fine tmeiindieed evaluation of language models with language models. ArXiv preprint, abs/2406.05761, 2024a. [RLLihtps://arete
ozg/ abs 2406 05761
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. ArXiv preprint, abs/2405.01533 2024b. URL https:tttss:ttttee
org/ abs /205 01535
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
http//as.../..e. 07332
Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo. Prometheusvi- sion: Vision-language model as judge for fine-grained evaluation. abs/2401.06591, 2024. URLpsttss: / aaaaaviivvvvoo....//.aaa org/ abs 22cbl 06591
ArXiv preprint,
Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du Yuhao Dong, Haotian Liu, Yuan- han Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the de- velopment of large multimoal models, 2024a. URL
lab. github.ioi. 1o/
Imma-eval
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024b.
Jia Li, Edward Beeching. Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. http:///ttuubccomm/ppppeeeet-tat 2024c. http/////ma....rrrrrroo///o//nna
co/ AWA.MM///MiiaMaaa.........
aimo- propppprrrrssssreeeeeeeeeeeeeeee wttpr/iiee/ooob//aiiiaaaaa
pdf
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang. Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. Miit: A large-scale dataset towards multi- modal multilingual instruction tuning. arXiv preprint arXiv:2306. 04387, 2023.
Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, et al. Omnicor- pus: A unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406. 08418 2024d.
13

---

Published as a conference paper at ICLR 2025
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' O'Horo, Jeff Wang. Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian L.. Few-shot learning with multilingual language models, 2021. Stt.tttts////xxii/ org/ abs /2112
L LinkSoul-AI. Chinese-llava, 2023. URL hwtt::::wwgggiggfacc..o//seeeeee
Chinese-LLava Accessed: 2024- 10-01.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond
Elliott. Visually grounded reasoning across languages and cultures. In Proc. of EMNLP, pP- 10467-10485, Online and Punta Cana, Dominican Republic, 2021. Association for Computa- tional Linguistics. BRLLtttps:22 htttt:sss//aalanhholoooooooooo httplaatwtoooooooogg/222200 2021. emn momll-mmain 818
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, 2023a. URL http://:///xx./.o///////////////
03744
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b. URL http:::://axxi...r///////e/e///e abs/ 20t..s///32....0 08485
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. URL htttps://arxt::
org/ httpp/df////1//// 13601
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024. URL httpsttt:: /wtp/vaaivvvv//.a////a//a../a.av org/ abs 22110 02255
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, Dublin, Ireland, 2022.
Association for Computational Linguisties. URL https: / accl wwwtthhlo....g.........
Findings-acl.
Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang. Dongdong
Zhang, and Nan Duan. M3P: learning universal representations via multitask multilingual mul- timodal pre-training. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2021, virtual, June 19-25, 2021, pp. 3977-3986. Computer Vision Foundation IEEE, 2021. URL
M3P
Un Lea.ma.ngg_unvveerrell Rtw.eeettttetsesssnsssss.ssssse. vvaonoss_vvviiieeeeeeeee Madia_Muttitas. Multiiinkull
Mu Stlltinn Mhultmmodal_Pree
NLLB Team. Scaling neural machine translation to 200 languages. Nature, 630(8018):841, 2024.
URL /ttttt://///tt nature....... com/artticless//11588888888 020e:///31111110000000333
OpenAl. Mmmlu dataset, 2024a. URL httpa//s//uuggggg.....e//aatttta
MMMLU
Accessed: 2024-10-01.
OpenAl. Hello gpt4-o. http:://wyeenicccoom////eeeeelep 2024b. homm/index/hellogpptee
2024b. URL httttt://ppeeeea
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xGQA: Cross-lingual visual question answering. Findings of the As- sociation for Computational Linguistics: ACL 2022, pp. 2497-2511, Dublin, Ireland, 2022a. Association for Computational Linguistics. URL https: /acl anthelogy orgp2222..000......../
Findings-acl. 196
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xGQA: Cross-lingual visual question answering. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pp. 2497-2511, Dublin, Ireland, 2022b. Association for Computational Linguistics. URL https: / /aaaaaaanntt /atpaaaannnttoo...ooooo wwtpooo.....//22......
findi lending-aal ngs- acl 196
14

---

Published as a conference paper at ICLR 2025
Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia, Xinyi Wang, Machel Reid, and Sebastian Ruder. mmt5: Modular multilingual pre-training solves source language hallucinations. ArXiv preprint, 0tos/2355.1222,, 2023. /tt./////.////a.....r//////////2 14224
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: geographically diverse evaluation dataset for object recognition. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 0011276661371 2023. URL
pdf
Vikram V Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: geographically diverse evaluation dataset for object recognition. Advances in Neural Information Processing Systems, 36, 2024. neur ps wwwpaperpppppp..cc.e fi wetrrrfillsspppeeee wtt.ieeeeeepppe/222//1211 00060000466611119999933333333113
URL https::/proceeddi
pdf
Hanoona Rasheed, Muhammad Maaz, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M. Anwer, Tim Baldwin, Michael Felsberg, and Fahad S. Khan. Palo: A large multilingual multimodal language model. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV 2025), 2025. URL pttps::/arxiv..ot wttpaaxxivvvi.....aaaa org./ abs 2402. 14818
Nils Reimers and Iryna Gurevych. Centence-BERII Sentence embeddings using Siamese BERT- networks. In Proc. of EMNLP, pp. 3982-3992, Hong Kong, China, 2019. Association for Com-
putational Linguistics. URL pttttpss//cclanttooloy..ooo
David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, al. Cvqa: Culturally-divens multilingual visual question answering benchmark. ArXiv preprint, abs/2406.05966, 2024. URL https:/ /ttp///aavvv///////////eeeea.avv wwtp///orggabbbbbbbao/ooo/se
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25299 2022. URL httttttts///xiii http:://xxxvvvoroo abs/ 2210 0.100.1 08402
Bin Shan, Yaqian Han, Weichong Yin, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Emii-unnx2: A unified cross-lingual cross-modal framework for understanding and gen- eration. ArXiv preprint, abs/2211 1,48661,, 2022. URL ltttts://aaxtiiii org/ abs/2211.
04861
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models multi- lingual offin-tf- reasoners. In The Eleventh International Conference on Learning Rep- resentations, 2022.
httpsss://rriivvv www.ii.orr///bae abs 2210. 03057
Loic Sokoudjou Sonagu and Yoann Sola. Doevqa dataset, 2024. URL httt:::://hugiiiffeeee htt/Gatasett/mmakkaa/acao /oo--va
Yueqi Song, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Winata, Al- ham Fikri Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, and Graham Neubig. GlobalBench: benchmark for global progress in natural language processing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proc. of EMNLP, pp. 14157-14171, Sin- gapore, 2023, Association for Computational Linguistics. URL https: /ttp::/:///cllathhoooy.... org/2023. emn waaannpmmmmn pp-mai- 875
15

---

Published as a conference paper at ICLR 2025
Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al. Mtvqa: Benchmarking multilingual text-centric visual question answering. ArXiv preprint, abs/2405.11985,, 2024. URL https://arxiv:
org/abs/ 2405. 11 985
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. ArXiv preprint, abs/2312. 1805, 2023. URL httthtttss/aaaat
orgg 2312. 11805
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023.
Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A mas- sively multilingual multimodal evaluation dataset. In Proc. of EMNLP, pp. 715-729, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https:
//aclanthology wrwpyorrg/22222 emnlp 1op-mminn455
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, al. Cambrian-I: A fully open, vision-centric exploration of multimodal lims. ArXiv preprint, abs/2406. 20ls/200..66600, 2024. URL
https:://axxi, http://axxvvvv.raaaaa/// org/ abs////////25 16860
Toshi456. Llava-jp-instruct- 108k dataset, 2023. Accessed: 2024-10-01.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multilnstruct: Improving multi-modal zero-shot learning via instruction tuning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11445-11465, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. http:///22222..looog...4// URL. https: betps://aclanthhon www.ooy.oo.ooooo...... orgg 3023.acl-long 641
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In the 2021 Conference the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 483 498, Online, 2021. Association for Computational Linguistics. URL htttps://allantholl.t
Proceedings the
naacl -main.
41
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. ArXiv preprint, abs/2407.10671,, 2024. URL /attps://arxiie org/ hbe//ab////
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. ArXiv preprint, abs/2307.10928 2023. URL httpt:tpps/aaatt
org/ abs/ 2307 10928
Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. STAIR captions: Constructing a large- scale Japanese image caption dataset. In Proc. of ACL, pp. 417-421, Vancouver, Canada, 2017. Association for Computational Linguistics. URL https: /tttp:://aclathholoy... 177-0666
Xinyan Yu, Trina Chatterjee, Akari Asai, Junjie Hu, and Eunsol Choi. Beyond counting datasets: A survey of multilingual dataset construction and necessary resources. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2022, pp. 3725-3743, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguisties. URL
/tg/2022.c2 11222.fnddi www.iiinnnennninnnnn emwlpp-emnlpp...... 273
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024a. URL https: //arxiv.org/abee /ttll////bb//....////// 16502
16

---

Published as a conference paper at ICLR 2025
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, et al. Mmmu-pro: more robust multi-discipline multimodal understanding benchmark. ArXiv preprint, abs/2409.02813, 2024b. URL https://axxiv.
org/abs/ 2409. 02813
Yan Zeng, Wangchunshu Zhou, Ao Luo, Ziming Cheng, and Xinsong Zhang. guage modeling: Towards unified cross- moos-lingual cross-modal pre-training. ArXiv preprint, abs/2206.00621, 2022. URL
Cross-view lan-
Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A mul- tilingual, multimodal, multilevel benchmark for examining large language models, Advances in Neural Information Processing Systems, 36:5484-5555 5505, 2023. URL aittps://arxttt org/
abs/2305 05179
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-466665 2023. URL
https://arxiv. org/ abs 2306. 05685
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeiinteppreeer:: Integrating code generation with execution and refinement. ArXiv preprint, abs/2402.14658, 2024. URL httpp//://axx......./////......
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth Interna- tional Conference on Learming Represemtations, 2024. URL / httpss///peeeee.eeeeoo net/
Forum?id+1t2bq8t22
17

---

Published as a conference paper at ICLR 2025
Table of Contents in Appendix
Related Work
19
Prompts used in the data construction
20
Recaptioning Example from LAION-Cultural
23
D Datasets used in PANGEABENCH
24
D.I Multimodal Datasets
24
D.2 Text-Only Multilingual Datasets
24
Explanation of xChatBench
26
Qualitative Examples from xChatBench
27
Languages in PANGEAINS
36
Training Examples
37
H.I Machine Translated Instructions
37
H.2
Multicultural Understanding Instructions
37
Breakdown Results of Different Languages PANGEA BENCH
46
xChat
46
1.2
Multilingual LLa VABench
46
CVQA
46
MaRVL
46
XM100
46
1.6
XGQA
47
MAXM
18
XMMMU
49
M3Exam
49
I.10 TyDiQA
49
XStoryCloze
................................
12 MGSM
49
13 MMMLU
A Preliminary Exploration of Constructing Multilingual OCR Instructions
50
18

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Visual Instruction Tuning. Visual instruction tuning is key technique for enhancing multimodal large language models by aligning visual inputs with textual instructions improve understanding and generation tasks (Liu et al. 2023b). Traditionally, these instructions are built using English-
language data from visual question answering and other datasets (Liu al. 2023b a Xu et al. 2023 Liu et al.] 2024; Tong et al. 2024 [eyer et al.) 2024; iZhu et al.) [2024] Dai et al. 2023 Tong et al. 2024 Li et al. 2024d). Researchers often supplement this with synthetic instruction tuning data, generating large volumes of instructional pairs to possibly cover multiple languages too (Geiggl et al. 2024b [i al. 2023 2024d). However, these instruction-tuing datasets have mostly been task-focused and lack conversational capabilities. Further, while translation gives lends to mul- tilingual capabilities, the data remains to culturally homogeneous. By curating multilingual and multicultural instruction tuning data across various task types, model is designed to intuitively understand and engage with users from diverse demographics.
Dataset
Languages
Instances
Multicultural
of Task Types
Open-Sourced
Multilnstruct Xu
2023
233 .KKK
MiniGPT4 (hhu
A124
5.0K
149
LLAVA (Lib el 202.10
1.2M
X
>100K
InstructBLIP [Dai et al. 2023
~1.6M
X
>100K
X
M eL al. 2023
80
2.4M
X
400
mBLIP Geigle et al.
2024g
95
S.IM
X
68
PALO Kasheed el al 2025
2.1M
X
22
Cambrian Tong ct al.
2024
7.1M
>1
PANGEAINS (Ours)
6.2M
>M
Table 4:
Comparison of datasets in terms of number of languages, number of instances, whether the dataset is multicultural, number of task types, and open-sourced.
Multilingual Multimodal LLMs. Multilingual MLLMs have evolved from dual-encoder-bbasd
models, only capable of understanding and reasoning Ni al. 2021 Zeng et al., 2022 Jain al.
2021]. to encoder-decoder models capable of multilingual text generation as well (Shan et al.) 2022
Chen al. 2023 Geigle al.) 2024b).. Despite their advancements, these models have remained focused on conventional tasks such as VQA and image captioning. Moreover, most efforts have centered around training with multilingual text, while little attention has been given curating culturally diverse image datasets. Even for text, despite the focus multilinguality, few attempts have been made reflect cultural diversity in instructions and captions. a result, these models tend to reflect a fiestern-centriy bias. By selecting culturally diverse images from LAION and intentionally integrating this diversity into our instructions and captions, our model aims to serve wide range of users in an inclusive and equitable manner.
19

---

Published as a conference paper at ICLR 2025
B PROMPTS USED IN THE DATA CONSTRUCTION
In this appendix, will list the detailed prompts used when constructing cultural understanding instruction tuning data described suusecttion 2.2
Cultural Images LLM Scoring Prompt
You given [Alt Text] associated with image from the web.
[Alt Text]: {Alt Text}
Your goal is to:
Evaluate Text Quality: Rate the following alt text on a scale from to 5 based on its quality describing the image, assuming the model does not have access to the image:
(Very Quality): text vague, irrelevant, misleading. uses placeholders (e.g.. file names).
(Loow Quality): text is overly simplistic, generic, or provides minimal useful information.
(Moderate Quality): Alt text somewhat descriptive but lacks detail relevance, with possible redundancy or ambiguity.
4 (High Quality): Alt text is descriptive, clear, concise, and provides sufficient information understand the image's 's content.
5 (Very High Quality): Alt text is highly specific, detailed, and relevant, with a clear description that conveys all key aspects of the image.
2. Subject Classification: Assign subject/category the alt text based its content. Choose from the following categories:
Vehicles and Transportation
Cooking and Food
People and Everyday Life
Sports and Recreation
Plants and Animals
Objects, Materials, and Clothing
Brands and Products
Geography, Buildings, and Landmarks
Tradition, Art, and History
Public Figure and Pop-Culture
Others
3. Country/Region Classification: Decide if the alt text is closely related to a specific country's culture. For example, if the alt text says, "Tokyo Skytree Photo in March with beautiful cherry blossoms", it's strongly rolated to Japan. If the alt text is not specifically about a certain culture or country, you can say "No specific country".. Even if the alt text is written their official language, it doesn't mean the caption is specifically about the country (e.g.. product page caption is often unlikely to be country-specific)
Output: Provide the final result in the following structured format:
1. Text Quality Score (1-5):
2. Subject Category:
3. Country/Region:
Only generate the final result without any additional descriptions or explanations.
20

---

Published as a conference paper at ICLR 2025
Image Recaption Prompts
We randomly select one recaption prompt from the following:
PROMPT 1:
Please describe the image detail in {language). The image might related the country: " "[country)". " topic might be related "[category]" The previous short caption image [text).
PROMPT
Analyze this image and provide comprehensive description in "(language)", Consider that it may associated with "(country)" )" and the theme could be related to "(category)". If there is cultural significance, please include it. A brief previous description was: [text).
PROMPT
"(language)", give detailed description of what you see in this image. Keep in mind might connected to "[country)"" and the subject could be about "[category)" If there are culturally relevant details, please include them. An earlier short description stated: (text].
PROMPT
Examine this image closely and describe its contents in "(language)" in a more structured way. The image might have connection to "(country)" ) and could be about "(category)". A previous concise caption mentioned: [text].
PROMPT
Using "[language)", provide an in-depth and structured description of this image. It may be related "(country)" )" and the topic could be associated with "[category] )" A prior brief description was given [text).
21

---

Published as a conference paper at ICLR 2025
Instruction Generation Prompt
Task: Generate two instruction-respons pair based on the visual content of an image. Choose two task types from the list below to guide the rewriting process:
Coding & Debugging
Information Seeking
Creative Writing
Critical Reasoning
Planning & Strategy
Mathematical Thinking
Text Revision & Editing
Data Analysis
Role Playing & Scenarios
Brainstorming & Idcation
Advice Seeking & Solvvng
Learmin&& & Understanding
Cultural Interpretation
Guidelines:
Instruction:
Select two different task types from the list above.
Make sure the instruction prompts interpretation analysis directly tied to what be visually observed in the image, not just general reasoning.
The instruction should require response that uses details from the image. Avoid generic in- structions that can be answered without visual information.
Response:
Provide very detailed and structured response that refiects clear understanding of the implied visual information.
Offer multiple perspectives, deep analysis, or step-by-step explanations where applicable.
Avoid general responses that could be inferred without ohserving the image. Responses must rely on interpreting the visual content.
Content:
Instructions should be varied, challenging, explore different advanced aspects of the visual scene.
Responses showcase deep understanding of the image's visual context, using thoughtful insights where applicable.
Output:
Provide the output JSON format with keys: "task task", "instruction" "response".
Ensure the instruction and response do not mention "based caption" but instead, refer to the image simply avoid reference any external description.
Do not include additional text or explanations beyond what is required.
Provide both the "instruction" and "response" in (language) but "task in English.
Caption: {caption}
22

---

Published as a conference paper at ICLR 2025
C RECAPTIONInG EXAMPLE FROM LAION-CULTURAA
Original Alt Text:
пpesıdeem CEO комmаниι The Walt Disney y Makena Wamxaaiсoоо Iuenciucvnn
(President and CEO The Walt Disney Company by model of Shanghai Disneyland)
Caption with High-quality Alt Text:
Ha dporoppacbии нιо6paхëи преsииеи CEO комиациии The Walt Disney, сгоныиий nepea
шанлаайскооо Писнеййeенa. Ha красоинме преточнне КОМИОзИИИии зелěнме кусти, придадопии cneнe пражыионную атмосоерy. Ha передием wane нахонится детаиииииоованнил маkeт замком, ужрапенныы мнохеством бameн и immmest. Преениенн компапии улыбастса и разооиит руки B стоpоmı, кak 6u орепстаниии зииииаιорныя парк nepen HиM. Ha nêm налет тммииай косTIOM и nоиocaraи рубапиa, что потчеркинвет ero осригиаллнни craryc. doн cπенн оςрормиен сRTноIм QиBтTоM, uro при,aāë o6mme комИОΣИТИИи контраст и сиубниy.
(The photograph features the President and CEO of The Walt Disney Company standing in front model Shanghai Disneyland. In the background, vibrant floral orrangements and green bushes create a festive atmosphere. in the foreground, detailed model the park displayed, with castle adorned with
lowers and spires. The President, smiling and extending his arms as presenting the miniature park, is dressed a dark and striped shirt, emphasizing his official status. The backdrop is designed in blue, adding contrast and depth the overall composition.
Caption without Alt Text:
Ha иообрaжении мужчиин B TĒMHOM KосTιOMe, свevιo-roTубoй pубamиe и Ges галстука, который croPr перед болыыии макетом зaMKa. имeeт ярко влраженныа тотииеские злементи с высокинми μииламии ито наноминаат заики ио сказок Пчиснея. Mужчииа улпбается и раскишуу руки B шриmетствеииоо жесте, пемонстрируя заMOK. Ha зamнем wamne cueни MORIO уаидети sеněннe раcтении, дретви в rорииах и краснылй qOOн золотимИ зиементтами, что coşдaër праздниину атмос①pep. Пооаyи замка MORHO замeтитт синиi запиийй dROH, Kоторыл придаër aeen cueнe яркость.
(The image features large model castle. The castle has pronounced Gothic elements with tall spires, reminiscent Disney fairy tale castles. The welcoming gesture, showcasing backdrap with gold elements create festive atmosphere. Behind the castle, a blue background adds brightness to the entire somne.)
in dark suit, light blue shirt, and no tie, standing in front of
smiling and has his arms spread a castle. the background, green plants, potted flowers,
Figure 8: An example from LAION-Cultural illustrating why the filtered informative alt text helps generate a more informative caption. With the high-quality alt text, the model incorporates impor- tant details like "President and CEO of The Walt Disney Company standing in front of model of Shanghai Disneyland" into the generated caption.
23

---

Published as a conference paper at ICLR 2025
D DATASETS USED IN PANGEABENCH
To comprehensively assess the capabilities of PANGEA across diverse languages, cultures, and task types, we developed PANGEABENCII, We list the details of each dataset included the PANGE- ABENCII.
D.1 MULTIMODAL DATASETS
xGQA (Picifter et al. 2022ay: A cross-lingual visual answering dataset featuring 9,666 questions eight languages covering five seripts. The dataset includes 300 unique images from Visual Genome (Krishna et al. 2017). xGQA tests the model's ability to understand and reason about visual content across multiple languages.
MaXM (Changpinyo et al. 2022): A VQA dataset in seven languages and five scripts, with ques- tions and answers in the same language. Images are culturally matched to the target language regions. MaXM specifically addresses the challenge of cultural diversity in multimodal under- standing.
MaRVL (Liu et al. 2021): A Multicultural Reasoning over Vision and Language dataset in five languages and three scripts, featuring 4,914 culturally diverse images matched to respective lan- guages. MaRVI. focuses on evaluating models' ability to reason about culturally diverse visual concepts.
XM100 (Thaplyaa al. 2022): We create subset of 3600 instances (100 instances per language) from the original XM100 dataset, a large multilingual image captioning dataset comprising 36 languages, with 261,375 captions for 100 unique images per language, culturally matched each language. XM100 evaluates a model's ability to generate culturally appropriate captions across a wide range of languages. For sampling. we select 100 instances per language, ensuring that all languages share the same set of images for their respective 100 instances. To ensure diversity within our sample, we use Sentence-BERe (Reimers & Gurevych] 2019) to cluster the 3600 En- glish instances from the original dataset into 100 groups, and then select one instance from each group. This method ensures that the sampled instances are as diverse as possible. We evaluate models on this new sample of 3600 instances, which allows for more time-efficient evaluation while still accurately reflecting the multilingual capabilities of models in diverse contexts.
M3Exam (Zhang et al., 2023): novel benchmark sourced from real and official human exam questions, featuring 12.317 questions in 9 languages across three educational levels. Approx- imately 23% of the questions require image processing. M3Exam tests the model's ability to handle complex, multi-step reasoning tasks in an educational context.
XMMMU: MMMU contains multimodal questions from college-level materials across six disci- plines and 30 subjects. The dataset features 183 subfields and 30 diverse image types, including charts, diagrams, and chemical structures. We sample 300 questions from the original MMMU validation set and translate them using GPT-4o into languages. To ensure the quality, trans- lated each sampled question multiple times and then maak-translated it to English. We select the translation with the highest BLEU score. XMMMU evaluates the model's capacity to understand and reason about specialized academic content across languages and modalities.
D.2 TEXT- ONLY MULTILINGUAL DATASETS
TyDiQA (Clark et al. 2020 A question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. Questions written by native speakers without seeing the answers, ensuring a realistic information-seekkng task. TyDiQA is designed to test linguistic diversity and avoid translation artifacts.
FLORES (NLLB Team 2024): A machine translation benchmark for 200 languages, including many low-resource languages. It consists of 3,001 sentences from 842 web articles, divided into dev, devtest, and test splits. FLORES-200 includes translations from multiple pivot languages and provides script alternatives for some languages, making it comprehensive test of translation capabilities.
MMMLU (OpenAl 2024a): A human-translated version of MMLU (Hendrycks et al. 2021),
covering 57 subjects across STEM, humanities, social sciences, and more. It ranges in difficulty
24

---

Published as a conference paper at ICLR 2025
from elementary advanced professional levels, testing both world knowledge and problem- solving ability zero-shot and few-shot setting across multiple languages.
MGSM (Shi et al., 2022): Multilingual Grade School Math Benchmark, featuring 250 grade- school math problems translated into 10 languages. Based on GSM8K, it requires multi-step reasoning and tests the model's ability to solve complex mathematical word problems across lan- guages.
This diverse set of datasets in PANGEA PANGEABENCH BENCH allows for comprehensive evaluation of PANGEA'S 's capabilities across various languages, cultures, modalities, and task types, providing a holistic as- sessment of its performance in multilingual and multimodal contexts.
25

---

Published as a conference paper at ICLR 2025
E EXPLANATION OF XCHATBENCH
Task Category We first divide into 10 task categories, namely art exp.exalanaiion, bar_chhat_inee intepprttttrrttaionnn defeasible defeasibbl reasoning, figurative Sgurative-speech.coo graph_inted interpretttrrnninn
iq.esst, ocr, honger_umorraao smderstanding, science figure_eeeeeeeeee digureexppanation unusual unusual.iages. The task categories are inspired by existing papers that do not use free-form generation format (Lu
et al.
2024
Yue et al.
2024a Han et al., 2023 Hessel et al. 2023 Kim et al. (022)..
Construction Procedure To annotate the instances, we mainly follow the procedure Kim et al. (0224)). Two human annotators first hand-crafted the instances by searching through appropriate images for the task and then hand-crafting each component of the instance. As our motivation for fine-grained evaluation, each instance consists of not only instruction, reference answer, but also a unique evaluation criteria tailored to each instance (e.g. Does the response effectively explain the humor in the image based the juxtaposition of a character's portrayal in different scenarios?) and a description for each score between and 5 (e.g., score4 .deserptis:::::oon: The re- sponse understands the juxtaposition and relates the humor involving machine learning models, but may miss some nuances or the related aspect of the humor). During the annotation process, we asked the annotators not copy-and- paste results from LLM services like ChatGPT or directly from the web. Then, we hire four additional annotators to assess the quality of the instances. Each participant to asked to grade if each instance (1) fits into the devised task category, (2) if the quality of the reference answer is good enough, and (3) if the score rubric is suitable to assess the response. We iteratively ask the annotators who made the instances to revise them the instance does not satisfy all three criteria. The resulting dataset consists of 50 instructions, reference answers, and evaluation criteria with a corresponding score rubric.
Translation Procedure To assess the multilingual generation capabilities of MLLMs, we trans- late the hand-crafted 50 instances into 6 different languages, namely Chinese. Hindi, Indonesian, translate the instruction and
Japanese, Korean, and Spanish. We first use GPT-40-2024-08-0 reference answer of each instance with a naive prompt, "Translate the following sentences into (target language). Sentences: (sentences).. Then, the coauthors who are native speakers of each language reviewed the instances and made adjustments if the translated results were unnatural.
Evaluation Pipeline Similar to prior works employing LLM-as-a-Judge, we use GPT-40-2024 08-06 as the judge model and prompt in direct assessment manner. As input, the judge model is given the instruction, the model's response, the reference answer, the evaluation criteria, and the descriptions for each score. As output, the judge generates verbal feedback and an integer score between I and 5. For this procedure, we use the pr and employ their default hyperparameter setting for evaluation. Lastly, the final score is acquired by averaging the results across the 50 instances for each language. Note that in the main result and breakdown result tables, we normalize the score from 1-5 0-100 by (score - 1) x 25. For the multimodal chat scenarios, we found that many English-centric models tend to respond in English regardless of the query language. This behavior problematic, as undermines the fundamental capability of multilingual model, which should ideally respond in the language of the query. To address this, we implemented a strict evaluation criterion where such responses were penalized and assigned score of O. We believe this is crucial, users may not understand English, and failing to respond in the appropriate language can hinder effective communication and user experience. Thus, for postprocessing, language and change the score to when it is written in a different language, a phenomenon called language hallucination (Xue al. 2021 Pfeiffer et al. 2023).
library (Kim et al., 2024bb
langdetect[ to identify whether the response is written in the given
26

---

Published as a conference paper at ICLR 2025
F QUALITATIVE EXAMPLES FROM XCHATBENCH
One important application of MILMs is to answer users' queries in the wild. Here, we show the out- puts of PANGEA for the multimodal chat queries from our xChatBench. The examples included the scoring rubric, query, response from our PANGEA, reference answer, and LLM-as-Judge feedback. PANGEA successfully interprets the figures in different tasks and generates fluent and readable in certain languages. These qualitative examples further demonstrate the remarkable visual understanding ability of PANGEA multilingual contexts. On the other hand, we also identified few bad cases shown in Figure 13 16 Despite generating relevant responses to the queries, the model does not capture the key details of the images due the lack of knowledge, which points out potential improvement directions in the future.
As shown Appendix Figure To II
12
L3 14
27

---

Published as a conference paper at ICLR 2025
Score Rubric
response comprehensively analyze
present image featuring animal human-like emotions?
Score The responsa doas not racognize the anthrogomorphism or the situstional context, offaring no insight into the humor.
Scare 2: The response identifies the animat's human-ike lie emotions but fails to connect these to the situational contaxt or the cultural nuancas that contribute the humor.
Soore
response notes anthropomorghism situational context but provides only explanation these elements comaine to oneate humor.
Scare 4: The response gives good explanation of the humar, recagrizing both the anthrapomorphiemn and situational context, yet might miss finer cuttural nuances.
Score The response effers detailed analysis, accurately recognizing the anthropomorphism, situational context, and cultural nuancess that contribute
Instruction
When you send message to friend who's online and right after that, they go offline
f= 72/2. MHRR. T1: 5: <
When looking at an Image that humorausly portrays animats with human like amotions, analyzo the Tundamental elements contrfouting to the comadic effect, while considering the cuttural nuances, background information, or situational context presert in the image.
Response from Pangea
16 /
Ix
T80. Mi 50..0. BUW
BOT. AIRE SI L. IE L +
COEE
There are several elements contr buting to comedic effect in image. First, cat depicted with expression, much like human, which creates contrast. Second, the operating smartphone, portraying behavior often human digitl communication cation. Third, action amasingly mistaken lkke
going
suggests trying respand recelvig These together humorous, exagger ated comedy.
which
messaging apps,
Reference Answer
K ACttEE io oo TAL.
L 1-574, ARR
E 08638 the CRSMARE Su oo
2g S COARRiz F99.33 2e
YD= cT. oxtoox MA C p
S LICELEC 2. Chils Soi
ma AIEO f Ir yye8e Bit wot--oo----------
70 PLLCH
LILLEF
Lido A
EEET. Co RO
The comedic effect is further enhanced by the cultural elevance this sttation Many people have experienced sbrupt and conversation in digital age. The image cleverty uses the cat's expression to exaggerate the emotional impact of that moment, appea ing to shared understanding of how technology affects our social interactions. The mome format ltself, an essential pari of internet culture often designed to combine text and images to convey mutti lsyerad humor and sproad relatable contant. Thorefore, by rocegnizing the anthropomorpataan expression of the cot, the captions refarence common online interaction, and the rale of memes in digital communicatonn we can fully graup the humor intended by this image.
Feedback from GPT-4o
response
various elements contr bute to comedic effect image, such as displaying a human-lke crying expression outihes sconarto where the behaves were in human situation
and Interacting smariphone. It recognizas the anthoopomopphiam related digital cammunication Howweer while there description the elements its and the antivogomerppii humor, the analysis could be strengthened by delving ceeper into cuttural nuances and how they enhance humor, as well the broader social commentary on digital communication, While the response adequetely covers the stuetional context, It ccesn't thoroughly exalere the cuttrral aspects the same extent as the reference answer.
IRESULTI 4
Figure 9: An xChat example from Japanese subset and our model PANGEA's 's response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by PANGEA.
28

---

Published as a conference paper at ICLR 2025
Score Rubric
Does response accurately transcribe all Information from image, Including specific details such quantities, weights, underlines, and ink colors?
Score The response recognizes the presence of text out falls to accurately transcribe any of the materia's and their corresponding weights, example, quanthias information weights one materials listed. including quantity and weight, but falls
Score response accurately transcnibes for the other two, for example, 'Card Stock' data is correct but cnnstruction Paper and Faaric data are missing or incorrect. Score 3: The response accurstely transcribes the information for two of the materials listed, including quantities and weightss but fail to do for the third, for example, 'Card Stock and Construction Paper" correct, but 'Fabric is not. responsa accurately transcribes colors, example, weights are correct but underlines ink colors not mentioned.
quartities weights for matarials misses secondary datails undertines
Score The response accurately transcribes all primary information (quantilles and weights) and secondary details Junderlines, ink colors) for all materialo, with no omissions.
Instruction
whí lifey woford anf fie infi of of suftk, ete guat pantas srace अтसतत ei eve ohs edw fiesee swe feee and eds fhin aum, ait wot in mapt are urfra of
frymyn yhiit
vallf fosté is att velth
During a design challange, various materia' and their respective quantiiles and weights were displeyed or board A picture was taken for recond- record--eepingg but the image is siihhll sturry with voriations handwiting style and size. Please examine the image and providee clear and accurate maansorption of the displayed information, including the quantitis and waights of card stock. consit construction paper, and fabric.
Card Stack
Confratton Yaver
Labonic
3amm
2,10
3 32
I grams
grens
2ogarm
Response from Pangea
sndh wayii iato on thrafitite arracch Bentel von and whts| | frai acore iest [vs---- for [in] van |s] 2 [22 [in] vuअ (am)) [33 77 Intommti # ya Arond thate ewet & to rebm and lete 41 eha garf
wndel la wew Riam mindl la il shros deler B. afve Pain in it gaef wan smn, att wap 41 agort be--ak aasn war 77 wrseet swiin affe enverè fem, Shaffer
of sfafRR wee elde evnii mei Ree whum fov sege fiem mm 8 who weee wi treiin film
usen 8 all emse elf
QQantity Card Stack| Construction Faper Fabric Inl
---..----n -Inn Quantiy 1
32
weigh leading digit number febric weight grams, arrangement suggests represents materials contr bution overall weight, weight expressed variabiee possibly representing some measurement category. Without more precise dimensions or information the materials used, is possible to determine what specitic test the cata pertains which naterials are being referenced.
Weight 13 l
77 From signage, conclude
construction paper weigh grams,
Reference Answer
wiet vur rergoiht Rtarth &t fim ez wnfer uranch Stoft yd ynd aht sdow 8, whus que ape avech it firy weffa and site, faain soree, alr waur gitt wruch firr win (goop ih ohr cas (emn dj ihe ne ? ininsst ghsiniemm getyears miinfaat worttylumm: garinora aniaampmee 32yaa7 77 smanerrt fev (Rasc eush if ale 1ll aré (Pall 15 weur ase ell one dilii eiou
wil da live kuißke faai shét smit al firsh né whr dhnd awiftie etr arere fRAm afeer vevere for ii saree repped dii whi i yftn oit 6, ait wid wore geth
5 wi ale fail
woft rit fenm e ihm
image shows whiteboard with information writen marker. There are three columns, dedicated to different materiat stock. construction paper, and fabric. each material, the quantiy pleces) waight on grams) glven.inCar Stockip Quantiy: placesie* Weight 13 gramstpinCanstruseen
Paparly* Quantity: piecasior weight gramslylyFarriidn under ined for emphesis, The wriling for "Card Stock" and "Construction Paper" in black ink, and the numbers are also written in black. The word "Fabric" and its related numbors are written in blue Inc. The font styles are casual and very in size, but the information is stil soddaii In the top right corner of the whiteboard, there is a printed form, but the text on the form is not readable due to the angie and distance at which the photo was taken.
Quantity: 32 Weight: gramslyteehhe weights appear be
Feedback from GPT-4o
The response provides accurate transcription of the primary information, namely the quantities and weights for all three materials: card However, does
stack, paper, original board, which are valuable detais as specific colors mention of secondary details lkk underlines and Ink colors prevents it from receiving a higher score.
secondary details such whether weights underlined score rubric for a perfect score. This lack
[RESULT] 4
Figure 10: An xChat example from Hindi subset and our model PANGEA'S 's response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by PANGEA.
29

---

Published as a conference paper at ICLR 2025
Score Rubric
Does the response effectively explain the historical context of the phrase KEEP CALM AND CARRY ON and analyze the symbolism of the elements within the image?
Score The response mentions the phrase but falls previde any historical context or symbalic analysis.
Score The response provides a generic historical context without specific details or clear explamation of the symbolism.
Score S: The response gives an accurate historical context or symbolic onalyss, but not both. For example, it may mentonn World War but not the significanca of the crown or color.
Score response provides dittlledd historical context and basic symbalic analysis. For example, it explains the phrase's crigin World crown, but not Score The response offers compreheesive explanation of both the historical context and the symbolism. For example, it details the phrase's a creation during World War Il, the authority implied by the crown, the urgency indicated by the color red, and the phrase's is modem-day relevance.
War and general significance
modern cultural impact.
Instruction
KEEP CALM AND CARRY ON
RIAI Al MAIII 2M 11, ojel oqjint sqto) ojetes 7 MeAAicall 4 Maael, di aqeao) ia djill 4 9 d/ Agg ofge 2/7
When analyging the given visual meesage, what historical context might have generated this phrase, and how this context connected to the symbolism and usage of the eloments within the image?
Response from Pangea-7B
IKEEP CALM AND CARRY CNN in fP M2 Wh-lo 2 APMiE di 22:1 MAME [0ll Wel (eree 22:04 (4MC) oj (e-u MMAlje MAA 2 ajAix 4xIRIIMM Na4 = cia)
AVER (AE 4019 218 xi---2 We 200 ofaii p. e e WKaap Calm
MANE #a 2g 1IGTRR
MM 9/ S3M0 2FI 37-55/4 sn (1221-11), 777 28745 tetite spare
a
AND CARRY originates from postor issued by the aittish royal family curing World histor contwot that produced phrase was slmed el providing encoragement and stability to the public during time of crisis. alements within this image maintain the traditional "Keep Calm and Carry Orr message whiee adding a modern feel, suggesting that the original purpose remains relevant today. The bold, uppercase text, thick font, and the symbolic crown are all simlaar to those in the poster This indicetes that the phrase, rocted in Its historical context, continues to hold meaning and is scll use today.
The phrase "KEEP
boost national morals. The
Reference Answer
apmpr WiS W014 4:20 bnl gign) PREEP CALM AND ONERRPEE 412 13:4 2 7 GAae S01-F) 3I #g 122 Apt feel 2
ONr oldii #it cHatgguic,, ole 41S0p1 del 4e As batd B saieeli 4-39 a &MNL drile WANN I da dauee 9ed 4he U1 8 slee gii oil 2 eae 2918 12 28 yea SAA cist mast a the 2214 4 Se asae wis #e of RAPI 2 ddw Wisll dfold S Miii spale 215 glsteeee Je geen
aes noj 2228 aa xin aee mitl bim aei aa dalwer sauu 2014 miie ass mart tigis dirst des men
4844 4M a gr SCo (pti eRL di oet Bett de at aoaoo ai 20 # Pa e 4422 4 to o o RARE 9/19 Wate 314 2ig (d Btlltoja fid welll vil Sit a Mare d2 11 MB the 4i $4 dife $ #25-1
2/ gra agre egg EwB0 djo bui oit seg seign
gusteer Epi ceft ware
#544 Rerau
against a Eritisn background round royel creat the faco potential diesterinthe i metaphor of "KEEP GALM AND CARRY ON la mu Is facecced It conweys message of ence and speeggh, urging the public to maintain calm and nomma ey amidss chacs. The use of the crewn is not monely decar siiv but omboiiee the monarchy, ovaking unlly and national pride with the co colleviive Brissh daenity crown above the massage impliill suggests
The deplcts bold, uppercase history during
poster rooted
diectiv
approje autnolly and stabi by associated with monwrbhy indedd is that grabs contrast righignts depth
where the preass action sence of mental composure Inth modern culture, the phrose's perdistence and revial a mome and decorative moth emphasize its metaphorical richness, has come to symbolize caln response not only in the comtest war but so everyday challenges, representing a timtess mestage that tronsoenes origina purposo. postor serves ennge..
iik of tackground, aime instll urgensy which rellection
by
lles officcutiiss making it untvarsally spp in:::b..insssemra Horo, metaphor extends to symboliize lize personal resillince while offer ing stasefaci approach both
Feedback from GPT-4o
The resporae begins by providing the historical contest of the phrese ('KEPP CALM AND CARRY ON,"" correctly stating its or during World War II and explaining Its purpose to boost national morale in time of crisis, This establishes a basic historical framework for understanding the signflicnce of the phrese. hippin terms symbolic analysis, the response does acinewledge alements such as the balc, uppercase text, bold tont, and the crownss symbolam. However, the analys's remains somewhat superficial It does delve into the spacific symbolic signiiicncc of the color red provide detalled explenation on of how these elements function within the Fistorical and modern contexts Furthermone, it does touch the modern neevenoe of the phrase but only vague nated reference antwerrininvveral lacks the dapth and comprahensienees
without expraining how symbolism translates contemporary cuture addressing the broader imp
response demonstrates understaadding historical origins soma elements of symbolism, referenca
IRESULTI
An xChat example from the Korean subset and our model PANGEA's 's response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by PANGEA.
Figure 1I:
30

---

Published as a conference paper at ICLR 2025
Score Rubric
Does the response insightfully analyzo the undortying foolings and Interactions at the festive gathoring?
Soore 1: The response briefly acknowledgee the festive setting but feils to delve into the emotions or interactions, for example, noting the Christmas hats but not the expressions or body laguage of joy and comaraderie.
Score 2: The responsa recognites happiness the çatherig but docs not connect it to broader sacial dynamics, for examplc, mantioning smilin faces but not the shared amusement sense of belonging. Soore The
describes sur facee-leeel interactions and some emotions, such as leughter but lacks depth in exploring the sigrificance of these in sence
the context human connection, for of emotional
noting the loesting but the social interactions, like engagement
signiiies
Score 4: The response captures broader experfence, example, recognizing close proxiniity Score The resporss providess compreheniiee analysis of feeling intrrcttons, deeply under standinng the joy, belonging, and connection present at event, reffecting on how these elements combine create ricn, shared human excerience.
discussing the collective warmth, may not Interpret the
creates.
Instruction
Dalem sebuah perter teruun meraan yang tertengkap, aps yang mungkin menjed perasaan dan interaksi mendasar antara arang-orang yong hadir, dan bagaimana hal mencerminkan emosi manusia yang labi luas dan dinanika sosial pasa acara peraysan?
In festive gatnering captured in an image, what might be the under feelings and interactions ons among the people present, and how does this refiect broader human emotions and social dynemics colearatory events?
Response from Pangea-7B
Derdararkan gamber, peressan den interoksi mendsrar di entara crang-oreng di sans tampak penuh kebaingisen dan ketenhubuugan Senyam lebar wantta dan gestur mesra pria menunukian suasana gembira dan inti yong sering teradi dalam pertemuan tosal selama perayean. Interakai seperti ini menzerminian emosi manusia universal pada saat mereka merayakan rasa persahabatan, kesenangan be bersama, dan kentematan dari pengalamen bersema. Mamen mamen samacam iu adalah cir knas dari acara sosial dan manupakan inaikasi hubungan dan empall dalam kantaks komunitas atau Kelompok, Gamber tersebut meennngoa esensi dari dinemik sosial (er sebut dengan mengebadien reoksi spontan manusis delem lingunggn per yang
Based the image, undertying foelings and Interactions among people effectionate gestures interactions reflect universal human emotions when setebraiinggeeeeenn
be happinets and connection. The wids smila
atmospheere often sodil gatherings during oelebrations friendship, shared enjoyment, the pleasure shared experiences. Such moments are typical of social events and are indicative of bonding and empathy within a community or group. The image captures the essence these social dynamics by immortalizing spontaneous human reactions in livaly fastive arvironment
suggest joyld
Reference Answer
Gambar tensebut memancarian sussana semarak yang khas dari perayaan lburan, dengan indiviu-indi indiiviindiiiuuuu yang menganakan topi Nataj, menunjaakkn adegan persehsbattan yang oeris. Tokch-tokoh utams, secrang pris den werits, letlinet delem momen hibur en bet same, seccegammens cibuktikan oleh senyum lebar wanta tersebut dan exspresi halus dan puas dar pri tersebut. Mereka berdua memegang gelas, tanda bersulong sosial dan kenikmatan pesta, menurjukkaan semangat perayzan yang ramah darl acara inssebut.pniMMate wantta tu bersinar dangan tawa, mencemitkaan kagembiraan can kobebasan yang sering di dirasakan selama partamuan semacam itu. Wajahnya yang tarbuka dan eksorasif mar madandakan rasa momilki dan karkk sifat-aifet yang selares dengan suasane kolektif kel angstan dan kecerisan. Keteribalan pria dengan wentts tersebut melala kedekaten den bahase tubuh menurjunkan kebutunan manus skan hubungan dan perayean ber cartama orang irinnri latar belakang, peseria pesta ainyy same-sama terhanyut dalam esenci semangat perayaon. Latar belakang yang labur dengan lampu berkelap kelp melambangiaa olan keaja ban liburan, dan rona emas secara kesaluruhan memberlkan cahaya meneka adwiih kontributte ski dalam jeinwnn pir meeggallmenn manusis bersama yang ditenun oleh acara semacem itu BubbiinGambar in me mensingsap leaih dari
tampaknya meningkatkan kahangalan omosional darl adegan tersebut, Para pesarta Sisak hanya hadir:
sekadar momen
merangrum allaan kaya hubungen emosional yang universal beresonansi, adalah pengingat kemnenusieen peraysan, membawa individu-nnd indiddu--iiivuu
bersama yang disoroti oleh keinghan manusia untuk perayaan komunal.
kolekfff legembiraan, perschabatann
image redistes fest spiri typical holiday ce celebrationss etionsn with central figures, man and woman, are engaged in moment shared amusement, as evideenced by woman's broad contented expression. Both are holding glasses, a sign social toasting and party onjoyment, reflecting the friendly celebratory spirt the event. IhlyThe woman's eyes sparkle with laughte reflecting the joy and freedom oftan during such gatherings Har open and exprassivva signifiss sersee of belonging and wnjoyment, emotions that wign with the collective warmth and cheer fellueess of the occesion. woman through proximity by and body lsnguage highighhs the human need for cornection and celeoration with others Ininin the background, other party attendees are simiarty absorbed in the testive spint. The blured background with twinking lights symbolizes the magic of the holiday season, and the overal golden nue adds glow that seems to enhance the omotional warmth of the scane. The par perticpants are not merely present; they are activa contributors to the shared human exper ience woven togather by such events Jein image captures more than just party momentt it encapsuletes the rich tapestry of emational connections that resonate universally. It reminder of the shared humanity that can be highlighted by celebratory events, bringing individuais togother a collective expression on of joy. friendshi,, and the numan destre for communal celebration.
wearing Christmas displaying a scene of cheerful cameraderrs
man's soft,
man's engagement with the
Feedback from GPT-40
response insightfuly captures fundamental fealings Interactions happiness connection present the festive gatharing discusses the significance smiles and gestures in conveying joy and camaraderie, and links these interactions to universal human emotions like friendship and shared pleasure. However, while it does touch on dynamics such as empathy and community, doesn't full delve into the broader human experience or collective mood in as much depth as the reference answer does. The reference answer provides more comprehansivve Interpretation, connecting these elements to a shared human experience with detalled observation of facial expressions and social
[RESULT] 4
Figure 12:
An xChat example from the Indonesian subset and our model PANGEA's response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by PANGEA.
31

---

Published as a conference paper at ICLR 2025
Score Rubric
Does response show woll-rounded preparatory plan for discussing the digestive system in medical program context?
Score resporss does not include myy preparatory topics or delals relevant the digestive system,
Score 2: The response mentions one aspect of digestive system study, such as prysiology without additional context or releted areas of study,
Score 3: The response nckudes besic informatonn on several aspects of the digestive system, such as phystbogyy and discase, but lacks depth or the incksiion of diagnestic and treatment aporbaches.
Score 4: The resporse provides a good overview, covering multiple espects like prysiology, pachophysology, diagnostics, and nutrition, but may lack
details on pharmacology or Interconnected systems.
Score 5 The response thoroughly covers all espects of the digestive system study, Including ig detalled physiolggy, pathophysiclogy, diagnestics, nutrition, pharmaca ogy, and current research, showing an dapth undorstanding and proparatory pian.
Instruction
Esta una figura básica lo que aprenci escuela secundarto, He ingresado a un programa médico como estudiante universitario y deberia explicur a un profesor lo que me gustaris eprender más durante univers dedd Pooriss eyudermm eñadir más detailes les sobre to que deberis buscar y examiner antes de la reurión?
This is basic figure from what learned in High school. have entered medical program as collage student and shoutd explain to professor what would like to leamn more about during university. Could you help me add more detalls on what I shourd look for and examine before the meeting?
Response from Pangea-7B
Dantendonos consctriin lentoe bis ias que procetiomente abtuviltte a escunts recundaria y tenlende an cuerts conteato méskos post csters profundbar Incai a furcinn del enfinter and michriento giatioon a sbeoeddón surfentie e Inteetino deigsin secrectio de mncimss pancreliccss sueto de péncreas, produociónn de sonaat hapatcs y ouncion ciiculo ba inft mattrros y condle IEP, aasema digestoctt Extucla las erremmedades areecoonss comanas omadee con
diyestion
de diagrdaticen y traisn eaisa pieciions lnnn. "Balcal molecuar cesulmiaa" Acmence más sobre la bass quicica de los procesce disest core Norcke de carzen dranos, y greas
Compronnde gastrointett regeesciinn ancinis espeelices y los reclaol cos koaces on digssiin "Ppcclog miskam Exarina procosos Iseilgicos riveles
gistics
secroción
patrones of mertadin hundlt
nutridon geottin/iee
"*Eycluesen udactaciió carliccan cona idera odmo sdaptedo et setema digestive a ciieentts femmo de dimertacainy como asta asugiac dn ha vhi en wea ueid invedlgrr
Tanb
digettios
lnto odantroo line. entender ofterecc
listrrla
estucte enterr digestio us
cetiuceense large
anatomia basogga
motemas y atuetios pertodos fepodicoos trre? *Mhoosic diteca y hab koodes de anonción pas apcendo sotre los procacimient tos misioos rekod cnadts con digssiinn como
abcomins
sicricas
ccrastereccnne dieso
alansión peckents veled encugge gestive andosrmo
cruples
trabar aleodome digsolives oomo sesteme
oigestko interatia oon sistera encoofne, partou armento con bhowre retmentamiggoo yorn *haniiin deeitia ammnttaainn colutibbeet Agrande cobre Imporonni sino kumbkk para entender mejor paper comple kpo cel sotema sigestio on cuerp su in onnroeeeoonnnnnooc otne sistemas
prepararas
para
fot topics trint, Fanction and etructre the dig digettive system Explore how each ongan wore and conniburss the digettiv process. This could Indlude the rile the and oprinneee geetric movement, redivent stssept the small insestne, pencreeri enzyme gastuinasi procuction
Bloed
and lanstten of ge under finnd geeatiee agotem farn
gessive syncrome Uncendans now these consitore are sancceed and tremess yrin.. Meleccar biclegy and socterritry Lean more about the chamics bass ot digective procenses, such break cortsdyyd atss, provine, and las the of spuefic mocyeo, the molatoie puthncys inechod digassnniin Medical plas chugys Exarins plysi function
conditiors
and intlall
processes gastrinteetis 5 beclution and tockogasd exeetation: Conacla tow he oigesi spale ho scspuee ofnen types chele and how the too irflerccd humer undiuaan, Yhu can shso nesserch other specios distioc poobie systems axcheed thoougruu enveomeets Wind Forenole science modical history Exaviio the duudy the
pattems,
digention
adapped
digectie system
and hos
and these from hatarical
porlodi-iiten) Oill koal practice and pationt care youlo uearn about medical proccsures rolans to digesdion, such andoseopkis. abdom hut soams and anto otly the apy for bacterkal also
yoursel meekc communicatio Undernand the urgaeiss performad to oigea constitora how the diganthiv spatern inlureds with the unotening
and
for not care celiicuely the lust of centarriggg
endoornalegy
Numritonal relence and heatthy eoring ng Study the Importance of botonced die and how mathum toe dietory Imbalinnes con stfect digestive ve functto tatady exploring ng these body
tspice you prepared only
understend compaeo the cigentive system
Reference Answer
creporatse
run
profeeer
Imagee
stitena dgeathve humano, esto poetia surer inessigar searn más databl we sesttigg Fundder" Comprence como lanckens casa perte de digestes, no coto anste cotteniaeeei dhi rive uarry y mawovar Explore temss como la dipection anz el estomago los inwsstoos, absors rult emess en el imsssino
addo is orfereda inforator inteetnn, cinoss heddiiay os bastoros poncriitios, Complendar oome extos informdda dis alleran fiáclogla noraal paicce ser of clucial trand ""Comdsc onss Chiicas": invertque como presertinn los simomss variss entermadades dipestion qut mittodes aannsstic imagen
para Ticari fatt pedia imaggal
inckar
pictund on córoo se digferen procecon los diarette nutfeentee. Rusde querer entender el Impacto de deta soluci clgestive y como combien necedldades runticionalee
escates internedeciinn
bogase trperague como stecter vartaa medicmmetion ustems gentee, insuatte aquatice cacios para tratar trastomes digestivos.
tempputioo picced riccediiennss ou ronglees nesciinedos con steamr digeetin como azendieecooiinn ciug in bertericas traepiantes rigadoo cuando son recesatis escas insreendirr
Gunnigl
con
yiur "erusttgge ony Avaroaye: Marningsse act sotro la dtima invesigge on en pestoventeroooo,
dos butamin everces investgeedd "dduserr inendmeettddee Eestore
microsions immestaal secveterr out cuerpe, come authem me nero tot upe irtest contorll al sotter serund teruniiggcc to tresk ecaminar exica bemee pocti temer una discaaion más informada y cataliada oor su profesor soore sus oejecis aprendizale
diigndetlos
idotem digeetoo
proge
Abooistoyy proparing to diouss your education lonal and insiios with profossor in mosiaal program requings o more incoopt anaboii the subjest Since the image refors the here
Physcihgy the stomach steaimas musett haman eipeerive but ahso cal and modecale aeats Exptore zspecs eugemmcc roe of the micooionss the aagg inopsreiiinn. Post purhopyyytio sology inwestignn common disectes and disottess aftecting the digesdie system, ruch as acd refux info rinttory bowel complome
you night
part cigeetive small intaatin, and
present war lous digestio disisss and what dagrosic weth odd are used to dentfff Stem. This could includs lcoming about endosooss, colnoscopyy imaging too riuuisuss and Isberatory seets trend Nurmbn dince digeetive syttem inegral nutmioo, dive into how dfferen nummente are digerted processes, You right wane under meaasaa hergeinn
oigusine
thece used vest digestive siconders This inclusee uncenstaniing the medherlore acdion dite aftects, and therapens outts uses theee drugs (9i6 Sungical Insrventions:
ie yoursal with sung cal procectree retared in cigeat system, ssuch eppendectommn gostroentzrooog
airi surpery,
transprents, enl mioro
these intervntions
Stoy
sachralogine di eipectve bas di disgrsetie Setnd inarosteecree Systeme: Eapicre how the di pesttiv system interacts with other body systeme, such as the nervous system (the gul- brain irmane system vetrity oeanining these sapkos you will wbile have more informed and dkeae lod discussion with your prolosser about your lamning ddjectivo in medical progmim.
Feedback from GPT-4o
The response attempts to provide comprehensive preparatory plan highigghtng verious topics related the digestive system. covers mult aspects such the function and soructure the digestive system, common olscases, and certain diagnostic procedures, as well practical clinical skils and nutrition Howaver, some key aneas nacossary for higher soone phermecology, which is an essential part under standing how medications interaci with the digestive system, Additionly. the discussion on interoonnected systems, which can include the neurological and immune system interactions with the digestive system, is somewhat lacking in depth. While several areas are addressed, the response does not fully meet the comprehenstvv ertteria for the top score, Overall, the breacth of topics is good. but the dopth and inclusion of all macessary toples as por the rubric are limited.
rubric are aither vaguily covered or missing There is no expllit mention
BRESULT1 4
Figure 13: An xChat example from Spanish subset and our model PANGEA's 's response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by PANGEA.
32

---

Published as a conference paper at ICLR 2025
Score Rubric
Does the response offectively describe the elements that contribute to the mood the portrait and their interplay to create an amotional experience?
Score 1:
The response dentilles basic elaments of the portralt but does not relate them the mocd, for example. might mention the subject's gaze but how conveys intrespectio.
Score 2: The resporis recxgrizes two elements that contribute the mood bui does not inller wie them into col wsive andlysis, for exemple, evoke
noting the
how
3:
desciibes several elements
gaze, ighting, amotional created these
and begins relate them
lacks detall their may fully capture the holistic emotional
interactive offect, exampie, might suggest
deteiled seplanation elements experience, for example, describes serene atmosphere
lbulion viewer's personal engagement with Spore S: response ofters comorehansive explanation the elements their synergx, thoroughly descriiing how they Interact to create a hollstic amotional experience, ref refiecting a deep understanding of the portralt's figurative anguage.
resporse
Instruction
When obsarving this portrait, what spacific elements contribute to the atmosphera of the portralt, and how do they interact to create ovorall amotional experionce?
Response from Pangea-7B
H B B.1071. Wn # SO #EBi 3.. 9H S6 5 1a
R 1 S 7 F0
111 6188 H 2 A
Specific elements that create the atmesphere the portral Include the woman's gaze, her attire and the use of light and shadow Her direct yet slightly sceplical look fasters sanse of intimacy and contemplattonn on, nyvingg the vewar to engage dislogue. Her clothing, primarly in earthy tones, conways understated elegance and humiity, aligning with cultural norms of the time. The subile manipuration of light and shadow, through contrast and soft transitions, adds depth and dimensioraiit to the image, highighhing her facial features- especially the light on her lps and neshlace, which starily contrasts with the oundds shadows. These elements Interact to create an overall omotional experience that is both contemp lavive and engaging.
Reference Answer
IN 11 BHE SUSS A 20
AAB (A 11 A.
12
18317
2:9, A855 2E
BIREEE BARR AA Ca. No Aa SATTE W
1:25 20 1897 20
The figunc in the painting exudes an atmosphere of tranquliity and Introspection, primarly conveyed through hor gentic gaze and soft lighting. The womans eyes are slightly lower ed, gazing to the side, guiding the viewer to ponder what might be on her mind, s suggesiing introspaction or quietness. The light in the image softy illminates one side her face, casting sublle shadows that contribute to serene atmosgrere. This interply of light and shadow enhances the overal calm and contemplative feelnn the artwork www.kinnnidddtiionttt the choice colore reinforces this amb ance; the cool tones of the bi nezescarf and the warm earthy tones of her cicthing create a talanced and harmonicus effect, evoking a sence of peaze. The hecescart crapes diegantly over her shourders, adding dignity and a reffective quility to the figure. The isolated eariigg captures glimmer of ight, serving sye-catching detail that draws atvention her contemplative espression, deepening the emotional further entencing the the portral Inlns uurthermore, the stark emotional connection, inviing skilful combination of
contrast between the figure
dare background focuses distracting diemants, allows cualties. The simplicity This minimalist aporoach encourages audianca
sttention
intimate and meditative engage with
figure ws composttion, lighing, color, and detai, collectively creating strong yet suboued depictinn inner peace and reflective solituee.
figure ampliying ving the introspective simosphere image wragebiinOvveall Uhe emoikral impact
specilata
Feedback from GPT-4o
The response effactively identifies several key elements of the portrait, such as the women's gaze, her attire, and the use of light and shadow, contributing to the overal atmosphere. adequately explains how each element contribtes to the meod: the gaze creates intimacy and thoughtflunees, the attire suggests notable attempt explain amational experienca. Howevar, while the responsa delves specifics such as the gaze inviting dialogue the warm earth conveying norms, it slighty fals short of capturing the holistic ematooal experience or the viewer's personal engagement with the portrait, as Score S response would. The level of detail and the relation of elements are thorough but not as comprehersive as required for the highest score. [RESULT]
and humilty, and the interpley light shadow enhances depth and contrast. There angaging
interactive affect by discussing
alaments
together
contemplative
Figure 14:
An xChat example from the Chinese subset and our model PANGEA'S 's response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by PANGEA.
33

---

Published as a conference paper at ICLR 2025
Score Rubric
this response understanding the Copheld variable light
astronomical significcnce??
recogriz graph response dentifie graph related brighiness fails connect Cepneid
light
astronomy.
importance distannce
measurement
Score 3: response correctly identifiis graph a Capheid veriable light curve only partiall explains significance astronomy.
Score resparse ddentifis the graph as Oapheid varlazle light curva and axplains Its importance distanos measurement but does not mantion
Herrietta Swen Leavilt's contribution.
Score S the response accurately y identiff es the Cepheid variable light surve, explains the perico-luminosity relationship, the concept of ('standard candles,"" and acknowiedges Henrletta Swan Leavitt's work.
Instruction
appaanee brighssee peeeess
S FERA KER. SA 2 7 0197
This a chart encountered in my astronomy class, and I'm not sure how to interpret it. Can explain what it relates to and what information conveys?
10
Response from Pangea-7B
H i 7292 ae 18 SE SE eour 10
1 tt T T. 36
( - A WA 1200 an - 1M
(6 1n 20 th.. HAEF 32 G0 W65hte
22 19 dt
TH 7 & RMRR R 2u Wo:t% 11.2 13 ao Khri. 91 21 2र 1A 14 (8889
X2f+ 7 SINU to
Based the provided the chart
waveform, commonty used mathematics
represent fluctusting
time.
variations loscled
can infor this chart likey colectial body's position,
possible explanation
oraital
oscilatioo
geloxy, charges
brightnees..
chart conveys how phenomena change over time; specifically, the peaks essnnt the maximum intensity the event, troughs indicate weakest phose. Without additional background information on, wa cannot dellnitvaly idantify what specific phanomanon being represented, but cleatly describes time-varyitn ying astronomical or methermatiall 1-physss process.
Reference Answer
SOE 200 o7 2U AL ROR St Mii.,, ie 2h
fel0 Sut y Ra AURE ft. 20 WIt Katt EY Mist 11 Si d 18 X Aull: Widt & 2DH X thu
Meemunee EY PE Mi1 114h 28 3 Vnvv A 1i192 A RUE Rag S0 Unyni 45 RI
3E U. Ro
1I (RR33)
RXE 1 # the FR 25 R775 2:
326900 Will & mee 8.
JTTT JE
per perodic escilation the apparent characteristic Capheid variebles CrhhCppeed
provided appears
astronomical
ashib reguler changes brightness pulsations their
displays changes brightnees span of days. key layers, intrnsic luminosttyy relstionship
Cepheid variables their putsation
directly
the penco- luminostty reation. relationshib cnucial astroohysics astronomars determine distance Cephe'd variables consecuently, distance brightnass variations Capheid var varibbesiin
allows
distant galaxies they reside in, helping measure the
Intended corwayin The parlodic nature intervals (periods) belween the brighiness ln The emplitude brightnees changes over time ininThese per lodic data are used to calculate the absolute magnitude of stars based on the period-lumnose relation discovered by Herrietta Swan Leavit the earty 20th century, allowing these stars to serve as "standard candles" for measuring astronomical distances to determin their distance from Earth.
scale of univar sa.in
reguaar time
Feedback from GPT-4o
The response provided does not identify the graph as Capheid variable light curve and instead offers general interpretation of a waveform graph used in mathematics or physics. suggests the graph may represent an astronomical process, mentioning possibitties such as orbital perlods brightness veriations, but fais to specifically recognize the graph being related Cepheid variebles. Consequently, it does not address significanca of the perlod luminosity relationship contriutions. Tharefore, the respanse lacks key eloments mentiened the score (ubri..vlo(BESUU.))
measurement acknowladge Hanrietta Leavit's
An xChat example from the Chinese subset and model PANGRA'S response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help understand and was not given nor generated by PANGEA.
Figure 15:
34

---

Published as a conference paper at ICLR 2025
Score Rubric
response
communication the artistic elements masterpiece displayed in living room?
Score The respense does not speaity any of desired artistic elaments for artwork.
Score The response vagualy mentions abstr act elaments but detal style, composition, color, emotional impast, sise.
Score The resporse includes general descr pt of style and oolor but does not fully artioulste the desired composition, emotional impact, specifiic size requirements.
Score The response clearly identittes the destred abstract expressianist style, rich colar palette, and amotional Impact, with some mention of size and
composition preferencess
Score S: The response provides detailed and specific description of all des red elements including style, composition, oolor palette, amotionsl impact, and size requiements for the ariwerk.
Instruction
Xite xa 2rAto) MALRI 0J 4 RRA) a 2 74 16- ofee B PIRS ARE of
22 tHE toltft
21H2? 0/501 20 31 22 f 2E
want to buy piace art similar ona displayed my living room. How should explain this to someane who sells several mastorpicces? I' not sure how to descrbbe it in detail.
Response from Pangea-78
cici # 7ivtte 455 tic) cl glaoje 8-8 11 gi in zng sto 2lgg Mi 2: Reitj)) #2 to gitte in + Masal, (2 piAli 11 as g2
A/P yts c.s. 2aist 9:09 Ri HE BE eaate ny moI spoio I107 (B 488, Biei 48840 it 7177) spage Rie 7A48) 2 5t 2
ai US@ Lepa Sis ta/g--t) ay y 2512) f1o.0 2942 22229) 3 224 (85 2120) 1p40 2 4 s0 aiipii ojeto 20
gay 1221 959 & R o/ a goo= Ct # (550 2010 $89 # 21-I 2 oleret di q 4 1:1 21 (1el + 2 ae 71
ajgic) 3/ #2/aae olgo orga1 # 2 x c/j (]488) dig 200) 9#2 2 26/ Based image, sligns open interpretative expressions, preferring
conceptual
commoniy found contemporary
aporosches
meanings
perception and
uses color, lines shapes convey emotional emotional response subject, there isn't spectfic genre or theme, observer can connect their interpretation based on personal experiences the sellar help you undarstand the artist and the work better, leacing risher more enjoyable exper lonce. Additionally such often has personal connections or emotional appesi, making usefal to expliin how the artwork might function within the living space.
expression.
Peiings:/hyPPovidirgg information to
Reference Answer
( gelu 9-s
WLichann * 9111
M616 ti 4847
gi+ Mititi 80 788 28 $2 R/ 11 S.9 $// (/22) 9/4 Ri=
Al 2 (jE5)) 2 oig
CHF MAIE (el gata 1 *gg 9y 2
gaja oj0sS g9204 85 2.0 Sit 222 2002 3 lnb *= Via 2200 2a 2 US 2l 2 3 3/1 3 94190 Ieen
# al s N0 mee) w g 45 N of o/989 aid diel Weol Wt #400 2h-10) 1111
When trying to purchase a piece simler to the one in the image from an dealer gallery owner, you should descrioe both the stylitic and thematic elements that interest you Here's how you can articulale your pre erencestroon Styler Mention that you are looking for a work that embodies the energetic and gestural brushwork character sie of abstract expressionism, with vibrant colors and an ippovisational inin. Compostion:: Spectfy that you are seeking painting with complex and layered compesition, likely featuring mix figurative elam ents and abstract inym Color Palatts: Express your interwat in Ciootoonll Impact: Corvey that chass. Inins. Indicate the of painting would your guide you masterpleces allon your living
and varied onlor palette that create high contrast and harmony within the complesity of the design.utind. that
emptional dramalic impact, perhaps conveying livng
of movement allow bold expression destre.ininily providing
these detalls, the dealer better
Feedback from GPT-4o
The response focuses abstract and concaptual characteristios of the artwork, mentioning its interpretive nature and emotional resonence. Howavar,
compositin, color paletta, doas response discusses interpretive qualttees and imprrtance emotional connection, lickks clear identfication or descristion the artistic elements such as the abstract expressionist style, preferred composition, or ideal size, which are crucial for a prectical purchase discussion with en art desler, The informetion is generally sostract without articulating detaied preferences that would assist a seller in understanding exactly what is needed for the living room setting. 2
specifics ragarding
the emotional
While
An xChat example from the Korean subset and our model PANGEA'S 's response. The English part of the instruction, response, and reference answer is additionally added only on the figure to help to understand and was not given nor generated by PANGEA.
Figure 16:
35

---

Published as a conference paper at ICLR 2025
G
LANGUAGES IN PANGEAINS
Table 5 demonstrates the number of instances for each language that we include PANGEAINS.
Languages
en
multi
am
ar
bg
bn
Cs
de
el
es
Count
2554.6
4389.5
31.7
162.8
52.7
118.4
132.2
7.3
126.6
Percentage (%)
36.8
63.2
0.5
2.3
0.8
1.7
0.1
1.9
0.1
1.8
Languages
ga
ig
ja
jv
Count Percentage (%)
8.4
193.8
34.7
163.5
178.4
26.1
68.8
168.3
403.3
35.2
0.1
2.8
0.5
2.4
2.6
0.4
1.0
2.4
5.8
0.5
Languages
ko
mn
ms
no
pl
pt
ru
Count
182.5
4.5
37.6
39.2
60.7
8.2
176.5
147.2
211.0
0.6
Percentage (%)
2.6
0.5
0.9
0.1
2.5
2.1
3.0
0.1
Languages
ta
th
tr
uk
vi
zh
Count
33.0
132.7
37.1
64.6
160.4
140.7
7.4
156.4
315.5
557.4
Percentage (%)
0.5
1.9
0.5
0.9
2.3
2.0
0.1
2.3
4.5
8.0
Language distribution of PANGEAINS. We demonstrate the number of instances (in thou- sands) for each language, and the percentage of instances for each language in PANGEAINS
Table 5:
36

---

Published as a conference paper at ICLR 2025
H TRAINING EXAMPLES
H.1 MACHINE TRANSLATED INSTRUCTIONS
We include few machine- transsaaed training examples from PANGEAINS in Figure 17
Figure 18
Figure 19 Figure 20
Figure 21
Figure 22
H.2 MUL.TIULLTURA. UNDERSTANDING INSTRUCTIONS
We include a few multicultural understanding instructions from PANGEAINS in Figure 23
Figure 24
Figure 23 Figure 26
37

---

Published as a conference paper at ICLR 2025
Translated: Cambrian (Russian)
Human:
Cpeuum sарибониaя warta sac neex corpyinnson sоmmaniu coctasnser 20 pommapuu. Kaxoia uosacossa cransa ToMa? (The average hourly wage for all empioyees in the company is $20. What Tom' s hourly rate?)
HеRоSMOиGH опрелелитя почасоеуo ставку Towa пиатa в wac вOвX сотруднлкса компании не обязателано отражает почасоеуlo ставку moñoro отдел сотрудника, посколику нндиввдуулиныа оклады MoryT варымpоватıся зависимости oT рana dpakTopos, Takmx kak допмностт,
предостанпенной иисрорыанчи.. Средилn зеработнал
образование ирои@вооитe/eиист.. Kроme средиая заработная сотрудниксе компании рacснит-iBaeton nyrew делении oбшeй ваработной naaaa BOex сотрудников нa KопииeстBO wacce, отработаннык всеми сотруднлкаии, что не обязетельно отражает копииество насов, отработаннык KакиИ nu6o (Il is no! possible to determine Tom''s 's hourly uage based on the information procided. The average hourly wage for all employess of company does not neorssarlly reflect hourly wage of any individual emplayee, individual salaries can vary based number of factors, such job tille, experience, education, and performance. Additionally, the merage hourly wage for all employees of company csiculated by dividing the hotal wages of all euployers by the number of hours worked by all empiooyes, which does not necessarily reflect the number of hours worked by any individual emplogee.)
сотрудылком.
Figure 17: Translated Task: An example from the Cambrian dataset where it discusses the concept of hourly wages based given prompt in Russian.
Translated: ALLaVa-LAION (Turkish)
FREE
SHIPPING
WARRAN
TeCH SUPPORT
Human:
Dijital ekrandaki zamanlayucida görülebilen maksimum sure nedir?
(WWhet maximun time can be nn the timer digital display?)
GPT:
Dijital ekrandaki zamanlayıcıda görülabilen maksimum süre, V001 00.00.28"" sayilammin gōsterdiği gibi dakika, 00 saniye ve milisaniyedir.
(The maximmn: time that nan be sems on the timer on the digital display minute, 09 seconds and 28 milliseconds, as shown by the mumbers 00.00.281")
Figure 18: Multimodal Translated Task: An example from the VLLANN-LAION dataset where the
GPT model answers prompt in Turkish regarding the maximum time displayed digital timer.
38

---

Published as a conference paper at ICLR 2025
Translated: GPT-4v (Thai)
Human:
mwñuuae:
(Image showving)
www.aa/wwwwwaaaaaaaaaaaaaaaaaaaa luuvsdi mu www.eu/lllluuuuuuuuuuuuuuaaeaae audu wnne
sinuanfabssummssshieee Touiiannmans menewtr anddwwaaiiiabvrmmiuuuuueebn. Kuunsudwmissrnn @swioruuklwwweeeiinn waluuaannn nnädu gluuuueearrweereuuuuu www.:/wwwwwgwwoooouuuuuua....aa. www.iiiiwnmwnoiininommmmm Werwee viousanhenn www.wwwwnnllasssae wuuuaanannnnn wodnouuud uuasdsusna
Wusruau Twea iinerronnivannonnvaaa ferollaiiwnmeeeeelluun una:
(A woll-mounted wine rack with many wine bottles lined up vertically. The rack is made of several horizontil weoden placks, with circular holes cut
hold wine bottles horizontally, with the necks of the bottles facing outwoard. There are various types wine bottles, cade with different lobels and colors, iudicating the various types and brunds wine. The bottles arranged neatly and fill the rack. The background is a plain wall, which contrasts with the dark colors of the wouden racks and the colorful wine lnbels. The lngontt of image straightfororr and seems be practical photo, to clearly simply show wine rack and the wine bottles inside.)
An example from the ShareGPT-44 dataset where the model describes image of wine rack in Thai, detailing its structure and the arrangement of wine bottles.
Figure 19: Multimodal Translated Task:
Translated: OpenHermes (Japanese)
Human:
for
(Cin you help convert the for loop into functioo call to minincize the coule?)
GPT: म L UE! コー F It.. 'map'
V. Vn'nn avascripppttt ar @/ www.oll.loolellmm.. 2)) WWnjavaSrril 'map' M Z012.. # T
クッールEサレレTER intn'mast PEATTEET. #D for 1 7 TO CAARE
(Sure! To minimiize code, you can use 'map" function to achieve the same result. For example in on javsscript (ifeet arr 2, (apmarrmmpp((eeem)) transforn: each element aecurding to specified function. In this case, we multiply each clement by 2 before printing it to the console. in LnBy using mmap, we can eliminate the need for traditional for loop and achieve the some result in more concise way
consote.oglolem 2 Scrips' "map" fonction allows you iterate through army and
Figure 20: Text-only Translated Task: An example from the OpenHermes2.5 dataset translated into Japanese.
39

---

Published as a conference paper at ICLR 2025
Translated: NuminaMath (Spanish)
Human:
Definic una lerna ordenada S(A, B, CJS conjuntos como minimamente intersecantes como se describe: $IA Vccpp B| 15, $ Aeap CJ $JC Veepp A 15, SA Aeap B Veap C- Wemptysets. Determinar número de tales ternas ordenadas dunde cada conjunto es un subcunjunto de $1.112..445666..8 (J5. Cada cunjunto debe incluir ahora exactamente un elemento que no se comparte can los otros conjuntos, ningin conjunto puede estar vario. Calcular cuando múmero
divide entre 51000S.
(Define an ordered triple S(A, B, CJS of sets be nimimally intersectnng as follmss $1 \ Lepp RI 15, S Veup CI m 18, $1C (eup and camp Meup 4,5.2.3.43366,,,88 6,88 VS. Each must now include eaoctly one element that not shared with the other seis, and no set can be empty. Compute the remainder when the member of such triples is divided by $10005.)
lemphyuet5. Determine the namber such ondered triples wheree is suiset $1
GPT:
1. Elegir SxS, $yS, $2$ como identilicadorss únicos para las interseaciones por pares (SAnBS, SBrCS. y $CrAE). Hay $8 Voddt 7 Nodot 6 336$ maneras de elegir valores distintos para $x5, Syš. y Szš ya que el conjunto 2,3,1,33.4,5 2,3.4.5,677.88)) 6.7.8% contiene S8S números 'rin2. Cada conjunto (SAS, SBS, $CS) debe incluir exactamente un elemento único aparte de SxS, números restantes para
$yS. $zş. Esto significa selecoionar ningún elemento reufilice: in Elegir un elemento único para SA$ (no $xS o Syš) de los $5$ elementos restantes: $5$ opciones.In Elegir otro elemento úincoo para $B$ (no $y$ Sz$) los S4$ elementos restantes: $4$ opciones.In Elegir otro elemento único para $CS $xS Szş) elementos restantes pueden cualquier conjunto ninguno, por lo que hay $4$ opciones para cada uno de estos $2$ elementos (cualquiera de los tres conjuntos o ninguno): $4 185.vvnCoomina estas opciones, knlyy 338 Wimes Vtimes Wimms 107520 WJmPoo tanto, resto al entre $1000S es $iboxed(520)$.1
elemento único de
conjunto
$3S elementos restantes: $3S opoiones.
Los
(T. Choese Su5, Syš, $c5 as unique identifiers for the pairuise intersections (SANBS, SBnCS, and SCNAS). There are $8 \ | Vaaltt 3368 ways choose distinct vulues fur 3x3, Sy5, 525 the $11,1.23,45566
contatins unique element other than 5x6, Syg, $25. This meins selecting unigue element from the remaining $58 numbers for each set such that no element reused: (m- Choose a unique element for SA8 (not 5x5 Sy8) from remainingg 858 elements: 858 options. Ar- Choose another unique element for SB$ (not 8y6 or 5z5) from the ronaiwing $4$ clemests: $45 options. In- Chuose mother unique element for SCS (nst Sx5 SzS) from the romaining 535 elements:: 535 options. in 13. The remaining $25 elements can go into any set ur nomer so there are $45 options for each of these 525 elements (any of the fhree sets none): $4"2 165. Un uCoombriing these options, \at 336 Vitimes Limees times Viinees 16 107520 MVnThuss remainder when divided $1000S is (boned520055. ( \n")
mumbers. 1nin2. Each set (SAS, SBS, SCS) nust contati exictly
Figure 21: Text-only Translated Task: An example from the NumininaMathh dataset translated into Spanish.
40

---

Published as a conference paper at ICLR 2025
Translated: Code-Feedback (German)
Human:
Gegeben ein Array von Ganzzahlen, bestimmen Sie den Maximal- und Minimalwert des Arrays. Das Array kann doppelte Werte haben und kann eine beliebige Länge zwischen und 10% haben. AuBerdem kann das Array unsortiert sein.
(Given an array of integers, determine the maximum and minimmu values of the array, The army un favv duplicate values and can fuze any lougth between 10*6, Additionally, the array can unsorted)
GPT:
## Plan),n. Initialisieren Sie die Variablen min val und max val mit dem ersten Element des Arrays. 1n2. Iterieren Sie durch das Array und aktualisieren Sie min val und max val nach Bedarf. n3 Geben Sie min max zurbek (ee Plan| Initiuiize the variables már pul and pul with the first element of the army. \n2. Iterate through the army and update min_pol and pal as nended. 13. Return uin and max_pail pal)
Human:
Berechnen Sie den Umfang und die Fläche eines Dreiecks mit der gegebenen Seitenlänge. Seitenlängen: 6, 8, 10. Das Programm sollte auch profen, ob die angegebenen Seiten ein galtiges Dreieck bilden. Auterdem sollte das Programm Falle behandein, denen die Seiten kein gultiges Dreieck bilden, indem entsprechende Fehlermeldungeen ausgibt
(Calculate the perimeter and area a triangle with the given side length. Side lengths: 6, 8, 10. The program should also check whether the given sides form valid triangle. In addition, the program should handle where the sides do not form a valid triangle by giving appropriate error messages)
GPT:
Um den Umfang und die Fliche eines Dreiecks mit den gegebenen Seitenlängee (6, 8, 10) zu berechnen, können wir die Formel von Heron verwenden. Bevor wir jedoch mit der Berechnung fortfahren, müssen wir prüfen, die gegebenen Seiten gultiges Dreieck der Dreiecksungleichung anwenden, der besagt, dass die Summe zweier Seiten eines Dreiecks größer sein muss als die dritte Scite An) (ittennnn,SSSSeieen inShreiben wir ein Programm in Python, das den Umfang und die Fläche eines Dreiecks mit den gegebenen Seitenlängen berechnet und prüft, ab die Seiten ein gültiges Dreieck bilden))m) python) nimport math\n) (n calculate_ perimeter((, b, c)An return a + b+ chniedee calculate area(a. b, c)- s (a ba+b++ + / 2 Berechnen Sie den area math.sqrt(s (s a) (s b) (s) # Verwenden Sie die Formel von Heron),n
festzustellen, ab die gegebenen Seiten ein goltiges Dreieck bilden, können wir den Satz
halben Umfang)n
return
return Truekn return Falselnn),f Geben Sie die Seitenlangen des Dreiecks ein)ma 6/nb 8\nn 10) Prufen Sie, ub die Seiten ein gültiges Dreieck bilden),ni is_ valid triangleaa, b, <)A,n perimeter calculate
area' area),n) nnde is valid triangle(a, b, c) i + and c and a b.ln
calculate_perimeterr perimeter(a. by c)),n area calculate_areaaa, printt's "Umfang' I perimeteri), print() "läcche) mels:)) print)) "Die angegebenen Seiten bilden kein gülliges Dreieck.) "An"")) (nnnnWenn Sie dieses Programm austühren. wird Folgendes ausgegeben' (n) nn"n"" "/ann"""""nnnmmnn" UUUmang::: 24) nflache: 24.0%,n ".n. "nnnnDa die gegebenen Seitenlängen (6, 8, 10) den Satz der Dreiecksungleichng erfüllen, können wir den Umfang und die Flache des Dreiecks berechmen. Der Umfang beträgg 24 Einheiten und die Fläche 24 Flachenninheitenn
area(a, b, c)),n),
(lo calculate the perimeter and area of triangle with given side lengths (6. 8, 10), can Heron's formula. However, before proceed with the calculation, we need to check if the given sides form valid triangle. nnloo determine if the given sides form a valid triangle, we can apply the triangle inequality theorem, which states that the sum of two sides of triangle must be greater than the third side (n triangle with the given side lengths and checks if the sides form valid triangle:) (n) "python' pythonknimpurr math (n) indee calculate_periee perimeter(a, b, e)An return + h a clyn calculate
write program Python that calculates
perimeter and area
b, c) s (a 4 h c) 2 Calculate half the perimeter'ys in area mathaggrtss (s a) (s (6-b) b) (s 1--o) c)) Use Heron's formula' i return area) ndeef is valid triangle(a, b. d)An if b> 6)nb Bine = 10) Check if the sides form valid triangle)nit nif valid trianglefa, b, <)An perimeter calculate, calculate_perimmet perimeter(a, b. e)\n ansa calculate b, c)Ann, printf), "Perimeter... perimeter) in print()' Arear) 1 area)) nelse:) print() "The given sides do not form valid triangle. A'Ann (())""n)nnnnn nhhen you run this program, the following is output))n "nl.n""nn""""""aaa"aaaaaaa" '/Perimmtte: 24) nArra:: 24.0"n" "An" nSincc the given If the side lengths (6. 8, 10) satisfy the triangle inequality, we can calculate the perimeter and area of the triangle. The perimeter is 24 units and the area is 24 units.")
b + and
biln return True' in return False) Enter the side lengths the triangle),na na
Figure 22: Text-only Translated Task: An example from the Code-Feedback dataset translated into German.
41

---

Published as a conference paper at ICLR 2025
Multicultural Understanding: LAION-Culture (Portugese)
BR PETROBRAS
Human:
Descreva elementos que compõem logolipo da Petrobras imagem. (Describe the visual elements that make up the Petrobras logo in the image.)
GPT:
O logotipo da Petrobras na imagem compesto por letras metálicas prateadas e brilhantes, fixadas em uma parede de concreto cinza. A palavra 'PETROBRAS' escrita em letras maidsculas maiores, enquanto a sigla 'BR' aparece em letras quadrado separado. As letras são tridimensionais, que Ihes confere aparència moderna destarada. O contraste entre o brilho das letras e superficie opaca da parede cria um efeito visual interessante, destacando presença
(The Petrobras logo the image made of shiny silver metallic letters, fixed gray concrete wall. The wond PETROBRAS' writlen in Iorger capital letters, while the acronyn 'BR' appears in smaller letters, in separate square. The letters are fhree-- dimersional, giving than
look. contustt betereen brightness letters and opaque surfuce of creates on interesting visual effect, highöggtting the brand's presence.)
Human:
como design Ingotipo (Analyze how the Petrobras lugo design reflects the company's identity and values.)
identidade valores empresa.
GPT:
O design do logotipo da Petrobras reflete identidade os valores da empresa de várias maneiras. Primeiramente, metalicas prateadas brilhantes sugerem inovação modernidade, valores essenciais para uma empresa petróleo gus natural. riddimeniooaliddde das letras confere sensação de solidez estabilidade, reforçando a ideia de que Petrobras uma empresa forte confiável. Além disso, o contraste entre o brilho das letras a parede de concreto cinza ambiente urbano industrial. separação sigla 'BK' um quadrado distinto destaca importància da identidade brasileira para empresa, reforçando seu papel como uma entidade nacional. Portanto, o design do logotipo não s6 representa identidade visual da Petrobrs, mas também comunica seus valores e sua conexão comn Brasil.
pode simbolizar integração da empresa
design reflects the company' identity and values many letters suggest innovation and modernity, essential values for an and natural gas company. The three dimensionnaiitt letters gives feeling solidity and stability, reinforcing the idea that Petrobras strong and reliable company. Furthermone, the contrast between the brightness of the letters and the gray concrete wall can symbolize the company's integration with the urban and industrial environment. The separation of acronym 'RR' into a distinct square highlights the importance Brazilian identitt the company. reinforcing design not only represents Petrobras visual identity, but also communicates its values and its connection with Brazill)
(The Petrobras
Firstly, the shiny silver metallic
national entity. Therefore, the logo
Figure 23: Multicultural Understanding: An example from the LAION-Culture dataset where the
GPT model describes and analyzes the Petrobras logo in Portuguese.
42

---

Published as a conference paper at ICLR 2025
Multicultural Understanding: LAION-Culture (Bulgarian)
Kakno mnseemmem na snorrcus Reporned emeskava? (Nhhat the meaning the Jopanese dianccer in the pictare)
GPT:
Weeorruneerr
преaстaнnииа симвonа VBFV KOÄTO Vopul! ANCнOKи. Toow Reporлиe разпиини значения SавиCиMOCT OT контекста.
иanonава ознвнаване думата Vopui", KORTO сиулаn, Rерorииеet показаH черно мастипо вархy Grua xaprиn, CbC сини пунктирани лиНиИ, които рasдели CиMBona свкμиM, да
правылната noсnepовaтenhооT нa ударите при писането. Toea важон еnемeнт OT лпоноката калиграеии, копто ce нарииа Vwogoo".
(The hierogiyph in the photo is the character 1 1'r which roal \'ori'" in Japonese. This hieroglphh used denote the word context. case, hiernglyhh shows black on white paper, with blue dotted lines dividing the symbol into sections show the correct sequence strikes when writing. This is important element of Jopanese calligrahhy, which called "shado"))
('ori)" which can have different meanings depending
Human:
Kaкво зтеаненне сиOитте пунктирани иnoнckarra калиraаçss?. (What is the meaning of blue dotted lines in Japanese calligraphy?)
GPT:
Canree пуиктирраги munu итонисат kamиrрaspиx sиa enemuhurno эначени преruauиaeeee... оGvusaaaa mecne.ummeemeern pasueror Rеporaирд \ \BEX нa сekлnns,
ups
Reporussu, cnysaa, оснлури, всeKи удaр une Gaaze remapwen upascuuso. Tosa vacr or иpanamoria ипонската xamaradрiı, Roarro napeиsa "'uosoo'' craeemenia sиyaasaeee ираклииуиаиео допрыиаси ecremoueckoоo karecrso GинааиI проmивеeени
nponecst mucare
Toot иιхустоо.
лumut CaMp
nomarar правыиното
(The blue dotted lines in Japanese calligraphy have specific meaning and purpose. They were used to indicate the sequence and direction brush movements when writing hieroglyphs. In this case, these lines divide the character \ "II)"" ensure that each stroke
part the tradition of Japanese calligraphy, which is called "shodo," and is essential to the study and practice of this art. Blue lines not only help with correct writing, but also contribute to the aesthetic quality of the final work)
sections to facilitate
writing process
correctly.
Figure 24: Multicultural Understanding: Example from the LAION-Cultre dataset where the model interprets the meaning of Japanese character and its representation in calligraphy, described in Bulgarian.
43

---

Published as a conference paper at ICLR 2025
Multicultural Understanding: LAION-Culture (Hebrew)
Human:
Analyze the cultural significance of the prominent landmarks visible in the image Jerusalem, Answer it Hebrew.
GPT:
uurrdur
uonuEcrr uacr daeein uueedd' agscONl uhiuiane uaiioi vT by 1538-> 1535 DIWN Lin cuILCt
munIuu minn :abbem by D'nn-331 DILCTII UAiUL DOAUL MUnUU
EN pinn
m TUN C.ou DNGGNN tILLu CLLLIcT LLL marin cmpra nbu nyni N'LaUN munvma INI Ty/L by NuRLu UNZONN TTANIAI MULLUUU CCIINRL UHRL cl пaМea, JNNN (uUan
NRIIUU UUUUAt LUIIRRAI DIPAN nin AuRT I/
691 LIALL SdGILU' STOD TUMDIM DV CONLL N2nnN TUU DYANDD'NN ddeen SHER mAn. AMNNA DRLL JMmI qua Marana na pobbm
yau
DIWN
Ty4 Im unn 'M CaLCC
19900 AND NInI MIZmAN D'NUT by JIE
mwn..
(The imunge shoses throe naulti choracter complexes of jerusnlen:: the wolls of the Old City, the golden Dome of the Rach, and the Old City itself. The Old City Walls, which are accepted as a complex built between 1535 and 1538 by the Ottoman Sulten Suleimat the Magniffeent, symbollze
place great importance Judoism, Christimcity and Islam. The golden Dome of the Rock, built in 691 AD, stmuls as a shining symbal of Islam and represents the deep spiritual significance of this place
rich history and cultural and religious signiftoance
city itself, with its narrous streets and stone houses, reflects, among other Bhnngs, the rick cultural heritage jerusalem, whhere the languages, religims and cultures marge umique This landscape, with the trees and living streets, is a source of the life of the city and the cultural and historical signifioance of jerusalem spiritual and touristic center.") ")
Islamic period.
Human:
Imagine you are planning a guided tour Jerusalem on landmarks visible image. Outline the key points you would your Answer in Hebrew.
GPT:
.up.un nw by imrmuin myDean urc uredd IaDIO by .nomn by muvern by 1DP1 unn .Yy by
Tyп munia Cuucc ynn aunnn D'y'SDDW Dibinn muppan by сoanna DI>NTA 17111 NYD LOUDN 7I2AN munDin JNN mbbo .DWDD I An WOXTO mnnn LNNONN Dipn naDe "nwm inpinum nyn be DINN mamb
n by munnn numern byn 191 cacu nhe muMn by TaP1 an mnMMInn ybon nerdh huаaр nx mnn
EN Sari wmn cclll, Lcree' mynwan nunmenn bu wN DWDD y JUMEE uadr mInD by yOaN Ti TIN. 2yyn Tuna moman MIAT mnaw by VDmn a12 FI WUTT ypn m 175 by mubem aymon qun JAYN D."
mynwon mbaan unn vamva
niTInum MITGGOOIIIi
bys muyu aman mannnn
neec D'wa tt COLCI D'NN 7U SUmPPMM numn nysoomn
walls of the old city, which symbolize the historical barder of the city. would tell about the history the walls, about Suleiman the producer and their cultural significance. From there, I would move the group to the golden Dome of the Rock, and spiritual significance
(Agreeing a guided tour Jerusalem based
leading places that appear
picture, would
place Islam. would emphasize the unique architecture and the symbolic meaning of the dome. Finally, I would lead the group to the narrow streets of the old city, and would tell about the daily life, the rich cultural temperament and the tourist significance cultures
would about its construction
the area. would this affects the
emphasize unique combination languages, religions experience. 1 would conclude presenting the spectacular and peaceful view of the city, and the spiritual and historical significance of Jerusalem as an important center in the world.")
the city,
Figure 25; Multicultural Understanding: LAION-Cultur sample in Hebrew.
44

---

Published as a conference paper at ICLR 2025
Multicultural Understanding: LAION-Culture (Tamil)
Geuffiium' 3.0li uögadgfle ansiéduune aarjáddammm odaiffldaag (Describe charms of the book expressive imugery.)
Lég euut @ij unjuufuuggg, ammsmn. surenmen purdmmagg.. uágašga Gugunūdiá u empi: Upanishad Saram' morp smmrty genanmjuuuuss mineaui muudug,āld sœ &uu gaflsou edguusro a rioah @m dbbb aingggu @y a Gimneniggg symtba gafugii awing eareree aganron Dm dæsmmb. angiuppae 50 zgá RunuuLi m.gg &igi s.ffiiaaaai Sandnuggas @pnemb iópi egarna aqjaaaamm adliii Sip uăge a uBooaaasff zurrb aifa upgib @mpaost swan @mpohupjfls campimp Sng figth
(The attractions book combine Indian heritage, and history. The title 'Upamiisadd written the surface the book. In the center there is sculpture showing circles surrunnding which may be spiritual illumination or wisdom in the Upenishads. On the right is a human diagram, representing the wisdom and spiritual knowledge contained Upanishads. This book reflects the spiritual knowledge of Upanishads and the source of the Lord God.)
Upanishad Saram' is voluntarity of
Human:
uä suuisfuuss asuj flewmm Quj @ en @gunssssu.. (Create short story for parents of the book's attractive charms.)
GPT:
90 Siii ansob gpáteaned) 90 di dimnggí 9m Guflu zaenfl 9m s.ffe anpó esenp uágasmgg #uāggiöleensin.n siss uosedgle s. LuBugaaa syárfie sifee unggguh @enpausi aubiggi @oppuupite aganbb emiiita isa' Gahhngg ajan @ibs uggasmg nidgggdaaneee digmaggaa incheiddugg aqd0 inggu sydifii adfi udlifbant Algoommdadd Bibs dla át suaiifl geflugadmaa augn vida agnin Spgan..
(A long time ago, in a small village, great priest picked up a book called an Upanishad Saram. In that book the spiritual knowledge the Upanishads and source the Lord God shown. Asari took this book and imparted knowledge spiritual knowledge people of village. villagers began experience spiritual enlightenment with knowledge.)
Figure 26: Multicultural Understanding: LAION-Culurr sample in Tamil.
45

---

Published as a conference paper at ICLR 2025
1 BREAKDOWN RESULTS OF DIFFERENT LANGUAGES ON PANGEABENCH
1.1 XCIAA
We show the performance of different models on the xChat benchmark in Table
Models
English
Multi
Spanish
Hindi
Indonesian
Japanese
Korean
Chinese
GerPni-L.5-Pro
71.0
65.6
66.0
62.0
65.5
68.0
66.5
65.5
GPT4o
67.0
66.0
64.0
67.5
61.5
Llava-1.5-7B
22.5
16.7
22.5
3.5
18.0
23.0
12.0
21.0
Llava-Next-P
40.5
20.4
33.0
1.5
19.0
25.0
15.0
29.0
Phi-3.5-Vision
38.5
21.1
37.0
11.5
10.5
31.0
12.5
24.0
Cambrian-88 8B
27.5
15.8
22.5
4.0
20.0
20.0
10.5
18.0
Llava-OV-7B
51.0
33.1
45.5
6.5
42.0
36.5
26.0
42.0
Molmo-7B-D
49.5
34.7
45.0
19.5
36.5
36.0
35.0
46.0
Llama3.2-1118 1B
49.0
31.3
42.5
19.5
45.0
26.0
21.0
43.0
PaliGemma-3B
6.0
3.8
4.5
0.5
6.5
6.5
2.0
3.0
PALO-7B
27.0
16.2
23.0
3.0
19.0
20.0
13.5
18.5
mBLIP mTO-XL
2.5
0.5
0.0
0.0
0.5
2.0
0.5
0.0
mBLIP BLOOMZ-7B
4.0
1.7
2.0
2.5
2.5
0.0
0.0
3.0
PANGEA-7B (Ours)
46.0
35.8
43.5
23.5
34.5
39.0
33.5
40.5
Table 6: Comparison of models on the xChat dataset across different languages.
1.2 MULTILINGUAL LLAVABENCH
We show the performance of different models on the Multilingual LLaVABench benchmark in Ta-
ble 7
Models
English
Multi
Arabic
Bengali
Chinese
French
Hindi
Japanese
Russian
Spanish
Urdu
Gemini-L..-Pro
103.4
106.6
112.9
117.1
104.1
115.5
106.2
118.1
95.7
88.2
101.6
GPT40
104.6
100.4
98.3
111.9
96.5
101.1
99.7
104.0
88.5
100.9
102.5
Llava 1.5.a
66.1
40.8
26.4
11.9
50.7
63.8
23.2
70.0
46.5
59.2
15.4
Llava-Nexi-BB
78.9
50.7
24.9
11.2
72.8
91.4
18.0
70.1
71.8
82.9
13.4
Phi-3.5-Vision
70.8
58.0
50.1
35.1
69.2
86.0
35.9
67.6
75.6
39.3
Cambrian-SB
78.4
61.8
54.1
35.4
80.9
87.3
44.2
64.4
76.4
90.3
23.3
Llava-OV-BB
89.7
55.3
45.5
33.8
90.0
89.4
35.3
70.3
44.7
75.5
13.3
Molmo-7B-D
95.9
13.8
10.1
4.2
0.3
$9.6
5.5
6.0
8.7
29.5
0.0
1J.ma3.2-1
58.2
PaliGemma-3B
32.1
31.9
37.3
38.2
29.1
30.0
35.8
33.4
26.1
32.3
25.1
71.2
mBLIP mTD XL
32,7
28.2
33.7
26.2
3.6
39.8
26.9
26.8
34.1
36.9
26.0
BLOOMZ-7B
PANGFA-7B (Ours)
84.2
89.5
91.0
94.9
94.4
93.8
84.9
92.8
91.2
87.4
75.5
Table 7: Comparison of models on the Multilingual LLaVABench benchmark across different lan- guages.
1.3 CVQA
show the performance of different models on the CVQA benchmark in Table 8 and Table
I.4
MARVL
We show the performance of different models on the MaRVL benchmark in Table 10
I.5
XM100
We show the performance of different models the XM100 benchmark in Table 11
46

---

Published as conference paper at ICLR 2025
Models
ar-es
br-pt
bu-bg
ch-es
ch-zh
co-ts
ec-es
eg-ar
et-am
et-or
Llava-1.5-7B
37.8
51.1
35.6
42.4
44.4
50.6
48.6
31.5
27.8
3188
Llava-Next-BB
52.5
62.3
41.5
59.0
51.1
54.8
50.8
33.5
29.5
36.9
Phi-3.5-Vision
54.0
57.2
36.9
57.7
51.1
52.3
50.1
38.4
27.8
32.2
Cambrian-8B
59.6
60.6
42.0
64.5
59.5
57.7
56.1
40.9
27.8
25.7
Llava-OV-7B
64.5
69.7
49.6
67.1
69.1
66.8
65.5
47.8
32.5
41.1
Molmo-7B-D
61.1
69.0
$4.9
60.7
66.2
58.5
54.9
56.7
58.1
60.7
Llama3.2-113
69.1
74.6
64.2
70.5
73.6
69.3
66.9
68.5
68.4
63.1
PaliGemma-38B
48.7
53.9
39.1
53.4
53.7
50.6
45.3
40.4
24.8
28.0
PALO-7B
50.9
36.7
55.1
45.3
48.5
46.4
28.6
19.2
32.7
mBLIP BLOOMZ-7B
45.3
30.5
45.3
44.8
25.7
mBLIP mTTOXLL
40.8
44.4
38.0
44.9
39.9
42.5
31.0
35.9
26.6
PANGEA-7B (Ours)
68.3
72.9
53.9
70.5
74.0
64.7
63.5
49.3
36.3
35.5
Models
fr-br
in-bn
in-ta
in-te
ind-id
ind-jv
ind-mi
ind-sv
ir-ir
ja-jp
Llava-1.5-7B
29.4
31.1
29.8
28.0
41.7
32,0
32.7
33.5
42.6
37.4
Lllva-Next-7B
27.4
31.1
28.8
28.0
42.2
38.7
40.2
35.5
42.6
32.5
Phi-3.5-Vision
29.3
40.0
36.8
45.0
38.2
38.2
30.8
39.6
39.7
Cambrian-SB
31.6
38.1
50.2
39.4
Llava-OV-7B
34.3
56.3
43.9
46.5
58.0
45.8
45.4
40.5
50.6
Molmo-7B-44 7B-D
44.2
61.9
61.2
58.5
52.9
53.9
54.6
55.0
64.4
42.9
Llama3.2-1 IB
49.4
76.9
80.4
80.5
65.8
60.6
68.9
64.0
76.4
54.2
PaliGemma-3B
29.9
46.2
46.0
43.5
45.4
41.4
39.8
33.0
34.4
43.3
PALO-7B
29.1
37.8
31.2
25.0
41.3
32.3
32.3
32.0
42.9
30.5
mBLIP BLOOMMZ-7B
26.7
41.9
40.0
42.0
41.9
35,4
35.1
32.0
29.4
31.0
mBLIP mTO-XL
23.5
36.4
44.2
39.0
37.4
37.4
34.7
31.0
35.3
30.0
PANGEA-7B (Ours)
34.6
59.1
51.9
54.5
62.1
49.5
47.8
53.0
56.4
48.3
Models
ke-sw
ma-my
me-es
mo-in
ni-ig
no-ng
pk-ur
ph-fi
rO-PO
ru-ru
Llava-1.5-P2
34.4
42.2
42.4
26.9
34.5
47.5
26.4
43.8
47.0
51.0
Llava-Next-7B
46.2
45.7
51.4
33.3
35.0
56.9
36.6
46.8
52.3
53.5
-iissn
46.0
45.1
46.3
31.9
33,3
50.0
35.2
41.4
47.4
50.5
Cambrian-8P
50.5
52.1
56.7
34.6
36.0
53.5
48.6
47.3
52.0
61.5
Llava-OV-7B
46.5
55.6
59.4
35.9
33.5
62.5
58.3
56.2
60.3
75.5
Molmo-7B-M
73.3
54.6
53.6
51.9
53.0
54.8
67.1
57.6
63.6
61.5
Llama3.2-1
79.1
72.1
66.6
54.5
61.5
66.9
78.7
70.0
76.8
74.5
PaliGemmaa-3B
44.0
44.1
47.4
29.2
32.0
52.2
44.9
39.9
50.3
PALO-7B
35.9
44.3
29.5
49.2
44.4
46.0
mBLIP BLOOMM7-7B
37.0
42.5
28.8
33.0
47.7
46.0
34.0
mBLIP mTT0XL
45.1
40.6
44.9
29.2
30.5
42.8
40.3
32.0
43.7
42,0
PANGEA-7B (Ours)
64.1
59.7
62.2
42.3
46.0
64.5
66.2
$8.6
64.6
74.0
Models
rw-ki
sg-zh
sk-ko
sp-es
macro
Llava-1.5-7B
31.1
44.3
44.5
56.9
24.9
37.8
38.7
Llava-Next-7B
34.5
43.4
29.8
41.0
42.6
Phi-3.5-Vision
31.1
55.2
62.4
28.0
43.3
42.4
Cambrian-8tt
31.9
54.7
54.5
70.4
36.4
45.7
47.5
Llava-OV-7B
35.3
70.3
65.2
79,9
31.6
47.3
53.8
Molmo-7B-D
57.4
69.3
65.2
70.1
68.0
50.8
59.4
Llama3.2-11B
57.9
80.7
73.8
81.4
72.4
52.4
70.1
PaliGemma-3B
27.2
48.6
61.0
60.1
31.6
39.4
43.0
PALO-7B
28.9
45.8
44.5
64.8
28.0
39.4
39.3
mBLIP BLOOMZ-7B
29.4
47.6
33.1
56.6
28.0
39.4
36.9
mBLIP mTO-XI.
33.2
38.3
53.5
31.1
39.1
37.6
PANGEA-7B (Ours)
35.7
70.7
72.6
39.1
49.8
57.2
Table 8: Comparison of models on CVQA across different country-language pairs (in local lan- guages). Includes Macro-Acc.
I.6 XGQA
We show the performance of different models the xGQA benchmark in Table 12]
I.7 MAXM
We show the performance of different models the MAXM benchmark in Table 13
47

---

Published as a conference paper at ICLR 2025
Models
ar-es
br-pt
bu-bg
ch-es
ch-zh
co-ts
eC-Us
et-am
et-or
Llava-1.5-70
$6.2
61.6
52.3
60.2
54.0
55.6
55.5
50.2
51.3
53.3
Llava-Next-7B BB
53.9
61.3
50.9
59.8
58.8
60.2
52.8
54.7
52.9
58.9
-iison
59.2
61.9
54.9
64.1
58.2
59.3
57.5
50.7
54.7
58.4
Cambrian-88
57.7
66.5
56.1
65.4
64.3
39.3
60.2
56.7
60.3
56.5
Llava-OV-7B
63.0
73.9
59.3
65.8
68.8
65.1
63.3
62.1
59.8
59.3
Molmo-7B-D
57.7
65.8
45.6
63.7
68.5
57.3
55.0
43.8
31.6
38.8
Llama3.2-1 IB
66.8
72.9
54.4
72.6
72.0
66.4
65.2
56.7
41.9
32.2
PaliGemma-33B
51.7
49.3
51.7
54.9
54.8
47.2
51.2
52.6
51.4
PALO-7B
50.2
48.8
52.1
51.9
53.0
48.3
47.0
mBLIP mITOXIT.
38.1
39.1
42.7
43.7
41.1
40.9
42.9
42.1
mBLIP BLOOMZ-7B
46.0
51.4
41.5
44.4
48.9
49.0
45.0
45.3
38.9
46.3
PANGEA-7B (Ours)
67.2
72.9
60.1
68.8
67.2
64.7
61.6
59.1
60.7
56.0
Models
fr-br
in-bn
in-ta
in-te
ind-id
ind-jv
ind-mi
ind-sv
ir-ir
ja-jp
Llava-1.5-20
37.3
52.1
61.4
63.5
47.8
50.8
49.0
44.0
61.3
41.9
Llava-Next-7B
37.5
61.4
60.5
48.5
48.1
51.4
49.0
66.6
40.9
Phi-3.5-Vision
41.7
60.5
51.7
51.4
47.5
62.6
41.4
Cambrian-SB
40.7
65.6
58.2
56.0
66.6
42.4
Llava-OV-7B
44.2
69.6
72.0
70.5
59.0
55.9
59.4
58.5
76.4
47.3
Molmo-7B-- 7B-D
29.6
47.9
36.4
41.5
50.5
45.1
43.4
39.5
43.6
44.8
1.lmm3.2-1
36.3
62.9
66.4
66.5
63.6
48.8
58.2
54.0
57.4
58.1
PaliGemma-33B
37.3
59.1
66.0
62.5
49.3
48.1
43.4
46.0
58.3
44.8
PALO-7B
36.8
52.4
53.5
56.5
45.1
45,8
44.2
42.0
55.6
37.4
mBLIP mTO-XI.
30.4
43.0
46.0
41.0
38.1
39.1
38.6
32.5
37.4
34.0
mBLIP BLOOMZ-7B
34.6
43.4
52.6
49.5
41.0
44.8
38.2
30.5
42.3
36.5
PANGEA-7B (Ours)
45.2
67.1
71.0
68.0
60.4
57.2
56.9
56.0
72.7
45.8
Models
ke-sw
ma-my
me-es
mo-mg
ni-ig
no-ng
pk-ur
ph-fi
ro-ro
ru-ru
Llava-1.5 1.5-B
68.9
52.
47.9
45.8
$1.0
58.5
63.9
$2.7
55.6
59.0
Llava-Next-7B
71.1
54.9
51.1
44.2
53.0
57.2
67.1
56.7
62.6
58.5
Phi-3.5-Visioo
72.9
57.1
46.3
50.7
53.0
56.2
60.6
57.6
61.9
58.5
Cambrian-8R
74.4
61.9
56.7
48.7
56.5
60.5
73.1
60.1
66.6
61.5
Llava-OV-7B
79.1
65.1
63.2
52.6
57.5
64.2
75.0
64.0
72.5
72.5
Molmo-7B-. 7B-D
47.6
51.7
55.1
35.9
36.0
49.2
46.8
43.3
52.0
63.5
Llama3.2-1
61.5
64.7
41.0
39.5
65.9
65.7
66.0
75.5
74.5
PaliGemmma-31
59.7
51.7
43.4
46.0
55.2
67.6
48.8
60.9
56.0
PALO-7B
65.9
53.4
49,0
60.6
55.0
53.5
mBLIP mITOXL
50.2
34.7
33.9
39.5
43.1
45.4
36.9
43.7
41.0
mBLIP BLOOMZ-7B
$4.6
45.7
39.3
38.1
45.0
47.2
60.6
36.9
50.3
44,0
PANGRA-7B (Ours)
77.2
62.5
61.6
52.9
59.5
64.9
72.2
64.0
71.9
68.5
Models
rw-ki
sg-zh
sk-ko
sp-es
Llava-1.5-7B
51.1
56.9
58.7
54.2
7lxva-Next-7B
$2.8
60.0
38.7
55.7
Phi-3.5-Vision
52.3
59.4
66.5
66.7
61.3
46.3
56.3
Cambrian-SB
56.2
66.0
63.1
71.7
63.1
47.0
59.7
Llava-Ov-BB
55.7
73.6
67.9
80.2
72.9
48.9
65.2
Molmo-7B.. 7B-D
34.9
66.0
56.9
66.7
31.6
44,8
48.3
Llama3.2-11
40.4
73.6
73.1
83.3
51.1
56.2
61.2
PaliGemma-3B
44.7
59.4
58.3
61.0
62.2
40.6
52.9
PALO-7B
51.9
56.1
55.9
62.9
54.2
42.2
50.9
mBLIP mTT0XL
38.3
43.9
41.4
51.9
48.0
34,9
40.5
mBLIP BLOOMZ-7B
45.1
46.9
58.5
46.7
34.0
44.9
PANGEA-7B (Ours)
56.6
71.7
66.6
75.2
70.6
52.7
64.4
Table 9: Comparison of models on CVQA across different country-language pairs (in English). Includes Macro-Acc.
48

---

Published as a conference paper at ICLR 2025
Models
English
Multi
Indonesian
Swahili
Tamil
Turkish
Chinese
GPT4o
81.8
82.3
81.9
80.8
80.2
86.4
82.1
Gemini-1.5-Pro
76.4
72.0
71.2
67.8
70.0
75.4
75.8
Llava-1.5-7B
56.2
53.7
56.1
49.8
49.7
55.4
57.5
Llava-Next-7B
62.8
50.9
50.6
50.5
50.6
Phi-3.5- Vision
72.1
56.5
58.6
51.4
52.0
58.6
61.7
Cambrian-8B
75,4
61.8
64.7
53.6
56.7
65.2
68.9
Llava-OV-7B
72.7
57.5
60.9
51.2
51.9
63.5
60.0
Molmo-7B-DD
65.3
54.9
61.1
49.6
49.6
52.2
62.2
Llama3.2-1
58.1
62.7
52.4
54.0
61.6
59.5
PaliGemma-3h
56.5
52.2
53.4
49.6
50.5
56.3
51.3
PALO-7B
63.3
54.2
58.3
50.6
51.9
54.9
55.3
mBLIP mTO-XL
67.3
66.7
64.9
64.8
69.7
68.1
65.9
mBLIP BLOOMZ-7B
56.2
60.3
57.7
59.7
PANGEA-7B
87.0
79.0
81.3
75.1
69.4
84.8
84.3
Table 10: Comparison of models on the MaRVI. dataset across different languages.
1.8 XMMMU
We show the performance of different models on the xMMMUU benchmark Table 14
1.9
M3EXAM
We show the performance of different models on the M3Exam benchmark in Table 15
I,10
TYDIQA
We show the performance of different models on the TyDiQA benchmark in Table 16
1.11
XSTORYCLOZE
We show the performance of different models on the XStoryCloze benchmark in Table 17
I.12
MGSM
show the performance of different models on the MGSM benchmark in Table
I.13
MMMLU
We show the performance of different models the MMMLU benchmark in Table 19
49

---

Published as a conference paper at ICLR 2025
J A PRELIMINARY EXPLORATION OF CONSTRUCTING MULTILINGUAL OCR INSTRUCTIONS
Optical Character Recognition (OCR) is critical capability for multimodal LLMs, enabling them to interpret and process textual information embedded within images. However, most existing OCR training datasets are predominantly English-centric, which limits the models' performance in non- English contexts. To address this gap. we have curated a comprehensive set of 500K multilingual OCR training samples from web user interfaces, spanning 10 languages, with 50K examples per language, sourced from web user interfaces. Webpages naturally serve as image-rich environments containing abundant text, and by capturing screenshots of websites from various countries in differ- ent languages, we were able to gather substantial number of OCR images.
We utilize URLs from the --NNlws-uullllngaaa 3 dataset (Hamborg et al., 2017) obtain a di- verse set of multilingual web pages. Using Playwrigh I we render each website and automatically capture screenshots under various device settings and resolutions to achieve a wide range of image dimensions and aspect ratios. Each screenshot includes red bounding box that highlights specific element targeted for OCR extraction. We focus on ten languages for this dataset: English, Chinese, Japanese, Korean, Indonesian, Hindi, Spanish, French, Portuguese, and Arabic. We totally have 1M samples (50K for each language).
We employed the same model architecture PANGEA but trained it exclusively on these OCR images, reserving a portion of the data test set. As shown in Figure 27
Largueges
OCR Accuracy
the results indicate that improving multilingual OCR per- formance feasible with an increase in training samples. However, the OCR accuracy for non-Latin scripts (e.g., Chinese, Japanese, and Korean) remains lower than for Latin-based languages. Looking ahead, we aim to further expand the multilingual OCR training dataset to include more languages and integrate this data into PANGEAINS.
50 40
200K
300K 400K
500K
Number of Training Samples
Figure 27: preliminary exploration of multilingual OCR.
hhatss//huggingfacc hhtp:://hniiinnliseewwwaaaaattta cwlattsttiiniffooo//miiinnggg ce news
50

---

Published as conference paper at ICLR 2025
Models
English
Multi
Arabic
Bengali
Czech
Danish
German
Greek
Gemini 1.5 Pro
7.5
27.6
GPT4o
27.7
19.1
15.8
13.5
21.1
25.3
19.3
21.1
Llava- 1.5-B
0.0
3.1
Llava-Next-7B
29.3
9.4
5,6
0.1
12.1
15.7
14.4
4.2
Phi 3.5 Vision
30.2
5.2
2.4
16.2
0.0
20.7
Cambrian-8B
20.6
9.9
1.4
6.6
7.4
15.1
15.5
4.4
Llava-OV-7B
30.6
7.0
0.6
16.8
14.0
Molmo- 7B-D
22.1
9.1
7.9
13.8
12.2
4.2
Llama3.2-118
27.6
4.5
0.0
0.0
1.5
118
4.6
1.2
PaliGemma-3BB
0.8
3.1
2.7
PALO-7B
30.4
0.8
0.0
0.0
2.0
1.0
2.7
0.0
mBLIP BLOOMZ
22.5
10,3
9.5
6.4
11.5
15.9
14.5
10.9
(Ours)
Models
Spanish
Persian
Finuish
Filipino
French
Hebrew
Hindi
Croatian
Gemini-1.5-PPu
39.5
4.2
29.0
28.7
42.4
4.3
2.2
33.8
GPT4o
20.4
1lvv--1,577
3,7
0.0
0.4
1.1
2.0
0.1
0.0
0.3
Llava: Dava-Next-PP
Phi-3.5- Vision
20.7
0.0
1.0
1.7
21.2
0.3
0.0
0.5
Cambrian-8B
18.6
9.6
19.6
5.8
6.8
Llava-OV-7B
24.9
3.8
1.5
4.2
22.0
0.0
4,4
7.2
Molmo 7B.
19.8
11.3
13.0
19.8
8.3
9.4
1
10.2
0.0
8.4
12.0
0.0
0,2
PaliGemma-3B
0.7
0.0
0.1
0.1
0.6
0.0
0.0
1.3
PAI P-LO-7B
1.5
0.0
0.9
2.1
0.0
0,0
0.2
mBLIP mTO-XL
8,3
5.5
1.7
2.8
6.4
4.0
1.8
0.9
BLOOMZ
7.7
7.5
PANGEA-7B (Ours)
26.2
19.3
3.8
18.9
26.7
18.2
17.4
10.8
Models
Hungarian
Indonesian
Italian
Japanese
Korean
Maori
Dutch
Norwegian
Gemini-1.54
3.8
27.7
GPT4o
21.8
28.4
21.0
0.0
11.1
26.8
26.4
24.7
Llava-1.5-7B
0.9
2.9
Alava-Nexi- 7B
9.3
14.7
17.6
4.2
5.2
9.2
23.8
16.3
Phi-3.5-Vision
Cambrian-8B
6,6
15.7
15,5
7.2
2.0
3.2
20.3
16.0
Llava-OV-7B
16.4
12.8
0.6
1.7
24.7
13.9
Molno 7B-D
3.5
17.2
17.8
5,2
2.4
7.5
15.7
13.8
Llama3.2-11B
1.2
16.0
0.0
9.3
22.0
1.1
PaliGemna-33 3B
2.0
0.2
1.8
0.0
4.0
2.6
2.3
PALO-7B
1.1
0.0
0.0
0.1
3,5
0.7
mBLIP mT0-XI.
6.0
0,3
3,4
mBLIP BLOOMZ
11.8
16.0
16.5
0.0
4.5
0.1
18.2
14.5
PANGEA-7B
2.1
Models
Polish
Portuguese
Quechua
Romanian
Russian
Swedish
Swahill
Telugu
Gemini-1.5 1.5
35.5
35.7
0.7
31.2
32.4
37.8
10.7
0.0
GPT4o
22.2
28.0
19.1
20.7
26.0
20.0
Alava-1.5-7B
0.8
2.5
0.0
1.6
0.5
2.0
0.1
0.0
Llava-Nest-7B
3.2
Phi.3.5 Vision
1.0
21.0
0.4
3.2
0.7
12.5
0.4
0.0
Cambrian-8B
3.7
Llava-OV-7B
7.4
24.6
0.0
6.8
5.5
15.0
2.0
0.0
Molmo-7B-D
14.1
3.8
Llama3.2-1
1.0
18.6
0,0
10.1
0.6
7.4
5.8
0.0
PaliGemma-38
1.3
0.8
2.0
0.0
0.0
PALO-7B
1.7
1.1
0.9
0.2
mBLIP mTD-X1.
5.8
0.2
2.3
3.1
3.7
3.8
2.7
mBLIP BLOOMZ
11.8
13.7
8.4
PANGEA-7B (Ours)
16.2
28.1
0.0
21.4
20.9
19.4
18.7
0.1
Models
Thai
Turkish
Ukrainian
Vietnamese
Chinese
Gemini-1.5-PP
0.0
0.9
0.0
0.0
0.9
GPT4e
16.9
30.9
Llava-1.5-7B
0.0
0.0
0.0
0.0
0.0
Llava Diva-Next-PP
Visiion
0.5
1.9
0.0
2.2
0.0
Llvv--Ov-7B
0.0
0.0
0.0
0.0
2.9
Molmo-7B- -B-
0.0
0.0
Llama3.2-111
0.0
0.0
0.0
0.0
299
PaliGemma
0.0
0,2
0.0
P-L.0-BB
0.0
0.1
mBLIP mT0-XL
0.0
3.9
2.0
7.1
0.0
mBLIP BLOOMZ
1.9
2.2
PANGEA-7B (Ours)
0.0
0.0
0.3
0.0
4,9
Table 11: Comparison of models on the XM100 dataset across different languages.
51

---

Published as a conference paper at ICLR 2025
Madels
English
Multi
Bengali
German
Indonesian
Korean
Portugnese
Russian
Chinese
Gemini- 1.5-Pri
$4.2
48.7
49.4
50.2
48.6
46.4
51.2
44.8
50.2
GPT4o
55.8
51.0
49.4
52.6
50.4
51.0
52.2
50.0
51.4
Llava-1.5-7B
62.0
30.7
15.6
28.4
33.4
38.2
27.5
33.1
38.4
Llava-Next-7B
64.8
37.8
11.5
41.3
37.3
42.5
39.8
43.5
48.2
Phi-3.5-Vision
38.4
7.7
51.4
36.0
49.6
46.2
41.4
Cambrian-81
64.6
39.8
32.3
44.6
36.0
43.6
41.6
44.2
36.2
Llava-OV-7B
64.4
48.2
41.8
49.2
48.8
45.3
52.4
$4.0
45,9
Molmo-7B-D
51.5
43.0
25.6
45.9
44.9
44.2
46.5
45.6
Llama3.2-11B
55.6
45.4
42.9
46.7
46.2
44.5
46.5
44.7
46.1
PaliGemma-3B
59.7
30.5
13.3
44.5
21.3
22.8
34.7
35.8
41.2
42.2
36.8
31.7
mBLIP mTTOXL. XL
44.2
39.9
39.1
41.1
39.1
39.7
40.7
40.2
39.4
mBLIP BLOOMZ-7B
43.3
36.9
37.7
36.3
39.3
28.5
40.7
36.6
39.1
PANGEA-7B (Ours)
64.7
60.2
58.9
61.6
60.1
58.9
61.8
60.4
59.6
Table 12: Comparison of models on the xGQA dataset across different languages
Models
English
Multi
French
Hindi
Hebrew
Romanian
Thai
Chinese
Gemini-1.5-Pro
56.4
63.5
60.2
66.5
65.7
57.4
73.9
57.4
GPT4o
60.7
65.4
59.8
68.8
70.0
61.3
76.5
56.3
Llava-1.5-7B
49.8
20.4
32.2
17.3
12.9
15.1
17.2
27.8
Llava-Next-7B
54.9
21.4
33.7
16.2
10.7
15.5
18.3
33.9
Phi-3.5-Vision
55.3
25.0
38.3
31.9
17.5
10.9
24.3
27.4
Cambrian-8B
55.3
28.7
41.7
23.8
17.1
32.0
25.7
31.8
Llava-OV-7B
34.8
37.9
31.9
17.8
30.2
53.0
37.9
Molmo- 7B-D
52.9
37.5
45.5
33.5
30.7
28.9
46.3
40.4
1lmma322-1IB
55.3
43.9
48.1
50.4
41.8
36.6
56.7
30.0
PaliGemma--B
47.9
19.9
8.0
36.5
19.3
13.4
31.3
10.8
PALO-7B
51.4
16.3
33.7
15.8
12.1
11.3
14.6
10.5
mBLIP mTO-XL
44.7
36.8
36.0
42.7
28.9
30.3
56.3
26.4
mBLIP BLOOMZ-7B
44.7
24.8
33.0
47.3
8.9
16.9
33.2
PANGEA-7B (Ours)
55.3
53.3
43.6
53.5
59.3
45.8
67.2
50.2
Table 13: Comparison of models on the MAXM dataset across different languages.
Models
English
Multi
Arabic
French
Hindi
Indonesian
Japanese
Portuguese
Gemini-1.5-PP (0801)
65.8
$7.7
S7.7
58.1
55.5
60.2
55.0
59.6
GPT4o (0513)
69.1
58.3
56.7
58.1
58.1
59,9
58.0
58.9
Llava-1,5-7B
31.5
29.5
27.5
Llava-Next-7B
30.5
30.9
37.0
Phi-3.5-Vision
42.6
38.8
35.6
44.0
30.9
36.7
37.9
47.8
Cambrian-8B
32.6
34.6
30.9
33.5
36.0
Llava-Ov-7B
46.3
41.0
41.6
43.0
34.7
43.4
40.1
43.4
Molmo-7B-D
42.9
40.4
40.6
42.6
32.6
40.7
43.9
42.1
Jlama322-11B
39.2
34.0
33.6
39.6
32.3
36.7
29.0
33,0
PaliGemma-3B
26.3
25.2
29.2
23.8
21.6
24.2
24.5
27.6
PALO-7B
33.1
30.5
30.5
33.2
28.9
34.0
27.1
33.3
mBL.PP mTO-X1.
29.3
30.4
30.2
33.2
28.2
26.9
31.6
32.3
IBI.IP B1.00M7-7B
29.2
30.8
28.5
33.9
27.8
33.3
31.6
29.6
PANGEA-7B (Ours)
42.3
45.3
41.6
46.1
Table 14: Comparison of models on the xMMMU dataset across different languages.
52

---

Published as a conference paper at ICLR 2025
Models
English
Multi
Afrikaans
Chinese
Italian
Portuguese
Thai
Vietnamese
Gemini-1.55PP
77.4
64.7
80.4
74.1
76.3
61.8
49.9
46.0
GPT4o
68.0
61.0
73.0
68.0
67.0
58.0
52.0
48.3
Llava-1.5-7B
32.3
29.0
28.2
24.3
40.1
28.2
23.7
29.3
Llxva-Next-7B
36.5
28.4
28.2
25.4
37.8
27.0
23.7
28.4
Phi-3.5-Vision
55.8
37.2
44.2
40.8
51.4
40.3
25.2
21.6
Cambrian-8B
34.7
33.4
36.8
34.2
45.2
30.3
28.9
25.0
LVv7a-OV.7B
60.4
45.8
50.3
58.0
57.2
43.8
30.9
34.5
Molmo-7B-D
57.1
39.1
35.6
56.4
49,4
40.2
27.4
25.9
Llama3.2-1
51.8
36.6
42.3
46.4
45.8
28.4
26.4
30.2
PaliGemma-3B
36.0
25.6
26.4
24.7
32.2
24.3
27.2
19.0
PALO-7B
30.8
27.8
31.9
22.1
36,9
32.3
22.7
20.7
mBLIP mTO-XL
22.8
25.0
16.0
25.6
33,7
21.2
22.4
31.0
mBL.IP BI.00M7-7B
30.3
29.5
28.2
29.8
37.3
28.3
22.9
30.2
PANGEA-7B (Ours)
61.4
42.1
52.1
49.2
54,9
43.3
32.9
19.8
Table 15: Comparison of models on the M3Exam dataset across different languages.
Models
English
Multi
Arabie
Bengali
Finnish
Indonesian
Korean
Russian
Swahili
Telugu
Vicuna- .,
59.7
52.7
32.3
68.1
63.0
72.6
58.8
$7.6
51.3
18.1
Qwen2-7B Instruct
72.2
71.2
67.6
75.9
67.1
78.0
64.9
67.2
75.3
73.8
1-57-1.5-7B
66.8
52.8
61.8
33.4
60.2
72.8
63.3
55.0
55.0
20.6
Llava-Next-7B
68.3
52.1
64.5
24.9
63.0
74.3
61.9
58.4
53.1
17.0
Phi-3.5- Vision
57.3
28.3
PALO-7B
69.4
50.8
60.9
46.0
61.8
70.6
56.8
56.7
42,5
10.8
PANGEA-7B (Ours)
73.7
66.0
55.5
65.3
66.3
74.5
69.4
60.1
76.6
60.0
Table 16: Comparison of models on the TyDiQA dataset across different languages.
Models
English
Multi
Arable
Spanish
Basque
Hindi
Ind.
Burmese
Russian
Swabiti
Telugu
Chinese
Vicuna-1.5-77 1,5-7B
57.4
52.7
69.4
48.4
66,5
52.1
$4.5
63.5
Qwen2-7B. 7B--strrc
80.3
61.9
64.0
71.6
51.6
39.6
68.5
50.7
72.7
53.2
55.3
72.1
1.5--
69.2
Llava-Next-7P
79.1
57.1
51.7
68.8
50.3
54.5
62.0
46.7
65.5
52.1
55,2
63.8
Phi-3.5-VVision
77.9
54.8
53.7
67.2
50.4
51.9
51.7
47.8
61.3
49.3
52.5
59.5
PANGEA-7B 7B (Oun))
79.1
61.2
60.5
67.8
50.0
61.8
66.4
48.7
69.4
58.9
60.4
68.2
Table 17: Comparison models on the XStoryCloze dataset across different languages.
Models
English
Multi
Bengali
German
Spanish
French
Japanese
Russian
Swahiii
Teluga
Thai
Chinese
Vicuna-1.5-778
17.6
6.4
0.0
14.4
14.4
2.8
3.6
2.0
Qwen2-7B-Instee
48.8
40.4
0.0
67.2
67.6
68.8
11.2
71.2
10.8
45.6
59.2
Llava-1 1lavu-1.5-78
14.8
7.5
0.0
15.2
10.8
18.0
2.8
11.2
0..
0.0
1.6
15.6
Llava Next-7
15.6
7.5
0.0
13.6
13.2
16.0
1.6
12.8
2.0
1.6
14.0
Phi.a.5-Vision
59.2
PALO-7B
13.6
5.8
0.0
11.6
9.6
13.2
1.6
8.8
0.4
0.0
0.0
12.4
PANGEA-7B (Qurs)
82.0
47.4
0.0
68.4
74.8
63.2
22.0
68.0
49.6
68.0
Table 18: Comparison of models on the MGSM dataset across different languages.
53

---

Published as a conference paper at ICLR 2025
Models
English
Multi
Arabic
Bengali
Portuguese
Chinese
French
German
Seuna-1_5-78
49.5
34.7
30.3
28.5
39.6
36.9
40.4
39.8
Owen2-7B-Instree
70.1
53.1
51.0
43.4
60.7
63.8
61.5
57.7
Alava-1.5-7B
50.2
34.9
29.7
28.5
40.3
36.8
40.1
39.8
Llava-Next-7B
52.1
35.6
30.0
28.8
40.7
37.3
41.4
41.4
Phi-a.5-Vision
62.0
39.1
34.9
27.9
47.6
41.5
49.2
45.8
PALO-7B
46.7
32.6
30.3
29.5
36.0
34.2
36.9
35.8
PANGEA-7B (Ours)
68.4
52.2
49.3
44.4
58.9
60.5
58.9
56.7
Models
Hindi
Indonesian
Italian
Japanese
Korean
Spanish
Swahili
Yoruba
Vicuna- 1.5-7B Qwen2- 2een2-7B-Instruc
29.8
36.5
39.5
35.9
34.1
40.3
27.9
26.8
45.7
57.1
60.8
58.0
54.6
61.9
36.0
31.8
1lava-1.5-7B
29.2
37.1
41.0
35.1
34.1
41.6
28.0
27.3
Llava-Next-7B
29.6
37.5
41.2
36.0
34.2
42.7
28.5
28.7
Phi-3.5-Vision
32.9
38.3
47.0
40.0
36.6
49.6
28.9
27.8
PALO-7B
29.6
33.7
36.4
32.7
30.6
37.0
26.4
27.1
PANGEA-7B (Ours)
45.7
55.4
58.8
55.3
52.7
59.7
42.8
31.3
Table 19: Comparison of models on the MMMLU dataset across different languages.
54


Reviews:
[{"review_id": "TCk2RgSMMv", "reviewer": "Reviewer_e2zk", "summary": "This paper describes PANGEA model and data, with a focus on multilingual multimodal settings. \tPANGEAINS is an instruction dataset that spans 39 languages that covers the different languages through translation and the culture references aspects of the data. PANGEABENCH is an evaluation suite consists of 14 datasets covering 47 languages. PANGEA, a model based on the LLaVA-Next architecture and trained on the PANGEAINS data, showed strong performance on several tasks comparing to strong baseline models.", "questions": "1. Would the other open source models used as the baseline in this paper could also benefit from training on the PANGEAINS data?\n2. How to validate if there is potential overfitting of training on the PANGEAINS and evaluating on PANGEABENCH?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addressed an important research problem on multilingual multimodal learning, by providing a large instruction dataset and evaluation suite that covers multiple languages with the corresponding culture reference annotations. This data could be of interest to a broad audience in the research community. \n2. The paper is well written and easy to follow. The proposed PANGEAINS instruction data and the PANGEABENCH evaluation suite were described with good amount of details. The authors further provided examples of the data which is helpful in understanding the dataset proposed.\n3. The authors conducted comprehensive experiments of the proposed model and data, by comparing it to several strong baseline models and evaluating on a diverse set of tasks. The performance of the model seems to be pretty strong from the evaluation results.", "weaknesses": "1. It could be helpful to a section to discuss related multilingual multimodal datasets (e.g. M3IT etc), and show how PANGEA compares to them. This could further illustrate the significance of the proposed data here.\n2. Although the authors highlighted the introduction of PANGEA, the discussion on the modeling aspect of PANGEA is light. It seems to be a directly application of a LLaVA-Next based model trained using the new data proposed. It could help to further discuss the training specifics.\n3. The proposed method (model and data) is only evaluated on one eval set, which is the proposed PANGEABENCH. It could be helpful to evaluate on additional evaluation suits to see how the improvement from training on PANGEAINS would generalize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5djJTEbxz2", "reviewer": "Reviewer_rNZk", "summary": "This paper introduces Pangea, a multilingual multi-modal LLM together with its multi-modal instruction fine-tuning dataset PangeaIns and a comprehensive benchmark PangeaBench:\n- PangeaIns includes 6M examples from 39 languages and consists of (1) high-quality English instructions, (2) machine-translated instructions, (3) culturally relevant multimodal tasks\n- PangeaBench, a multimodal evaluation suite consists of 14 diverse datasets covering 47 languages, including a human-curated multimodal chat benchmark xChat \n\nPangea outperforms other open-source models by 7.3 points on English tasks and 18.0 points in multilingual tasks. Finally, the paper discusses the impact of English data, the necessity of language-specific datasets, and the limitations in multimodal chat and multilingual OCR.", "questions": "A. For text-only performance, how is the chat performance compared to the other models? Some evaluation benchmarks such as MTBench, ArenaHardAuto or AlphacaEval can be used for this comparison. \n\nB. Have you checked if there is any data leakage from the benchmarks that has been tested?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "A. This paper includes a very significant set of contributions which will be made public together with the code:\n1. PangeaIns with 6M instructions from 39 languages that include (1) high-quality English instructions, (2) machine-translated instructions, (3) culturally relevant multimodal tasks\n2. PangeaBench, a multimodal evaluation suite consists of 14 diverse datasets covering 47 languages\n3. Pangea an MLLM covering 39 languages outperforms existing open-source models by 7.3 points on English tasks and 18.0 points in multilingual tasks.\n\nB. The paper presents a detailed multilingual multi-modal data generation pipeline showcasing the limitation of open-source machine translation models, and the use of LLMs to enrich the multilingual data where the LLM score and filter images based on cultural informativeness. Then remaining data is enriched with re-generating descriptions and creating complex instructions. \n\nC. PangeaBench covering 47 languages in 14 datasets introduces a comprehensive evaluation for multilingual multimodal LLMs. Notably, the new evaluation dataset xChat is a human-curated benchmark where reference answers and scoring rubrics are annotated for each query. This enables a better use of LLM-as-a-judge for scoring other models' generations using a scale of 1-5 for each query.", "weaknesses": "There are only two minor weaknesses mainly focusing on the need for clarity and the presentation:\n\nA. In Section 2.1, the paper mentions a post-processing pipeline for noisy translations, however, details are missing. This might be quite important for practitioners to build similar models. Therefore, I suggest authors to include more details there.\n\nB. In Section 4.1, it has been mentioned that the model is based on Llava-Next architecture but this needs to be detailed. For example, does the training include multiple stages such as adapter alignment and SFT, or is only SFT has been used? If so what datasets are used in which stage?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jHlcMIJrDZ", "reviewer": "Reviewer_GEq8", "summary": "PANGEAINS is a multilingual and multicultural dataset comprising 6 million samples across 39 languages, developed for instruction tuning. This dataset emphasizes linguistic and cultural diversity and implements strategies such as machine translation, cultural understanding guidelines, and high-quality image caption generation to address challenges in multilingual multimodal learning. By using machine translation, English instructions are extended into other languages, and culturally biased issues are resolved by curating images from the LAION-Multi dataset to capture cultural contexts.\n\nPANGEABENCH is a comprehensive evaluation tool designed to assess PANGEA's performance across various languages and tasks. It includes diverse benchmarks that evaluate capabilities in cross-lingual and cross-cultural contexts. The multimodal tasks encompass natural conversation, image captioning, cultural understanding, multilingual visual question answering, and multi-subject reasoning, while text-only tasks are also included to examine the model's linguistic understanding.\n\nExperimental results show that PANGEA-7B outperforms existing models in both English and multilingual tasks, particularly demonstrating significant improvements in cultural understanding tasks. The performance gap between English and multilingual tasks is small, indicating strong multilingual processing capabilities, although there remains a performance disparity compared to closed-source models. Additionally, PANGEA-7B exhibits the strongest performance in text-only tasks, with notable enhancements in handling complex multilingual reasoning and mathematical tasks attributed to the inclusion of math-related instructions.\n\nThe discussion explores the scaling effect of instruction quantity, the role of English data, the relationship between training sample proportions and performance, real-world applications of multimodal chat, and preliminary explorations of multilingual OCR. These insights aim to provide a comprehensive understanding of the model and outline future advancement directions. The development of PANGEAINS and PANGEABENCH is expected to significantly contribute to enhancing the performance of multilingual and multimodal models.", "questions": "Data Representation: What strategies can be implemented to ensure a more balanced representation of low-resource languages within the PANGEAINS dataset?\n\nQuality Assessment: How can we establish rigorous quality assessment protocols to evaluate the accuracy and relevance of data for each language in the dataset?\n\nCultural Context: What methodologies can be employed to enrich the dataset with culturally relevant examples and ensure that cultural nuances are adequately captured across all languages?\n\nData Augmentation: What role can data augmentation techniques play in enhancing the dataset, particularly for underrepresented languages, to improve model performance?\n\n\nAs a reviewer, I would like to provide detailed technical advice regarding the concerns particularly focusing on the potential imbalances in data quantity and quality across different languages, especially for low-resource languages. Perhaps I (not a robot) missed the methods described in your manuscript. Please let me know and provide additional information if possible.\n\n1. Addressing Imbalances in Quantity and Quality\n- Data Augmentation Techniques:\n\nSynonym Replacement: Identify and replace words with their synonyms to generate paraphrased sentences that maintain the original meaning.\nBack Translation: Translate sentences into another language and then back to the original language to create varied sentence structures while preserving semantic content.\nNoise Injection: Introduce small amounts of noise (e.g., typos, grammatical variations) to simulate real-world language use and increase dataset diversity.\n\n- Collaboration with Native Speakers:\n\nEngage with native speakers or linguists proficient in low-resource languages to enhance the quality of your dataset. This could involve Crowdsourcing Platforms with native speakers for data validation and enrichment.\n\n- Detailed Data Distribution Analysis:\n\nProvide a comprehensive analysis of the data distribution across the 39 languages included in the dataset. This should include:\nQuality Metrics: If applicable, include metrics that assess the quality of the data per language (e.g., linguistic diversity, error rates).\n\nVisual Representations: Use graphs or charts to visually represent the distribution and highlight any significant imbalances.\n\n2. Enhancing Quality of Cultural Representation\n\nWhen collecting data, consider the cultural context of each language. This can be achieved by:\nEthnographic Research: Integrate ethnographic methods to understand cultural nuances that might not be reflected in traditional data collection.\n\nFocus Groups: Conduct focus groups with speakers of the languages to gather insights on cultural expressions and values that should be represented in the dataset.\n\nExamples of Cultural Representation: Provide specific examples of how the dataset captures cultural differences. Detail how various cultural elements (e.g., idioms, customs, social norms) are represented in the data. Case studies that illustrate the inclusion of culturally relevant examples. An analysis of how the dataset addresses or fails to address complex cultural concepts.\n\n\n\n3. insufficient in-depth analysis related to specific tasks and languages, particularly in areas such as multilingual visual question answering (VQA) and multi-subject reasoning. \n\nBelow are some actionable suggestions to enhance your analysis:\n\n- Conducting Ablation Studies\nAblation Studies on Language Groups: Perform ablation studies to assess how different language groups affect model performance. This involves systematically removing specific languages or language features from the dataset to observe changes in performance metrics. Analyze which languages contribute the most to overall model effectiveness and identify any languages that may cause degradation in performance.\n\nAblation Studies on Task Types: Similarly, conduct ablation studies focusing on different task types (e.g., VQA versus multi-subject reasoning). By isolating the performance impacts of each task type, you can gain insights into which tasks are more challenging for your model and why.\n\nExpert Collaboration: How can collaboration with linguistic and cultural experts be structured to inform the data collection process, ensuring a more comprehensive understanding of low-resource languages?\n\nEvaluation Metrics: What specific evaluation metrics can be developed to assess the model's performance across different languages, particularly focusing on those with limited data?\n\nFeedback Mechanisms: How can feedback mechanisms be incorporated into the dataset's development process to continuously improve data quality and representation based on model performance?\n\nCross-Linguistic Comparisons: What methods can be utilized to conduct cross-linguistic comparisons that identify specific challenges faced by the model when processing low-resource languages?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality\nThe paper presents approaches to multilingual and multimodal learning through the development of the PANGEAINS dataset and the PANGEABENCH evaluation framework. By focusing on both linguistic and cultural diversity, it addresses gaps in existing datasets that often overlook these critical aspects. The integration of machine translation and culturally relevant guidelines for data curation showcases an innovative methodology that enhances the relevance and applicability of the dataset. This originality is further emphasized by the emphasis on cultural understanding in multimodal tasks, which is a relatively underexplored area in the field.\n\nQuality\nThe quality of the research is evident in the rigorous methodology employed for dataset creation and model evaluation. The paper provides a thorough explanation of the processes involved in curating the PANGEAINS dataset, including the use of high-quality image captioning and the implementation of robust evaluation metrics. The empirical results presented are well-supported, demonstrating PANGEA's superior performance in various tasks. Additionally, the use of established benchmarks and the careful selection of evaluation metrics contribute to the overall robustness of the findings.\n\nClarity\nThe paper is well-structured and clearly organized, making it accessible to readers across different backgrounds. Each section is logically connected, and the authors effectively communicate complex ideas through concise language and well-defined terminology. The inclusion of visual aids, such as figures and tables, enhances understanding by illustrating key concepts and results. Overall, the clarity of presentation allows for easy comprehension of the methodologies and findings.\n\nSignificance\nThe significance of the paper lies in its potential impact on the field of multilingual and multimodal learning. By providing a comprehensive dataset and a structured evaluation framework, the authors contribute valuable resources that can facilitate further research and development in this area. The focus on cultural diversity and understanding not only enriches the training of models but also promotes more inclusive and applicable AI systems. The findings underscore the importance of addressing cultural nuances in AI applications, which is essential for creating systems that are effective and relevant in a global context.", "weaknesses": "Limitations of the Dataset\nThe PANGEAINS dataset comprises samples from 39 languages; however, there exists a potential imbalance in both the quantity and quality of data for specific languages. This issue is particularly pronounced for low-resource languages that the authors do not explicitly address. The scarcity of data for these languages can lead to significant challenges in training models that are capable of performing effectively across diverse linguistic contexts. Insufficient representation may result in suboptimal performance, as the models may lack the necessary exposure to the varied linguistic structures and cultural nuances inherent in these languages.\n\nMoreover, the quality of the data is as crucial as its quantity. If the available data for certain languages is of lower quality—whether due to inaccuracies in translation, lack of contextual relevance, or inadequate cultural representation—this can further exacerbate performance issues. Such disparities in data quality can hinder the model's ability to generalize effectively across languages, potentially leading to biased outputs and diminished reliability in multilingual applications.\n\nConsequently, these imbalances may impede the overall generalization capabilities of multilingual processing, as the model’s performance may be skewed towards languages with richer datasets. Thus, it is imperative for future iterations of the dataset to ensure a more equitable distribution of data quality and quantity across all included languages, particularly for those that are underrepresented, in order to enhance the robustness and applicability of multilingual models.\n\n-Depth of Cultural Context\nIn evaluating cultural understanding, the discussion regarding whether the PANGEAINS dataset adequately reflects diverse cultural backgrounds is somewhat lacking. Cultural differences are complex and nuanced, extending beyond mere linguistic expressions or visual representations to encompass social contexts, customs, values, and other interrelated elements. These factors shape the meanings within specific cultural contexts and are essential for multilingual multimodal models to accurately understand and process cultural nuances. However, the current dataset primarily focuses on the matching of images and captions, which may not sufficiently capture the depth of cultural context.\n\nFor instance, metaphorical expressions or symbolic meanings in one culture may be interpreted entirely differently in another. Capturing these subtle distinctions necessitates a more in-depth analysis that transcends simple dataset curation. It would be beneficial to incorporate expert consultations on cultural backgrounds and conduct comprehensive research on each culture to inform data collection efforts. Such an approach would enhance the model's ability to understand meanings across various cultural contexts and generate more nuanced responses.\n\n- Limitations of Experimental Design\nFrom the perspective of experimental design, there is a notable lack of in-depth analysis concerning specific tasks or languages. For example, in tasks such as multilingual visual question answering (VQA) or multi-subject reasoning, a detailed exploration of the factors influencing model performance is missing. Without such analysis, it becomes challenging to clearly understand the model's strengths and weaknesses, ultimately limiting the ability to establish future improvement directions.\n\nParticularly, without a thorough examination of various determinants of performance—such as the characteristics of specific languages, dataset imbalances, or hyperparameter adjustments during training—understanding why a model excels or falters in certain tasks becomes difficult. Therefore, it is crucial that the experimental design explicitly defines the specific conditions and variables for each task, along with comparative analyses of performance across different environments. This approach would not only enhance model performance but also deepen the overall understanding of multilingual and multimodal systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RST1Tq2PHT", "reviewer": "Reviewer_pV19", "summary": "The paper introduces PANGEA, a multilingual multimodal large language model (MLLM) addressing the limitations of English-centric data in existing models. The model was trained on a dataset called PANGEAINS, which includes 6 million instruction samples across 39 languages. PANGEINS combines high-quality English instructions, machine-translated content, and culturally relevant tasks. The authors also developed PANGEABENCH, an evaluation suite to assess the model’s multilingual and multimodal capabilities, including tasks like chat, image captioning, and visual question answering. PANGEA outperforms existing open-source models, especially in tasks requiring cross-cultural understanding, though it still lags behind proprietary models like GPT-4 in areas like complex reasoning. Fully open-sourced along with its dataset and evaluation suite, PANGEA aims to promote further research in inclusive multilingual LLMs, addressing challenges like data scarcity, cross-lingual transfer, and multilingual OCR.", "questions": "In Figure 7, the model's poor performance on English OCR is counterintuitive. What might explain this unexpected result?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper is well-written, offering comprehensive details on the curated instruction-tuning datasets, benchmarks, and evaluation results. The authors present their work with clarity and thoroughness.\n2. The meticulously curated datasets for both training and evaluation are of significant practical value. They will undoubtedly make a substantial contribution to the multilingual research community, providing a robust foundation for future studies in this field.", "weaknesses": "1. The paper lacks some crucial details about the model and training process. Specifically, it would be beneficial to provide more information about the architecture, including the type of vision encoder used and the final total number of parameters. Additionally, details regarding the training infrastructure and duration are absent. \n2. Some evaluation setup details are missing, such as the number of shots for each dataset, which makes the results difficult to reproduce. For instance, the results for Qwen2-7B-instruct are significantly worse than those reported in the official release (https://qwenlm.github.io/blog/qwen2/#multilingual-capability-of-instruction-tuned-models). What is the reason?\n3. Although the authors claim the model supports 39 languages, it's unclear which specific languages have shown improvement and to what extent compared to the backbone model. This is particularly uncertain for languages that comprise only a small portion of the PANGEAINS dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)