Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
Authors: name: Xiang Yue; name: Yueqi Song; name: Akari Asai; name: Seungone Kim; name: Jean de Dieu Nyandwi; name: Simran Khanuja; name: Anjali Kantharuban; name: Lintang Sutawika; name: Sathyanarayanan Ramamoorthy; name: Graham Neubig
Published: 2024-10-21 16:19:41

Abstract: Despite recent advances in multimodal large language models (MLLMs), their
development has predominantly focused on English- and western-centric datasets
and tasks, leaving most of the world's languages and diverse cultural contexts
underrepresented. This paper introduces Pangea, a multilingual multimodal LLM
trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages.
PangeaIns features: 1) high-quality English instructions, 2) carefully
machine-translated instructions, and 3) culturally relevant multimodal tasks to
ensure cross-cultural coverage. To rigorously assess models' capabilities, we
introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets
covering 47 languages. Results show that Pangea significantly outperforms
existing open-source models in multilingual settings and diverse cultural
contexts. Ablation studies further reveal the importance of English data
proportions, language popularity, and the number of multimodal training samples
on overall performance. We fully open-source our data, code, and trained
checkpoints, to facilitate the development of inclusive and robust multilingual
MLLMs, promoting equity and accessibility across a broader linguistic and
cultural spectrum.