MELLA: Bridging Linguistic Capability and Cultural Groundedness for
  Low-Resource Language MLLMs
Authors: name: Yufei Gao, name: Jiaying Fei, name: Nuo Chen, name: Ruirui Chen, name: Guohang Yan, name: Yunshi Lan, name: Botian Shi

Multimodal Large Language Models (MLLMs) have shown remarkable performance in
high-resource languages. However, their effectiveness diminishes significantly
in the contexts of low-resource languages. Current multilingual enhancement
methods are often limited to text modality or rely solely on machine
translation. While such approaches help models acquire basic linguistic
capabilities and produce "thin descriptions", they neglect the importance of
multimodal informativeness and cultural groundedness, both of which are crucial
for serving low-resource language users effectively. To bridge this gap, in
this study, we identify two significant objectives for a truly effective MLLM
in low-resource language settings, namely 1) linguistic capability and 2)
cultural groundedness, placing special emphasis on cultural awareness. To
achieve these dual objectives, we propose a dual-source strategy that guides
the collection of data tailored to each goal, sourcing native web alt-text for
culture and MLLM-generated captions for linguistics. As a concrete
implementation, we introduce MELLA, a multimodal, multilingual dataset.
Experiment results show that after fine-tuning on MELLA, there is a general
performance improvement for the eight languages on various MLLM backbones, with
models producing "thick descriptions". We verify that the performance gains are
from both cultural knowledge enhancement and linguistic capability enhancement.
Our dataset can be found at https://opendatalab.com/applyMultilingualCorpus.