Parrot: Multilingual Visual Instruction Tuning
Authors: Hai-Long Sun
Da-Wei Zhou
Yang Li
Shiyin Lu
Chao Yi
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang
De-Chuan Zhan
Han-Jia Ye
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Large Language Models; Multilingual MLLM; Mixture-of-Experts
Primary area: deep_learning->large_language_models

Abstract: The rapid development of Multimodal Large Language Models (MLLMs), such as GPT-4, marks a significant step toward artificial general intelligence. Existing methods typically align vision encoders with LLMs via supervised fine-tuning (SFT), but this often deteriorates their ability to handle multiple languages as training progresses. We empirically observe that imbalanced SFT datasets, largely English-centric, degrade performance on non-English languages due to the failure in multilingual token alignment. To address this, we propose Parrot, a novel approach that leverages textual guidance for visual token alignment at the language level. Parrot conditions visual tokens on diverse language inputs and uses Mixture-of-Experts (MoE) to align multilingual tokens. By computing cross-attention between initial visual features and textual embeddings, we select the most relevant experts, converting visual tokens into language-specific representations. Additionally, we introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions, to assess multilingual capabilities. Parrot achieves state-of-the-art performance on both the multilingual benchmarks and a wide range of multimodal tasks. Code and dataset are available at: \url{https://github.com/AIDC-AI/Parrot}.

Parrot: Multilingual Visual Instruction Tuning
Hai-Long Sun 123*
Da-Wei Zhou
Yang Li
Shiyin Lu
Chao Yi Qing-Guo Chen Zhao Xu 3
Weihua Luo
Kaifu Zhang
De-Chuan Zhan Han-Jia Ye!
Abstract
The rapid development of Multimodal Large Lan- guage Models (MLLMs), such as GPT-4o, marks significant step toward artificial general intel- ligence. Existing methods typically align vi- sion encoders with LLMs via supervised fine- tuning (SFT), but this often deteriorates their abil- ity handle multiple languages as training pro- gresses. We empirically observe that imbalanced SFT datasets, largely English-centrie degrade per- formance on non-English languages due to the failure in multilingual token alignment. To ad- dress this, we propose PARROT, a novel approach that leverages textual guidance for visual token alignment the language level. PARROT con- ditions visual tokens diverse language inputs and uses Mixture-of-Experts (MoE) to align mul- tilingual tokens. By computing cross-attention between initial visual features and textual embed- dings, we select the most relevant experts, convert- ing visual tokens into language-specific represen- tations. Additionally, we introduce the Massive Multilingual Multimodal Benchmark (MMMB), new benchmark comprising 6 languages, 15 categories, and 12,000 questions, to assess mul- tilingual capabilities. PARROT achieves state- of-the-art performance on both the multilingual benchmarks and a wide range of multimodal tasks. Code and dataset are available at: https: //github.ccm com AIDCAAADCAAAAA AI/Parrot
CLituse-CLIPP QweeZ5 Chat 7B
Opedti-CLIP Quvnt.5-Chat
@8 8P8 RREH 12
The person the image n young man who standing in stadium, holding trophy. He is wearing red and while sports uniform which suggest that member sports
GALAAMBAL 8 AAUt F),
The image portrays a man who standing an arena, holding a trophy.
Figure I. The output of OpenAI-CLIP-bosl based and Chinese-CLA CLIP- based models using the same Chinese prompts. We observe that the OpenAI-CLIP-ael Chinese and English responses.
model exhibits confusion between
OpenAL, 2023a; 2024), has gained significant attention. However, LLMs are limited to processing only text-only data. The integration of visual modalities has endowed LLMs with multimodal capabilities (Wang et al., 2024a; Liu et al., 2024), driving the emergence of Multimodal Large Language Models (MLLMs). These models combine pre-trained LLMs with vision encoders, bridging the modality gap by aligning visual features with language embeddings. Current research predominantly uses either Q-Former (Li et al., 2023b) or MLP projector (Liu et al., 2023b) to align vision encoders with LLMs, enabling models to process multimodal.
1. Introduction
The rapid development of Large Language Models (LLMs), such as GPT-4 (Radford et al., 2018; Brown et al., 2020;
'Work done during his internship at Alibaba Group 'chooo of Artificial Intelligence, Nanjing University "National Key Labo- ratory for Novel Software Technology, Nanjing University 'AI Business, Alibaba Group. Correspondence to: Han-Jia Ye <ychj @ samda.jju.cdu.cn..
Multilingual capability a crucial aspect of MLLMs, enabling them to generate responses in the same language the input, accommodating linguistic diversity. This feature is vital for ensuring equitable access to technology across different regions and cultures (Chen et al., 2022; Hu et al., 2023). Many LLMs (Grattafiori et al., 2024; Yang et al., 2024; OpenAI, 2023b) exhibit multilingual capabilities,
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Parrot: Multilingual Visual Instruction Tuning
generating diverse language responses based on prompts. However, after multimodal alignment training. MILLMs often lose their ability to effectively understand, process, or generate non-Engle non-English languages. This phenomenon, which we refer to as multilingual erosion, results in models like LLaVA (Liu et al., 2023b), which tend to respond predominantly in English, even when the input another language. Therefore, is essential address multilingual erosion for improving MLLM's multilingual abilities.
tential erros and noise. Extensive experiments validate the PARROT's state-of-the-art performance across multi- lingual benchmarks, surpassing Qwen2-VL and LLaVA- Vision in multiple languages. Additionally. we evaluate our model across a broad range of multimodal benchmarks (e.g., MME (Fu et al., 2023) and ScienceQA (Lu et al., 2022), and SEED-Bench (Li et al., 2024a)), demonstrating its competitive performance in diverse tasks.
The primary cause of multilingual erosion is the imbalanced nature of multimodal alignment data, which is overwhelm- ingly English-centric. While models align visual and textual tokens well in English, their performance in other languages suboptimal. Through empirical analysis, we observe that multilingual erosion stems from the misalignment between visual tokens and textual tokens in non- Enggligish languages. For example, in our ablation study using OpenAl-CI.IP and Chinese-CL.I (Yang et al., 2022), we find that model using OpenAI-CLIP struggles with Chinese inputs, while the Chinese- Misosse-CII--cee equppede model effectively understands and generates Chinese responses. As shown in Figure 6, the t-SNE visualizations further demonstrate that the visual LLaVA are more closely
2. MMMB: A Massive Multilingual Multimodal Benchmark
In this section, we first outline the limitations of existing benchmarks and identify the key characteristies ideal multilingual benchmark should exhibit. We then provide a detailed explanation of how to construct a new benchmark.
2.1. Limitations of Existing Benchmarks
There are several existing multilingual benchmarks (e.g., Multi30K (Elliott et al., 2016), M3Exam (Zhang et al., 2024b), MMBench (Liu et al., 2023c), and LLaVA- Bench (Liu et al., 2023b; Hu et al., 2023)) for MLLMs, but they have notable limitations: 1) Outdated Benchmarks. Multi30k is designed for image-text retrieval tasks, and its performance has nearly reached its upper bound due to rela- tively simple problems. 2) Non-Standardized Evaluations. Benchmarks like LLaVA-Bench rely on evaluations using GPT-4. This dependence on GPT-4 as a de facto "Ground Truth" may hinder reproducibility. Additionally, LLaVA uses a deprecated version (GPT-4-0314), which introduces potential inconsistencies if other versions are used, leading to unfair comparisons. Moreover, M3Exam lacks consistent test samples across different languages, making it difficult to determine whether poor performance results from the difficulty of the problem or the model's limited multilin- gual capabilities. 3) Limited Languages. MMBench and LLaVA-Bel Bench are restricted to English and Chinese, limit- ing their ability to assess multilingual capabilities across a broader range of languages.
features of aligned with the Chinese prompts. Therefore, an intuitive question is: how to transform the visual features into language-specifii embeddings to enhance the MLLM's multilingual capabilities.
Due to the scarcity of non- non-English multimodal data (e.g., the lack of large-scall serge,scale,aae high-quality image-text datasets), it is necessary to use little multilingual data as possible hance the model's multilingual capabilities. To this end, we propose a novel method, PARROT, which uses textual guid- ance to align visual tokens at the language level. PARROT leverages a Mixture-of-Exxee Experts (MoE) module (Jacobs et al., 1991) to convert visual tokens into language-specific em- beddings. Specifically, we first calculate the cross-attentinn between the class token of visual features and the text embeddings. The resulting features are then passed through the MoF. router to activate a probability distribution for each language expert. Based on the input language, visual tokens biased towards English are transformed into language- specific embeddings using the appropriate expert. This approach not only enhances the multilingual capabilities of the MLLM but also bridges the multimodal gap effectively.
2.2. The Key Characteristics of an Ideal Benchmark
To more accurately evaluate the multilingual capabilities of MLLMs, an ideal benchmark should exhibit the following characteristics:
To address the scarcity of current multilingual benchmarks, we introduce a new benchmark encompassing six languages: English, Chinese, Portuguese, Arabic, Turkish, and Rus- sian. This includes an extension of the MMBench dataset to six languages and Massive Multilingual Multimodal Benchmark (MMMB) featuring 2,000 evaluation ques- tions per language, totaling 12,000 questions. Through semi- autimatttmaiic construction process, mitigate the po-
1) Languages with Significant Differences. The bench- mark should cover a diverse range of language families, selecting languages that are distinct and non-repetitive. This ensures a comprehensive assessment of MLLMs' ability to adapt across linguistic variations.
2) Problems with Medium Level of Difficulty. The prob- lems should not be overly challenging (e.g., logical rea-
2

---

Parrot: Multilingual Visual Instruction Tuning
the output of the code "b is lower than 50*?
Question: <inugee 1> The region bounded by the graph shown above. Choose an integral expression that can be used to find the area of R.
What the capital Michigan?
A. Pierre
Options:
B. Charleston
(A)_ (r(0) o(x]dxx
C. Grand Rapids
58: print to is lower than Se")
(8) [0'))()) -o00dxx
Lansing
(9) (( - go0
print "p is greater than se")
(D) [[([(x)) x(x]]dxx
(a) Code reasoning
(b) Logical reasoning
(c) Low relevance between image and text
Figure 2. Some bad cases for the existing multilingual benchmark. Left: code reasoning is strongly related to English. Middle: logical reasoning too challenging. Right: lack relevance between image and text.
Accnninng imags,
(ar), Turkish (tr), and Russian (ri). These languages rep- resent a diverse range of linguistic families. Detailed infor- mation and multilingual examples are provided in Figure 4. Regarding dataset requirements and consistency, our bench- mark constructed with two key considerations: 1) Since MMBench (Liu al., 2023c) officially includes English and Chinese versions, we extend
primunly
Manual Calibration
Fishemnen
B. Surfess
Initial transiate
C. Swimmas
D. Souba Divero
GPT4
Refinement x
Géracl dnczikli olank
abealisart
the other four languages. 2) For the creation of the new massive multilingual multi- modal benchmark, we select and curate relevant data from the ScienceQA (Lu et al., 2022), MME (Fu et al., 2023), and SEED-Bench (Li et al., 2024a) datasets, adhering to established guidelines. These datasets are then transformed into Visual Question Answering (VQA) format, resulting in a total of 12,000 samples across all six languages.
Balkkalar
Select the best
11. Shriptier
C. Vazzetler
Voting x
D. Tiyli delgglar
3
Figure 3. The calibration process for constructing multilingual henchmark consists of two stages: translation and calibration.
soning), as the primary goal is to assess the multilingual understanding, processing, and generating capabilities of MLLMs, rather than their reasoning abilities.
To mitigate potential errors and noise in the data acquisition process, we employ the following strategies to enhance the quality of our translations, as illustrated in Figure 3. First, we use GPT-4 to translate the original problem into the target language. Next, the initial translation is re-entered into for re-check and refinement. This step helps identify and correct any immediate inconsistencies or inaccuracies. For manual calibration, we engage two groups of professional translators for each language involved in the study:
3) Tasks with Multilingual and Multimodal. As shown in Figure 2, datasets should not be overly reliant on En- glish (e.g., code reasoning). These tasks should not inher- ently translate into multiple languages if they are composed mainly of English words. Moreover, images should be an indispensable part when MLLMs answer the question. For instance, when shown a map of the United States and asked to identify its capital, relying solely on text-based abilities would be insufficient. Therefore, it is crucial that questions exhibit a significant interplay between images and text.
First Group for Refinement. This group consists of three language experts who independently review and refine the translations generated by GPT-4. This results in three distinct versions of each translation.
4) Content Consistency across Languages. To fairly as- sess the multilingual capabilities of MILMs, content across languages must remain consistent. For instance, if English questions focus primarily on addition within one hundred, while Chinese questions emphasize calculus computation, it would be difficult to determine whether poor performance in Chinese stems from the complexity of the problem or from the model's limited multilingual capabilities. Ensuring content consistency across languages is thus essential for a fair and accurate comparison.
Second Group for Voting. The second group eval- uates these refined translations and selects the most accurate one, ensuring captures the intended mean- ing and nuances of the original text.
This calibration process significantly improves data quality by reducing errors and ensuring that translations are contex- tually accurate across languages. As result, our benchmark achieves higher linguistic precision and cultural relevance, which we believe enhances the robustness of our research findings. Future versions will include additional details to further enhance readability and completeness.
2.3. Construction Pipeline
Following the above criteria, we select six languages for in- clusion: English (en), Chinese (zh), Portuguese (pt), Arabic
3

---

Parrot: Multilingual Visual Instruction Tuning
Can you ace any clouds in the sky?
erysnet seponnze?
Quai das seguintos opplis cock terosin collin??
A. No, it's complesely clear
F. Them is no sky visble
A. biblieteco
C. Yo, and
D. Yes, but they spanse
donse
D.4
deparamenos polizis
restauranee
Eaglish
Randiax
merccario
Portsgunc
wite of hayt
2: 9,00
Gäosele göre bucelikli
pöriityer!
A
Stetquler
B. y
Anaics
f 2,00
Chince
10pi8 dolgeçer
Yirkak
Englishi
Chinews
Lamgunge
Language fimilly
Langragge group
Geography
Maps
English
Indo Buro-eu
Germanic
Rassion
Partuguase
Chinese
-ibean
Sinitic
Attribates
Languages
12,000 samples
Portugesse
Eurupean
Romance
15 categories
:
Arabic
Aftu-Asiatic
Semitic
Arabic
Turkish
Turkish
Altaie
Turkie
Count
Cotor
Kussian
Indo-Buropean
Slavic
Figure 4. The overview of MMMB benchmark. It incorporates 6 languages, 15 categories, and 12,000 questions.
2.4. Evaluation Strategy
denoted by W. This projector converts Z, into language embedding tokens H., effectively enabling the integration of multimodal information within the LLM's framework.
Since random guessing can lead to 25% Top-1 accuracy for 4-choice questions, it may reduce the discernible perfor- mance differences between various MLLMs. Additionally, MLLMs may have tendency to favor particular choice among the options (Liu et al., 2023c), further amplifying evaluation bias. To address these issues, we implement circular validation strategy inspired by MMBench. Specifically, MMMB is adapted to the Yes/No question format, where each image is paired with two questions, requiring 'Yes' and 'No' answers, respectively. As shown in Figure 10 in Appendix, an answer is considered accurate only if both questions are answered correctly; failing either results in marking the entire instance as incorrect. This strategy ensures more rigorous evaluation of MILMM,, reducing the impact of random guessing and promoting more robust comparisons across different models.
H. W Zy, with Z, = g(X.).
(1)
Finally, we input H. and H2 into LLM to generate the model's responses. However, after the modality alignment training, LLaVA loses its ability to process in non- non-English languages.
3.2. Pilot Study
To address the challenge of multilingual erosion in MLLMs due to the dominance of English image-text data, we empirically observe an inherent mismatch between visual tokens H. and textual tokens H,, which biases the model to- wards English semantics and increases the likelihood of En- glish outputs. Specifically, the widely-used OpenAI-CLIP vision encoder (Radford et al., 202 l), pre-trained large English-cenn ceriss- image-text corpus, yields visual representa- tions more aligned with English.
3. Methods
3.1. Preliminaries: Visual Instruction Tuning
representative work in MLLMs LLaVA (Liu et al., 2023b), which introduces simple yet effective method for aligning the vision encoder and the pre-trained LLM. Specif- ically, for given input image X., LLaVA uses the pre- trained CLIP vision encoder ViT-L/14 (Radford et al., 2021) to extract visual features Z, = g(X.). It then employs Vicuna (Chiang et al., 2023) as the LLM to generate textual embeddings H,, To align the vision encoder with the LLM,, projector, implemented as multi- mulei perceptron (MLP)
investigate this, we conduct the ablation study using OpenAI-CLIP and Chinese-CLIC (Yang et al., 2022). As shown in Figure the OpenAI-CLIP model struggles with Chinese inputs, while the Chinese-Cli CLIP model effectively processes and generates Chinese outputs. Additionally, to explore the distance between the visual features generated by different encoders and the Chinese prompts, we plot t- SNE visualizations for a more intuitive understanding. As illustrated Figure 6, the t-SNE further reveal that the

---

Parrot: Multilingual Visual Instruction Tuning
visual features from Chinese-CLIP are more closely aligned with the Chinese prompts.
voting, producing P E RRe:
p = (oftmax((ineer((H..))
3.3. Textual Guidance to Drive Visual Token Alignment
which selects and activates the appropriate experts. Next, we process the English- embeddings H, through the se- lected experts to convert them into language-specific visual representations:
First of all, we have address key challenge: due to the limited availability of -on-English multimodal data, we cannot rely on extensive multilingual datasets to enhance the multilingual performance of MLLMs. Therefore, it is necessary to design a method that efficiently aligns visual and textual features at the language level, rather than relying on large multilingual data. Motivated by the pilot study, we aim to directly align visual tokens with textual embeddings at the language level. To this end, we propose PARROT, novel method that leverages textual guidance to facilitate the multilingual alignment of visual features. PARROT enables the transition of English-biase biased visual features acquired through the OpenAI-CLIP to accommodate other languages, ensuring sagguage-specffii visual tokens are generated for LLMs based on multilingual inputs, thereby enhancing mul- tilingual abilities.
k MoB(H.) Σ i=1
PL E(H,).
(4)
This approach effectively aligns English-biased embeddings with multiple languages, ensuring a more accurate and com- prehensive representation across different linguistic con- texts. To stabilize training and reduce the variance in visual- semantic information, ensuring the model performs well beyond the multilingual multimodal domain, we employ MoE reweighting to obtain the final language-specific vi- sual embeddings Gy;;
First, we extract visual features through the vision encoder and transform them into language embedding tokens H, using a projector. We obtain the embeddings H, E NxC derived from text inputs via the word embedding table. Subsequently, to convert the English-biased features into language-specifte ones, we employ mross-modal cross- attention mechanism to obtain H RC:
G = H. + aMoE(x),
(5)
where a is a trade-off parameter. In conclusion, we first fuse the visual and textual inputs via Eq. 2 to transform the visual embeddings with textual guidance. The fused result is then input into the MoE module, which selects and acti- vates the most relevant language experts via Eq. obtain language-specifee embeddings as shown in Eq. Finally, MoE reweighting is applied to convert visual embeddings information 5,
H vC
H. = Attention Q,
K, V) = Softmax
H.,
with less variance in original visual-semant This approach enables the MLLM to gain multilingual capa- bilities using minimal multilingual data. Figure illustrates the architecture, the detailed MoE module, and the training stages of PARROT.
(2)
where Q H.. and both K and V are equivalent H,-
Hels E RCC represents the [CLS] token of H,.. This process allows the visual features to be dynamically adjusted and transformed into language-specific semantic embeddings based on multilingual inputs.
3.4. Training Stage
Our goal is to enhance the multilingual capabilities of MLLMs with minimal multilingual data. The training pro- cedure is divided into two distinct stages:
Since the projected language embedding tokens H, are biased towards English, we need to convert them into language-specific embeddings for different languages. To -xpeerssss
this end, we introduce lightweight (MoE) module, which consists of router and several lan- guage transformation experts. The MoE router is a linear layer that generates a probability distribution over the set of experts E = [e1. c.. ... vating specific experts. Each expert is an MLP designed to convert English-biasee
Stage 1: Modality Alignment. In this stage, we freeze both the vision encoder and the LLM weights, focusing solely on optimizing the projectors to bridge the modality gap. Notably, the MoF. module is bypassed entirely, meaning the image tokens do not pass through the MoE since the primary goal of this stage is to train the projector using a large number of image-text pairs. This enables the projector to align image tokens and textual tokens effectively without interference from the untrained MoE module.
ee., effectively selecting and acti-
embeddings into language-specifis embeddings. The inputs to experts E is H., and the outputs have the same dimensions as the inputs.
Subsequently, to obtain a normalized probability distribution for activating language-specins experts, H', is fed as input to the router. The router network contains a linear layer that computes the normalized weight matrix using H. for
Stage 2: Instruction Tuning for Multilingual Alignment. We continue to freeze the vision encoder weights while training all other modules. In this stage, we introduce multi- lingual training data and randomly initialize the parameters
5

---

Parrot: Multilingual Visual Instruction Tuning
pictume rabbit 2. Bi 4 da th - R 3. Resimdekl hayean bir tavşandır
Pre-trainting
Weighted-Sum
LLM
Projector
Large Language Model
MLP3
MLLP
MILP
Vision Encoder
Multilinguaa MoE
Instruction Tiuning
Lineer
Cross-Attention
LLM
Projection
Word Embedding
Router
Linear
MoE'
Expert
Projector
Vision Encoder
1. What animal is in the picture? B R 30 the &t &
Multilinguul MoE block
Vision Encoder
3. Fotograftaki hayvan nedir?
Figure 5.
The overall architecture of PARROT. It converts English-biase features to language specific features based on the multilingual MoE module, aiming to improve the multilingual capabilities. The training details within each stage are presented the right.
of MoE. The MoE is optimized with textual guidance, which drives the alignment of visual tokens while leveraging the well-trained projector. The prior alignment achieved in the pre-training stage facilitates efficient optimization of the MoE during this phase. The entire training process of PAR- ROT is outlined in pscudocode, shown in Algorithm I in the Appendix.
2021) as the vision encoder. To validate the effectiveness of PARROT, we select both Qwenl.5 and Qwen2 (Bai et al., 2023a) as the backbones. The initial learning rates for the two stages are with the batch size of 256 and 128. The entire training process is optimized to 21 hours on the 16x A100 GPUs setup, benefiting from the relatively small training datasets. Additionally, BF16 and TF32 precision formats are employed to balance speed and accuracy throughout the training process. As defined in Eq. we set the number of experts to six to correspond with the number of languages. Each expert is MLP composed of two linear layers with SiLU (Elfwing et al., 2018) activation function. More details are provided in Table 13.
1e and -e-3, respectively,
To address data scarcity non-English languages, we use a semi-automatic method similar to the one depicted in Figure 3 to acquire image-text data. We randomly parti- tion the ShareGPT4V dataset (Chen et al., 2023b) for each language, extracting non-duplicate, non-parallel image-text pairs for training, ultimately obtaining nearly 10K samples per language. This two-stage training approach ensures effective modality and multilingual alignment, even with limited non-En non-English data, aligning well with the challenges of data scareity in low-resource languages.
Evaluation Benchmark. Our evaluation consists of two parts: one assessing the multilingual capabilities MLLMs, while the other evaluating its overall performance. The first part is conducted on two datasets: multilingual MM- Bench (Liu et al., 2023c) and newly developed bench- mark MMMB. The second part of the evaluation spans wide range of multimodal tasks, such as MME (Fu et al., 2023), MMStar (Chen et al., 2024), ScienceQA (Lu et al., 2022), RealWorldQA (x.ai, 2024) and SEED-Bench (Li et al., 2024a), with performance visualized in a radar chart in Figure 7b.
4. Experiments
In this section, we begin with an overview of the experi- mental framework, providing details on specific implemen- tations, evaluation benchmarks, and MLLMs used for com- parative evaluation. Following this, we conduct a compre- hensive comparison of PARROT with the state-of-the-art ap- proaches using multilingual benchmarks. We also compare PARROT with leading models across range of multimodal tasks. Finally, we conclude with ablation studies and visual- ization of multilingual cases, highlighting the exceptional ability of PARROT in handling multilingual tasks.
Comparison Models. For comprehensive comparisons, select leading open-source models in MILMs, including VL. (Li et al., 2023a), LLaVA-NeXT (Liu et al.,
2024), Monkey (Li et al., 2023d), VisualGLM (Du et al.,
2022), VisCPM (Hu et al., 2023), GLM-4v (GLM et al.,
2024), ShareGPT4V (Chen et al., 2023b), InstruetBIIIP (Dai et al., 2023), mPLUG-Ow12 (Ye al., 2023), Idefics3 (Lau- rençon et al., 2024), LLaVA-One Vision (Li et al., 2024b), and Qwen2-VL (Wang al., 2024a). For the evaluation
4.1. Experimental Setup
Implementation Details. In this study, we configure
PARROT with the pre-trained CLIP ViT-L/14 (Radford et al.,
6

---

Parrot: Multilingual Visual Instruction Tuning
Table 1. Accuracy performance comparison on multilingual benchmarks. We report all compared methods with VLMEvalKit (Duan et al., 2024). The best and second results are shown in bold and underline, respectively.
MMMB
MMBench
Method
IIM
zh
pt
zh
VisualGLM (Du et al., 2022)
ChatGLM-6B
31.05
18.07
19.42
15.38
22.81
19.77
23.2
17.18
11.43
2.92
6.62
5.33
VisCPM- Chat (Hu et al,, 2023)
CPM-Bee- 10B
53.10
47.54
28.19
26.90
26.78
26.84
45.88
46.39
15.81
1.46
9.19
1.20
Qwen-VL. (Bai et al. 20235)
Qwen-7B
56.02
57.77
46.37
43.04
41.05
48 65
54.29
56.52
43.12
35.73
39.17
42,86
InstructBLIP (Dai al., 2023)
Vicuna-7B
39:47
32.92
35.67
23.80
28.36
36.37
27.83
18.81
27.14
3.26
8.50
20.87
mPLUG-OW12 (Ye ul., 2023)
LLaMA2-7B
67.25
60,99
59.70
45,78
45,43
62.63
66.15
59.36
58.24
37,88
47.68
60.39
Monkey (L. al., 2023d)
Qwen-V1. VL-7B
66.02
58, 18
46.31
38.83
37.66
48.59
58.07
53.52
49.57
31.01
31.35
45.18
LLaVA-1.5 (Liu et al.,, 2023a)
Vicuna-v1.5-7B
67.07
58.83
59.76
43,50
46.43
$9.06
65.37
58.33
59.02
36.16
43.90
56.95
LLaVA-NeXT (Liu et ul., 2024)
LLMMA3-8B
70.92
64.33
63.21
48.34
48.02
66.35
69.80
63.31
61.83
47.6 64
47.03
64.99
DeepSeck-VL.12 (L. et al., 2024a)
Deepseck-7B
72.66
65.95
64.41
49.70
49.07
67.57
70.71
64.03
62.61
48.05
47.95
65.53
GLM-4v-9B (GLM et al., 2024)
GLM-4-9B
69.26
62.83
61.59
47.23
46.9
64.35
67.91
61.31
60.01
46.13
45.7
63.09
ShareCiPT4V (Chen al., 2023b)
Vicuma-v1.5-7B
69.24
60.23
60.29
43.57
45.26
61.23
69.59
61.6
59.62
37.37
43.38
59.45
Idefics--8B (Laurengon et al., 2024)
LLaMA3-8B
74.50
67.70
66.08
50,91
50.41
69.66
73.50
66.79
65.53
49.85
49.79
68.68
Qwen2-VL (Wang et al., 2024a)
Qwen2-7B
80.51
80.23
78.11
74.07
71.72
79.33
78.59
78.39
75.93
74.70
73:45
75.88
LLaVA-One (Li et al., 2024b)
Qwen2-7B
79.03
78.23
75.91
73.36
67.79
76.37
76.74
75.30
73.45
70.44
64.85
73.14
PARROT
Qwenl
70.00
68.13
67.31
62.69
58.01
66.26
70.70
70.36
65.12
57.82
58.43
64.00
PARROT
Qwen2-7B
80.11
80.03
79.62
76.55
75.02
79.94
78.02
77.16
76.76
75.92
74.05
77.71
LLaVA with Chinese-
cat' from ImageNet, and perform t-SNE (Van der Maaten & Hinton, 2008) visualizations of both visual and textual features in LLaVA and PARROT. As shown in Figure 6, for LLLVV,, the Chinese-CI.I--B CL.d--baeed visual features are significantly closer nigh-dimensional space compared to OpenAI-CLIP. However, thanks to the architecture designed, PARROT bridges the gap with textual features effectively, regardless of the vision encoder used.
Purrot with OpenAI-CLIP
Chinese Prompt
LLaVA with OpenAI-CLIP
A
Purrot with Chinese-CLIP
4.3. Further Analysis
In this section, we explore several critical questions to com- prehensively analyze PARROT.
Figure 6. t-SNE visualizations of LLaVA and PARROT using dif- ferent vision encoders.
Are all components of PARROT equally effective? As shown in Figure 7a, incorporating multilingual data enhances performance across all languages. Addition- ally, the MoE module contributes significantly to per- formance improvements, validating the effectiveness of our proposed method.
process, we employ the VLMEvalKit (Duan et al., 2024) in OpenCompass, ensuring consistent configuration settings across all methods to maintain fairness in comparison.
4.2. Main Results
As shown in Table 1, PARROT-Qwen2-778 achieves state-of- the-art (SOTA) performance across four languages in both the MMBench and MMMB benchmark, with English and Chinese in second place. Figure 7b demonstrates that PAR- ROT excels not only in multilingual capabilities but also in handling complex multimodal tasks, such as MME (Fu et al., 2023), MMStar (Chen et al.. 2024), and SEED-Bende (Li et al., 2024a). In Figure 7c, we infer PARROT using a Chi- nese prompt and visualize the expert distributions within the MoE, revealing the dynamic activation of different ex- perts for different languages. Additionally, as illustrated in Figure 4, PARROT achieves competitive performance in existing multilingual benchmarks, utilizing less than 1 % of the data compared to other multilingual MILLMs.
How does PARROT handle the quality and imbal- ance challenges? Large-scale translated multilingual data, while seemingly abundant, often suffers from quality issues (e.g., translation errors, cultural mis- matches, noisy artifacts), particularly for low-resource languages. Even with massive data volumes, low- resource languages often remain underrepresented as high-resource languages dominate training distribu- tions (e.g.. 90%+ of tokens in typical datasets). Forc- ing higher proportions of low-resource data can risk triggering the "curse of multilingualism.. where mod- els sacrifice high-resource language performance to accommodate low-resource languages, as observed in prior work (Conneau et al., 2019; Chang et al., 2023; Blevins et al., 2024). Our approach strategically priori- tizes high-quality alignment signals over raw data quan- tity. This avoids the pitfalls of noisy translation and
To validate the effectiveness of the PARROT architecture, we use a Chinese prompt (translated to English as "please describe the image in detail"), sample 50 images of 'tiger
7

---

Parrot: Multilingual Visual Instruction Tuning
70-
Bosclin
Mulilingua Duta
Muhilingull Data
WrelWeringa
MNZ
Lager
Accurccy (%)
65
500
(7K
60
Lager
SQA-MM
SEED IMG
2h
M Languages
Mine-Grannd
ElatiA-NAT
MMSSter
Monbry 9AB1
0%
25%
Export
50%
Exprott
75%
Exprrtt
190%
Expert
Expert
Expect
(a) Ablation study.
(b) Multiple multimodal tasks.
(c) Expert distributions.
Figure 7. Left: The ablation study of multilingual data and the MoE modulc using the MMBench benehmark. Middle: The performance of PARROT broad range of multimodal tasks compared with existing models. Right: Expert distributions of MoE. We summarize the activated experts during the feed-forward process using Chinese Prompts.
the imbalance inherent to brute-force multilingual scal- ing, ensuring stable performance across all languages without compromising high-resource capabilities.
4.4. Visualization of Multilingual Conversations
To enhance the intuitive understanding of the PARROT's multilingual capability, we prepare comprehensive case study accompanied by illustrative visuals. For instance, as depicted in Figure 8, our framework demonstrates re- markable multilingual capabilities. This underscores the PARROT's versatility in navigating different languages and presents its potential in bridging linguistic gaps across di- verse domains. Through careful analysis and visualization, we aim to provide a deeper insight into the mechanism driving this capability, illustrating its practical implications and potential applications real-world scenarios. This vi- sualization serves as a strong indicator of the PARROT's solid architecture and its exceptional ability understand, process, and generate multiple languages with remarkable efficiency. More multilingual conversation cases shown in Appendix F.
How does PARROT perform compared to a naive, translation-based baseline? To assess this, we imple- ment baseline using the Google Translation API that translates non-English queries to English, processes them, and translates responses back to the original lan- guage. As shown in Table 10, the results reveal "see- saw effect": while this naive approach yields improve- ments in certain languages like Chinese, it simultane- ously causes performance degradation others, par- ticularly Russian and Portuguese. This phenomenon highlights the fundamental limitations of relying solely translation services for addressing multilingualism in multimodal tasks, underscoring the need for more sophisticated approaches like PARROT.
Are the scaling laws effective in PARROT? To ex- plore the effectiveness of scaling laws in multilingual settings, we conduct experiments where the multilin- gual data (excluding Chinese and English) gradually expanded until it matches the volume of Chinese data (70K). As shown in Tablel 1, the results indicate that PARROT continues to follow the multilingual scaling law. For example, the performance on Portuguese in- creased by 3.0 points, and Arabic saw 5.2-point gain. Additionally, PARROT benefits from model size scaling, as shown in Table 12.
5. Conclusion
This paper addresses the critical challenge of improving the multilingual capabilities of MILMs and investigates the misalignment of visual features across languages. We intro- duce PARROT, novel approach that leverages textual guid- ance to align visual tokens at the language level, enabling the conversion of English-biased visual embeddings into language-speciffe ones through an MoE module. Extensive experiments conducted on a newly introduced benchmark, the Massive Multilingual Multimodal Benchmark (MMMB), across six languages demonstrate that PARROT achieves state-of-the-art performance compared to existing methods, with particularly notable improvements in Turkish and Ara- bic. PARROT not only advances the frontier of MLLMs but also highlights the importance of equitable access to tech- nological benefits across linguistic and cultural contexts.
Do the main performance gains come from the mul- tilingual dataset? As shown in Table and Figure 7a, LLaVA shows limited improvements with the addition of multilingual data. In contrast, PARROT achieves substantial gains, significantly outperforming LLaVA. Therefore, ensure that the primary performance gains come from the design of PARROT.
8

---

Parrot: Multilingual Visual Instruction Tuning
Arable: سلريا
الحمس 4e- ا و
[Arabio-Fnglish Trans: What sport is person the picture doing?]
Output: التب ياص الصررةة ب التديي (Arabic-Faglish Trans: The person in the picture is playing tennis.]
Turkish; Spereu hangi etkinligi kazandi ve görsele gåre kac Grand Slam sampiyonlagu kazandı?
[Turkish- Fnglissh Truns: What event did the athlete win and how many Giund Slam titles has the athlete according the image?)
CHAN OON
Output: Sporcu, 2023 ABD Açik'i kazandı ve görsele gêve Grand Slam sampiyonlogu kazandı. (Turkisht-Englin Englishh Trans: The athlete won the US Open 2023, and according to the image, he has won 24 Girund Stam titles]
Portaguese: Descreva us ernzçães expressões cupturdas rosto atleta imugem. que isso sugere sohre sentimentos dele relação sua vitória? [Portugaase-Engll Trans: Describe the emotions or expressions captured in the athlete's lace the image. What does this suggest about their feelings regarding their victory"!
Output: jogador de ténis, que está em pē segumundo miuetee de ténis, parece expressando felicidade orgulho em relação sua vitória. Flle está somindo, que sugere que ele está unimado feliz com resulado de seu esforvo. A imagem captura um momento de realização conquista, com jogadar no centro do quadru, destacado sensução de
por vibrunte [Portuguse--Enge Trans: The termis player, standing and holding tennis racket, appeans to be expressing huppiness and pride about his victory. He smiling, which suggests excited and pleased with the outcome of his effort, The image captures moment of achievement and conquest, with the player at center of trame. highlighted by a vibrunt color and promiment fonl that says "Champion" . Overall, the image conveys sense of peide, accomplismment, and fulfillmmntt)
fonte proemimente que "Campeño".. No geral, imagem transmite
realização realização.
English::
overall colors compusition imagee?
Output: The image feutures painting of deer and standing river. The deer positioned the right side of the painting, while the swan is on the left side. The river is surrounded by lush greenery, with many different types of flowers scattered throughout scene. painting has vibrant colortal composition, with various shades of green, blue, and white. The painting is done in watercolor style, which adds to its artistic and artislic qualities.
Chinese: (E) 56-
[Chinese- English Trans: What
are present the image!!
Output: i2 S8 It + #4 af R a &M * +# AAA 2.
(Chinese-Englis) Trans: The picture features two animale a swan swimming the water and standing on shore.]
Russian: Torrsa im modipeemme arr comeernaii nem.
[Russian English Trans: What is the weather shown this picture?]
commeusii. Hefo waenoe, moaysum neñom Gemann odumaaaus spиcи соыиеиaа CаEToM, окружлнонрий neiisus apsstil nexon aorau. Bea oбeτанsииa кавется 092046 cιosoñioi spactıooi,
Output: Hooccp svlr sapramme mareggu
nescoome активаеоо oraaaa eacacM возпуxe.
[Russian-Fagge Faglish Trans: The wenther in this picture lookss very sunny. The sky is clear, with blue sky and white clouds, bright sunlighi, and the suroonding scenery bright and full of life. The overall envirommentt appears very pesceful and beuutifuul, suitable for outdoer activities.
Figure 8. Multimodal conversation cases PARROT in multiple languages.
Acknowledgments
diversity and inclusivity. To address the challenge of the imbalanced language data in SET datasets and improve non- English visual tokens alignment, this approach contributes to breaking language barriers and facilitating cross-cultural communication, thereby fostering understanding and collab- oration across diverse linguistic communities. Additionally, the creation of the Massive Multilingual Multimodal Bench- mark (MMMB) fills a crucial gap in evaluating multilingual capabilities, enabling researchers to assess and improve upon models' performance across different languages and cultures. This paper presents work whose goal is to advance the field of Machine Learning and the potential societal impact of this work is largely positive.
This work is partially supported by National Key R&D Program of China (2024YEE0202800, NSFC (62376118, 18, 62476123), Key Program of Jiangsu Science Foundation (BK20243012). Fundamental Research Funds for the Cen- tral Universities (2024300373, 14380021), the Al & AI for Science Project of Nanjing University, Collaborative Innovation Center of Novel Software Technology and In- dustrialization.
Impact Statement
PARROT leveraging MoE to enhance multilingual alignment presents a positive social impact by promoting linguistic
9

---

Parrot: Multilingual Visual Instruction Tuning
References
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang. J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. arXiv:2311.1223 127993, 2023b.
AI, M. Build the future of ai with meta Ilama 3. Technical report, Meta AI, 2024. URL
meta.
com/llama3//
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330 2024.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., al. Flamingo: visual language model for few-shot learning. NeurlPS, 2022.
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Gryener, A., Mustafa, B., Beyer, I.,, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint NMw:2209906694. 2022.
anthropic. Introducing the next generation of claude. Tech- nical report, anthropic, 2024. URL https: /www. anthropic. www.///ww////eemmmmmmmmmm httee////laddee 3-famlly.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang. F., et al. Qwen technical report. arXiv:2309. 16609, 2023a.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Muyan, Z., Zhang, Q. Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistie tasks, arXiv:2312. 14238, 2023c.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P.,
Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A fron- tier large vision-language model with versatile abilities. arXiv:2308. 129666 2023b.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, 2023. URL https://vicnss Imsys. org.
Bavishi, R., Elsen, E., Hawthorne, C., Nye, M.. Odena, A., Somani, A., and Taşillar, S. Introducing our mul- timodal models, 2023. URL https:////// /www://wwww//o./o adept ai/b ai/blog//e fuyu-8b.
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross cross-lingual represen- tation learning scale, arXiv preprini arXiv: 190p,22110,11 2019.
Blevins, T., Limisiewicz, T., Gururangan, S., Li, M., Go- nen, H., Smith, N. A., and Zettlemoyer, L. Breaking the curse of multilinguality with cross-lingual expert lan- guage models. arXiv preprint arXiv:2401. 10440, 2024.
Dai, W., Li D., Tiong, A. M. H., Zhao J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurIPS, 2020.
Dong, Y., Liu, Z., Sun, H.-L., Yang. J., Hu, W., Rao, Y., and Liu, Z. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. In CVPR, 2025.
Chang, T. A., Amett, C., Tu, Z., and Bergen, B. K. When is multilinguality a curse? language modeling for 250 high-and low-resource languages. arXiv preprint wwXiv:2311.09255. 2023.
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang. J. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3200-33,,1 2022.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, I.., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vimevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:2407.11691 2024.
Chen, G., Hou, L., Chen, Y., Dai, W., Shang, L., Jiang, X., Liu, Q., Pan, J., and Wang, W. melip: Multilingual clip via cross-lingua transfer. In ACL, 2023a.
Chen, G., Liao, M., Yu, P., Wang, D., Qiao, Z., Yang, C. Zhao, X., and Fan, K. C-3po: Compact plug-and- play proxy optimization to achieve human-like retrieval- augmented generation. arXiv preprint arXiv:2502.06600, 2025.
Elfwing, S., Uchibe, E., and Doya, K. Sigmoid linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
10

---

Parrot: Multilingual Visual Instruction Tuning
Elliott, D., Frank, S., Sima'an an, K., and Specia, L. Multi30k: Multilingual anglish-german image descriptions. arXiv preprint
Li, B., Wang, R., Wang. G., Ge. Y., Ge, Y., and Shan,
Y. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. In CVPR, 2024a.
2016.
Fang. Y. Sun, Q.. Wang, X., Huang, T., Wang, X., and Cao, Y. Eva-02: A visual representation for neon genesis. arXiv:2303. 11331, 2023.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint WrXi:2240.033226 2024b.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Qiu, Z.,, Lin, W., et al. Mme: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv:2306. 13394, 2023.
Li, C., Wong, C.. Zhang, S., Usuyama, N., Liu, H., Yang. J., Naumann, T., Poon, H., and Gao, J. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. vrXiv:2306.00890 2023a.
GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., Yu, H., Wang, H., Sun, J., Zhang, J., Cheng, J., Gui, J., Tang, J., Zhang, J., Li, J., Zhao, L.,, Wu, L., Zhong, L., Liu, M., Huang, M., Zhang, P. Zheng, Q.. Lu, R., Duan, S., Zhang, S., Cao. S., Yang, S., Tam, W. L., Zhao, W., Liu, X., Xia, Zhang, X., Gu, X., Lx, X., Liu, X., Liu, X., Yang, X., Song, X., Zhang, X., An, Y., Xu, Y., Niu, Y., Yang, Y., Li, Y., Bai, Y., Dong, Y., Qi, Z.., Wang, Z., Yang, Z.,, Du. Z.. Hou, Z. and Wang, Z. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv:2406. 12793, 2024.
J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, arXiv:2301 12597, 2023b.
Li, J., Wang. X., Zhu, S., Kuo, C.-W.. Xu, L., Chen, F., Jain, J., Shi, H., and Wen, L. Cumo: Scaling multimodal Ilm with co-upcycled mixture-of-experts arXiv preprint arXiv:2405.00 0094999 2024c.
X.,
Li, L., Yin, Y., Li, S., Chen, L.,, Wang, P., Ren, S., Li, M., Yang, Y., Xu, J., Sun, X., Kong, L., and Liu, Q. Mt A large-scale dataset towards multi-modal multilingual instruction tuning. wrXiv22366.0388) 2023c.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The Ilama 3 herd of models. arXiv preprint Ax 2024.
Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403. 18814, 2024d.
Hinck, M., Holtermann, C., Olson, M. L., Schneider, F., Yu, S., Bhiwandiwalla, A., Lauscher, A., Tseng, S., and Lal, V. Why do llava vision-language models reply to images in english? arXiv preprint arXiv:2407.02333, 2024.
Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., and Bai, X. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv: 231 2tl.:231..0607 2023d.
Hu, H. J., Shen, Y., Wallis, P., Allen-Zhu, 7.,, Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv:2106609683: 2021.
Li, Z., Cen, J., Su, B., Huang, W., Xu, T., Rong, Y., and Zhao, D. Large language-geometry model: When 11m meets equivariance. arXiv preprint arXiv:2502.11142 2025.
Hu, J., Yao, Y., Wang, C., Wang, S., Pan, Y., Chen, Q., Yu, T., Wu, H.. Zhao, Y.. Zhang, H., et al. Large multilin- gual models pivot zero-shot multimodal learning across languages. arXiv:2308.1200 2023.
Liu, H., Li, C., Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023a.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. Adaptive mixtures of local experts. Neural computation, 3(1):79 87, 1991.
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tuning. arXiv:2304.0848; 2023b.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL 2024-01-30-11avva-nex
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C. Chaplot, D. S., Casas, D. d. I., Bressand, F,, Lengyel, G., Lample, G., Saulnier, I.., et al. Mistral 7b, arXiv:2310.0682 2023.
io/blog/
Laurençon, H., Marafioti, A., Sanh, V., and Tronchon, L. Building and better understanding langunee mod- els: insights and future directions. arXiv:2408.1266 2024.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mm- bench: Is your multi-modal model an all-around player? WwXiv2237.06628. 2023c.

---

Parrot: Multilingual Visual Instruction Tuning
Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Z., Sun, Y., Deng, C., Xu, H., Xie, Z., and Ruan, C. Deepseck-vl: Towards real-world vision- language understanding, arXiv:2403.05525, 2024a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
Lu, J., Zhang, D., Wu, X., Gao, X., Gan, R., Zhang, J., Song, Y., and Zhang, P. Ziya-visual: Bilingual large vision-language model via multi-task instruction tuning. arXiv preprint arXiv:2310.08166. 2023.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap. T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1,5: Unlocking multi- modal understanding across millions of tokens of context. arXi::2403.05530 2024.
Lu, P., Mishra, S., Xia, T., Qiu, L.,, Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurlPS, 2022.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
Lu, S., Li, Y., Chen, Q-G., Xu, Z., Luo, W., Zhang, K., and Ye, H.-J Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint voXiv:2405.20797 2024b.
Sun, H.-L., Sun, Z., Peng, H., and Ye, H.-J. Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning. In ACL, 2025a.
McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P. Shah, D., Du, X., Peng, F., Weers, F., al. Mml: Methods, analysis & insights from multimodal Ilm 2024.
Sun, H.-L., Zhou, D.-W., Zhan, D.-C., and Ye, H.-J Pilot: A pre-trained model-based continual learning toolbox. SCIENCE CHINA Information Sciences, 68(4):147101, 2025b.
pre-training. arXiv:2403.090
Ni, M., Huang, H., Su, L., Cui, E., Bharti, T., Wang. L., Zhang, D., and Duan, N. M3p: Learning universal rep- resentations via multitask multilingual multimodal pre- training. In CVPR, 2021.
Sun, H.-L., Zhou, D.-W., Zhao, H., Gan, L., Zhan, D.-C., and Ye, H.-J. Mos: Model surgery for pre-trained model- based class-incremental learning. In AAAI, volume 39, pp. 20699-20707, 2025c.
OpenAL. Chatgpt: A language model for conversational ai. Technical report, OpenAI, 2023a. URL https: www.aaa..ooooooooooeeeeeeeeoeeo //www, wtwww.wpenai.coooommomom/m com/ wwwoo//eeeearcccooooo chatgpt...
Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip at scale. arXiv:2303. 2303:2203./3333 15389, 2023.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
OpenAI. Gpt-4 technical report. arXiv:2303.08774 2023b.
OpenAI. Gpt-4o: Hello gpt-4o. Technical report, Ope- nAI, 2024. URL htttps://opente wtw.eenni..oomeeeeeommmmmmeeeeee /nne//nex////// hello- fello-gptt-oe gpt-gptt-oooo 40/.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. JMLR, 9(11), 2008
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv:2304.07193 2023.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12111 2024a.
Pires, T. P., Schmidt, R. M., Liao, Y.-H., and Peitz, S. Learn ing iogguage-speciffe layers for multilingual machine translation. arXiv preprint wxXiv:3305.02665 2023.
Wang. Y., Chen, S., Jiang, W., Yang, W., Wan, Y., and Zhang, I.. Online composite optimization between stochastic and adversarial environments. NewrIPS, 37: 94808-94850, 2024b.
Purason, T. and Tättar, A. Multilingual neural machine trans- lation with the right amount of sharing. In Proceedings of the 23d Annual Conference of the European Association for Machine Translation, pp. 91-100, 2022.
Wang. Y., Jian, Y.. Yang, W., Lu, S., Shen, L., Wang, B., Zeng, X., and Zhang, L. Towards unbiased information extraction and adaptation in cross-domain recommenda- tion. In AAAI, volume 39, pp. 12757-12765. 2025.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training, 2018
12

---

Parrot: Multilingual Visual Instruction Tuning
x.ai. Realworldqa dataset. https://x.ai//:
Zhou, D.-W., Sun, H.-L.,, Ye, H.-J., and Zhan, D.-C. Ex- pandable subspace ensemble for pre-trained model-based class-incremental learning. In CVPR, pp. 23554-23564, 2024b.
grok-1.5v, 2024.
Yang. A., Pan, J., Lin, Men, R., Zhang, Y., Zhou, J., and Zhou, C. Chinese clip: Contrastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01335. 2022.
Zhou, M., Zhou, L., Wang, S., Cheng, Y., Li, L., Yu, Z., and Liu, Uc2: Universal cross-lingual cross-modal rision-and-lange
pre-training. In CVPR, 2021.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv preprint arXiv:24/2.15135 2024.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304. 10592. 2023.
Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv:2304. 14178, 2023.
Zhu, J., Wang, W., Chen, Z., Liu, Z.,, Ye, S., Gu, L.,, Duan, Y., Tian, Su, W. Shao, J., al. Internv13: Exploring advanced training and test-time recipes for open- multimodal models. arXiv preprint arXh::2504.0044 2025.
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi: Open foundation models by 01. ai. arXiv:2403.04633 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 11975-11986, 2023.
Zhang, B., Bapna, A., Sennrich, R., and Firat, O. Share or not? learning to schedule language-specific capacity for multilingual translation. In ICLR, 2021.
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., and Shum, H.-Y. Dino: Detr with improved de- noising anchor boxes for end-to-end object detection. arXiv:2203.088 03605, 2022.
Zhang, L., Jin, Q. Huang, H., Zhang, D., and Wei, F. Re- spond in my language: Mitigating language inconsistency in response generation based on large language models. In Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Pupers), pp. 4177-4192, 2024a.
Zhang, W., Aljunied, M., Gao, C., Chia, Y. K., and Bing, L. M3exam: A multilingual, multimodal. multilevel bench- mark for examining large language models. Advances in Neural Information Processing Systems, 36, 2024b.
Zhang, Y.K.., Lu, S., Li, Y., Ma. Y., Chen, Q., Xu, Z., Luo, W., Zhang, K., Zhan, D.-C., and Ye, H.J. Wings: Learning multimodal 1lms without text-only forgetting. volume 37, pp. 31828-31853, 2024c.
Zhao. X., Chen, X., Cheng, Y., and Chen, T. Sparse moe with language guided routing for multilingual ma- chine translation. In ICLR, 2024. URL https:// openreview net/forum's iiddiiddype d=yddyhhy1hhssss
Zhou, D.W.., Sun, H.-L., Ning, J., Ye, H.-J., and Zhan, D.-.. Continual learning with pre-trained models: survey. In IJCAI, pp. 8363-8371, 2024a.
13

---

Parrot: Multilingual Visual Instruction Tuning
A. The Details of Training Datasets
In this section, we analyze the multilingual data in LLaVA (Liu et al., 2023b). From Table and Figure 9, it evident that during the pre-train stage. LLaVA solely utillizes multimodal image-text pairs data for training, comprising 558K of English data. During the SFT stage, both multimodal and text-only data are incorporated into the training process. Multilingual data appear only in the text-only dataset. Apart from English, the most prominent won-English data is Chinese, amounting to just 3.1K, constituting 0.25% of the total dataset. Therefore, it is evident that LLa'A's datasets are English-centric and imbalanced. The specific language and abbreviation are as follows: English (en), Chinese (zh), Korean (ko), Spanish (es), French (fr), Japanese (ja), German (de), Portuguese (pr), Traditional Chinese (zh-tw), Italian (if).
Table 2. The detailed information about LLa LA's datasets.
(a) The language information in two stages.
Training Stage
Type
Total Size
English
Other Languages
Multimodal Text-only
558K
558K
Stage (Pre-train)
Multimodal
624K
558K
Stage (SFT)
Text-only
41K
31K
10K
(b) The top- 10 multilingual information
Language
en
zh
ko
es
fr
ja
de
pr
zh-tw
Size
31K
3192
1219
1123
1049
551
435
422
305
234
Total Size
No
Is this an apple?
Yes
Yes
Is this a pear?
The answer wrong.
No
The answer is correct.
zht
ko
fr
ja
de
pt 2ht-tw it
Figure 10. An example of circular evaluation strategy.
Figure 9. The pie chart of LLaVA's multilingual data.
B. Related Work
Multimodal Large Language Models. The domain of MLLMs has witnessed significant advances, particularly in the enhancement of visual and language processing. Current MLLMs are usually combination of visual encoders (Radford al., 2021; Sun et al., 2023; Fang et al., 2023; Zhang al., 2022; Oquab et al., 2023; Zhai et al., 2023). LLMs, and fusion modules. Innovations like Flamingo (Alayrac et al., 2022) have advanced visual representation by integrating Perceiver Resampler with vision encoders. BLIP-2 (Li et al., 2023b) and InstructBLIP (Dai et al., 2023) employ Q-Former to connect the frozen LLM and vision encoder. Intern (Chen et al., 2023c) trains huge ViT and QFormer to integrate visual modalities through a multi-stage training method. MiniGPT4 (Zhu et al., 2023) leverages both a Q-Former and a linear projector to bridge the gap between the vision module and L.M.. Furthermore, LLaVA (Liu et al., 2023b) adopts
14

---

Parrot: Multilingual Visual Instruction Tuning
Table 3. Details on the PARROT's training data, which is derived from publicly available datasets.
Training Stage
Datasets
Samples
Total
L.LaVA-I.--reetiie (Liu et al., 2023b)
558K
Stage 1
Laion- (Schuhmann et al., 2022)
12K
1.2M
CC CC12M-Caption (Changpinyo et al., 2021)
645K
LLaVA-1.5-finetume (Liu et al., 2023b)
665K
ShareGiPT4V-Zh (Chen et al., 2023b)
71K
ShareGPT4V-pt (Chen et al., 2023b)
14K
Stage 2
793K
ShareGPT4V-ar (Chen et al., 2023b)
12K
ShareGPT4V-Ir (Chen al., 2023b)
17K
ShareGPT4V-ru (Chen et al., 2023b)
14K
simple MLP projector to promote the alignment between the LLM and vision encoder. mPLUG-OWl (Ye et al., 2023) introduces an approach that begins to finetune the vision encoder and align visual features, followed by tuning the LLM using LoRA (Hu et al., 2021). Qwen-VL (Bai et al., 2023b) improves visual module resolution to 448, aiming to refine the model's visual processing capabilities. Fuyu-8B BBy (Bavishi et al., 2023) directly projects image patches before integration with L.M. MM1 (McKinzie et al., 2024) has conducted ablative studies on connector design choices, revealing that the modality adapter type is less critical than the number of visual tokens and the resolution. MiniGemini (Li al., 2024d) utilizes high-resolution visual tokens and high-quality data narrow the performance gap with GPT-4 and Gemini. With the rapid advancements in open-source models, proprictary models such as GPT-4o (OpenAI, 2024), Gemini (Team et al., 2023; Reid et al., 2024), Qwen- Vonn-VL--ennn VL-series (Wang et al., 2024a), and Claude3 (anthropic, 2024) have achieved outstanding results in evaluations and practical applications. Some other recent works (Lu et al., 2024b; Zhu et al., 2025; Sun et al., 2025c; Li et al., 2025; Sun et al., 2025a; Dong et al., 2025; Zhang et al., 2024c) provide valuable insights and future directions for building and understanding adopt a framework similar LLaVA to design our model.
language models. In this work, owing to the simplicity of the LLaVA architecture, we
Multilingual Multimodal Models. Recent years have witnessed rapid progress in the expansion of multimodal models to include wider variety of languages. M (Ni et al., 2021) leverages English as a pivot and alternates between English-only vision-language pre-training and multilingual masked language modeling. In contrast, UC² (Zhou et al., 2021) translates English captions into various languages and uses images as the anchor. mCLIP (Chen et al., 2023a) enhances the CLIP model by aligning it with a multilingual text encoder through knowledge distillation. Thanks to the expansion of the overall capabilities of large language models (AI, 2024; Bai al., 2023a: Jiang et al., 2023; Young et al., 2024), their multilingual capacities have significantly improved. Integrating multilingual LLMs with visual abilities has increasingly become a research focus. In the domain of LLMs, PaLII (Chen et al., 2022) develops a 17B multilingual language-image model that spans over 100 languages. Ying-VLM (Li et al., 2023c) discovers that instruction tuning in English can extend its applicability to other languages. Ziya-Visual (Lu et al.. 2023) illustrates the translation of English image-text datasets into Chinese, using in-context learning for instruction-respun paradigm that fine-tunes the MLLM in a quasi-zero-shot manner based on a strong multilingual large language model. Despite these advancements, they are primarily confined to two languages or rely on the massive translated corpus. On the other hand, there no suitable multilingual benehmark for MLLMs evaluate the performance of multiple languages. There are also some multilingual research studies in other domains, such as multilingual machine translation (Zhao et al., 2024; Pires et al., 2023; Purason & Tättar, 2022; Zhang et al., 2021).
generation. VisCPM (Hu et al., 2023) introduces training
C. Additional Experimental Results
In this section, we present additional experiments and ablation studies to further validate the generality and capability of PAR- ROT across various tasks. Additionally, we elaborate on the training details for Figure 1 to provide a clearer understanding.
C.1. Bilingual Evaluation on LLa VA-Bech
VisCPM (Hu et al., 2023) extends the LLaVA VA-Bench dataset to the Chinese version for bilingual evaluation. To comprehen- sively compare PARROT with other multilingual models, we conduct experiments on this benchmark. Due to the deprecation
15

---

Parrot: Multilingual Visual Instruction Tuning
Table Experimental results L. Test Set accessed by GPT-4. Con: Conversation, DD: Detailed Description, CR: Complex
Reasoning, AVG: the average score of three tasks. The symbol denotes that the data are judged following the version of GPT-4-1 1-166-1106 preview because the GPT-4-0314 version deprecated by OpenAI.
LLM
English
Chinese
Model
Backbone
Con
DD
CR
AVG
Con
DD
CR
AVG
MiniGPT-4
Vicuna-
65.0
67.3
76.6
69.7
English Model
InstructBLIP
Vicuna-131
81.9
68.0
91.2
80.5
LLaVA
Vicuna-13B
89.5
70.4
96.2
85.6
mPLUG-OWL
BLOOMZ-7B
64.6
47.7
80.1
64.2
76.3
61.2
77.8
72.0
En-Zh Bilingual
VisualGLM
ChatGLM- M-66
62.4
63.0
80.6
68.7
76.6
87.8
83.6
82.7
Qwen-VL-Chal
Qwen- 7B
82.4
76.9
91.9
83.8
82.3
93.4
89.5
88.2
Model
VisCPM-Balance
CPM-Bee-100
75.5
64.7
91.3
77.3
85.4
81.4
96.6
88.0
Multilingual
PARROT
Qwenl.5-7B
82.5
71.0
89.3
81.1
82.
88.6
92.3
87.7
Model
of the GPT-4-0314 version by OpenAl, we test PARROT in LLaVA-Bench following the version of GPT-4-1106-prevw for comparison. As shown in Table 4, PARROT not only demonstrates exceptional ability in the English version of this benchmark but also presents competitive performance in the Chinese version.
Notably, as shown Table 5, VisCPM uses 140M English data and 1M Chinese data to train the model. comparison, Owen-VIL-Chat uses 1.1B English data and 300M Chinese data, whereas PARROT only utilizes approximately 2M data in total. Despite using less than 1% of the training data, PARROT achieves remarkable performance in both the English and Chinese versions LLa VL-Bnehn Owing to the architecture proposed, significant improvement in the model's multilingual capability can be achieved with minimal data usage.
Table 5. Comparison of vision encoders, LLMs, and training data in different models.
Model
vision encoder
LLM
Training Data
mPLUG-OWl
ViT-L/14 (0.3B)
BLOOMZ-7B
VisualGLM
Q-Former (1.6B)
ChaiGLM-6B
English: 300M; Chinese 30M
Owen-VI--Che
ViT-bigG (1.9B)
Qwen-7B
English: I.IB; Chinese: 300M
VisCPM
Muffin (0.7B)
CPM-Bee-Ill 10B
English: 140M; Chinese: 1M
PARROT
ViT-L/14 (0.3B)
Qwenl.5-Chas
English: 1.8M; Chinese: 71K
C.2. Further Ablation Studies
Ablation study each component. We conduct ablation experiment the multilingual data and the MoE module. As shown in Figure 7a, using multilingual data improves performance in each language. Moreover, the MoE module significantly improves performance, demonstrating the effectiveness of our proposed method.
Ablation study on different datasets. As shown in Table 6, it evident that the inclusion different multilingual datasets continually improves performance on the MMBench benchmark, and models with 7B parameters are used for this experiment. This highlights the robustness and scalability of our approach to handling multiple languages effectively.
Ablation study monolingual fine-tuning datasets. The ablation study presented in Table 14 evaluates the performance of different monolingual datasets added incrementally to the baseline dataset LLaVA-I 1.5---.--fintunee It highlights the significant impact of adding different multilingual datasets to a baseline model. Each dataset incrementally improves performance in its respective language and, when combined, leads to overall enhanced performance across all evaluated languages. This indicates the robustness and effectiveness of the proposed method in handling multilingual data, making a scalable solution for multilingual tasks.
16

---

Parrot: Multilingual Visual Instruction Tuning
Table Ablation study different multilingual training datasets in MMBench henchmark. Models with 7B parameters are used for this ablation.
Dataset
English
Chinese
Portuguese
Arabic
Turkish
Russian
LLaVA-1.5-finetu
69.4
66.6
60.3
55.3
52.1
60.7
zh
69.2
-0.2
68.6
+2.0
64.1
+3.8
59.1
+3.8
50.9
-1.2
61.6
+0.9 I
zh pr
71.1
+1.7
70.4
+3.8
65.4
+5.1
57.9
+2.6
52.1
+0.0
62.9
+2.2
zh pt
71.0
+1.6
68.6
+2.0
65.7
+5.4
58.6
+3.3
52.2
+0.1
62.2
+1.5
zh pl ar
70.4
+1.0
68.7
64.9
+4.6
61.2
+5.9
59.7
+7.6
62.0
+ zh pt ar ir ra
70.7
+1.3
70.4
+3.8
65.1
+4.8
57.8
+2.5
58.4
+6.3
64.0
+3.3
C.3. Comparison of Different Vision Encoders
We also compare the different vision encoders within the PARROT framework in Table 7. shows that the Chinese-CLIP- based model maintains comparable multilingual performance to the OpenAI-CLIP-baall one. This demonstrates that framework can be compatible with different vision encoders and achieve multilingual alignment through the MoE module.
Table 7. The comparison of various vision encoders within the PARROT framework.
MMMB
MMBench
Method
I.MM
Vision Encoder
pt
zk
LLaVA-LS
Vicuna-v1.5-77
OpenAI-CLIP
67.07
58.83
59.76
43.50
46.43
59.06
65.37
58.33
39.02
36.16
43.90
56.95
LLaVA-L5
Vicuna-v1.5.44
Chinese-CLIP
66.45
59.23
59.22
42.68
46.11
58.89
65.92
57.85
58.45
36.90
44.82
56.32
ShareGPPTAV
Vicuna-v1.5-27
OpenAI-CLIIP
69.24
60.23
60.29
43.57
45.26
61.23
69.59
61.60
59.62
37.37
43.38
59.45
ShareGPT4V
Vic-7a-v1.5-7B
Chinese-CLn
68.65
60.85
59.49
44.33
44.90
61.88
70.28
61.91
58.83
37.00
42.55
58.97
PARROT
Qwenl.5-7B
OpenAI-CLIP
70.00
68.13
67.31
62.69
58.01
66.26
70.70
70.36
65.12
57.82
58.43
64.00
PARROT
Qwenl.5-7B
Chinese-CLIP
69.22
69.24
66.32
62.15
57.77
64.31
69.95
70.87
64.92
56.57
57.13
63.15
Table The performance of different vision encoders and LLMs on MMBench and MMMB. MMB refers to MMBench. "En/en" represents the English version, and "CN2h"" represents the Chinese version.
Method
Vision encoder
LLM
MMB-EN
MMB-CN
MMMB-en
MMMB -zh
LLaVA
OpenAI-CLIP ViT-L/14
Vicuna 7B
65.4
58.3
67.1
58.8
LLaVA
OpenAI-CLIP ViT-L/14
Owenl.5-Chan 7B
68.8
66.4
68.2
62.4
LLaVA
Chinese-CLIP ViT-L/14
Qwenl.5-Chat 7B
68.1
68.3
67.6
66.
PARROT
OpenAI-CLIP ViT-L/14
Qwenl.5-Che 7B
70.7
70.4
70.0
68.1
C.4. Comparison with LLaVA using the Same Data
To validate the effectiveness our proposed approach, we conduct further experiments with an ablation study. Specifically, we expand the baseline LLaVA method by incorporating the same multilingual data used in PARROT. Both models are evaluated the MMMB dataset, and the results are presented in the Table 9, From the results, observe that while LLaVA shows slight improvement with the addition of multilingual data, the increase in performance is limited. In contrast, our PARROT model demonstrates substantial improvement when multilingual data is included, significantly outperforming LLaVA. This highlights that simply adding multilingual data is not sufficient to bridge the multilingual gap, further emphasizing the effectiveness of our proposed design.
C.5. Data Scaling and Model Size Scaling
To further investigate the scaling law in multilingual settings, we have conducted experiments where we progressively expanded the multilingual data (excluding Chinese and English) until it reached a volume comparable to the amount of Chinese data (~70.). The results, shown in the Table 11, demonstrate that PARROT still satisfies the multilingual scaling law. For instance, the performance on Portuguese improved by 3.0 points, and Arabic saw a gain of 5.2 points. As we
17

---

Parrot: Multilingual Visual Instruction Tuning
Table The comparison of the baseline LLaVA and PARROT ing the same multilingual training data. LLaVA shows limited improvement with multilingual data, while PARROT achieves sig- nificant gains, gready outperforming LLaVA.
Table 10. The comparison of the translation- based baseline and PARROT. While the naive baseline shows some improvements certain languages, such as Chinese, leads to performance degradation in others, such as Russian and Portuguese.
Method
MMMB
Method
MMMB
en
zb
p
ar
ir
n
pt
LLaVA w/o Multilingual data
67.1
58.8
59.8
43.5
46.4
59.1
LLaVA
67.1
58.8
59.8
43.5
46.4
59.1
LLaVA w/ Multilingual data
67.0
59.1
60.3
44.2
48.1
59.7
LLaVA w/ translation
67.1
60.7
58.6
47.3
48.6
58.9
PARROT
70.0
68.1
67.3
62.7
58.0
66.3
PARROT
70.0
68.1
increase the multilingual data, the model's performance on the MMMB benchmark continues to improve, suggesting that our model can handle imbalanced multilingual data while still achieving effective scaling and performance gains.
Table 11. The performance comparison on MMMB when scaling the sample sizes of each language.
Table 12. The performance comparison on MMMB when scaling model sizes of Qwen-series LLM.
MMMB
Sample Size
MMMB
en
zh
pt
ar
Ir
ru
Method
zh
pt
Ir
10K
70.0
68.1
67.3
62.7
58.0
66.3
PARROT-7B
70.0
68.1
67.3
62.7
58.0
66.3
30K
70.1
68.0
67.6
64.1
59.9
66.7
PARROT-14B
73.9
71.6
69.8
68.1
64.3
70.1
50K
69.9
67.9
67.8
64.8
61.4
67.2
PARROT-32B
76.3
75.4
73.8
72.1
71.2
73.5
70K
70.3
68.4
68.3
65.7
63.2
67.4
Additionally, we extend PARROT's LLM backbone from Qwen1.5-7B to Qwenl.5-333 using the same model design and configuration, and evaluate them on the MMMB dataset. As shown Table 12, the results indicate that PARROT continues yield better performance even with larger LLM backbone. This finding validates the idea that the scaling law for model parameters still holds, and design remains effective as the model size increases. While Qwenl 1.-3BB model, these results suggest that our approach scale well with model size, and we believe similar trends would be observed with even larger models, such as those with 30B parameters or beyond.
currently limited the
D. More Implementation Details
D.1. MoE Training Strategy
During the first pre-training stage, the MoE module is initialized with random parameters but activated or included in the training process. Instead, we focus exclusively on training the projector. This avoids the issue of training good projector under randomly initialized MoE. In detail:
1) Pre-training Stage. In this stage, the MoE module is bypassed entirely, meaning the image tokens do not pass through the MoE. The primary goal of this stage is to train the projector using large number of image-text pairs. This enables the projector to align image tokens and textual tokens effectively without interference from the untrained MoE module.
2) SFT Stage. Since SFT stage requires the participation of MoE modules, we randomly initialize the parameters the MoE components prior to the SFT phase. Once the projector has been trained and achieves robust alignment capabilities in the pre-training stage, introduce multilingual training data and activate the MoE parameters. At this stage, the MoE is optimized with textual guidance, which drives the alignment of visual tokens while leveraging the well-trained projector. The prior alignment achieved in the pre-training stage allows the MoE to optimize efficiently during this phase.
We present the entire training process of PARROT in the form of pseudocode, as shown Algorithm It is clear from the algorithm that during the pre-training phase, only the projector is trained. Before the start of the SFT phase, the MoE modules are randomly initialized and incorporated into the training process during the SFT phase.
18

---

Parrot: Multilingual Visual Instruction Tuning
Algorithm I PARROT for multilingual MLLM
Input:
Pre-training datasets: D SFT datasets: D²;
1: Construct the training data in LLaVA format:
2: Activate the parameters of the projector and freeze others;
3: for each data in D do
Pre-training stage
Optimize the projector to effectively bridge the modality gap;
end
6: Randomly initialize the parameters of MoE.
7: Activate the parameters of the projector, LLM, and MoE;
8: for each data D do
SFT stage
Select multilingual experts based textual guidance:
10: Optimize the projector, LLM, and MoE;
11: end for
Table 13. The detailed training hyperparametens.
Config
Stage I
Stage
Experts
6
MLP expert network
2 Linear layers with SiLU
Deepspeed
Zero2
Zero3
Image resolution
336x336 Clip- ViT-1/11--366
Image encoder
Feature select layer
-2
Image projector
2 Linear layers with GeLU
Epoch
Optimizer
AdamW
Learning rate Learning rate scheduler
1c-3
20-5
Cosine
Weight decay
0.0
Text max length
2048
Batch size per GPU
GPU
16 x A100-80G
Precision
Bf16
Gradient checkpoint
True
D.2. More Experimental Details about Different Backbones
In the following, we provide detailed information to explain Figure 1. Firstly, to ensure fair comparison between the OpenAI-CLIP-based model and the Chinese-CLiP-brr model, we train distinct models using the same training data as LLaVA, as shown Table 2a. The hyperparameters are listed Table 13 without the MoE hyperparameters. As depicted Figure 1, the OpenAI-CI.IP-base model struggles to generate Chinese outputs when given Chinese prompts due the English-centric training data. In contrast, despite the extremely scarce amount of Chinese training data, the Chinese-CLIP CLiss--sciPbb model naturally acquires zero-shot capability understand, process, and generate Chinese texts. Furthermore, compare both models on MMBench-CN and MMMB-zh to evaluate their Chinese capability. As shown in Table 8, the performance of the Chinese-Cr -Lies CL.IP-basee --seed model is significantly higher than that of the OpenAI-CTLIP-base model. On the other hand, we empirically find that different LLMs have significant impact on performance. Qwen (Bai et al., 2023a) demonstrates superior Chinese capability compared to Vicuna (Chiang et al., 2023), yet its English capability remains competitive.
D.3. Implementation Details of PARROT
As shown in Table 13, we provide the training hyperparameters for PARROT. Throughout all stages of training, we consistently train for one epoch, with batch size of 256 for the first stage and 128 for the second stage. We maintain an image resolution of 336x336 for all two stages and enable the gradient checkpoint mode for each training stage.
19

---

Parrot: Multilingual Visual Instruction Tuning
D.4. Analysis of the Translation-base Baseline
There is a naive baseline where first translate the question into English and then translate the English answer back to the target language. On the one hand, experimental setting follows recent work in multilingual and multimodal large language models (Hu et al., 2023; Zhang al., 2024a; Hinck et al., 2024), where such naive bascline has not been commonly considered. While the translation-best based approach could be straightforward alternative, it faces some significant challenges.
First, it is highly susceptible to translation noise, particularly issues related to polysemy and meaning ambiguity between languages. Moreover, our benchmark includes substantial number of cultural-specifi questions, which require deep cultural context knowledge that translation alone cannot effectively capture. In practical use, adding additional translation step would also introduce extra overhead, increasing both the time and computational cost.
Despite these challenges, we acknowledge the importance of evaluating this baseline and conducting experiments to assess the performance of this translation- based baseline by using the Google Translation API. As shown in the Table 10, the results reveal a "seesaw effect"-while the naive baseline shows some improvements in certain languages, such as Chinese, leads to performance degradation in others, such as Russian and Portuguese. This highlights the difficulty addressing multilingualism and multimodal tasks solely through translation.
E. Discussion
PARROT is a novel approach that leverages textual guidance to align visual tokens at the language level, enabling the conversion of English-biased visual embeddings into language-specific ones through an MoE module. In future work, we plan to incorporate more culture-related samples in various languages. This will enhance the representation of diverse cultural contexts and ensure that our benchmark accurately reflects the complexities multilingual interactions. Additionally, we will focus on developing tasks that not only assess linguistic capabilities but also evaluate cultural nuances, which are crucial for effective communication in multilingual settings. By doing so, aim to provide comprehensive evaluation of multilingual models and their performance across different cultural backgrounds.
F. More Visualization Results
In this section, we include additional visualization results between users' questions and PARROT's responses using multiple languages. These pictures are selected from LLaVA (Liu al., 2023b) and CuMo (Li et al., 2024c). As depicted in Figures Figures 12 to 17, it is evident that possosoossesses superior multilingual capabilities for understanding, processing, and generating multilingual texts. In certain specific cases, PARROT may also experience hallucinations. As depicted in the upper case of Figure 12, it misidentifies Xiaomi SU7 Porsche Taycan.
Table 14. Ablation study on monolingual fine-tuning dataset in MMMB benchmark. The table shows an effect of performance on six languages when using fine-tuning data from different languages. Models with 7B parameters are used for this ablation.
Dataset
English
Chinese
Portuguese
Arabic
Turkish
Russian
LLa
72.69
67.60
65.61
57.72
48.30
63.80
+ zh 71k
69.18
69.06
63.92
58.13
48.95
63.63
pt 14k
69.94
68.83
65.67
58.65
51.11
63.04
ar 12k
70.47
68.36
64.39
60.79
51.11
63.16
+ tr 17k
70.82
69.01
64.85
60.76
60.70
64.39
+ ru 14k
69.59
68.07
64.27
60.35
53.92
64.15
+ zh pt ar tr ru
70.00
68.13
67.31
62.69
58.01
66.26
20

---

Parrot: Multilingual Visual Instruction Tuning
DISAGESS
According comic, country most direct way expressing rejection?
favor,
imagem, selecione
A. #
descrição que melhor se encaixe.
B. P
A. France
A, Jacaré entre peixes.
Predader calma.
C Chinese
C. em rio tem piranha, Jacaré nada de costas
Chiiess
Eighish
Convivéndia aquática insólita.
Portiguerr
Beijing 2008
What the character
the picture?
#b 60
Black Widow
A_ #
Captain America
B. 2
C. 3/
D. 24
Zuglisk
Chisssc
Figure 11, Several culture-related samples in different languages.
21

---

Parrot: Multilingual Visual Instruction Tuning
English: What car in this image?
Output: The turquoise blue car is Porsche Taycan.
English: What animals are in this image?
Output: A cat and dog are in the image.
Figure 12. More visualization results between the user and PARROT using English prompts. We highlight the hallucinations from the responses of PARROT.
Chinese: f/ 2/2 d/?
(Chinese-English Trans: Do you know who drew this painting]
Output: 7p
3 +
(Chineso-English Trans: Leonardo da Vinei]
36
Chinese: i6 18 tM 2 H6 & 2
[Chinese-Fnglishh Trans: Please describe the main content of this painting.]
Haa - -
t 2,0 SEE &
# *
Ehiniss-English Trans: This painting is portrait, depicting the head and shoulders of young woman. hair dark brown and styled into
She is dressed a black dress with gold embellissments. There are brown spots her face, and her eyes are brown. She smiling, indicating she is in good mood. The style of the painting portraiture, characterized by the depiction of the subject's head and shoulders. Portraits often use realistic techniques capture the features and expressions of the person. The attire and setting of the figure in this painting suggest may depicting an artist from the Renaissance period.]
Figure 13. visualization results between user PARROT using Chinese prompts.
22

---

Parrot: Multilingual Visual Instruction Tuning
Portuguese: Qual é final deste filme?
[Porlugucse-English Trans: What's the ending of this movic?]
Output: O filme "Titanic" é um romance histórico de 1997 que apresenta uma narrativa fictícia de uma mulher chamada Rose e um homem chamado Jack. narrativa se passa durante o final do século XIX, quando navio "Titanic" está se preparando para sua viagem de luxo para Nova York. Jack Rose enamoram casam, mas viagem é marcada por um desastre inesperado.
[Portuguese- Englishh Trans: The movie "Titanic" is a 1997 historical romance that features a fictional narrative about a woman named Rose and man named Jack. The story is set during the late 19th century, the ship "Titanic" is preparing for its luxury voyage New York. Jack and Rose fall in love and get married, but the journey is marked by an unexpected disaster.]
Figure 14. More visualization results between the and PARROT using Portuguese prompts.
Arabie: fúi زجزتي عند منهاا حزتا أكون أن بجب الكي الأشياه ما
AArabicEEgglist Trans: What's the ending of this movie!
Output: عراتق أهي ار المانية الأشهاه أن المبالف الأأطع معل مدتقةة مخاطر أهي حززا بكون أن بجب الموقي هنا زبارة أااا ftm اليينف امل بالعر علم علي يككورا أن الزوارر علي بجب ناله إبس باإصصاةة المانية الأشياء إر الصخور أو السخور سنل محالة رمربية أمنة تجربة لطمان المليية الططرفف أو الحرارةة مرحات أو الطقسب اك تفور
MArabic-Englins Trans: During the visit to this site, one should be cautious of any potential hazards, such as surfaces, or water-related objects, or any potential obstacles, like rocks boulders aquatic objects. Additionally, visitors should be aware of environmental factors, such as changes in weather, temperature, or water conditions, to ensure a safe and satisfying experience.]
Figure 15, More visualization results between the user and PARROT using Arabic prompts.
23

---

Parrot: Multilingual Visual Instruction Tuning
Turkish: Bu malzemelerle hangi yemekleri yapabilirim?
Murkish-Engliss Trans: What are the meals that I can cook with these?]
Output: Buzdolabindaii çeşitli malzemelerle, çilekli bir salata, havuçlu bir sandviç, meyveli bir yemek ve süt ve yogurtla bir yemek gibi çeşitli yemekler yapilabilir. Buzdolabı, havuç, çilek ve yogurt gibi diğer gıda maddelerini de içerir; bunlar çeşitli yemekler yaratmak için kullanılabilir. Birden fazla yiyecek maddesi ve kapların olması, buzdolabmm iyi stoklandığım ve çeşitli yemek seçeneklerine olanak tanıdigımı gösterir.
Turkish-Englinn Trans: With the various items in the refrigerator, one can cook variety of meals, such as salad with strawberries, a sandwich with carrols, meal with berries, and meal with milk and yogurt. The refrigerator also contains other food items, such as carrots, strawberries, and yoguri, which can be used to create a diverse range of dishes. The presence of multiple food items and containers suggests that the refrigerator is well-stocked, allowing for variety of meal options.]
Figure 16. More visualization results between the and PARROT using Turkish prompts.
Russian: Πooaayyïcaa, помогите MHC прсдставить SIr MсCTO Ha ςотorоaарии.
Clussian-Englän Trans; Please help introduce the place in this picture.]
Output: MecTO, ивображснное на картинкс, 3тO co6op Василия Влажсиного, обннцнаььн
извсстнььй как cooop Покровв Пресвитой Fогороииииы pay. Tro знакоbоe стросние находится MосKве, Pоссня, на Kpachoй ииомааи.
[Russian-English Trans: The place depicted in the picture is the Saint Basil's Cathedral, officially known the Cathedral of the Intercession of the Most Holy Theotokos on the Moat. This iconic structure is located in Moscow, Russia, on Red Square.
Figure 17. More visualization results between the and PARROT using Russian prompts.
24


Reviews:
[{"review_id": "8XlGVn0440", "reviewer": "Reviewer_852q", "summary": "The paper is addressing what authors call as \"multi lingual erosion\" in multimodal large language models (MLLMs) - a phenomenon where post multi modal alignment the model loses ability to respond in or process non-English inputs. The authors identify that existing vision-language alignment methods (LlaVa) use English centric data, thus resulting in visual features that are biased towards English and thus is generally poor towards other languages\n\n\nTo overcome the issue of multilingual erosion, the paper introduces Parrot, a novel multilingual visual instruction tuning method that uses textual guidance with Mixture of Experts to align visual features with multiple languages. Parrot, extracts image features via frozen encoder and then projects these into LLMs embedding space. Then, it performs a cross modal cross attention between visual features and token embeddings on input text. The cross attention output is fed to MoE router that activates language specific experts which then produces a language specific visual token representation by transforming English biased visual embeddings. Subsequently MoE re-weighting merges transformed features with original ones, there by retaining visual information. Thus image embeddings are first aligned with visual information prior to feeding them to the LLM.\n\nParrot is  then trained in a multi stage fashion - first, a modality alignment phase using a large English centric data keeping the vision encoder and the LLM frozen. The second stage is instruction tuning phase. for multilingual alignment where projector, MoE and LLM are tuned based on a multi-lingual instruction image text data set ~10k samples for each of the 5 language used in stage 2. These data sets were obtained using ShareGPT4V and GPT-4 translations with human calibrations. The base LLM used in Qwen-1.5 Chat which has strong multi-lingual capabilities. \n\nAuthors introduced Massive Multilingual Multimodal Benchmark (MMMB) to evaluate multilingual performance, comprising 12,000 visual QA instances spanning 6 languages (English, Chinese, Portuguese, Arabic, Turkish, Russian) across 15 categories​. They further extend the existing MMBench dev set to all six languages by translating questions via GPT-4 and manually verifying them​. Authors used a circular evaluation strategy where each multiple-choice question is converted into two yes/no questions to mitigate random guessing biases​. Beyond these multimodal ability is tested on a range of standard tasks.\n\nThe paper shows that this approach state-of-art (SOTA) on multilingual benchmarks, notably 14B obtains highest accuracy on all languages in MM Bench, and all non English languages on MMMB. 7B has a strong performance as well and exceed LLaVa-NeXT-13B. The gains have come without regressions to English performance. Similarly authors also show strong performance on general multimodal tasks performing competitively across tasks. The ablation studies demonstrate that MoE and the multilingual instruction tuning data set are critical to the performance of Parrot.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for the reviewer’s thorough and thoughtful assessment of our work, as well as their recognition of Parrot’s contributions.\n\n> **Q1: Scalability to many languages.**\n\nA1: Parrot’s MoE framework is inherently designed to support seamless integration of new languages. The addition of language-specific experts requires only minimal data (e.g., ~5k samples per language) and incurs a linear parameter overhead (one expert per language). Empirically, we observe robust knowledge transfer **within language families.** We will extend Parrot to encompass broader linguistic diversity in future work and present more case studies given an unseen language to the model.\n\n> **Q2: MoE re-weighting parameter.**\n\nA2: The trade-off parameter $\\alpha$ in Eq. 5 balances the preservation of original visual semantics ($\\alpha=0$) against language-specific transformation strength ($\\alpha=1$). Through grid search on MMMB validation data with $\\alpha \\in \\{0.1, 0.3, 0.5, 0.7, 1.0, 1.5\\}$, we found $\\alpha=0.5$ optimally maintains visual fidelity while enabling robust multilingual alignment (>5% gain over $\\alpha=0.1$), as excessive reliance on original English-biased features hindered language adaptation. Higher $\\alpha$ values (>1.0) caused degradation in general visual tasks (e.g., -2.2% on MMBench object counting). We will include full parameter sensitivity curves in the final version to clarify this design choice.\n\n> **Q3: MoE gating mechanism.**\n\nA3: As described in Eq. 4, we use soft gating (weighted combination of experts) rather than hard routing during inference. This ensures smooth transitions between languages, avoids overfitting to dominant experts, and dynamically handles multilingual prompts (e.g., mixed-language queries). We will add a discussion in §3.3 highlighting this design choice.\n\n> **Q4: Base LLM impact.**\n\nA4: We ablated this by replacing Qwen1.5-7B with Vicuna1.5-7B (weaker multilingual support). As shown in the table below, Qwen1.5 outperforms Vicuna1.5 by +4.4% on Turkish and +8.5% on Arabic. This highlights that Parrot’s effectiveness is somewhat reliant on the base LLM’s multilingual ability. **Notably, while using the weaker multilingual LLM, the performance gain compared to the baseline LLaVA-1.5 is remarkably large.**\n\n|Method|LLM|MMMB_en|MMMB_zh|MMMB_pt|MMMB_ar|MMMB_tr|MMMB_ru|\n|-|-|-|-|-|-|-|-|\n|LLaVA-1.5|Vicuna1.5-7B|67.1|58.8|59.8|43.5|46.4|59.1|\n|Parrot|Vicuna1.5-7B|68.2|65.4|64.3|54.2|53.6|63.0|\n|Parrot|Qwen1.5-7B|70.0|68.1|67.3|62.7|58.0|66.3|\n\n> **Q5: Inference efficiency.**\n\nA5: Parrot maintains inference efficiency comparable to LLaVA by executing cross-attention and MoE processing **once per image** during visual token projection, not per generated token. The lightweight MoE module adds <0.5% parameters to the LLM, resulting in almost no extra runtime latency increase versus LLaVA-1.5 under identical hardware. Frozen CLIP encoding and single-pass visual-language alignment ensure the computational overhead remains negligible despite enhanced multilingual capabilities.\n\n> **Q6: High-resolution image handling.**\n\nA6: Thank you for your valuable suggestion. Parrot currently uses CLIP ViT-L/14-336 that is the maximum input size version. Crucially, our current implementation prioritizes fair comparison with LLaVA-1.5 baselines that share the same CLIP backbone. In future work, we plan to integrate dynamic-resolution approaches like NaViT's flexible patching or SigLIP's high-res pretraining to better handle high-resolution images while maintaining multilingual alignment capabilities.\n\n> **Q7: Minor revisions.**\n\nA7: Thanks for the detailed review and helpful comments. We will incorporate these updates into the final version.\n1) **Figure 2 caption**: We have clarified the caption to \"Examples of suboptimal multilingual benchmark design in existing works, as evaluated against MMMB's principles.\"\n2) **Reference**: we shall include a comprehensive comparative analysis of multilingual MLLM's recent advancements in the final version.", "claims_and_evidence": "Overall the claims are generally well supported with convincing evidence. \n\n1. Multilingual erosion: This is empirically well demonstrated using Chinese pre-trained CLIP yields better outputs (66.4 vs 68.3) on MM Bench-cn validation English biased encoder was an issue. \n\n2. Parrot improving multi lingual alignment with little multilingual data: This is one of the strongest claims of this work and is well supported. Parrot is using ~1% data that competing models use (ex: Parrot 7B on 2M examples with ~10k per non-English language) outperforms Qwen-VL-Chat (1.1B English + 300M Chinese examples).  The ablation studies in table3 clearly establishes that using Parrot architecture adding small multilingual datasets incrementally can generate accuracy improvements\n \n3. SOTA performance and improvements in underserved languages : Parrot's performance is SOTA across both benchmarks except English MMMB. This is well supported through comparisons in Table 2. Authors shows significant improvement on Turkish and Arabic exceeding prior SOTA by at least 10 points. These are substantial and adequately represented.\n\n Lastly authors show that the performance is maintained across multimodal tasks. This claim is supported by measuring on standard MM tasks - MME, MMStar etc. The data is represented in figure 5b and can be represented in a table for better clarity.", "methods_and_evaluation_criteria": "1. Methods are well motivated, and the approach looks sound. The idea of using language-specific experts to condition visual token on input language makes sense -  text conditioned transformation of visual-linguistic token in another language. Using MoEs and letting the experts choose specialized transformation seems appropriate. The modular approach, and two-stage training while not new are a reasonable strategy. Overall this methodology isn't significantly novel but extends current approaches to solve the multi lingual problem. There do not appear to be any fundamental flaws with this approach.\n\n2. The MoE re-weighting is good design choice, that ensures visual features are constrained by original semantics. This is likely a contributor for Parrot's strong performance on generic tasks. However detailing on how the reweighting param was set and its impact would be useful to have.\n\n3. Evals and benchmarks - This is a strong suite of the paper. They evals are thorough and appropriate. Specifically MMMB is well designed benchmark and fills prior gaps - wider array of languages and consistency. The authors also cover typologically different languages ensuring results are meaningful for a range of linguistic scenarios. \n\nBeyond these authors evaluate on multimodal benchmarks to show that there  no notable regressions while comparing against an array of both open source and closed models. Using VLMEvalkit to evaluate all models ensures consistency and fairness.\n\n This is a strong, rigorous evaluation criteria.", "theoretical_claims": "There are  no theoretical claims that are put forth here.  This is primarily a empirical work.", "experimental_designs_or_analyses": "The experimental design is comprehensive and sound. \n\n1. Baselines - Parrot is compared a wide range of existing models open-source MLLMs (LLaVA 1.5, LLaVA-NeXT, Qwen-VL, mPLUG-Owl 2, InstructBLIP, MiniGPT-4v2 and closed source ones GPT-4V, Gemini, Qwen-VL-MAX. Authors used a unified eval framework which is VLMEvalKit. They have onboarded all models onto this eval framework which is a fair experimental evaluation approach. \n\n2. MMMB and MM Bench:  The benchmarks seem comprehensive and robust. MMMB which was described in detail, consists of moderately difficult, multimodal questions across languages. They are not biased towards a specific language or task. \nFor MM Bench, GPT-4 was used for translation followed by human verification. Human verification followed by circular evaluation is  a strong design choice for evals which is fair and consistent \n\n3. Ablations:  The ablations are well designed. \na) Multilingual data vs MoE-  (i) baseline (English-only), (ii) +multilingual data (but no MoE), (iii) +MoE (but maybe no multilingual data), and (iv) full Parrot. The authors show clear evidence that adding MoE improves the performance. \nb) Incremental data ablation: Table 3 provides an analysis of how adding each language’s fine-tuning data affects performance​\n\nThere appear to be no flawed experiment design choices. The experiments are comprehensive.", "relation_to_broader_scientific_literature": "This work is well-positioned within existing literature - effectively builds on prior prior multilingual vision-language modeling, MoE applications, and multilingual instruction-tuned models. The key is the integration of MoEs with modular approaches to improve multilingual MMLLMs with minimal computational and data resources\n\n- Multilingual Text Encoders for Vision: There have been attempts to retrofit models like CLIP to multilingual text. For example, mCLIP  learned a multilingual text encoder aligned with CLIP’s vision space via knowledge distillation​. Parrot’s approach is different in that it keeps the text input to the LLM multilingual, and instead adjusts the visual input. \n\n- Large Multilingual Multimodal Models: The paper references PaLI (Chen et al., 2022)​  which is a 17B parameter model jointly trained on image-language data for 100+ languages.  Parrot’s contribution can be seen as a more data-efficient and parameter-efficient. Parrot adapts a 7B or 14B LLM with a small MoE module and achieves strong multilingual performance with a tiny fraction of data\n\n- Instruction Tuning in Multilingual Context: The paper cites Ying-VLM-  instruction tuning an MLLM in English can indirectly generalize to other languages. Parrot’s work aligns with the philosophy that leveraging a multilingual LLM is key and goes further - explicitly tuning the visual features for each language, which yields far better results than hoping for zero-shot transfer. Parrot’s MoE module can be seen as bridging the gap between a multilingual instruction-tuned LLM and the monolingual vision features.\n\n- Mixture-of-Experts in Multilingual Systems: There is relevant work in NLP that uses MoE for multilingual or multi-domain adaptation - Zhao et al. on language-guided MoE routing for machine translation. Parrot directly draws from this concept, as they mention in related work and implement a similar idea in the vision-language domain. Parrot’s novelty is applying it to align visual embeddings with multilingual text", "essential_references_not_discussed": "Paper is comprehensive addresses all major directions. No omission of essential work\n\nSome recent notable references could be include - Pangea (Yue et al, 2024) is very recent work. Introduces fully open multi lingual MMLLM (39 languages) with instruction tuning dataset which 6M. Here authors show it significantly outperforms existing models in multi lingual settings. It is likely Pangea was published after the submission.\n\nIt would be interesting for authords to compare Parrot with Pangea, though Pangea has an alternate, complementary approach which is huge data and end-end training. \n\nSimilarly Maya - instruction tuned using a multilingualCLIP could be another reference to cite\n\nPali has been extended to PaliX with more capabilities, which is another reference to cite.", "other_strengths_and_weaknesses": "Strengths\n- Strong Empirical Results and Data Efficiency - As stated previously significantly outperforms open models and most closed models with a fraction of data\n\n\n- Originality of Approach: While built on existing components, the combination of cross-modal attention + MoE for language-specific visual token transformation is a novel idea in this space. To my understanding, no prior multimodal LLM work has used a mixture-of-experts to handle multiple languages dynamically. This is a fresh perspective compared to simply increasing training data or training separate models.   This is a non-trivial innovation that extends ideas from multilingual NLP into multimodal alignment.\n\nWeaknesses\n- Scalability to Many Languages: The paper doesn’t address how the method scales beyond the chosen languages. In principle, MoE could scale but there are challenges of requiring data in each language.  The authors do not test or discuss what happens if an unseen language is input to the model. There is strong dependency on Base LLM’s Language Ability:", "other_comments_or_suggestions": "Minor comments\n\nClarify Figure 2 Caption: The caption for Figure 2 is a bit unclear: “bad cases for multilingual benchmarkperceive”​\n\nMoE Re-weighting Parameter: It’s mentioned that a trade-off parameter is used to blend original and expert outputs​. How is this chosen? Add more details to discuss this further.", "questions_for_authors": "1. How is the MoE re-weighting parameter determined?\n\n2. MoE Gating Mechanism: Do you use a hard gating or a soft combination of experts during inference?\n\n3. Base LLM Choice and Multilingual Strength: You chose Qwen-1.5 as the LLM due to its strong multilingual capability. Have you tried Parrot with a base LLM that is less multilingual? How doed the base LLM impact performance?\n\n4. Inference Efficiency: During inference, what is the computational overhead of Parrot compared to a standard pipeline (e.g., LLaVA)? Since you use a CLIP encoder (frozen) and an MoE, do you run the cross-attention and expert forward pass for every token, or just once per image?\n\n4. High-Resolution Image Handling: In the limitations, you mention CLIP’s limitation with high-res or detail rich images. Have you considered straightforward mitigations like using CLIP-ViT-L/14 with a larger input size?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cNLHjBgAm2", "reviewer": "Reviewer_nHXy", "summary": "The paper proposes Parrot, an MLLM targeting to handle multilingual tasks. Parrot based on the LLava architecture, and employ an MoE module to enhance multilingual VQA ability. The paper employ a new alignment method that aligns a english-biased clip encoder to various languages modality. Moreover, it proposes a benchmark MMMB to test the multilingual ability of different MLLMs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful and candid feedback.\n\n> **Q1: Prior work and drawback.**\n\nA1: **In section B in Appendix**, we have discussed prior multilingual MLLM methods like mCLIP, VisCPM, and M3IT. Most prior work relies on large-scale multilingual multimodal data (e.g., M3IT uses 2.5M+ samples), which is impractical for low-resource languages (more details are referred to in Response A1 of the reply to Reviewer QoYJ). Other multilingual CLIP methods improve performance for specific languages but sacrifice generalizability across diverse languages due to fixed encoders. For more clarity, we will expand this section in the revision to explicitly outline their key drawbacks.\n\n> **Q2: Benchmark and method co-design.**\n\nA2: MMMB is not specifically designed for Parrot. In contrast, MMMB and Parrot are co-designed to address two limitations in multilingual multimodal research: (1) existing benchmarks (e.g., M3Exam, LLaVA-Bench) not only lack standardized linguistic coverage, typically limited to 2-3 high-resource languages with inconsistent annotation protocols, but also exhibit systemic limitations that we have discussed in Section 2.1; (2) conventional models trained on imbalanced SFT data exhibit performance erosion across languages, yet lack evaluation frameworks to diagnose such failures. Additionally, we conducted experiments not only on our designed benchmark but also validated results on other multilingual benchmarks, such as MMBench (Table 1) and LLaVA-Bench (**Table 4 in the Appendix**).\n\n\n> **Q3: Using multilingual CLIP and LLM.**\n\nA3: In our preliminary experiments, we explored the use of multilingual CLIP as a baseline. However, this approach exhibited critical limitations: (1) It failed to balance performance across all target languages, as reliance on the inherent multilingual capacity of CLIP led to inconsistent generalization. In other words, it is hard to generalize beyond typologically similar languages covered in multilingual CLIP's pretraining. (2) Multilingual CLIP showed inferior visual perception capabilities compared to the original OpenAI-CLIP, degrading performance on general visual tasks. As shown in the table below, while LLaVA equipped with multilingual CLIP shows improvement in Chinese (+1.5), Arabic (+2.7), and Turkish (+1.1), performance in other languages declines (e.g., English -1.7).\n\nIn contrast, Parrot leverages a single OpenAI-CLIP alongside a multilingual LLM backbone (Qwen-LM) and achieves balanced, state-of-the-art performance across all 6 languages in two benchmarks. This demonstrates that our MoE-driven alignment paradigm effectively resolves language bias without requiring language-specific or multilingual visual encoders, while preserving strong general multimodal perception capabilities.\n\n|Method|ViT|LLM|MMMB_en|MMMB_zh|MMMB_pt|MMMB_ar|MMMB_tr|MMMB_ru|\n|-|-|-|-|-|-|-|-|-|\n|LLaVA-1.5|OpenAI-CLIP|Qwen1.5-7B|67.1|58.8|59.8|43.5|46.4|59.1|\n|LLaVA-1.5|M-CLIP|Qwen1.5-7B|65.4|60.3|58.1|47.2|47.5|58.8|\n|Parrot|OpenAI-CLIP|Qwen1.5-7B|70.0|68.1|67.3|62.7|58.0|66.3|\n\n> **Q4: Lack of comparison to the latest MLLM.**\n\nA4.1: Current MLLMs are primarily data-driven or architecture-driven. Our work aims to achieve data-efficient multilingual adaptation by enhancing multilingual capabilities with minimal training data while preserving general ability, rather than developing a model with exceptionally strong general capabilities. Many leading methods do not release their data. **Therefore, direct comparisons of general capabilities with strong models risk unfairness due to vast data disparities.** Despite this, we also compare Parrot to general-purpose MLLMs (e.g., Qwen2-VL (2024.09) and LLaVA-OV (2024.08)) on the MMMB and multilingual MMBench dataset in the table below and **Table 12 in Appendix**. Although Qwen2-VL and LLaVA-OV are trained with **over 10x the amount of data used by our model**, Parrot still outperforms them on the multilingual benchmark. This comparison spans two datasets and six languages, demonstrating Parrot's superior performance over these recent approaches.\n\n|Method|LLM|MMMB_en|MMMB_zh|MMMB_pt|MMMB_ar|MMMB_tr|MMMB_ru|\n|-|-|-|-|-|-|-|-|\n|Qwen2-VL|Qwen2-7B|80.5|80.2|78.1|74.0|71.7|79.3|\n|LLaVA-OV|Qwen2-7B|79.0|78.2|75.9|73.3|67.8|76.4|\n|Parrot|Qwen2-7B|80.1|80.0|79.6|76.5|75.0|79.9|\n\n|Method|LLM|MMB_en|MMB_zh|MMB_pt|MMB_ar|MMB_tr|MMB_ru|\n|-|-|-|-|-|-|-|-|\n|Qwen2-VL|Qwen2-7B|79.6|79.6|75.9|71.7|70.9|76.0|\n|LLaVA-OV|Qwen2-7B|77.1|76.6|73.2|66.9|65.5|71.3|\n|Parrot|Qwen2-7B|78.7|78.4|76.3|75.2|74.1|77.8|\n\nA4.2: For the radar chart in Figure 7, we have compared Parrot with the SOTA in 2024 (e.g., Mini-Gemini and LLaVA-Next), which shows Parrot’s competitiveness even with limited data. **Crucially, Parrot’s goal is not to surpass general-purpose MLLMs but to enhance multilingual ability with minimal data.**", "claims_and_evidence": "Motivations are clear and well-supported with empirical analysis. \n\nOne problematic claims are that while there maybe potential problem when prompting MLLM with non-english query, what are the previous solutions for a multilingual MLLM? And what is the drawback of those method that motivate this method? The author need to highlight this part to differentiate the work from others.", "methods_and_evaluation_criteria": "1. It will be a little nonsensical to propose a benchmark and a novel model at the same time. \n\n2. Have the author consider using a multilingual CLIP and a multilingual LLM as a starting point and train them on multilingual dataset? What would be the performance of it compare to the MoE alignment paradigm?\n\n3. Lack of comparison to latest MLLM (open-source and potentially close model).", "theoretical_claims": "There are no theoretical claim in the paper.", "experimental_designs_or_analyses": "The overall design is good. My comments are on the baselines:\n\nThe multilingual ability assessment seems to be reasonable to me - it contains some latest baselines and parrot demonstrates a superior performance among them.\n\nHowever, in the Radar plot, where the paper access the general ability of the MLLMs, the baselines are outdated (all from 2023). I would like to comparison against more latest MLLMs.", "relation_to_broader_scientific_literature": "The idea is effective when dealing with limited available data when one tries to train a multilingual MLLMs. If one is able to scale the data via AI-generated data, distillation or human labeling, the contribution of the paper is less useful.", "essential_references_not_discussed": "Some SoTA baselines are missing, which may have solve the multilingual problem to some extent.", "other_strengths_and_weaknesses": "See Claims And Evidence and Methods And Evaluation", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See Claims And Evidence and Methods And Evaluation", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YqLkhQSem", "reviewer": "Reviewer_Gwu1", "summary": "This paper introduces PARROT, a novel approach to enhance the multilingual capabilities of MLLM, using language specific embeddings to fuse to visual embeddings and multilingual MoE.  It addresses the issue of multilingual erosion, where MLLM loses proficiency in non-English languages after multimodal alignment training (e.g. answering in English while asking in non-English). For better evaluating multilinguality, this paper also introduces a new Massive Multilingual Multimodal Benchmark (MMMB). Experiments show SOTA results on MMMB, MMBench and other multimodal tasks.\n\n## update after rebuttal\n\nThe rebuttal has clarified my concerns. I am happy to maintain my original recommendation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thorough and constructive feedback, as well as their endorsement of our work.\n\n> **Q1: MoE vs. Simpler Methods (LoRRA).**\n\nA1: We explored the LoRRA-based (abbreviated as L-based) adaptation shown in the table below but found it insufficient for two reasons.[1]\n1) L-based method introduces fixed language-specific parameters, which struggle to handle multilingual prompts dynamically (e.g., mixed-language queries).\n2) Its low-rank updates are less effective for aligning diverse languages, especially when training data is sparse.\n\nIn contrast, MoE dynamically routes visual tokens to language-specific experts based on textual guidance, enabling flexible adaptation with minimal parameters. Additionally, due to the small number of parameters in MoE (<0.5% of the parameters in the LLM), it is not an expensive method.\n\n|Methods|MMMB_en|MMMB_zh|MMMB_pt|MMMB_ar|MMMB_tr|MMMB_ru|\n|-|-|-|-|-|-|-|\n|LLaVA-1.5|67.1|58.8|59.8|43.5|46.4|59.1|\n|LLaVA-1.5 w/ LoRA+multilingual data|66.9|61.1|60.9|47.2|50.4|61.3|\n|Parrot|70.0|68.1|67.3|62.7|58.0|66.3|\n\n[1] Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020.   \n\n> **Q2: Whether each language expert indeed handles its language-specific embeddings.**\n\nA2: In Figure 7c, the router activates the Chinese expert most strongly, but experts are not strictly language-exclusive. Instead, they learn cross-lingual synergies (e.g., English/German share morphological features). To further present the activation using other language prompts, we will add t-SNE visualizations of expert activations across languages and include expert distributions of MoE for each language in the final version.\n\n> **Q3: Evaluation fairness.**\n\nA3: To validate Parrot's effectiveness, we conduct an ablation study by expanding LLaVA with **the same multilingual data** used in Parrot. Both models are evaluated on MMMB, with results in **Table 13 in Appendix.** While LLaVA shows slight improvement, the gains are limited. In contrast, Parrot achieves substantial improvements, highlighting that simply adding multilingual data is insufficient to bridge the gap. Moreover, the findings from the ablation study in Figure 6a further support this conclusion, reinforcing the validity of our design.\n\n> **Q4: Ablation studies.**\n\nA4: In the previous response (A1), we provided comparisons between Parrot and L-based alternatives. Due to time and word constraints during the rebuttal phase, we will include expanded ablation studies on expert counts, sizes, and frozen vs. trainable configurations in the final version.\n\n> **Q5: Suggestions regarding the figure revisions.**\n\nA5: We thank your valuable feedback. We are committed to implementing these revisions and will incorporate them into the final version.\n\n> **Q6: Non-English data scarcity.**\n\nA6: While non-English web data is abundant, high-quality multimodal data remains scarce. 1) Most non-English alt-text is noisy or misaligned (e.g., social media images with irrelevant captions). 2) Parrot’s semi-automatic curation (Figure 3) ensures linguistic and cultural precision, which raw web data lacks. Further details are provided in **Response A1** of the reply to Reviewer QoYJ.\n\n> **Q7: Mitigating multilingual erosion via pre-training.**\n\nA7: Incorporating more multilingual data during pre-training could help mitigate multilingual erosion, but it may not fully resolve the issue. The main goal of multimodal pre-training is to learn a projector for cross-modal alignment, which often remains biased toward dominant languages (e.g., English) due to data imbalances. Without explicit mechanisms like Parrot's MoE-based language-specific alignment, subsequent SFT stages would still be affected by English-centric bias. Due to word constraints, we will include experiments with multilingual pre-training data in the final version.\n\n> **Q8: Translation baseline explanation.**\n\nA8: The translation baseline uses Google Translation API to translate non-English queries to English, feeds them to LLaVA, then translates responses back. \n1. Translated prompts often lack cultural-specific context. **Image**: A traditional Chinese red envelope (红包) with handwritten characters \"福到\" (upside-down \"福\", symbolizing **\"fortune arrives\"**) and \"岁岁平安\" (\"peace every year\"). The translated-based method cannot perform Glyph-aware visual grounding (recognizing 福's inverted form as intentional)\n2. When a Portuguese user asks about a Russian meme with the text \"Почему программисты любят кофе? Потому что Java!\" (where Java is a pun on both coffee culture and programming language), machine translation to English collapses the dual meaning into \"coffee brands,\" stripping the programming-language humor.\n\nWe have presented examples (Figure 10) where translation easily fails due to ambiguity or cultural nuances.", "claims_and_evidence": "The evidence is clear to support the claim.", "methods_and_evaluation_criteria": "Methods\n- The method looks to make sense overall, but I am not sure whether it’s necessary to introduce an expensive MoE module to fuse language-specific embeddings – would a simpler LoRRA approach work as well?\n- Also it’s unclear whether each language expert indeed handles its language specific embeddings? (Figure 7c is just an example for Chinese prompt)\n\nEvaluation\n- The new benchmark is well designed and addresses the limitations (section 2.1) of existing multilingual benchmarks for MLLM.\n- Evaluations are comprehensive, conducted on a wide range of tasks.\n- However, it might be unfair to compare PARROT with others on MMMB/MMBench, as PARROT has been trained on these specific 6 languages and on the data constructed with the same approach as MMMB?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Some ablations might be worth adding:\n- MoE vs. other approaches to distinguish prompt language, such as LoRRA and language as task prefix. \n- Number of MoE experts.\n- Size (parameters) of MoE experts.\n- Frozen vs. unfrozen ablations.\n\nAlso qualitative analysis on baselines and PARROT would be insightful.", "relation_to_broader_scientific_literature": "Multilinguality is a key capability for MLLM. This paper addresses this domain with a new approach to improve multilinguality  and a new benchmark to better verify multilinguality – all these are good for the community.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "The new MMMB benchmark should be beneficial to the community.\nThis paper is well written and easy to read.", "other_comments_or_suggestions": "suggestion: Figure 5 shows the core approach so could be moved forward (instead of in page 6)\nsuggestion: add legend for Figure 6; otherwise it’s unclear without reading the corresponding paragraphs", "questions_for_authors": "Given there are much more non-English image-text (e.g. alt-text) pairs on the web than English (≈3:1), why this paper claims “Due to the scarcity of non-English multimodal data (e.g., the lack of large-scale, high-quality image-text datasets), it is necessary to use as little multilingual data as possible to enhance the model’s multilingual capabilities.”?\n\nCan the multilingual erosion issue be simply mitigated if we mix more multilingual data in the pre-training instead of the SFT stage?\n\nTable 14 shows the comparison of the translation-based baseline and Parrot. It’s unclear to me why Parrot can be much better than translation baselines. Could you please explain the experiment’s settings (such as how the baseline eval was conducted), and give some qualitative examples to show why translation doesn’t work in some cases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3te1XXhAd7", "reviewer": "Reviewer_QoYJ", "summary": "This paper proposed an MOE architecture to handle multilingual multimodal tasks in vision-language models. And created a new multimodal understanding benchmark including 6 languages translated by GPT-4 with human post-edit.\n\nNote: this paper uses an incorrect template, which might have risk of getting rejected. According to ICML's author instructions: \"All submissions must closely follow the formatting guidelines in the provided templates; otherwise, they will automatically be rejected.\" Please correct it as soon as possible.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind comments and constructive feedback on our paper.\n\n> **Q1: Motivation of data efficiency.**\n\nA1: While large-scale translated multilingual data may seem abundant, its quality (especially for low-resource languages) is often critically compromised due to translation errors, cultural mismatches, and noisy artifacts. Even with massive data volumes, low-resource languages remain underrepresented in practice **because high-resource languages inevitably dominate training distributions** (e.g., 90%+ of tokens in typical datasets). Forcing higher proportions of low-resource data risks triggering **the curse of multilingualism**, where over-parameterized models sacrifice high-resource language performance to accommodate low-resource languages, as observed in prior work [1-3]. Our approach strategically prioritizes high-quality alignment signals over raw data quantity, avoiding both the pitfalls of noisy translation and the imbalance inherent to brute-force multilingual scaling. This ensures stable performance across all languages without compromising high-resource capabilities. We will further discuss and refine the motivation, taking this issue into careful consideration in the final version.\n\n[1] Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020.   \n[2] When Is Multilinguality a Curse? Language Modeling for 252 High- and Low-Resource Languages. ICLR 2024.   \n[3] Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models. EMNLP 2024.\n\n> **Q2: Translate-train/test baselines.**\n\nA2: We appreciate the insightful suggestion to include translate-train and translate-test baselines. While these are valid approaches, they face critical limitations:\n1. Translate-Train:   \n    - **Data Quality:** Machine-translated data often contains semantic distortions, syntactic errors, and cultural mismatches (e.g., idioms, region-specific references), especially for low-resource languages. For example, as shown in the table below, our experiments showed that training with 70K translated multilingual samples for each language achieved limited improvement.    \n    - **Imbalanced Optimization:** Merging large-scale translated data amplifies the dominance of high-resource languages (e.g., English/Chinese), as models tend to overfit to their syntactic patterns, further marginalizing low-resource languages. This phenomenon is called the curse of multilingualism.\n    \n|Methods|MMMB_en|MMMB_zh|MMMB_pt|MMMB_ar|MMMB_tr|MMMB_ru|\n|-|-|-|-|-|-|-|\n|LLaVA w/ 0K|67.1|58.8|59.8|43.5|46.4|59.1|\n|LLaVA w/ 10K|67.0|59.0|60.3|44.1|47.2|59.4|\n|LLaVA w/ 30K|66.8|59.4|60.7|44.6|47.9|59.7|\n|LLaVA w/ 50K|67.1|59.3|61.2|44.4|47.6|60.1|\n|LLaVA w/ 70K|66.7|59.7|61.3|44.8|48.1|60.4|\n|Parrot|70.0|68.1|67.3|62.7|58.0|66.3|\n\n2. Translate-Test:   \n    - **Latency Overhead:** Translating inputs introduces **2× latency (translation + inference), making real-time applications impractical.** For instance, translating a 100-token Arabic query to English adds ~500ms latency (Google Translate API), which is prohibitive for interactive systems.   \n    - **Limited Gains:** **As shown in Table 14 in Appendix**, our preliminary tests with LLaVA on MMMB showed that the translate-test improved Arabic accuracy by only 3.8% (vs. Parrot’s +19.2% gain), while degrading Russian performance by 0.2% due to translation errors. Further details are provided in **Response A8** of the reply to Reviewer Gwu1.\n\nThese results align with prior findings [4-5], where translate-train/test paradigms underperform dedicated multilingual architectures in both efficiency and robustness. Parrot circumvents these issues by directly aligning cross-lingual semantics without relying on noisy intermediate translations.\n\n[4] Lost in Translation: Analysis of Information Loss During Machine Translation Between Polysynthetic and Fusional Languages. ACL 2018.   \n[5] Why do LLaVA Vision-Language Models Reply to Images in English? EMNLP2024.\n\n> **Q3: Template formatting issue.**\n\nA3: We sincerely appreciate the reviewer’s attention to detail. Upon re-examination, we confirm that the manuscript was prepared using ICML’s official template. However, subtle discrepancies in visual formatting emerged during PDF compilation due to technical nuances in rendering tools. Regardless of the compilation method, the page limits and other requirements are fully in line with ICML’s guidelines. While we cannot directly update the PDF during the rebuttal phase, these unintended inconsistencies have now been fully resolved, with corrections to be incorporated into the final version.", "claims_and_evidence": "Yes, their experimental results shown the effectiveness of their method compared with other baselines.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "There are previous works on multilingual instruction tuning and multilingual multimodal evaluation benchmarks, and the authors discuss their relationships in this paper.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper proposes a novel MOE architecture integrated into current VLMs, which can help enhance model's multilingual ability during visual instruction tuning in a natural and easy-to-understand way.\n2. This paper points out the limitations of previous literature regarding multilingual multimodal benchmark curation and creates a new benchmark.\n\nWeaknesses:\n1. I have some doubts about the motivation of this paper, where the authors claim in the introduction: \"it is necessary to use as little multilingual data as possible to enhance the model's multilingual capabilities.\" In fact, we can relatively easily obtain large-scale multilingual data using translators. Although the quality is not guaranteed, the quantity is sufficient, which might make the performance gains from the new architecture trivial compared to using more translated training data. While this requires additional cost, in my view, the cost is not too significant compared to pre-training.\n2. This leads to another question about the experimental setting in this paper: a strong baseline that needs to be considered for comparison is using machine translation to obtain translated large-scale datasets, e.g., 200K samples per language, then training VLMs on this merged translated dataset (commonly known as the translate-train setting). Another strong baseline would be to first translate test data to English, then test performance on the translated data (commonly known as the translate-test setting).", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the issue of "multilingual erosion", where MLLM loose proficiency in non-English languages after multimodal alignment training (e.g. answering in English while asking non-English questions). PARROT presents a novel approach to enhance the multilingual capabilities of MLLMs, using language specific embeddings to fuse to visual embeddings and multilingual MoEs. To be able to  evaluate multilinguality, the paper also introduces a new Massive Multilingual Multimodal Benchmark (MMMB). Experiments show SOTA results on MMMB, MMBench and other multimodal tasks. During rebuttals, authors clarified and resolved reviewers' concerns. While this is primarily empirical work, the paper solves an important problem and makes a substantial contribution to the field of multi-modal LLMs.
Decision: Accept (poster)