Parrot: Multilingual Visual Instruction Tuning

Authors: Hai-Long Sun, Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, Han-Jia Ye
Primary area: generative models
Keywords: Multimodal Large Language Models; Multilingual MLLM; Mixture-of-Experts

Abstract:
The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V has marked a significant step towards artificial general intelligence. Existing methods mainly focus on aligning vision encoders with LLMs through supervised fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs' inherent ability to react to multiple languages progressively deteriorate as the training process evolves. We empirically find that the imbalanced SFT datasets, primarily composed of English-centric image-text pairs, lead to significantly reduced performance in non-English languages. This is due to the failure of aligning the vision encoder and LLM with multilingual tokens during the SFT process. In this paper, we introduce Parrot, a novel method that utilizes textual guidance to drive visual token alignment at the language level. Parrot makes the visual tokens condition on diverse language inputs and uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens. Specifically, to enhance non-English visual tokens alignment, we compute the cross-attention using the initial visual features and textual embeddings, the result of which is then fed into the MoE router to select the most relevant experts. The selected experts subsequently convert the initial visual tokens into language-specific visual tokens. Moreover, considering the current lack of benchmarks for evaluating multilingual capabilities within the field, we collect and make available a Massive Multilingual Multimodal Benchmark which includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our method not only demonstrates state-of-the-art performance on multilingual MMBench and MMMB, but also excels across a broad range of multimodal tasks.

Decision: reject
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730085676313

thread_type:
official_review

<thread_object>
creator: reviewer_iikz

<content>
summary:
The paper aims to strengthen the multilingual ability of vision-language models.
Due to the data imbalance problem, vision-language models often perform better on English-based data while suffering from low performance on language with scarce data.

To address the problem, the paper proposes a routing strategy with textual guidance.
The router makes the image embeddings language-aware.
Moreover, the paper collects new data for non-English languages with the assistance of GPT-4.

Experiments on MMBench show reasonable improvements over baseline models, like LLaVA.

strengths:
(1) The paper is clear and easy to follow.     
(2) The data imbalance problem in vision-language models is interesting.

weaknesses:
(1) How to train the language transformation experts for the MoE module?        
     Are there explicit constraints to optimize the routing?                
     As the paper mentions training the baseline LLaVA with the multilingual data suffers from data imbalance problem.                 
     However, the proposed training strategy can also suffer from the problem:                  
          I. The routing strategy can be dominated by English-based data.                        
          II. The language transformation experts should perform worse for languages with fewer data.                  
     The authors should discuss how the above two problems are addressed.  

(2) How does the MoE module affect model performance as the data size of each language increases?
     Also, how does the increased data size affect the baseline model performance, like LLaVA?

(3) The proposed method actually injects language information into the image embeddings, making $H_v$ language-aware.
      Another straightforward baseline is to train several translation expert models, translating other languages into English.

questions:
(1) Carefully study how the performance of language transformation experts affects the overall performance.        
(2) Inspect how the router works, like looking into the classification accuracy for different languages.         
(3) Compare with the baseline models fairly, like using the same training data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730522053159

thread_type:
official_review

<thread_object>
creator: reviewer_cbkd

<content>
summary:
This paper addresses the imbalance in the quantity of SFT data for different languages within training datasets used in MLLM's SFT process, which results in suboptimal alignment performance for various languages with limited data. To tackle this issue, the authors propose a novel structure that combines the cross-attention and MoE structure, enabling the input visual tokens to MLLM to be conditioned on the language input. Furthermore, to address the current lack of comprehensive benchmarks for multilingual multimodal tasks, the authors propose a new benchmark, named MMMB, which provides a more extensive evaluation for multilingual MLLMs. Extensive experiments validate the effectiveness of the proposed method on multilingual benchmarks.

strengths:
- The author's observation regarding the lack of balance among different languages in the SFT data of MLLMs is insightful.
- The proposed method is simple and efficient.
- The benchmark proposed by the author is rigorous and highly relevant for evaluating multilingual MLLMs.
- The paper is well-written and easy to follow.
- The experiments are comprehensive, demonstrating the effectiveness of the method proposed by the author.

weaknesses:
- **Regarding the issue of alignment:** I agree with the author's observation that the imbalance of SFT data among different languages may lead to poor alignment between vision tokens and multilingual tokens in MLLMs. However, it should be noted that the alignment data in the pretraining phase consists of English-only data, and the amount of data in the pretraining phase is significantly larger than that in the SFT phase. Would the impact of alignment between visual tokens and different language tokens be more severe in the pretraining phase?
- **Lack of comparison for the latest MLLM models**: Some of the VLMs the author compares are outdated. Could the evaluation include the latest VLMs, such as Qwen2-VL [1] and LLaVA-OV [2]?
- **About the scalibility**: Introducing a language-aware structure is an effective approach, but if similar structures are not introduced, would simply increasing the proportion of different language data in the SFT data yield a similar improvement in model performance? In larger models, such as those with 30B or larger model, is the performance gain from this model design consistent?

[1] Peng Wang, et al. "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution" arXiv preprint arXiv:2409.12191 (2024)

[2] Bo Li, et al. "LLaVA-OneVision: Easy Visual Task Transfer" arXiv preprint arXiv:2408.03326 (2024)

questions:
Please see the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692570843

thread_type:
official_review

<thread_object>
creator: reviewer_f1m5

<content>
summary:
This paper proposes Parrot, an MLLM designed to handle multilingual tasks. Parrot follows the LLava architecture, introducing an additional MoE module after the visual projector to enhance multilingual understanding. During training, Parrot translates public datasets into multiple languages and adopts a two-stage training scheme similar to LLava. To assess multilingual capabilities in MLLMs, the paper introduces MMMB (Massive Multilingual Multimodal Benchmark), encompassing 6 languages, 15 categories, and 12,000 questions. Parrot demonstrates strong performance on both MMBench and MMMB.

strengths:
1. High reusability—the proposed approach is simple and highly portable.
2. Extensive experimentation—the paper validates multilingual capability across multiple benchmarks.
3. Open-sourced multilingual benchmark dataset (MMMB) to support multilingual evaluation for MLLMs.

weaknesses:
1. The in-house dataset is not discussed in detail, such as whether it will be open-sourced, and whether the manual calibration process mentioned in lines 365-366 follows the same methodology as MMMB construction.
2. There is no experimental analysis on Parrot's performance loss in a single language, for example, whether the use of multilingual data and the MoE module reduces the model's English proficiency.

questions:
1. As mentioned in Weakness #1, will the in-house dataset be open-sourced? The construction process should be explained in more detail, particularly regarding noise control and data diversity.
2. As mentioned in Weakness #2.
3. Parrot’s training consists of two stages, largely following LLava’s approach. However, the inclusion of the MoE architecture raises questions about its integration in stage 1. Specifically, how are the MoE weights initialized? If initialized randomly, is it optimal to include the MoE in stage 1, given that this stage focuses on aligning multimodal features? Additionally, if the MoE is indeed included in stage 1, an ablation study on whether to freeze or not freeze the MoE module would be insightful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717543557

thread_type:
official_review

<thread_object>
creator: reviewer_folk

<content>
summary:
This paper aims at the multilingual issues in recent MLLMs and proposes a MoE-based alignment layer to this end. First, the authors find that existing MLLMs are not friendly to non-English queries and analyze this from the imbalanced training datasets. A simple solution would be to train a new adapter for each language, but this approach is not feasible due to the limited number of training pairs. This paper thus introduces Parrot, which trains a soft adapter under the MoE framework. To fully test the multilingual ability of MLLMs, this paper also releases a Massive Multilingual Multimodal Benchmark(MMMB). Results on MMMB and MMBench show that Parrot has a better multilingual alignment with limited training data.

strengths:
1) The problem that improves the multilingual abilities of MLLMs is an open and challenging problem. This paper develops a simple and efficient insight for the community.

2) This idea that uses MoE-based adapter is interesting. It can learn from the limited image-text pairs and show good performance empirically.

3) The collected new benchmark MMMB would be useful for subsequent research.

4) Extensive comparison and ablations show the efficiency of the proposed model.

weaknesses:
1) Parrot  depends on balanced datasets and does not consider unbalanced situations, which are the most common in practice. That is saying that Parrot may not satisfy the scaling law. When considering massive training pairs, unbalanced cases occur, which could lead to sub-optimal MoE learning, sticking in the same predicament as existing MLLMs.

2) Lack of a strong baseline. I wonder about the performance of a naive baseline where we first translate the question into English and then translate the English answer back to the target language.

questions:
1) At the first pretraining stage, is the MoE initialized with random parameters? If yes, how can we learn a good Projector under a random MoE?

2) An open question: What is the most real benefit for today's MLLMs? Recent MLLMs can be divided into two groups: 1) dataset-driven, these models adopt simple adaptor to map images into the language space (Qwen-vl, llava, gpt-4o, gemini Pro) and jointly train MLLMs on massive image-text data. 2) tokenization-based models, these models believe a good image tokenization can align the image and text well (Parrot, [1][2]). In my opinion, the simple structure and target datasets fine-tuning may have better robustness and improvement than small structure-based modifications.

[1] https://arxiv.org/abs/2408.05019
[2] https://arxiv.org/pdf/2405.01926
</content>
</thread_object>

</thread>
<thread>
creation date:
1732018600534

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to express our deepest gratitude to the reviewers for the meticulous examination of the paper and their insightful and valuable comments. We acknowledge that all the reviewers observed the shining point, saying our work is **well-written, interesting, easy to follow, and comprehensive** (foLK, F1M5, cBkD, iiKz). They also consider our proposed method as well as the new multilingual benchmark meaningful (foLK, F1M5, cBkD), shedding light on the MLLM community (cBkD), and being useful for subsequent research (foLK). They agree on the insightful observation of the lack of balance among different languages, too (foLK, cBkD, iiKz). Additionally, all the reviewers acknowledge that extensive experiments validate the improved performance of our proposed method (foLK, F1M5, cBkD, iiKz).

In this rebuttal, we have given careful thought to the reviewers’ suggestions and made the following revisions to our manuscript to answer the questions and concerns:

- In **Supplementary Section D.6**, we add numerical results about the multilingual data scaling and model size scaling;
- In **Supplementary Section D.7**, we add experiments about the baseline LLaVA using the same multilingual data as Parrot;
- In **Supplementary Section D.8**, we add the experiments to compare Parrot with Qwen2-VL and LLaVA-OV and extend Parrot with Qwen2-7B LLM;
- In **Supplementary Section E.1**, we add a more detailed description and pseudocode of the MoE training strategy;
- In **Supplementary Section E.2**, we add the experiments about the translation-based baseline and discussions about the challenges when using this approach;
- In **Supplementary Section E.3**, we add the description about the construction of our in-house dataset;
- We have uploaded the source code of Parrot as the **Supplementary Material**;
- We have uploaded the training dataset of Parrot to an anonymous GitHub repository (**[Code and Dataset](https://anonymous.4open.science/r/Parrot-Anonymous-FDC2)**).

We have highlighted the revised part in our manuscript in **blue** color. Please check the answers to specific comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732264861911

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Appreciating Your Reviews and Humbly Ask for Feedback

comment:
Dear Reviewers,

During the remaining time of the author-reviewer discussion period, it would be great if you could inform us whether our response has addressed your concerns regarding our paper. Your dedication to reviewing our work despite your busy schedule is genuinely appreciated. Lastly, we just want to say thank you for your evaluation of both our paper and our rebuttal.

Kind regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary:
The paper introduces Parrot, a Mixture-of-Experts (MoE)-based model to improve multilingual alignment in Multimodal Large Language Models (MLLMs). It addresses imbalances in non-English data and proposes MMMB, a multilingual benchmark. Empirical results demonstrate Parrot’s improved performance across benchmarks compared to baselines.

(b) Strengths:
Introduction of a new evaluation benchmark (MMMB).

(c) Weaknesses:
1) Baseline comparisons lack rigor; fair comparisons across pre-trained models and hyperparameters are missing.
2) The MoE's effectiveness in low-resource scenarios is not fully resolved.

(d) Decision:
Reject. While the paper presents a well-motivated study, the limited novelty, incremental technical contributions, and incomplete baseline comparisons lead to rejection.