EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

Authors: Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schuetze, Jörg Tiedemann, Barry Haddow
Primary area: datasets and benchmarks
Keywords: multilingual adaptation, large language model.

Abstract:
In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, with a focus on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset and enrich it with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks and PolyWrite, an open-ended generation benchmark developed in this study. Our results highlight the effectiveness of continual pre-training in expanding large language models’ language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability.

Decision: reject
Reviewer scores: 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729927556129

thread_type:
official_review

<thread_object>
creator: reviewer_yeqs

<content>
summary:
This paper proposes a multilingual dataset named "the MaLA corpus" by augmenting and merging existing datasets, a multilingual model by continue pre-training on the processed dataset using LLaMA-2-7b, and a open-ended multilingual benchmark generated by ChatGPT. The main contents of this paper are used to illustrate the common pipeline of processing data and report results on multilingual benchmark. The only finding in this paper is that the multilingual capacities of LLaMA-2-7b can be improved by continue pre-training on multilingual corpus, which is trivial.

strengths:
- The MaLA corpus compiled and PolyWrite multilingual benchmark will be beneficial to multilingual NLP researchers if they are public.
- Extensive experiments are conducted to show the effectiveness of EMMA-500.

weaknesses:
- [Addressed] Limited novelty: the corpus are generated by merging and cleaning existing datasets. The method to create multilingual benchmark is not novel, which is commonly adopted in the multilingual NLP community. It is an incremental work on augmenting the multilingual corpus and benchmark.
- Lacking baseline: some models of continue pre-training on LLaMA-2-7b are missing like ALMA[1] and Bayling[2].

[1]. Xu, H., Kim, Y. J., Sharaf, A., & Awadalla, H. H. A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR 2024.

[2]. Zhang, S., Fang, Q., Zhang, Z., Ma, Z., Zhou, Y., Huang, L., ... & Feng, Y. (2023). Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. arXiv preprint arXiv:2306.10968.

questions:
1. Will this corpus, model weight, benchmark and processing scripts be public?
2. Line 121-122: "This issue is resolved ...", Which issue has been resolved?
3. Line 310-311: Why is the global batch size set to 16M, rather than the commonly used 4M tokens like LLaMA-2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730144756575

thread_type:
official_review

<thread_object>
creator: reviewer_idtk

<content>
summary:
This paper presents a continually-pretrained multilingual LLM that covers 546 languages. The paper focuses on the data mix (MaLA corpus) and the evaluation. 
The model is based on Llama2-7B and compared to other Llama derivatives or multilingually pretrained models. It claims comparatively strong performance on commonsense reasoning, machine translation and open-ended generation. 
The data combines multiple existing sources and adds preprocessing steps.
The evaluation includes existing benchmarks and new intrinsic benchmarks and a new automated open-generation benchmark that is using BLEU as a metric for comparing generations.

strengths:
- Solid effort in including/focusing the data mix on low-resource languages that were excluded in previous works. This is ambitious and showcases that more languages are possibly “in reach” for more language-inclusive language modeling.
- Good highlight on iso code normalization and writing system treatment, which is often underestimated/undervalued in multilingual modeling but essential.

weaknesses:
1. No discussion of related work beyond listing models for comparison and data sources that are part of MaLA. It would be important to communicate how this work differs from past efforts beyond covering more languages in continual training. What are the main achievements that have been unlocked? Where is the scientific novelty?
2. Presentation: The result tables are overcrowded with text, and some of the results that are mentioned as main model strengths are hidden away in the appendix. It is hard for the reader to understand which numbers the claims of performance are grounded in. Furthermore, it is not clear what the main goal behind the model is: is it to be good at tail languages or beat other models in their supported languages or both?
3. Evaluation: 
    - Metric: Two of the three main evaluation results are based on BLEU, although it is known that BLEU is (1) not equally suitable for all 500+ languages (see e.g. https://arxiv.org/pdf/2205.03983, Section 4.2) (2) is purely surface-based, e.g. underestimates paraphrases, neglects semantics, etc. The evaluation with self-BLEU is questionable - what does high diversity mean in the absence of a metric of accuracy? 
    - Human: There is no human evaluation of the final model, nor the data pipeline, hence it is not clear how good the generation is especially in the newly covered tail languages. 
      - The only benchmark that covers 500+ languages is intrinsic modeling on Glot500-c, which might be biased as the same corpus was used during pre-training, hence data quality issues won’t show in evaluation.
      - The new PolyWrite benchmark is machine-generated and machine-translated, and machine-filtered. How reliable is it? Is there Western-centric bias in the prompts, or are they equally representative for all languages covered?
    - Safety: No safety-related evaluation. It is known that data quality especially in low-resource languages is often lacking, caused by quality issues in the data pipeline, which in turn can introduce unwanted biases (https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00447/109285) 
    - Comparison with other models: It is not fair to compare models that do not officially support certain languages on these specific languages. This distorts averages across languages and makes it impossible to compare where wins are due to covering languages at all vs truly outperforming other models that also include these languages. I would recommend separating these: evaluating against other models on their languages and benchmarks, and then evaluating on the newly covered languages, with a focus on quality of generations (some basic checks can be run: is the generation in the correct language? Does it have expected length?). 

  I understand that this represents a “best effort with limited means” - building reliable evaluation for 500+ languages is a huge enterprise by itself. However, the paper should at least prominently discuss these shortcomings and be more explicit about evaluation gaps and how to interpret the presented results, and which purpose the model can serve for advancing NLP in these languages. To be concrete, let’s say I want to use the model for Nogai - what can I expect? 
Expanding the set of languages in training without having reliable evaluation metrics for each of them (or at least discussing or being transparent about their shortcomings and evaluation gaps) is not sufficient for responsibly claiming an advance in multilingual modeling. I do believe that the effort put into data preparation and training is worth publishing, but needs to be evaluated and presented in a much more rigorous manner.

questions:
- What is the relation to the MaLA-500 model/work? What are key improvements over it?
- How was the data mix tuned? What was the process of setting the weights, based on intuition or prior experience or reports? Were any ablations run? This would be nice to describe in the Appendix (if no space in the main paper) to help others understand the deciding factors in setting up a successful data mix.
- Table 4: 
  - Why is tower-instruct performing so poorly (especially since it’s built for translation)?
  - How would the model perform with fewer or more shots? Is the number of shots optimized for EMMA-500?
- Since whitespace cleaning only applies to whitespace-tokenized languages, are the other languages cleaner per se or need cleaning in other ways? 
- How were existing datasets and their languages to be included chosen?
- Table 6: Are BLEU scores in this low range even reliable? Is the difference between 1- and 2-BLEU outputs significant? This should be supported by at least some evidence, e.g. significance tests or qualitative examples.
- How is self-BLEU evaluation influenced by sampling hyperparameters? Please report sampling parameters and discuss how they might affect the self-BLEU.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730504923467

thread_type:
official_review

<thread_object>
creator: reviewer_bad1

<content>
summary:
EMMA-500 outlines work around three technical contributions:
* EMMA-500 -> A multilingual model trained on 546 languages
* MaLA Corpus -> A multilingual corpus spanning 939 languages (A subset of which that had >100k tokens were used in the EMMA-500 training)
* PolyWrite -> Open Ended generation benchmark

strengths:
* Always great to see work that tackles the challenges in the long tail of languages.
* In-depth information for several preprocessing steps. This is really useful and important to allow future work to build on. 
* Several evaluations performed on a plethora of datasets.

weaknesses:
I would like to list the following weakness fully ensuring the authors that I am not unreasonable and am completely open to increasing my score if these are addressed/answered satisfactorily.

* The contributions section could be rewritten as bullet points and bold headings to emphasize the exact contributions. This was confusing for a first-time reader given that MaLA-500(I assume this is related to previous work) also exists.
* Evaluations lack benchmarking the recent Aya-23 model. They are even omitted when using the eval sets from the Aya paper. Surprisingly they are only benchmarked in the code evals without any further explanation.
* The categorization of “high” vs “mid” vs “low” resources is created by measuring token count in this specific training dataset, however there already exists widely accepted standards/taxonomies(like https://microsoft.github.io/linguisticdiversity/ – Joshi et al). No reference or explanation is provided as to why the authors deviated from this standard. 
* For n-shot evals ( like 3-shot on FLORES), it would be useful to also see 0-shot evals to understand how big a role ICL plays in the performance gains.
* I would edit the tables to be easier to read( for ex: bold the important numbers; say lower/higher is better; add arrows)

questions:
* Could you please explain why the Aya model was omitted from some evals despite being quite a popular model for multilingual benchmarking?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission introduces EMMA-500, a multilingual language model trained across 546 languages via continual pre-training on the MaLA corpus, a multilingual dataset compiled from various existing sources. The paper highlight improvements in multilingual performance, particularly for low-resource languages, and propose a new open-ended generation benchmark, PolyWrite, alongside evaluations on multilingual tasks. The claimed contribution lies in improving Llama 2’s multilingual capacity, especially for underrepresented languages. 

However, the submission falls short of significant novelty. While the compilation of the MaLA corpus and its integration into a training pipeline is valuable, the data cleaning and methodology lack rigorous analysis or clear advancements over existing multilingual approaches. Furthermore, the evaluation framework is incomplete and raises concerns: reliance on automatic metrics like BLEU and self-BLEU, known to be unreliable across diverse languages, is not sufficiently mitigated with human evaluations. PolyWrite, though a novel benchmark, lacks thorough validation, with unclear reliability and concerns about biases inherent in machine-generated prompts. Additionally, comparisons with state-of-the-art multilingual models, including Aya-23, are insufficient, and the justification for their exclusion in certain evaluations is unconvincing. The taxonomy for language categorization deviates from established standards (e.g., Joshi et al.), and the rebuttal failed to adequately address concerns regarding this choice. 

Overall, despite the considerable effort to create a resource and improve multilingual inclusivity, the work does not present a substantial scientific contribution beyond aggregating and cleaning datasets for continual training. Given these weaknesses, I recommend rejecting the paper.