Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a simple yet effective strategy to better balance different languages when pretraining a massively multilingual language model (such as mBERT, mT5, XLM-R, or XLM-E). The idea is quite straightforward but nicely motivated and convincingly executed: instead of the typically used temperature-based sampling, the authors propose the UniMax strategy which avoids or mitigates overfitting to low-resource languages as it caps the number of repeats in the skewed distribution of the languages. This capping is done by distributing a predefined character budget as uniformly as possible without using more than N epochs per language (where N is the hyper-parameter).

Using mT5-style pretraining as the case study, the paper then shows the benefits of the UniMax strategy over the standard temperature-based sampling at pretraining, focusing on two multilingual tasks: NMT (on WMT21) and QA on TyDi QA. The empirical findings support the proposed strategy, and the authors offer a series of additional experiments and ablations related to critical strategy choices (e.g., the value for N). They also demonstrate that UniMax works with different computational budgets and yields performance benefits in 'full-budget' as well as 'reduced-budget' pretraining regimes.

As clean language input is pivotal for the strategy to work well, as a side contribution, the authors also clean and filter the original mC4 corpus, yielding higher-quality pretraining data.
strength_and_weaknesses: Strengths:
S1. The proposed strategy is very easy to understand and to implement, it is motivated well (from the high-level hypotheses all the way to empirically showing why it should work, cf., Figure 1, and showing that it works in the actual experiments, cf., Figure 4 and Table 3).

S2. The work is well motivated and very well written - the reader can really follow the main motivation and the authors' line of thinking, which is supported by empirical evidence and insightful analyses throughout the paper.

S3. The proposed strategy holds promise to improve all currently available multilingual LLMs, given sufficient budget for computation - hopefully, this will inspire retraining of most of the current models.

S4. The side contribution (the cleaned mC4 corpus) might be useful for other researchers as well. 

Weaknesses:

W1. The choice of tasks is okay, but could be improved - I feel that more emphasis in evaluation should be given to low-resource languages, e.g., perhaps evaluating on the NLI task using the AmericasNLI dataset?

W2. This is not a major weakness as I am fully aware of computational restrictions - the paper focuses on a single model in this paper (mT5) so without providing any extra empirical evidence, it is not guaranteed that UniMax will work with other multilingual Transformers such as XLM-R or X-MOD. It probably will work, but this has to be empirically checked.
- Related to the X-MOD work (Pfeiffer et al., NAACL-2022) - I wonder if the UniMax strategy is equally important for that pretraining approach, where the curse of multilinguality is mitigated by modularising pretraining - is it then equally important to do capping of low-resource languages if each language is associated with its dedicated parameters? The authors should at least discuss this paper as part of related work.

W3. The paper should more clearly isolate the impact of cleaning mC4 versus the impact of the actual proposed sampling strategy.
clarity,_quality,_novelty_and_reproducibility: The paper is very well written, and clearly links its main hypotheses with the actual experiments, providing clear and easy-to-grasp motivation, with clear contributions as well.
- There are some minor clarifications that should be provided in the revised version. In Section 6, it is unclear whether the mT5 model used in comparisons in Table 3 is mT5 from the previous work (i.e., pretrained on the old/uncleaned mC4 corpus) or whether it is the mT5 model pretrained by the authors on the cleaned mC4 corpus).
- Ideally, the paper should also isolate the impact of cleaning the mC4 corpus by: (a) pretraining the full-budget mT5 on the cleaned corpus and comparing it with the old mT5; (b) also running their Unimax pretraining (in the 'reduced-budget' setting with the old mC4 corpus to isolate how much UniMax actually depends on having the cleaned corpus).

Novelty and Originality:
- Tackling the language sampling strategy for multilingual Transformer-based models is not a novel idea, and it has been tackled in some prior work (e.g., see this previous work: https://aclanthology.org/2021.findings-acl.106.pdf). However, doing experiments on pretraining has been limited, mostly due to computational constraints (as the ability to perform extensive comparative empirical analyses is limited when the core focus is put on pretraining). However, the authors did find a good balance between what can versus cannot be done (given the computational limitations), and the paper does a convincing-enough job in supporting the proposed UniMax strategy (which is quite simple in its core). As such, the paper might raise more awareness on improving language sampling in future work as well.

Reproducibility: the paper provides a large number of additional results and design choices in the main paper as well as in the appendix, so I don't have any concerns when it comes to replicating the main experiments from the paper.
summary_of_the_review:  This is a solid work with solid motivation and convincing experiments (also in light of computational demands required for this type of work). The idea is not highly original and not highly exciting, but the final deliverables of the work might have some impact in the field of multilingual representation learning. 

The coverage of related work is also really good, showing that the authors are knowledgeable in this field and have done a good job in systematising knowledge from prior work. 

I am open to adjusting my score if the authors provide additional information related to some of my questions above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes UniMax, a simple language data sampling strategy to provide a more uniform coverage of high-resource languages. This allows the training to be more resistant to overfitting and memorization on low-resource languages since they are repeated excessively on a standard upsampling strategy. The paper also performs an extensive experiments on a series of datasets on various multilingual benchmarks. The reported findings are useful for practitioners and researchers who are working on multilingual large language models.

**Methods**
- The proposed method is relatively simple and straightforward. 

**Efficiency**
- The paper also highlights the effective usage of computing budget, which is very important in building very large models.


strength_and_weaknesses: *Strengths:*
- A simple yet effective method. The approach is straightforward, and the results outperform the other sampling ratio baselines.
- The authors plan to release the datasets. It is a plus for reproducibility.

*Weaknesses:*
- The novelty is limited.
- Lack of analysis on how the sampling method affects the low-resource languages. I would suggest adding a specific section to describe the effects of sampling on low-resource languages. It would be great if the authors could also group the languages on the downstream tasks, so it is easier for the reader to get insights.
- There is a technical issue on the Algorithm 1. $p_l$ <- Normalize($U_l$) should be $p$ <- Normalize($U$) instead?
clarity,_quality,_novelty_and_reproducibility: **Clarity**
- Overall, the paper is well-written. The motivation is clear.
- Technical issue with the Algorithm, particularly on the normalization part. $p_l$ <- Normalize($U_l$) should be $p$ <- Normalize($U$) instead?

**Quality**
- The paper quality meets the expectation of a good submission. No major typograpical errors.

**Novelty**
- In terms of technical novelty, this approach is not essentially new.
- The paper focuses on analysis and empirical results. The study can be helpful in training a model with a vast number of languages.

**Reproducibility**
- It is possible to reproduce the results with some difficulties (i.e., some hyperparameters are not detailed mentioned). It would be great to release the code and dataset to help other researchers to reproduce the results.
summary_of_the_review: In general, the paper provides interesting idea to investigate the sampling ratio to effectively address the overfitting issue on languages with much less data, and, it is practically useful. Indeed, a good empirical paper; however, the novelty of the paper is very limited, and I think the analysis of how changing the ratio affects the training loss and final performance is one of the important contributions of this paper.

For now, I would give "marginally below the acceptance threshold". And I would be happy to adjust my score if the authors address the concerns mentioned above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new sampling method to solve the language-imbalance problem in multilingual pretraining. The new method generally upsamples low-resource languages to a fixed epochs and uniformly sample high-resource languages. Experiments are conducted following mT5 and show the proposed method is better than the widely-used temperature-based sampling.
strength_and_weaknesses: Strengths:

The proposed model is simple yet effective in experiments. Large-scale experiments on mT5 model and mC4 data are conducted to show the effectiveness. Various downstream tasks are verified.


Weaknesses:

The claim that the proposed UNIMAX outperforms the temperature-based sampling is not well supported by experiments. According to Section 5, the UNIMAX outperforms on low-resource languages while underperforms on high-resource languages, although the average performance over multiple languages is improved. Prior works (even the temperature-based sampling itself) already found that larger improvments on low-resource languages with smaller degradation on high-resource languages would result in better average performance. A method which for example improves on low-resource languages and keeps the performance on high-resource languages is expected.

Experiments are not convincing enough. Limited evaluation tasks (TyDi QA and Multilingual NMT) are selected without good reasons to ignore others. The comparison to mT5 is unfair given different data and settings. In addition, it is unknown whether the proposed method would improve on other pretrained language models like mBERT. And could it be used on parallel data for training like XLM, mBART?

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow. The novelty is limited. Data and codes will be released for reproduction.
summary_of_the_review: See the section of Strength and Weaknesses
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors explore an alternative method to temperature sampling for multilingual data sampling. The central premise is that existing methods can oversample low-resource languages resulting in overfitting and memorization. To alleviate this problem, they propose UNIMAX which caps the number of repetitions for low-resource languages while sampling high-resource languages uniformly. The experiments show that UNIMAX reduces overfitting for low-resource languages during pre-training and also improves performance on some downstream tasks. 
strength_and_weaknesses: **Strengths**

- Extensive experimentation over large models and long pre-training provide strong evidence for the results. 
- The UNIMAX method is simple to implement. 
- The results show that close to uniform sampling works well and repetition may not very useful which is promising for language equity and efficient use of compute cycles. 

**Weaknesses**

- The paper performs evaluation only on 2 tasks. Evaluation on a wider range of tasks would provide stronger evidence for the conclusions. 
- The cross-lingual evaluations have been performed in in-language/translate-train settings. To understand the impact of the sampling strategy on cross-lingual representations, it would be better to report results in the cross-lingual zero-shot setting. 
clarity,_quality,_novelty_and_reproducibility: - The paper is well-written and extensive experiments have been systematically reported. 
- The paper adds to the body of work on sampling methods for multilingual training. The sampling method provides some benefits in some downstream tasks. 
- An additional artifact of the work is the availability of a larger and cleaner version of the mC4 corpus. 
summary_of_the_review: The paper addresses the question of overfitting on low-resource languages while performing multilingual sampling. The results show close to uniform sampling holds promise, but evaluation on more tasks and settings as mentioned above is called for to make general claims.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:

This paper proposes a simple yet effective strategy for language data sampling for training a large multilingual pre-trained language model, to avoid oversampling low-resource languages, which often happens with the temperature-based strategy.  Experiments on mT5-style models and two multilingual tasks show benefits of the proposed method in various experiment settings.  A filtered subset of mC4 corpus is provided as a higher-quality praining data for researchers.


Strength:

The proposed strategy is very easy to understand and to implement.
The work is well motivated and very well written.
The approach is straightforward, and the results outperform the other sampling ratio baselines.
A new datasets for multilingual LM training will be released.


Weaknesses:

More evaluation could be helpful to make the paper more convincing, for example, with more models and datasets.

justification_for_why_not_higher_score: More evaluation could be helpful to make the paper more convincing, for example, with more models and datasets.
justification_for_why_not_lower_score: This paper addresses an important problem for the training of multi-linguil PLMs.  The proposed methods is effective.
The authors' response well addressed most of the concerns raised by the reviewers.
Decision: Accept: poster