UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining
Authors: Hyung Won Chung
Xavier Garcia
Adam Roberts
Yi Tay
Orhan Firat
Sharan Narang
Noah Constant
Venue: ICLR 2023 poster (2023)
Keywords: Keywords: multilingual
pretraining
language models
language sampling
language distribution
low-resource languages
overfitting

Abstract: Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages. However previous work has not systematically evaluated the efficacy of different pretraining language distributions across model scales. In this paper, we propose a new sampling method, UniMax, that delivers more uniform coverage of head languages while mitigating overfitting on tail languages by explicitly capping the number of repeats over each language's corpus. We perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale. We find that UniMax outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.

Published as a conference paper at ICLR 2023
UNIMAX: FAIRER AND MORE EFFECTIVE LANGUAGE SAMPLIGG FOR LARGE-SCALE MULTILINGUAL PRE- TRAINING
Hyung Won Chung, Noah Constant", Xavier Garcia" Adam Roberts, Yi Tay, Sharan Narang, Orhan Firat Google Research
h. W. ww..ngg227 7 gmaiil com, (nconstant, xgarcial@roggle.com
ABSTRACT
Pretrained multilingual large language models have typically used heuristic comperature-baser sampling to balance between different languages. However previous work has not systematically evaluated the efficacy of different pretrain- ing language distributions across model scales. In this paper, we propose new sampling method, UNIMAX, that delivers more uniform coverage of head lan- guages while mitigating overfitting number of repeats over each language's s corpus. We perform an extensive series ablations testing range of sampling strategies marks, while varying model scale. We find that UNIMAX outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UNIMAX sampling.
tail languages by explicitly capping the
a suite of multilingual bench-
INTRODUCTION
State-of-the-art multilingual models (Xue et al. 2021 2022 Goyal et al. 2021 inter alia) utilize large-scale supersupe learning. which involves jointly training on many languages. Because data availability varies greatly across languages, multilingual pretraining can be characterized as multitask learning (or multi-objective optimization) with severe data imbalance. Typically English is the highest-resource language (or task) with orders of magnitude larger size than lower-resource languages. For example, in the mC4 corpus (Xue al.) (202), English has roughly 9.7 trillion char- acters, which is over 92,000 times larger than the lowest resource language, Yoruba. As result, a key problem in designing such models is the "language balancing" problem: in what proportions should we balance the pretraining languages? Deriving the optimal balance is difficult open search problem due to the high cost of pretraining.
The standard approach to this problem has been to upsample languages with smaller datasets, using a temperature hyperparameter 7 (Devlin et al., 2019). However, one shortcoming of this approach that choosing т based the desired distribution among higher-resources languages may result in examples from the lowest-resource languages being repeated excessively. Figure II shows the number of epochs covered for each language in the mC4 corpus. When using 3.33 and a tril- lion token budget (the values used in popular models such as mT5 and ByT5), the lowest-resource languages are repeated over 100 times. This excessive repetition can have several unwanted conse- quences: (i) it leads to overfitting, which degrades performance on downstream tasks (Raffel et al.)
2020 Lee et al. 2022 Hermandez al. (2022), (ii) it increases the risk of memorizing private or sensitive content (Carlini et al. [2021] Lee et al. 2022), and (iii) it wastes training cycles that could have been devoted to unique examples. As models continue to grow in scale (Chowdhery et al.) 2022 Brown et al.) 2020; Smith et al. 2022), these issues with temperature sampling grow more pressing, as larger models benefit from longer training (Hoftmann et al. 2022). overfit more easily, and have greater capacity memorize.
"qqual contribution

---

Published as a conference paper at ICLR 2023
Pretraining sampling rate [%
50
100
r=3.33 (1 budget)
Number of epochs
UniMax (1 /8 budget)
50
3.33 (Ix budget)
25
UniMax (Ix budget)
UniMax (Ix budget)
UniMax (1 budget)
3
2
2
1
0
0
50 Language rank
100
0
50 Language rank
100
(a) Number of training epochs for each language. Temperature sampling results in large number of data repeats for low-resource languages, whereas UNIMAX explicitly caps repeats.
(b) Pretraining sampling distribution. Temperature sampling results in poorly balanced distributions, whereas UNIMAX provides more uniform distribu- tions without excessive upsampling.
Figure I: The x-axis is the rank of the language based on the character count. 1/8 budget refers to the 250,000 steps with sequence length of 512, which one-eights of the full-scaling training budget (IM steps with 1024 sequence length) referred to as Ix, matching that of mT5.
This paper proposes new paradigm for sampling across languages and datasets that ameliorates the above mentioned problems. We propose UNIMAX (uniform + max), a conceptually simple but highly effective two-pronged sampling approach that results in fairer and more effective language distributions for pretraining multilingual language models that work well across model scales. One of the main assumptions we make that practical large-scale training jobs operate with a fixed amount of compute, which is often translated into fixed training token budget (Raffel et al.) 2020). UNIMAX starts by pre-allocating training tokens underrepresented datasets based on the number of allowed max repeats (N). For the remaining budget, prioritize "linguistic utility" BBasi jet al. 2022) by allocating uniformly across all languages with sufficient data to avoid exceeding the prescribed number of per-language epochs. Unlike previous approaches, this means UNIMAX relatively resistant to distribution biases that arise due to artifacts of the corpus generation process (i.e., web crawlers). To take concrete example, the mC4 corpus contains 70x more English than Chinese text. While mT5's temperature sampling (r = 3.33) results in training on 3.4x more English than Chinese, UNIMAX will assign equal training tokens to the two languages, provided that this doesn't 't result in repeating the 39 billion available Chinese tokens more than N times.
Another key benefit of UNIMAX is that sampling strategies at scale, it is important to carefully control how many times a dataset can be repeated during training avoid overfitting and memorization. Our proposed method explicitly controls the extent of data repeats of any language, providing a direct countermeasure to overfitting on low-resource languages, without imposing any reprioritization on higher-resource languages.
robust model scaling. In considering language
Our key contributions are to: (1) Propose UNIMAX, simple but effective language sampling strat- egy that provides more uniform coverage of high-resource languages while mitigating overfitting on low-resource languages. (2) Perform extensive series of ablations testing a range of sam- pling strategies on a suite of multilingual benchmarks, while varying model scale. (3) Release an improved and refreshed variant of the mC4 multilingual corpus consisting of 29 trillion characters across 107 languages. (4) Release pretrained model checkpoints using UNIMAX sampling
RELATED WORK
While (onasively-)multiiingge models enjoy the benefits of positive transfer across languages, the sheer number of languages reduces the effective capacity of the model per task. This competition
s.md
2

---

Published as a conference paper at ICLR 2023
among languages for limited model capacity is the well-known problem of "capacity bottleneck" AAivazhagan et al. 2019). also known as the "curse of multilingualiity (Conneau et al. 2020).
In exploring the interaction between language balancing and model scale, previous work has ablated sampling temperature with models at or below 1 billion parameters (Conneau et al. 2020 Xue et al. ).21], but we believe our study is the first systematically explore balancing strategies scales encoder-decoder models, whereas most previous work used encoder-only models or targeted machine translation exclusively. Michel et al. (202T) explore adaptive mixing strategies in a similar setting, and also find that fixed uniform mixing strong baseline for multilingual pretraining.
above billion parameters. Additionally, our work targets general-
Within the context of machine translation" Jean et al. (2019) proposed using a bi-level optimization algorithm maximizing the validation utility to adjust the sampling weights per language. AArivazha gan et al. (2019) was the first testing various temperatures (T 5, 100) to train a multilingual translation model for 212 language pairs. The study suggested over-sampling low-resource while sub-sampling high-resouree languages to maximize uniform utility. The over-traininggooer-itttt due to over-sampling was alleviated by increasing the number of tasks in that study. WWan et al. (2020b) proposed another bi-level approach, using differentiable data selection (Wang et al. 2020a). The study includes a uniform language distribution baseline (T = oo), but this differs from UniMax in having no limit repetition of examples in low-resouree languages. Recently Wang et al. (202I) proposed using the gradient aligoment information across languages to adjust the language weights, but it is unclear if this can be practically scaled to 100 or more languages.
Work on cross-lingual transfer learning has repeatedly found that English may not be the "ideal" source language for transfer. Malkin al. (2022) conduct an analysis of zero-shot transfer, and calculate a "donation score" for each of 22 languages, measuring whether its inclusion in pretraining helps or hurts performance in other languages. They find English among the "least donating" one of only four languages whose inclusion results in an overall degradation in performance for other languages. In a similar exploration. Ture al. (202T) find that German and Russian are more effective source languages for zero-shot transfer than English. These findings support the idea of testing sampling strategies where English doesn't outweigh other languages.
We focus here on the problem of balancing languages with significant web data. In particular, we limit our scope to the approximately hundred languages that occur with some frequency in the CommonCrawl corpus, as detected by CLD3 language detection. Other methods are likely needed to scale NLP methods to thousands of languages, e.g., see Wang et al. (2022) and Bapna et al. (2022)
3 SAMPLING METHODS
Temperature-based sampling In order define sampling distribution over languages, an empirical distribution for each language
T4
Pt
Ever TW
(1)
where ny represents the "size" of language I, discussed in Section [3.1 A temperature sampling strategy uses distribution defined by exponentiating pt by the inverse of the temperature and renormalizing.
1/T
P
qi
EVeLPY 1/T
Commonly used temperature values in the literature are summarized in Table Higher temperature makes the distribution "flatter". approaching the uniform distribution as + do. In the other extreme with r = O, the entire probability mass is contained in the language with the highest probability. Note, there is no guarantee that there exists value of that achieves the desired balance on high- and mid-resource languages while still avoiding overfitting on the tail languages.
We view the translation literature as directly relevant for language balancing research. However one key difference is that translation models used in balancing studies have assumed that English is always either the source or the target language. This may lead to preferring language distributions that cover more Fnglish.
3

---

Published as a conference paper at ICLR 2023
Table I: Language sampling temperatures of recent multilingual LLMs trained on unlabeled data.
Temperature (7)
mBERT (Devlin et al. 2019
1.43
XLM
CCnnncau Lample 2019
2.00
XIMMR (Conccau et
3.33
m55 (uu et
3.33
XLM-E CChi al. 2022
1.43
UNIMAX sampling For UNIMAX sampling, start with predefined character budget C. In practice, this is typically defined by the training compute allocated for the training job. The goal of UNIMAX sampling is allocate the character budget languages
possible, without using more than N epochs of any language. The first step is to sort the languages based on the character count in the training corpus. We iterate over languages starting from the one with the lowest character count. At each iteration, we check if the remaining character budget can be split evenly among the remaining languages without using more than N epochs of any language. If so, we allocate the budget uniformly. If not, language / is allocated N epochs worth of characters and the remaining budget reduced. Algorithm formalizes this procedure.
uniformly
Algorithm 1: UNIMAX
Inputs: Character count & of each language / all the languages L of the training corpus Total character budget C'
The number of epochs per language N
Output: Sampling distribution iı of each language
// Sort the languages by increasing number of character counts
L sortByCount(L)
B + C // Initialize the remaining budget to the total character budget
i 0
for I in I. do
by +
Iea[L]-i // Compute the remaining budget per-language
B
if bj C X N then
// If per-language budget exceeds N epochs of I, use N epochs
Ui + x N
else
U + by // Otherwise use uniform per-language budget.
end if
B - B - U1 // Update the remaining budget
i
end for
normal parmalize(( iz))
return p
3.1 VOCABULARY GENERATION PROCESS
Following Xue et al. 2021 we use SentencePiece tokenization (Kudo & Richardson 2018), which consumes corpus to produce subword-level tokenizer. This corpus sampled from the available training data, typically using the same distribution used during model training. In the multilingual setting, however, this amounts to instance the language balancing problem. While many works explore varying the temperature of the training distribution, few perform the analogous change the vocabulary-leannis
distribution. This could have drastic consequences in certain situations. For example, if the vocabulary learning distribution heavily favors Chinese, but the training distri- bution heavily favors English, models using this vocabulary may experience poor performance due to insufficient vocabulary coverage (Chung et al. 2020).
Most sampling strategies require some notion of "size" (nz) of each language This is typically computed by counting words tokens using pre-existing multilingual tokenizer. But beyond the
4

---

Published as a conference paper at ICLR 2023
unwanted complexity of requiring one "pre-tokenizer" to train second tokenizer, such strategies do not generalize well to the massively multilingual setting, as tokenization across 100+ languages is non-trivial, particularly in languages written without spaces (e.g. Chinese, Japanese, Thai).
We propose to resolve the above issues by training separate vocabularies for each sampling strategy, using character counts as measure of sub-corpus size. As previous work has tended measure training budget in tokens, when comparing against such approaches, we assume a token contains characters on average, following Xue et al. (2022). We use this same conversion rate determine the total character budget C in Algorithm given real training budget in terms of tokens.
4 EXPERIMENTS
4.1 PRETRAINING CORPUS
To ensure meaningful results when comparing language sampling strategies, it critical that the examples in our pretraining corpus are correctly labeled for language. However language detection is far from a solved problem (Caswell et al.] (220). and audits have found low accuracy across a range of public multilingual datasets (Kreutzer al. 2022). As one severe example, only 40% of the documents in the Marathi language bucket of the popular mC4 dataset (3.0.1) were found to be Marathi, with 50% coming from other languages, and 10% being non-language.
To mitigate this problem, we construct a new multilingual corpus by filtering mC4 (Xue et al. 2021 remove documents whose language ID confidence is below 0.95. Compared the original mC4 corpus (which used a threshold of 0.7), this removes 6. % of documents; however only 5. % of characters are removed, as the filtered low-confidence documents also tend be shorter. The most filtered languages are Welsh (86%), Sindhi (85%) and Luxembourgish (84%). Inspection of several hundred examples across a subset of languages indicated clear reduction in mislabeled documents.
4.2 VOCABULARY
As discussed in Section 3.1 we train dedicated SentencePiece vocabulary for each language sampling method considered. We analyze these vocabularies in terms of token lengths and coverage of various writing scripts in Appendix [A] Overall, the expected trends are observed that (i) higher temperature, (ii) smaller UNIMAX training budget, and (iii) higher UNIMAX max-epoch threshold (N) all lead to vocabularies that allocate capacity more uniformly across languages. This more uniform allocation results in fewer tokens from higher- resoucee language scripts (e.g. Latin), more tokens from lower-resource language scripts (e.g. Devanagari), and shift towards shorter tokens.
4.3 EVALUATION TASKS
In selecting evaluation tasks, aim satisfy several key properties. First, tasks should be lin- guistically diverse, covering a range of languages from distinct families and regions, including both high- and low-resource languages. Second, tasks should be free of language bias. For example, the task training data and evaluation metrics should be well-balanced across languages. We also avoid benchmarks where English plays special role, including datasets constructed in English and trans- lated post-hoc to other languages, as well as zero-shot transfer tasks where English is the sole source language. Finally, to the degree possible, benchmarks should be realistic, such that performing bet- ter on the benchmark gives us confidence that model will do better on actual tasks facing language technology users. This is in contrast to "intermediate structure" tasks such as part-of-speech tagging.
TyDi QA (Clark et al., 2020) is multilingual question-answering benchmark covering a range of typologically diverse languages. Questions are written from scratch by native speakers each lan- guage, ensuring culturally relevant content and the absence of "translationesee. We use the "GoldP" task, which covers 9 languages. To evaluate candidate models, we use the "in-language multitask" setting (Hu et al. 2020) fine-tuning mixture of all available languages, and evaluating in each language separately. To maximize per-language performance, we select per-language checkpoints based on the validation performance, and report validation metrics,
test set is provided.
The WMT21 shared task on large-scale multilingual machine translation (Wezeek et al. 2021) tests the ability of single model translate across many languages. We focus on the "small track" tasks,
5

---

Published as a conference paper at ICLR 2023
each testing translation between 6 languages, in all 30 combinations. As with TyDi QA, fine- tune a single multilingual model on the mixture of all tasks and select per-language-pale checkpoints based on the test set performance To reduce heavy English bias, we restrict training data to 1,000 examples per language pair by randomly subsampling from the training data. Limiting the size of the fine-tuning dataset also increases the importance of transferring knowledge from pretraining, thereby making more apparent the differences between various pretraining sampling strategies.
We also evaluate on several widely adopted multilingual benchmarks: XNLI (Conneau al. 2018 XQuAD (Artetxe et al. 2020), MLQA (Lewis et al. 2020) and PAW'S- (Yang et al. 2019). While these do not satisfy all of our desiderata above (e.g., some are translated from English and some skew towards high-resource languages), their popularity still makes them valuable points of reference.
4.4 TRAINING SETUP
We closely follow mT5 (Xue et al. 2021 for model architecture and training procedure. Specifi- cally, we use encoder-decoder Transformer architecture. We use the span corruption pretraining objective from T5 (Raffel et al. 2020) on a multilingual corpus consisting of 101 languages plus Latin-script variants (e-g. ru- t.). We use batch size of 1024 sequences where each sequence is defined by selecting chunk of 568 tokens from the training corpus. This is then split into 512 input
and 114 target tokens. For the results in Section {5] the number of training steps is 250,000 unless
otherwise stated. Additional training details can be found in Appendix
5 RESULTS
5.1 PRETRAINING LOSS
One challenge evaluating multilingual models the relative paucity of benchmarks that cover broad range languages and are free from language bias (see Section (4.3). Numerous studies have established strong correlation between pretraining and fine-tuning performance (Devlin et al.) 2019 Narang al. 2021 Tay et al. 2022 inter alia). Thus, in addition to targeted downstream evaluations, it can be valuable to monitor and analyze performance on held-out pretraining data, which by definition covers all pretraining languages. Figure [ shows pretraining loss curves of models trained with three sampling strategies, for both English (en) and Yoruba (ye), the highest and lowest-resource languages in our study, We plot three model sizes, as model scale controls the tradeoffs between positive cross-lingual transfer and overfitting.
In Figure 2a we observe that the loss values for en are much lower than for yo, and model scale alone does little to close this gap. This suggests that downstream performance on low-resource languages will suffer if they are not upsampled. On the other hand, too much upsampling prob- lematic, as shown in Figure 2b) Crucially, overfitting only emerges with scale: La rge shows no obvious overfitting, XL shows weak overfitting, and becomes conspicuous at XXL size. Moreover, while the effects of repetition may appear to be limited to Yoruba, results from Hernandez et al. 2022) suggest that overfitting in one language is likely to hurt performance across the board; the authors find that even repeating 0.1% of data 100 times can be as harmful halving model size.
Figure shows the pretraining loss curve for a longer train, corresponding to 1/2 character budget. In this case, overfitting emerges even for the Large model after around 300,000 steps. Given that previous work (Conneau et al. 2020 Xue et al. 2021) studied sampling distributions at smaller model sizes or for shorter training steps, the importance of overfitting may have been overlooked.
Notably, across Figures 2c and 3c| UNIMAX closes the loss gap between high- and low-resource languages without showing any sign of overfitting.
5.2
DOWNSTREAM EVALUATIONS
Figure 4
shows the average TyDi QA performance for the three sampling strategies discussed in Our first observation that UNIMAX outperforms the other two consistently across
Section 3
We exclude results on language pairs with Serbian as the target language, due to unstable performance. The WMT21 Serbian training data covers Cyrillic and Latin scripts, and we observed high variance in eval metrics across checkpoints, depending on whether the model happened use the same script the references.

---

Published as a conference paper at ICLR 2023
Large XI.
4
XL en XXL
3.0
70 Large
50 Large
yo Large
on Large
3.0
XL.
XL
XXL
2.5
XXL.
<n XXL.
XXL.
<n XXL.
2.5
2.0
2.0
1.5
0
50 100
150
200
250
0 50
100
150
200
250
50
100
150
200
250
Pretraining steps [K]
Pretraining stops [K]
Pretraining steps [K]
(a) 7 =1
(b) 7 3.33
(c) UNIMAX
Figure 2: Pretraining cross-entropy loss the held-out data over the training steps. With too-low
temperature, low-resource languages are sampled too little, and their losses are relatively high. With higher temperature, overfitting becomes severe with increasing model size. Note that loss values are not directly comparable across sampling strategies due to the difference vocabulary.
3.5
Large
3.0
yn Large
Large
en Large
en Lang
3
2.5
Loss
2.5
2
2.0
2.0
1.5
1.5
1
0
200 400
600
800
1000
0
200
400
600 800
1000
200
400
600
800 1000
Pretraining steps [K]
Pretraining steps [K]
Pretraining steps [K]
(a) 7
(b) 7 3.33
(c) UNIMAX
Figure 3: Pretraining cross-entropy loss the held-out data over 1M training steps. With the sequence length of 512, this corresponds to 1/2 character budget. The overfitting behavior emerges only after sufficient number of training steps for 3.33.
Average EM FI
Average of EM and FI
81
83
Average EM
80
UniMax
3.33
UaMMx
UniMa
79
80
82
11 Number of parameters [B]
11
13
Number parameters [B]
Number of parameters [B]
(a) All languages
(b) Higher-resource
(c) Lower-resource
Figure 4: Average TyDi QA GoldP performance across three model sizes. Overall, UNIMAX out- performs both baselines all model sizes considered. Breakdowns higher-resource (top-5) and lower-resource (bottom-4) languages show UNIMAX outperforms 7 = 3.33 on both high- and low- resource, and only underperforms
on high-resource at large model scales.
model scales. We can get additional insights by correlating per-language performance with the pretraining portion of each language, shown in Figure Generally, languages seen more during pretraining tend to have higher performance, but this benefit seems to have its limit. For example, r sampling allocates 47.7% of training to en but this only translates to 1% better performance than UNIMAX, which allots only 1 1% en. We believe this surprisingly small delta is attributable to the benefit of positive transfer from non-Engie non-English languages onto English. Additionally, we note that 3.33 Swahili, despite seeing fewer Swahili examples during training. See Table 7 for full lan-uane metrics across sampling strategies and model scales.
UNIMAX outperforms
Figure 6 (left) shows that UNIMAX also outperforms temperature sampling WMT21, across all model sizes considered. Figure (right) shows that the benefit is spread out across the majority of language pairs, opposed only benefiting small subset. We also note that the benefit is more

---

Published as a conference paper at ICLR 2023
50
Average of HM and FI
UniMax
UniMax
90
3.33
Sampling rate (X6
25
33333
85
80
75
ko
id bn te
ko
id
bn
to
Figure 5: Left: Per-language TyDi QA performance of XXL models pretrained using three different sampling strategics. Right: Pretraining sampling rates of the three models.
48
100
UniMax
Best method [%)
3.35
chrF
44
I
42
UniMax
50
t= 3.33
38
|
1
5
7
11 13
0
Number of parameters [B]
Large
XL
XXL
Figure 6: Left: WMT21 performance averaged across all language pairs. UNIMAX outperforms
both baselines all model sizes. Right: The majority of the language pairs benefits with UNIMAX.
pronounced for languages pairs where the target language is non-English see Fig. 8 for details. Appendix pDroovidess evaluations on additional benchmarks, which show similar trends.
5.3 FURTHER ABLATIONS
Our core experiments described in the previous sections were done across three model scales, but limited the character budget to 1/8, due compute resource constraints. In this section, perform two additional ablations, but limiting to Large size.
First, in Table 2 (left) show the effect of increasing the budget by 4x. Note, longer training changes the "shape" of the UNIMAX distribution (cf. Figure [b]. as the increased budget means more languages will hit their max-N epoch cap. Overall, find that UNIMAX still outperforms temperature sampling, with the longer train boosting performance across the board.
Second, we ablate the UNIMAX max-epoch parameter N. So far, we have used example is repeated. Table (right) shows TyDi QA results for N E { 1. 5, 10 J. We observe the best performance when disallowing repeats entirely, but the effect small. We also note that the optimal setting of N likely depends on the character budget, as for large enough budgets, UNIMAX-1 will see vanishingly little of the lowest-resource languages.
1, i.c. no
(right)
Table 2: Average TyDi QA results of additional Lar Large-size models. Left: Comparison of sampling strategies at larger (1/2) character budget. Right: Ablation of UNIMAX max-epochs (N) parameter.
TyDi QA
TyDi QA
1.0
81.2
N
82.2
7 = 3.33
82.8
81.5
UNIMAX 83.1
N = 10
81.8
8

---

Published as a conference paper at ICLR 2023
Table 3: Comparison mT5. XNLI and PAWS-X show average per-language accuracy; the rest show average per-language EM/FI. We use the translate-train setting except for TyDi QA, which uses "in-language". We omit results for the Large configuration due to instabilities in training.
XNLI
PAWS-X
XQuAD
MLQA
TyDi QA
Model
mT5
um
m15
um/15
mT5
um
m'3
um
mT5
um
Small
72.0
76.2
79.9
87.2
49.4/64.5
60.5 / 74.0
38.8 56.6
41.8 60.7
62.7 / 74.0
56.6/ 70.0
Base
79.8
80.8
89.3
90.4
59.7 /9.7/75,3
67.3 / 79.8
48.5/67.6
68.4/79.7 51.6 70.6
68.4/ 81.0
XL
85.3
86.5
91.0
90.7
56.6/75.1
75.0 86.1
73.5
58.3/
74.1 78.4/87.6
XXL
87.1
87.8
91.5
91.2
75.
77.9/88. 88.2
57.4/76.0
70.5 78.6
79.5 88.77
81.2 89.7
6
UMT5 MODELS
We put our above findings into practice by training a suite of "umT5" models over a trillion tokens. For these final models, we also update the training corpus and add analysis comparing to mT5, which is the most direct point of comparison. As large language models are increasingly used for -eneeiggginttesive tasks (Petroni al., 2019 Roberts et al. 2020, Petroni et al., [022), it is important that training corpora are up-to-date. Given that the mC4 corpus is over two years old, we update the corpus (update version 3.1.0) to cover crawled documents through August 2022.
Beyond adding fresh documents, we make three changes to mC4. First, as in Section $4.1 we raise the language detection confidence threshold from 0.7 to 0.95, which we found increase accuracy and reduce documents with little or no natural language. Second, we adjust the mC4 bad word filters to be soft filters, allowing a random 0.1 1% of documents with bad words to pass through. This ensures that models trained our corpus will have least minimal exposure to any term, which is likely to help on tasks like toxicity detection. Finally, we remove from mC4's bad words lists any term that results in filtering 10% of documents from the language in question. Applying these changes, our resulting training corpus consists of 28.8 trillion characters from 9.0 billion documents 35% increase in documents over mC4. This increase is due primarily added documents, and persists despite the more aggressive filtering during language detection. See Table for full corpus statistics.
We closely follow the training setup and evaluation tasks from the mT5 paper for a fair comparison. Table 3 shows that umT5 outperforms mT5 on most tasks, across all sizes, and particularly at the largest size See Appendix for an additional ablation isolating the effect of the data refresh.
CONCLUSION
In this paper, introduced UNIMAX, language sampling strategy that comes close to being uniform across languages- as close as possible without introducing harmful repetition. We showed this method performs well across several benchmarks and model scales, up 13 billion parameters. Given the guarantees provides against repetition, we expect UNIMAX is also well-suited to even larger model sizes, where issues of overfitting and memorization are known to grow more severe.
Our study focused on a single pretraining paradigm. Future work is needed to test if UNIMAX confers gains in other settings, such as: (i) encoder-only or decoder-only models, (ii) models trained with parallel data, and (iii) models with dedicated parameters per language (Pfeiffer et al. 2022).
Beyond evaluation metries, there is an priori reason prefer more uniform language distributions: they can be seen as more equitable, in that they come closer to treating each language as an equally priority. However there an important remaining question about whether "language" is the right unit over which to normalize. One alternative that we think deserves further exploration is inspired by the notion of "demographic utility" discussed by Blasi et al. (2022), which treats each speaker as equal in balancing across languages. See Appendix B for further analysis.
We are eager to see if future research can formulate successful sampling strategies that take into account both linguistic and demographic utility. Ideally the language distributions output by these strategies can lead to stronger-performing and more equitable pretrained models that better serve a wider range of users and use cases.
'We exclude Large size, as umT5-LLage exhibited pretraining instability and underperformed Base on all metrics. We have retrained 15-5--aarge with a new random seed and will release the final checkpoints.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We are grateful to Yuan Cao, Noah Fiedel, Ben Hutchinson, Kathy Meier-Hellstern, Slav Petrov, Sebastian Ruder and Siamak Shakeri for helpful comments and discussion.
REFERENCES
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR, abs/1907.05019, 2019. htt.tt://aaxii..r////aa/a//
05019
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics. pp. 4623-4637, Online, July 2020. Association for Computational Linguis- URL http:s///aalaattoooooooooo//////
tics. doi: ael-main. 421
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Meng- meng Niu, Pallavi N. Baljekar, Xavier Gareía, Wolfgang Macherey, Theresa Breiner, Vera Axel- rod, Jason Riesa, Yuanbin Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanping Huang, Z. Chen, Yonghui Wu, and Macduff Hughes. Building machine translation systems for the next thousand languages. ArXiv, abs/2205.03983, 2022.
Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. Systematic inequalities in language technology performance across the world's languages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5486-5505, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. (0.18653/v1/002. aal- 376
acl-long.376. URL httpt:htt///cctc http://acclanttooo..orrogg
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models Few-Shot Learners. arXiv e-prints, art arXiv:2005.14166 May 2020.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lec, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raf- fel. Extracting training data from large language models. In 30th USENIX Security Sympo- sium (USENIX Security 2/), pp. 2633-2650 USENIX Association, August 2021. ISBN 978-1- 939133-24-3, URL https: / www. usenix ptsp://nffeeeeeeeeeeeeeeeeee//// presentation carlinarini 1ini ext eactini
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. Language ID in the wild: Unex- pected challenges on the path to a thousand- laaguuge web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6588-6608, Barcelona, Spain (On- line), December 2020. International Committee on Computational Linguisties. doi: 18653/ 01/2000.clingg-minn.77. URL
wwwp./....000..ooo...g co wtt.:ooooooggogooo inn-main
579
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. XLM-E: Cross-lingual language model pre-training via ELECTRA. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6170-6188 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.1186/3/vll 006653/v1/2022/ 1/2022 acl222acclloooo long. URL https:
Taaclannholll ogy www.hhoolly...oo//00000 ac lonel.ooogg.... 427
10

---

Published as a conference paper at ICLR 2023
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy MeiereHellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways. arXiv e-prints, art. arXiv:2204.0233 023111, April 2022.
Hyung Won Chung. Dan Garrette, Kiat Chuan Tan, and Jason Riesa. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Em- pirical Methods Natural Language Processing (EMNLP), pp.
Online, November 2020. Association for Computational Linguistics. doi: 10. 10l1186//1/2220.ccoom 00t555///22200..mmmmaaaaa org/ wwwp:////000....0mmmme.......eee cammlppmmiin 367
URL
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information-ssekking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454-470 2020. doi: 10.1 10.1162/tacl URL WBLLattpss///allataaaooooooo
Eacl-1.3
Alexis Conneau and Guillaume Lample. In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL
Cross-lingual language model pretraining.
http://as.......ppaee///111..... cctppppperr/21///11ee////// pdf
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475-2485, Brussels, Belgium, guistics. doi: 10.18653/v1/D18- 1269. URL httph: pestps://aclaantollee org. 1269
2018. Association for Computational Lin-
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 144008851, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/253//1/0220.cllmmm 747, URL https::::: //aclanthology. wwwpoogy..rgg/0000 org/ acl wecc-mmaiii 747
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19--11 1423. https:
Naman Goyal, Jingfei Du, Myle Out, Giri Anantharaman, and Alexis Conneau. Larger-seale trans- formers for multilingual masked language modeling. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP-2021), pp. 29-33, Online, August 2021. Asso- ciation for Computational Linguistics. doi: 10.18853//v/20 1/2021. 1.4. URL https: www.aaachooocy...go//00eeeeeeeee
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nel- son Elhage, Zac Hatfield- Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and inter- pretability of learning from repeated data. arXiv preprint 2rXiv:2055.004: 2022.
11

---

Published as a conference paper at ICLR 2023
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Henni- gan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy. Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models. arXiv e-prints, art. arXiv:2203. 15555566, March 2022.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingus general- isation. Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Con- ference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 4411 141114 PMLR, 13-18 Jul 2020. URL https: https: https::/ppooeeedings wml.eedddinnsm........ wtwpinnss...r....s..... wwt.sssss..p..ss1ss......
1020b.htmm
Sébastien Jean, Orhan Firat, and Melvin Johnson. Adaptive scheduling for multi-task learning. arXiv preprint (rXiiv1909.06444 2019.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orsikkh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Pa- padimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaven- ture F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balh, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayo- dele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 72, 2022. doi: a0047. URL fictps://aclantaee
wwwp//ggg.22... tac1-1.4
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v 11//8653/1//D1-000 1/D18-2012 URL http:: //www www aclwels http://........ttooo//////////// http//toooooyyll--eo 2012
org
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings ofe the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424 8445, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 11655////2022.alllooomo 2002.
URL https: http:://cclatthologyy orq/
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MIQQ: Eval- uating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 7315-7330, Online, July 2020. As- URL https:
sociation for Computational Linguistics. doi: 1/200 htwp:hhhoollyy..g/222000200
//aclanthology. wrg/2020. acl miel-mmain.. 653
Dan Malkin, Tomasz Limisiewicz, and Gabriel Stanovsky. A balanced data approach for eval- uating cross- cross-lingual transfer: Mapping the linguistic blood bank. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties:
Human Language Technologies, pp. 4903-4915, Seattle, United States, July 2022. Associa- tion for Computational Linguisties. doi: 10. 10.18653/v1/2022 waw.:aaacclmaa saaacc-main.361... URL. https:
//aclanthology.oom wtg. 2022. maaacl-main...
Paul Michel, Sebastian Ruder, and Dani Yogatama. Balancing average and worst-case accuracy in multitask learning. CoRR, abs/21 10.00838, 2021. URL https: arxiv. wttp/oo/g/aaiiiaaoooa
05438
12

---

Published as a conference paper at ICLR 2023
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding. Jake Mar- cus, Adam Roberts, and Colin Raffel. Do transformer modifications transfer across implementa- tions and applications? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5758-5777 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.1100....11 18653/v122222 http:////22111111111222 .ew.lee aopp-main.465.. URL https:
/tac/aaaanttti http://awaaannttooo.o.ooo wwtp:ttthwwoooo....//2g/o emnlp ma main.aii.....mm 465
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/1//D1--122 URL ort/D19-1/1D
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome,, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v wtt.l/..aaclll-aieeeaaa URL https: http:///aaaaathhhoo
org/ 2021. nwacc - main. m..in.2. 200
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li. James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis-
tics: Human Language Technologies, pp. 3479-3495, Seattle, United States, July 2022. Asso- ciation for Computational Linguistics. doi: 18.18653/ 1/2023/111222222ll-mma 255. URL https:
//aclanthology. hol.aathoooogy..rg/2222222 naacl - ma iaaainn... 255
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
074. html
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418 14416,,46,, Online, November 2020.
Association for Computational Linguistics. doi: 10.18653/v 1/202/ /2t/:/00000mmmminnn..a https://aclanttae
URL
www.ooogg//0..oooooo/o pmm in. 437
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Mare van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bu- lian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Ceneealy,Jnaahann H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling Up Models and Data with t 55x and seqio. arXiv e-prints, art. arXiv:2203.17188 March 2022.
Noam Shazeer and Mitchell Stem. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Con- ference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4594-4604. PMLR, 10-15 Jul 2018. URL httpt:: htaps:///rrceedinng ml w.w....rress.////
shazeer Ba. h
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song. Mohammad Shocybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using DeepSpeed and Megatron to
13

---

Published as a conference paper at ICLR 2023
Train Megatron- NLG 530B, A Large-Scale Generative Language Model. arXiv e-prints, art. arXiv:2201.11990, January 2022.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. In International Conference on Learning Representa- lions, 2022. URL htttttss///peeeeeiieee
Iulia Ture, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, and Kristina Toutanova. Revisiting the primacy of english in zero-shot cross-lingual transfer. CoRR, 11121106.1111, 2021. URL
auttps://arxx attp://///aaaioooorooo////////// http://aas2220..1112
Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards. ICML'20. JMLR.org, 2020a.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural ma- chine translation. Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 8526 8537, Online, July 2020b. Association for Computational Linguis- tics. doi: 10. 10.18853/v1222 122020.clmaa acl-main.754
URL
Xinyi Wang, Sebastian Ruder, and Graham Neubig. Expanding pretrained models to thousands more languages via lexicon-based adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 863-877, Dublin, Ireland, May 2022, Association for Computational Linguistics. doi: 10. 10.1186633/11/2222aaa--ongg 61. URL bttps::/aclaathhlogyy org/2022 acl- mocc-loog..6
Zirui Wang. Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Confer- ence on Learning Representations, 2021. URL sttts:://ppereeee
n watee/foruuee forum?idd
FIVE WH
Guillaume Wenzek, Vishrav Chaudhary, Angela Fan, Sahir Gomez, Naman Goyal, Somya Jain, Douwe Kiela, Tristan Thrush, and Francisco Guzmán. Findings of the WMT 2021 shared task on large-scale multilingual machine translation. In Proceedings of the Sixth Conference Ma- chine Translation, pp. 89-99, Online, November 2021. Association for Computational Linguis- tics. URL.Attps//aclatthooooo. wwwp:oy..rrg22200
wmt-l.1
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. Linnputttionallinuuiiiiiissss doi: 10.18653/v1/2021 maacc--ain..4 URL https: 7/aclanthology. wttp/2011..accl1maai.......
Linting Xue, Aditya Barua, Noah Constant, Rami AI-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022. doi: 10. 20.1662/accaa.0044 URL (otthtt:: / wewaccannhhooggg wwt//og//22222 tac1-1117
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversar- ial dataset for paraphrase identification. In Proceedings of the 2019 Conference Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3687-3692, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.19653/1/DD11-1333 URL https://aclaattin wtw.:laahhtooo...ggoooo.oo
logy. wow..oog/o/..11333/0/
A VOCABULARY ANALYSIS
We analyze the makeup of the sub-word vocabularies resulting from different language sampling strategies in Table 4 To see how capacity is allocated across languages, we measure the proportion
14

---

Published as a conference paper at ICLR 2023
Table Frequency (%) of tokens satisfying various conditions across vocabularies generated using different sampling strategies, as well as the m15 vocabulary.
3.33
UMI 4x
UM10 4x
UMI 1x
UM10 1x
mTS
Latin
67.0
54.0
56.3
47.9
49.5
46.6
Cyrillic (e.g. Russian)
17.7
13.0
12.0
14.6
14.1
12.6
10.7
Han (Chinese/Japanese)
3.2
4.3
4.9
4.2
4.5
4.0
7.2
Arabic
2.8
4.6
5.6
4.2
4.5
4.6
3.0
Greek
1.3
1.9
3.1
1.5
2.0
1.3
2.1
Hangul (Korean)
0.8
1.5
2.1
1.5
1.8
1.4
1.7
Devanagari (e.g. Hindi)
0.5
2.2
1.9
3.0
2.9
2.6
1.3
Thai
0.3
0.9
1.2
1.0
1.3
0.9
1.8
Contains whitespace
56.1
51.8
52,8
51.5
52.3
50.9
22.5
Contains punctuation
0.4
0.4
0.4
0.5
0.4
0.5
4.2
1-char
6.1
8.2
8.6
8.3
8.6
8.2
7.8
2-char
4.3
6.6
6.2
6.5
6.3
6.7
10.1
3-char
8.9
12.6
11.7
13.1
12.6
13.4
17.8
4-char
12.4
15.4
14.3
16.0
15.4
16.3
21.8
5-char
13.1
15.2
14.3
15.5
15.1
15.9
17.2
6-char
12.1
12.8
12.6
12.9
12.8
13.1
10.5
7-char
10.6
9.6
9.9
9.6
9.7
9.4
6.0
8-char
7.0
7.6
3.8
9-char
7.4
5.0
5.7
4.8
5.1
4.6
2.2
10+ char
15.9
7.5
9.2
6.7
7.5
5.9
2.7
of tokens using scripts associated with specific languages or language groups. We observe that increasing from 1 to 3.33 reduces the amount of Latin-script and Cyrillic tokens, presumably due to heavy reduction of English and Russian, the two most prevalent languages in mC4. However, as expected. raising temperature increases allocation lower-ranking languages (Arabic, Chinese, Greek, Hindi, Japanese, Korean, Thai). Overall, the UNIMAX vocabularies are fairly similar to 3.33, with the expected trend that increasing pretraining budget (1x + 4x) shifts more allocation to high-resource languages, while increasing the max-epoch threshold (UM1 + UM10) shifts more allocation low-resource languages.
Looking at token lengths, we observe three factors contribute to shorter tokens: higher temperature, smaller UNIMAX budget, and higher UNIMAX max-epoch threshold. This is expected, as if few high-resource languages dominate the vocabulary training corpus, the sub-word optimizer will assign more capacity to rare words from these languages, as opposed to covering more frequent (and shorter) words from wider range of languages.
Comparing to mT5, we note that our vocabularies have longer tokens overall, and include more tokens containing the SentencePiece meta-symbol indicating whitespace ).. We suspect these dif- ferences stem from mT5 using a whitespace-splinee splitting "pre-tokenizer" to derive preliminary word counts which fed the sub-word training algorithm. By comparison, on raw text, with no need for a pre-tokenizer. We also note that mT5 contains many more tokens in- cluding punctuation. This is most likely due the presence of low-quality non-language documents that are filtered by our higher language detection threshold, as described in section 44.1
vocabulary trained
B REPRESENTATION RATIOS
To shed light on how moving towards more population-based sampling strategy would reshape language distributions, define the "representation ratio" IR(1, f) of language within training distribution to the ratio between the language's ' rate of use within and its rate of native speakers (s) among the world population (w), shown in equation (3). We say a language is "overrepre- sented" (R 1) or "underrepresented" (R < 1) to indicate that its prevalence training is more or less than expected based its global I.1 speaker population.
15

---

Published as a conference paper at ICLR 2023
1200,0
European Europen
1000
315.2
mT5 rale 100/11 wwaleereeeeee ral
200.0
103.0
10.0
10.9
1.0
1.0
60 2224 01
gwg.agad SEF3502
Figure 7: Representation ratios (mT5 pretraining rate speaker rate) of m'55 training languages. Esperanto (eo) and Latin (1a) are clipped to 1,000; with few no native speakers, the actual values are much higher.
R(I,t) = ts/ EveLte tv $1/W
w
8i Eventr tv
Figure plots representation ratios for the mT5 language distribution, with native speaker counts taken from Wikipedia We observe wide range of representation, ranging from 7x underrep- resented to over 900x overrepresented. We also note a clear pattern whereby languages of Europe tend to have higher representation Mann-Whitney U=566, p<le-5, two-tailed), echoing general bias observed by Blasi et al. (2022) for NLP resources.
Overall, find that languages with large speaker populations, and particularly non-European lan- guages are heavily underrepresented in commonly used pretraining distributions. For example, the top underrepresented languages in mT5 training are all languages of Asia and Africa with 50 million or more native speakers: Chinese, Punjabi, Yoruba, Bengali and Hindi
C ADDITIONAL TRAINING DETAILS
The model architectures used in this study are the same as mT5 models, except that relative posi- tion embeddings are not shared across layers. In all of models, the vocabulary size is 256,000 subwords, and byte-level fallback is enabled, so unknown tokens are broken down into UTF-8 bytes.
We the T5X library (Roberts al. 2022) train the models using Google Cloud TPUs. For pretraining, we use Adafactor optimizer (Shazeer & Stern 2018) with constant learning rate of 0.01 in the first 10,000 steps and inverse square root decay afterwards. For finetuning, we use Adafactor with a constant learning rate of 5c-5. Unlike mT5, we do not use loss normalization factor. Instead we use the number of real target tokens as the effective loss normalization.
Finally, we do not factorize the second moment of the Adafactor states and we also use momentum, neither of which are used in T5 and mT55 studies.
D ADDITIONAL BENCHMARKS
As further comparison, fine-tune and evaluate the nine pretrained models from Sections 4 and 5) three additional benchmarks: XNLJ zero-shot, XNLI translate-train, and XQuAD. We observe UNIMAX performs the best overall, although 3.33 is close second.
E ABLATION ON MC4 REFRESH
To isolate the effect of refreshing the mC4 data, train two additional models 100, 000 steps using UNIMAX sampling, under the same "full-budget" setting as in Section (6) These models differ only in that trains the original mC4 data, while the other uses refreshed mC4 corpus.
II Languages not covered by mT5 are not plotted, but have representation ratio of zero. As one example, Odia is language of India spoken by 33 million native speakers, but not included mT5 training.
16

---

Published as a conference paper at ICLR 2023
Table 5: Additional benchmark results across sampling strategy and model scale. XNLI scores are average per-language accuracy. XQuAD is run in the translate-train setting and scores are average per-language EM/F1.
XNLI zero-shot
XNLI translate-train
XQuAD
Large (I.2B)
1.0
74.8
82.0
71.0/82.6
T= 3.33
78.2
82.3
71.6/83.2
UNIMAX
78.3
82.7
71.6/83.1
XL (3.7B)
T == 1.0
84.1
73.8/84.9
3.33
85.0
74.2/85.3
UNIMAX
85.0
74.5/85.5
XXL (13B)
1.0
85.1
75.1/86.0
T = 3.33
85.5
75.4/86.3
UNIMAX
85.5
75.6/86.4
Table 6: Ablation mC4 refresh. XNLI scores are average per-language accuracy in the translate-
train setting.
XNLI
mC4
77.6
Refreshed mC4
77.7
17

---

Published as a conference paper at ICLR 2023
100
Bext method (%)
60
20
Large
XL
XXL
UniMax
Temperature 3.33
Temperature
Figure 8: WMT21 results restricting to language pairs with non-English target. UNIMAX performs best on the vast majority of pairs, across all scales.
Table 7: Per-lang TyDi QA GoldP performance, as average exact-match and Fl metrics.
Numbers in brackets represent the rank of the language in the pretraining corpus (c.g. ru has the second largest character count).
Avg
[1]
[2]
[15]
[23]
ko [27]
id [33]
bn [37]
te [59]
[62]
Large (1.2B)
T 80.1
78.7
81.0
79.0
76.1
83.4
79.4
85.5
T == 3.33
82.0
76.2
77.3
81.5
80.1
77.6
85.9
84.2
87.7
84.7
UNIMAX
83.3
76.5
78.6
80.8
80.4
77.9
85.9
86.5
87.9
85.6
XL (3.7B)
1.0
81.7
78.4
81.0
82.5
81.7
77.5
87.0
81.7
86.4
82.2
3.33
83.4
79.0
79.5
83.1
81.2
79.6
86.3
86.5
88.3
87.2
UNIMAX
83.9
78.0
82.3
81.9
80.4
86.8
88.4
87.7
XXL (13B)
T == 1.0
82.2
79.1
82.9
82.7
83.1
80.0
86.7
83,8
86.4
85.2
== 3.33
84.0
77.8
81.3
83.4
82.7
79.9
88.4
86.6
88.8
86.2
UNIMAX 84,4
78.1
81.0
83.1
82.2
81.5
89.2
89.5
88.5
86.6
Table 6 shows results fine-tuning these two models on the XNLI task. We observe that the data refresh alone only gives a small boost (-0.1), supporting the view that the gains of UNIMAX over mT5 (+1.0) in Table 3 are primarily due improved language sampling. While the inclusion of more recently crawled documents does not help on this particular benchmark (which also several years old), expect that the refreshed data will be useful to practitioners, and should help tasks requiring up-to-date knowledge.
F ADDITIONAL TABLES
18

---

Published as a conference paper at ICLR 2023
Table Statistics for our improved and refreshed variant of the mC4 corpus, as well as the sampling rates (%) of the sampling methods studied in the paper.
Chars (B)
UniMax (1x)
UniMax (1/8)
Chars
UniMax (1x)
UniMax
Lang
3.33
Lang
(B)
3.33
(1/8)
on
13,396
5.75
46.58
3.22
1.48
af
7.4
0.61
0.03
0.16
1.27
ru
3,018
3.68
10.49
3.22
1.48
7.4
0.60
0.16
1.26
2,052
3.28
7.13
3.22
1.48
6,9
0.59
0.02
0.15
1.18
de 1,799
3.15
6.26
3.22
1,48
6.3
0.58
0.02
0.13
1.08
fr 1,433
2,94
4.98
3.22
1.48
tc
5.9
0.57
0.02
0.13
1.01
779
g
0.93
pt 658
2.33
2.29
3.22
1,48
5.2
0.54
0.02
0.11
0.89
2h
556
2.21
1.93
3.22
1,48
4.8
0.53
0.02
0.10
0.82
pl
527
2.18
1.83
3.22
1.48
la
4.5
0.52
0.02
0.10
0.78
409
3.22
1.48
4.4
0.76
nl
371
1.96
1.29
3,22
1.48
my
4.2
0.51
0.01
0.09
0.72
Ir
349
1.93
1.22
3.22
1.48
SW
4.1
0,51
0.01
0.09
0.70
ar
252
1.75
3.22
1.48
ky
3.7
0.49
0.08
0.64
250
1.74
0.87
3.22
1.48
gu
0.08
0.61
240
1,72
0.83
3.22
1,48
km
3,5
0.48
0.01
0.07
0.60
CS
236
1.71
0.82
3.22
1.48
3.3
0.48
0.01
0.07
0.57
202
ey
3.1
201
1.63
0.70
3.22
1.48
3.0
0.46
0.01
0.06
0.52
hu
185
1.59
0164
3.22
1.48
ru-l Ltt
2.6
0.44
0.01
0.06
0.44
uk
171
1,55
3.22
1.48
pa
2.2
0.42
0.05
0.37
166
3.22
1.48
ga
2.1
0.36
da
132
1.44
0.46
2.83
1.48
zh-Latn
1.9
0.40
0.01
0.04
0.33
120
1.40
0.42
2.58
1.48
1.4
0.37
0.01
0.03
0.25
no
116
1,38
2.49
1.48
0,36
0.03
0.22
bg
1.32
1.48
lb
0.36
0.00
0.03
0.22
th
92
1.29
0.32
1.97
1.48
ha
1.1
0,34
0.00
0.02
0.19
sk
79
1.23
0.27
1.69
1.48
ceb
1.1
0,34
0.00
0.02
0.19
1.48
0.17
kO
1.21
0.26
1.59
1.48
mg
0.9
0.33
0.00
0.02
0.16
63
1.15
0.22
1.35
1.48
am
0.9
0.32
0.00
0.02
0.16
IW
1.12
0,20
1,23
1.48
el-Latn
0.9
0.32
0.00
0.02
0.15
1.48
0.15
id
51
1.08
0.18
1.09
1.48
0.8
0.31
0.00
0.02
0,14
sl
47
1O05
0.16
L.00
1.48
ht
0.8
0.31
0.00
0.02
0.14
1.02
0.91
1.48
yi
0.02
0.13
lv
0.99
0.13
0.83
1.48
0.8
0.31
0.00
0.02
0.13
bn
34
0.95
0.12
0.72
1.48
hi-Latn
0.7
0,30
0.00
0.02
0.12
sq
18
0.79
0.06
0.39
1.48
zu
0.7
0.30
0.00
0.02
0.12
1.48
jv
0.12
18
0.79
0.06
0.39
1.48
hmn
0.6
0.29
0.00
0.01
0.11
ta
17
0.78
0.06
0.37
1.48
mi
0.6
0.28
0.00
0.01
0.10
ms
0.75
0.32
1.48
0.27
0.00
0.09
0.73
0.30
1.48
0.5
0.27
0.00
0.09
kk
13
0.72
0.05
0.28
1.48
0.5
0.27
0.00
0.01
0.08
mr
13
0.72
0.05
0.28
1,48
xh
0.5
0.27
0.00
0.01
0.08
ne
0.23
1.48
st
0.27
0.08
0.68
0.04
0.23
1,48
0.4
0.25
0.00
0.07
ka
10
0.67
0.04
0.22
1.48
0.4
0.25
0.00
0.01
0.07
hy
0.66
0.03
0.21
1.48
ig
0.4
0.25
0.00
0.07
1.48
ja-Latm
0.06
fil
9.5
0.65
0.03
0.20
1.48
haw
0.4
0.24
0.00
0.01
0.06
ml
9.4
0.65
0.03
0.20
1.48
0.3
0.24
0.00
0.01
0.06
mn
9.3
0,20
1.48
bg-Larn
0.1
0.16
0.00
0.00
0.01
8.8
0.64
0.19
1.48
19


Reviews:
[{"review_id": "aoB9uqLD9H", "reviewer": "Reviewer_JVbY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors explore an alternative method to temperature sampling for multilingual data sampling. The central premise is that existing methods can oversample low-resource languages resulting in overfitting and memorization. To alleviate this problem, they propose UNIMAX which caps the number of repetitions for low-resource languages while sampling high-resource languages uniformly. The experiments show that UNIMAX reduces overfitting for low-resource languages during pre-training and also improves performance on some downstream tasks. ", "summary_of_the_review": "The paper addresses the question of overfitting on low-resource languages while performing multilingual sampling. The results show close to uniform sampling holds promise, but evaluation on more tasks and settings as mentioned above is called for to make general claims. ", "strength_and_weaknesses": "**Strengths**\n\n- Extensive experimentation over large models and long pre-training provide strong evidence for the results. \n- The UNIMAX method is simple to implement. \n- The results show that close to uniform sampling works well and repetition may not very useful which is promising for language equity and efficient use of compute cycles. \n\n**Weaknesses**\n\n- The paper performs evaluation only on 2 tasks. Evaluation on a wider range of tasks would provide stronger evidence for the conclusions. \n- The cross-lingual evaluations have been performed in in-language/translate-train settings. To understand the impact of the sampling strategy on cross-lingual representations, it would be better to report results in the cross-lingual zero-shot setting. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_JojyGtQe9", "reviewer": "Reviewer_ZdPt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a simple yet effective strategy to better balance different languages when pretraining a massively multilingual language model (such as mBERT, mT5, XLM-R, or XLM-E). The idea is quite straightforward but nicely motivated and convincingly executed: instead of the typically used temperature-based sampling, the authors propose the UniMax strategy which avoids or mitigates overfitting to low-resource languages as it caps the number of repeats in the skewed distribution of the languages. This capping is done by distributing a predefined character budget as uniformly as possible without using more than N epochs per language (where N is the hyper-parameter).\n\nUsing mT5-style pretraining as the case study, the paper then shows the benefits of the UniMax strategy over the standard temperature-based sampling at pretraining, focusing on two multilingual tasks: NMT (on WMT21) and QA on TyDi QA. The empirical findings support the proposed strategy, and the authors offer a series of additional experiments and ablations related to critical strategy choices (e.g., the value for N). They also demonstrate that UniMax works with different computational budgets and yields performance benefits in 'full-budget' as well as 'reduced-budget' pretraining regimes.\n\nAs clean language input is pivotal for the strategy to work well, as a side contribution, the authors also clean and filter the original mC4 corpus, yielding higher-quality pretraining data.", "summary_of_the_review": " This is a solid work with solid motivation and convincing experiments (also in light of computational demands required for this type of work). The idea is not highly original and not highly exciting, but the final deliverables of the work might have some impact in the field of multilingual representation learning. \n\nThe coverage of related work is also really good, showing that the authors are knowledgeable in this field and have done a good job in systematising knowledge from prior work. \n\nI am open to adjusting my score if the authors provide additional information related to some of my questions above.", "strength_and_weaknesses": "Strengths:\nS1. The proposed strategy is very easy to understand and to implement, it is motivated well (from the high-level hypotheses all the way to empirically showing why it should work, cf., Figure 1, and showing that it works in the actual experiments, cf., Figure 4 and Table 3).\n\nS2. The work is well motivated and very well written - the reader can really follow the main motivation and the authors' line of thinking, which is supported by empirical evidence and insightful analyses throughout the paper.\n\nS3. The proposed strategy holds promise to improve all currently available multilingual LLMs, given sufficient budget for computation - hopefully, this will inspire retraining of most of the current models.\n\nS4. The side contribution (the cleaned mC4 corpus) might be useful for other researchers as well. \n\nWeaknesses:\n\nW1. The choice of tasks is okay, but could be improved - I feel that more emphasis in evaluation should be given to low-resource languages, e.g., perhaps evaluating on the NLI task using the AmericasNLI dataset?\n\nW2. This is not a major weakness as I am fully aware of computational restrictions - the paper focuses on a single model in this paper (mT5) so without providing any extra empirical evidence, it is not guaranteed that UniMax will work with other multilingual Transformers such as XLM-R or X-MOD. It probably will work, but this has to be empirically checked.\n- Related to the X-MOD work (Pfeiffer et al., NAACL-2022) - I wonder if the UniMax strategy is equally important for that pretraining approach, where the curse of multilinguality is mitigated by modularising pretraining - is it then equally important to do capping of low-resource languages if each language is associated with its dedicated parameters? The authors should at least discuss this paper as part of related work.\n\nW3. The paper should more clearly isolate the impact of cleaning mC4 versus the impact of the actual proposed sampling strategy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vas_LRiQQ-", "reviewer": "Reviewer_JYyD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new sampling method to solve the language-imbalance problem in multilingual pretraining. The new method generally upsamples low-resource languages to a fixed epochs and uniformly sample high-resource languages. Experiments are conducted following mT5 and show the proposed method is better than the widely-used temperature-based sampling.", "summary_of_the_review": "See the section of Strength and Weaknesses", "strength_and_weaknesses": "Strengths:\n\nThe proposed model is simple yet effective in experiments. Large-scale experiments on mT5 model and mC4 data are conducted to show the effectiveness. Various downstream tasks are verified.\n\n\nWeaknesses:\n\nThe claim that the proposed UNIMAX outperforms the temperature-based sampling is not well supported by experiments. According to Section 5, the UNIMAX outperforms on low-resource languages while underperforms on high-resource languages, although the average performance over multiple languages is improved. Prior works (even the temperature-based sampling itself) already found that larger improvments on low-resource languages with smaller degradation on high-resource languages would result in better average performance. A method which for example improves on low-resource languages and keeps the performance on high-resource languages is expected.\n\nExperiments are not convincing enough. Limited evaluation tasks (TyDi QA and Multilingual NMT) are selected without good reasons to ignore others. The comparison to mT5 is unfair given different data and settings. In addition, it is unknown whether the proposed method would improve on other pretrained language models like mBERT. And could it be used on parallel data for training like XLM, mBART?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8D-xq-HeV-Y", "reviewer": "Reviewer_9yE2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes UniMax, a simple language data sampling strategy to provide a more uniform coverage of high-resource languages. This allows the training to be more resistant to overfitting and memorization on low-resource languages since they are repeated excessively on a standard upsampling strategy. The paper also performs an extensive experiments on a series of datasets on various multilingual benchmarks. The reported findings are useful for practitioners and researchers who are working on multilingual large language models.\n\n**Methods**\n- The proposed method is relatively simple and straightforward. \n\n**Efficiency**\n- The paper also highlights the effective usage of computing budget, which is very important in building very large models.\n\n", "summary_of_the_review": "In general, the paper provides interesting idea to investigate the sampling ratio to effectively address the overfitting issue on languages with much less data, and, it is practically useful. Indeed, a good empirical paper; however, the novelty of the paper is very limited, and I think the analysis of how changing the ratio affects the training loss and final performance is one of the important contributions of this paper.\n\nFor now, I would give \"marginally below the acceptance threshold\". And I would be happy to adjust my score if the authors address the concerns mentioned above.", "strength_and_weaknesses": "*Strengths:*\n- A simple yet effective method. The approach is straightforward, and the results outperform the other sampling ratio baselines.\n- The authors plan to release the datasets. It is a plus for reproducibility.\n\n*Weaknesses:*\n- The novelty is limited.\n- Lack of analysis on how the sampling method affects the low-resource languages. I would suggest adding a specific section to describe the effects of sampling on low-resource languages. It would be great if the authors could also group the languages on the downstream tasks, so it is easier for the reader to get insights.\n- There is a technical issue on the Algorithm 1. $p_l$ <- Normalize($U_l$) should be $p$ <- Normalize($U$) instead?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster