Scaling Laws for Multilingual Language Models

Authors: Yifei He, Alon Benhaim, Barun Patra, Praneetha Vaddamanu, Sanchit Ahuja, Parul Chopra, Vishrav Chaudhary, Han Zhao, Xia Song
Primary area: foundation or frontier models, including LLMs
Keywords: scaling laws, large language models, multilingual language models

Abstract:
We propose a novel scaling law for general-purpose decoder-only language models (LMs) trained on multilingual data, tackling the problem of balancing languages during multilingual pretraining. A primary challenge in studying multilingual scaling is the difficulty of analyzing individual language performance due to cross-lingual transfer. To address this, we shift the focus from individual languages to language families. We introduce and validate a hypothesis that the test cross-entropy loss for each language family is determined solely by its own sampling ratio, independent of other languages in the mixture. This insight simplifies the complexity of multilingual scaling and make the analysis scalable to an arbitrary number of languages. Building on this hypothesis, we derive a power-law relationship that links performance with dataset size, model size and sampling ratios. This relationship enables us to predict performance across various combinations of the above three quantities, and derive the optimal sampling ratios at different model scales. To demonstrate the effectiveness and accuracy of our proposed scaling law, we perform a large-scale empirical study, training more than 100 models on 23 languages spanning 5 language families. Our experiments show that the optimal sampling ratios derived from small models (85M parameters) generalize effectively to models that are several orders of magnitude larger (1.2B parameters), offering a resource-efficient approach for multilingual LM training at scale.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729822512895

thread_type:
official_review

<thread_object>
creator: reviewer_czqd

<content>
summary:
This paper presents a scaling law for multilingual LLMs. It also argues for a hypothesis that the cross entropy loss for each language family in the LLM is determined only by the sampling ratio of the family. Experiments are performed on 23 languages (from 5 families) and model sizes range from 95M parameters to 1.2B parameters.

strengths:
1. Scaling laws are important objects of study; better understanding can help us inform modeling/training decisions. The paper examines scaling law in the context of multilingual LLM, which I believe is new. There are scaling laws on monolingual LLM and machine translation models, but this setup deserves its own study. 

2. One practical result is the prediction of good language sampling ratios. This is an important design decision.

weaknesses:
1. While the results are interesting, I believe the claims are too strong. It would be better to tone down. Here are some examples of strong claims: 
- page 1: "Our contributions are grounded ... making our analysis scalable to an arbitrary number of languages." - The experiments are performed on 23 languages. There are 6000+ languages in the world. 
- page 7: Half a page is dedicated to explaining how the work is better than previous work (i.e. generalized framework, multilingual scalability, incorporation of dataset size, simpler...) I think it suffices to keep it simple and say that your work looks at a different problem; I do not see a need to argue so much. Further, it is somewhat strange to argue that your equation is simpler than previous work -- if we view scaling law as a phenomenon to be discovered, then faithfulness is probably more important than simplicity perhaps?

2. I question whether the proposed argument about language families as the main factor. It is true that there is probably more crosslingual transfer within a language family and less across families, but this is a very simplified view of language. Specifically: 
- Language families are genetic relationships, but in practice there are significant areal effects (c.f. https://aclanthology.org/N09-1067/ ). There are also other kinds of sharing, for example Japanese is an isolate but there are many Kanji characters borrowed from Chinese. 
- The experiments are actually not very diverse in terms of language families: Of the 23 languages (5 families) used, 22 languages (4 "families") are actually from the Indo-European "family". Germanic, Romance, Slavic, Indic are considered different branches of the Indo-European "family". So only 1 language (Chinese) is truly from a different family in the linguistic sense.
- One complication is script, which is not discussed in the paper. Different scripts will lead to very different tokenization results, which likely impacts LLM training. In the languages considered, most use the Latin script and the Russian branch uses both Cyrillic and Latin, I assume.

questions:
1. This is not necessary a weakness, but I wonder if we should view the models to be on the small side compared to contemporaneous work. For example, training data is up to 50B tokens and model size is 398M parameters to 1.2B parameters. I think there is value in the experiments in any case, but can you comment a bit on the scale of some of the large multilingual LLMs in the literature? For example, EuroLLM-1.7B is trained on 4 trillion tokens. 

2. The citations are generally adequate. Here are a two more suggested papers to add - they discuss scaling laws for bilingual machine translation. 
https://aclanthology.org/2021.emnlp-main.478/
https://arxiv.org/abs/2109.07740

3. Regarding sampling strategy, the general rule-of-thumb I think is to up-weight lower-resourced languages. At the same time, some anecdotes seem to suggest it is better to make English data the largest (e.g. 50%) due to its potential to help LLM in reasoning and other tasks. Can you comment about the chosen sampling strategy in your experiments in the context of these guidelines? How similar or different is it? Or does it depend on the group of languages?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730123799765

thread_type:
official_review

<thread_object>
creator: reviewer_ig2e

<content>
summary:
The paper studies a scaling law for general-purpose decoder-only multilingual Language Models to reduce the need for resource-intensive data mixture selection in large-scale training. This paper focuses on balancing language families rather than individual languages during pretraining. The authors hypothesize that pretraining can leverage language family groupings to simplify the complexity of multilingual learning. From this hypothesis, they derive a power-law relationship that links dataset size, model size, and language sampling ratios, helping to optimize training across multiple scales (85M, 397M, 810M, and 1.2B parameters) on each of the 5 language families.

strengths:
S1. The proposed scaling law could simplify the complexity of cross-lingual interactions during training, offering a meaningful extension of existing work that focuses primarily on monolingual or bilingual models. This decoupling from individual languages could also reduce the computational burden.

S2. The paper also proposed an optimal sampling ratio as a practical data-mixing strategy, which could be applicable for real-world multilingual model training.

S3. The paper is clear in its presentation, with a well-structured flow of ideas that makes it easy to follow. The authors provide clear and well-presented tables, figures, and mathematical formulas, which support the text and make complex concepts more digestible.

weaknesses:
W1. The primary issue with the paper is its focus on the generality of the proposed scaling law without applying it to specific downstream tasks (beyond machine translation as it is claimed). Testing it on downstream tasks like summarization or question answering, on metrics that are widely used to assess the performance of each downstream task, would have strengthened the claim that this scaling law is for a general-purpose decoder multilingual language model. It also provides more solid evidence of its applicability across different use cases to interested readers.

W2. The improvement in normalized loss from the optimal sampling ratio appears minimal. For example, in Table 4 for the 85M model, the difference between the uniform sampling ratio ($5.912$) and the proposed sampling ratio ($5.895$) is just $0.017$, raising questions about the significance of this gain. It is unclear to me how beneficial such a small reduction in loss is, especially given the additional complexity introduced by the data-mixing strategy. Again, going back to W1, if these minor improvements do not translate into meaningful performance gains on downstream tasks, the value of this approach is not that convincing. The paper should showcase the practical advantages of achieving these marginal gains in normalized loss.

Additionally, in Table 6 for the 1.2B model, the total normalized loss for uniform sampling seems inconsistent: summing the individual values ($1.142 + 1.180 + 1.273 + 1.142 + 1.134$) results in $5.871$, which differs from the reported value of $5.946$. If the former is true, this could imply that the proposed sampling ratio doesn't generalize as well for larger models (1.2B), which is the main purpose of this proposed sampling ratio. Clarifying this would help substantiate whether the benefits of the proposed ratio are meaningful across different model sizes.

W3. While the paper provides some empirical evidence supporting Hypothesis 1, this evidence is limited to specific language families (Indic-Romance and Slavic-Sino-Tibetan, with Sino-Tibetan only represented by Chinese). This is a substantial claim and is referenced throughout the paper. However, there are numerous studies, particularly in monolingual settings, showing cross-lingual transfer outside of phylogenetically related language families [1]. For example, English (Germanic) or other high-resource languages are widely used for cross-lingual transfer in various multilingual setups [1, 2]. Language similarity can also be measured in multiple ways, such as typological or geographical distance, and not solely by language families [3]. It would be valuable to investigate whether the proposed hypothesis holds between languages like English and Romance languages, where there is known cross-lingual transfer despite family differences.

### References:

[1] Lin, Y.-H., Chen, C.-Y., Lee, J., Li, Z., Zhang, Y., Xia, M., Rijhwani, S., He, J., Zhang, Z., Ma, X., et al. (2019). Choosing transfer languages for cross-lingual learning. *arXiv preprint arXiv:1905.12688*. [Link to Paper](https://arxiv.org/abs/1905.12688)  

[2] Adelani, D. I., Liu, H., Shen, X., Vassilyev, N., Alabi, J. O., Mao, Y., Gao, H., Lee, A. E.-S. (2023). SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. *arXiv preprint arXiv:2309.07445*. [Link to Paper](https://arxiv.org/abs/2309.07445)  

[3] Littell, P., Mortensen, D. R., Lin, K., Kairis, K., Turner, C., Levin, L. (2017). URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*, 8–14. [Link to Paper](https://aclanthology.org/E17-2002/)

questions:
In addition to the points raised in the weaknesses, I do have additional questions:

- Could you provide more details by comparing your proposed scaling law with previous work in multilingual models? For example, can you fit previous works on scaling laws into your framework and compare their performance? This would give readers better clarity on the benefits and advantages of using the proposed approach.
- Have you tested your scaling law on out-of-distribution data as in data outside the CommonCrawl dataset? Since your test set is taken from the same dataset, I wonder if the scaling law would generalize well to datasets that are not part of CommonCrawl, especially as you included dataset size as part of the proposed scaling law. Exploring the model’s generalization ability on such data could provide additional insight.
- How well does the scaling law apply to low-resource language families like Niger-Congo? Also, could the current results on each language family have been skewed by the higher-resource languages such as English, Spanish, Russian, Hindi, and Chinese within their respective families? In other words, would the proposed strategy also benefit less-represented languages, such as Catalan and Kannada, which have fewer tokens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535756724

thread_type:
official_review

<thread_object>
creator: reviewer_j6wp

<content>
summary:
This paper introduces a method to calculate the multilingual scaling law and estimate the optimal sampling ratio for each language during multilingual pre-training. Experimental results on models with 85M and 1.2B parameters show that the optimal sampling ratios estimated are better than the three naive baselines.

strengths:
- The scaling law proposed is more fit for multilingual pre-training by considering cross-lingual transfer.
- There is an interesting finding that $\gamma_{i}(N, D)$ is independence from N and D.
- The optimal sampling ratios inferred can bring marginal improvement (< 0.1) than the three naive baselines.

weaknesses:
- **Missing important baselines**: The three baselines in this paper are naive. More baseline methods like the Equation (7) in Fernandes et al., 2023 are not incorporated.

- **More experiments needed**: The minimal cross-group transfer hypothesis (hypothesis 1) is hard to quantify and meet. It is still far from the real phenomenon. To support hypothesis 1, experiments are only conducted on (Romance, Indic) and (Sino-Tibetan, Slavic) language family pairs with varying three sampling ratios in {0.2, 0.4, 0.6}. How about the sampling ratio for other language family pairs varies? (For example, the results for more similar language families like Germanic and Romance or Romance and Slavic)

- **Overstatements**: 
  1. The first sentence in the abstract "We propose a novel scaling law for general-purpose decoder-only language models(LMs) trained on multilingual data, **addressing the problem** of balancing languages during multilingual pretraining." Actually, the problem is simplified into 5 language families and cases of low-resource languages are reduced in this paper (Line 190-191).

  2. The lines in Figure 4 are nearly parallel, and it is better to report the slope rate for each line.

- **Typos**:
  1. The x-y axis for Figure 4: $L(p)-p$ --> $log(L(p))-log(p)$
  2. Line 452: $q_j$ --> $q_i$
  3. Line 515: asses --> assess

questions:
a. (Hypothesis 1) How about the sampling ratio for other language family pairs varies? For example, the results for more similar language families like Germanic and Romance or Romance and Slavic.

b. As shown in Table 5, it can be found that the probabilities of Romance, Germanic, and Sino-Tibetan differ between models with 85M or 1.2B parameters. How to interpret this phenomenon?

c. (Line 531, 100 models) Does it refer to the number of models investigated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731126703195

thread_type:
official_review

<thread_object>
creator: reviewer_ojyu

<content>
summary:
This paper looks at the oft-neglected area of multilingual language model and in particular, scaling laws. They note that most prior work in scaling laws is focused on monolingual models or translation models (as opposed to broader generation models). The authors train more than 100 models varying in size from 85 million parameters to 1.2 billion (not counting embeddings) for 5 language families. Overall, they claim their sampling strategy is better at getting overall better multilingual loss.

strengths:
An overlooked area in the field. Multilingual Language models are very important.

Lots of experiments.

weaknesses:
A bit dense (especially some captions of figures and tables that took me a while to figure out.

For instance my thoughts about "Figure 1 Model size N figure is a bit confusing to me. What is N? 0,2,4,8 of what? Oh, 8 is 85Million parameters. It could be a bit clearer in the caption."

Figure 5 left and middle … a bit hard to know. Y-axis is loss (for a sampling ratio?) X-axis is sampling ratio? Colored lines are model sizes. Data size is just difference between left and middle graph? The caption should really be more explicit. Also, what am I supposed to take away from this?

questions:
Does better loss generalize to better downstream performance on other tasks? I would assume so I was just curious if you had tried anything?

Figure 1 appears to show that all models get better with more data, larger model size, and sampling ratio of 1. Why use a uniform sampling rate of 0.2 and not just 1.0? I assume that 1.0 implies 0.0 of others meaning 0.2 means equal sampling of all. If so, do you use all the data completely? And does this outperform other baselines? My guess is that this is shown somewhere in tables 3-6 but it is unclear to me.

What happens if you randomly group languages (not by family) and apply your scaling laws? Do the same conclusions hold? I'm trying to get at your claims that language family matter more - but I don't think you test languages apart from their language family ever.

I’d argue that MT is a type of generation and cache your claims in the intro a bit more.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732062120363

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We want to thank all the reviewers for their insightful feedback and the time invested in reviewing our paper. We are particularly grateful for the acknowledgement of several key aspects of our work:
- We study an important yet underrepresented problem in multilingual scaling law (Reviewer ojYU, Czqd).
- Our language-family-based setup simplifies the complexity of cross-lingual interactions during training, enhancing scalability across multiple languages (Reviewer ig2E, j6WP).
- Our proposed scaling law provides an effective way to optimize for the multilingual training data mixture, providing practical guidance for data allocation (Reviewer j6WP, ig2E, Czqd).

---

We hope that the following response can address the reviewers’ concerns:

**[More proof of hypothesis]** We have conducted additional experiments to further validate our hypothesis across more closely related language families as suggested by the reviewers. Using the same testing procedure from Section 3.2, we experiment with combinations of Romance, Germanic, and Slavic families. Since the reviewers note that high-resource languages like English (within Germanic) may transfer well to related languages in the Romance family (e.g., Spanish, French), we test settings where the Romance sampling ratio is fixed at either 0.2 or 0.5, while varying the proportions of Germanic and Slavic. The results, summarized in the table below, show that changes in the Germanic sampling ratio do not significantly impact the Romance loss either. This supports our hypothesis that the performance of each language family is primarily influenced by its own sampling ratio, with minimal dependence on inter-family interactions. We observe the same pattern when fixing Germanic ratios and vary Romance ratios.

|     Romance ratio    |     Germanic   ratio    |     Slavic ratio    |     Romance loss    |
|----------------------|-------------------------|---------------------|---------------------|
|     0.2              |     0.2                 |     0.6             |     2.782           |
|     0.2              |     0.4                 |     0.4             |     2.784           |
|     0.2              |     0.6                 |     0.2             |     2.787           |
|     0.5              |     0.125               |     0.375           |     2.606           |
|     0.5              |     0.25                |     0.25            |     2.607           |
|     0.5              |     0.375               |     0.125           |     2.601           |

The results highlight the distinctions in the analysis of transfers among **language families** compared with **individual languages**. It demonstrates that the language-family-based approach allows for sufficient intra-family transfer, reducing reliance on inter-family interactions. In fact, this is a key insight and contribution of our work. The results and explanations are updated in both Section 3.2 (Figure 2 (left), line 198-209) and Appendix C (Figure 6, line 695-700) in our revised manuscript.

---
**[Language grouping strategy]** We do not claim that grouping by language families is the only viable or optimal approach for all contexts, as noted in footnote 3 on page 4, where we suggest alternative grouping strategies as well. Instead, we define two key requirements for effective groupings: minimal cross-group transfer and data sufficiency. Among all the grouping methods, we find family-based grouping is a natural choice that aligns with our two requirements, and empirically we verify that this hypothesis holds well. The additional considerations brought up by the reviewers would be valuable for future explorations into grouping strategies that incorporate areal or typological factors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates scaling laws for multilingual language models (MLMs), proposing a method to predict optimal sampling ratios for languages within the training mixture by grouping languages into families. The authors argue that focusing on language families rather than individual languages simplifies cross-lingual transfer analysis and scalability. They claim minimal cross-family transfer and validate these hypotheses using test cross-entropy loss as a measure, conducting experiments across 23 languages from five families with models ranging from 85M to 1.2B parameters. Key contributions include a power-law relationship linking dataset size, model size, and sampling ratios, which generalizes small model findings to larger ones and provides guidelines for multilingual training.

Strengths include addressing an underexplored area in multilingual scaling laws, with rigorous experimentation on diverse language families. The proposed method offers a potentially cost-effective way to optimize data allocation, and the clarity of presentation was noted. However, there are also significant weaknesses: The absence of downstream task evaluations limits the paper's practical implications, and the claimed minimal cross-family transfer is not convincingly demonstrated across diverse settings. Improvements in normalized loss are marginal, questioning the significance of the proposed method’s benefits. Moreover, the reliance on language family grouping overlooks alternative groupings, and the scalability to low-resource languages remains unclear.

Despite revisions addressing some concerns, key limitations persist. The evidence supporting minimal cross-family transfer, a central hypothesis, remains insufficient, as confounding factors (e.g., relatedness within language families) are not adequately addressed. The marginal gains in performance, even when validated, do not substantiate the methodology's utility. Given these issues, the decision is to reject the paper.