ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality
Authors: Shayne Longpre
Sneha Kudugunta
Niklas Muennighoff
I-Hung Hsu
Isaac Rayburn Caswell
Alex Pentland
Sercan O Arik
Chen-Yu Lee
Sayna Ebrahimi
Venue: ICLR 2026 Poster (2026)
Keywords: scaling laws
multilinguality
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Scaling laws for multilingual pretraining, finetuning, and language transfer.

Abstract: Scaling laws research has focused overwhelmingly on English—yet the most prominent AI models explicitly serve billions of international users. In this work, we undertake the largest multilingual scaling laws study to date, totaling 774 multilingual training experiments, spanning 10M-8B model parameters, 400+ training languages and 48 evaluation languages. We introduce the Adaptive Transfer Scaling Law (ATLAS) for both monolingual and multilingual pretraining, which outperforms existing scaling laws' out-of-sample generalization often by more than $0.3$ $R^2$. Our analyses of the experiments shed light on multilingual learning dynamics, transfer properties between languages, and the curse of multilinguality. First, we derive a cross-lingual transfer matrix, empirically measuring mutual benefit scores between $38 \times 38=1444$ language pairs. Second, we derive a language-agnostic scaling law that reveals how to optimally scale model size and data when adding languages without sacrificing performance. Third, we identify the computational crossover points for when to pretrain from scratch versus finetune from multilingual checkpoints. We hope these findings provide the scientific foundation for democratizing scaling laws across languages, and enable practitioners to efficiently scale models—beyond English-first AI.

Published as a conference paper at ICLR 2026
ATLAS ADAPTIVE TRANSFER SCALING LAWS FOR MULTILINGUAL PRETRAINING, FINETUNING, AND DECODING THE CURSE OF MULTILINGUALITY
Shayne Longpre"! Sneha Kudugunta² Niklas Muennighoff I-Hung Hsu*
Isaac Caswell Alex Pentland' Sercan Ö. Arik' Chen-Yu Lee" Sayna Ebrahimi
MIT UUniversity of Washington
Sttanford University 'Google Cloud AI Google DeepMind
ABSTRACT
Scaling laws research has focused overwhelmingly on English- the most prominent AI models explicitly serve billions of international users. In this work, we undertake the largest multilingual scaling laws study to date, totaling 774 mul- tilingual training experiments, spanning 10M-8B model parameters, 400+ training languages and 48 evaluation languages. We introduce the ADAPTIVE TRANS- FER SCALING LAW (ATLAS) for both monolingual and multilingual pretraining, which outperforms existing scaling laws' out-of-sample generalization often by more than 0.3 R². Our analyses of the experiments shed light on multilingual learning dynamics, transfer properties between languages, and the curse of multi- linguality. First, we derive a cross-lingual transfer matrix, empirically measuring mutual benefit scores between 38 x 38 1444 language pairs. Second, we de- rive a language-agnostic scaling law that reveals how to optimally scale model size and data when adding languages without sacrificing performance. Third, we identify the computational crossover points for when to pretrain from scratch ver- sus finetune from multilingual checkpoints. We hope these findings provide the scientific foundation for democratizing scaling laws across languages, and enable practitioners to efficiently scale modell--byond English-first Al.
INTRODUCTION
Scaling laws research has focused overwhelmingly on English (Kaplan et al., 2020; Hoffmann al., 2022: Li et al., 2025a). However, most of the major closed and open models now explicitly target massively multilingual uses (OpenAI, 2025; Anthropic, 2025; Google DeepMind, 2025; DeepSeek- AI, 2024; Team OLMo et al., 2024). Nonetheless, multilingual scaling laws research in the public sphere is limited. Among proprietary lab reports, only Llama-3 briefly discuss their multilingual scaling laws, though they train on only 8% non-English tokens (Dubey et al., 2024). Prominent public work investigates scaling laws for data mixing (Goyal et al., 2024; Ye et al., 2024; Ge et al., 2024a), scaling laws for machine translation (Fernandes al., 2023; Gordon al., 2021), multilin- gual instruction tuning/adaptation (Shaham et al., 2024; Lai et al., 2024; Weber et al., 2024), and only a couple of recent rigorous contributions investigate scaling for smaller multilingual models (<45M and <1.2B respectively) (Chang et al., 2024; He et al., 2024).
Extending these prior works, we seek to fill the ample remaining knowledge gaps with compre- hensive examinations of the following research questions. First, we explore how the properties of different languages' scaling laws differ. Second, we measure the cross-lingual transfer benefits be- tween 38 languages. To our knowledge, this represents the most comprehensive available empirical resource to explicitly measure language transfer across 38 x 38 1444 language pairs. Third, we succeed in modeling the curse of multilinguality the phenomena where adding languages to the training mixture can degrade loss for each language, due to limited model capacity. Lastly, we measure when it is more efficient to pretrain from scratch, or finetune from a general-pun purpose mul- tilingual checkpoint, resolving an unaddressed practical question. In pursuing these research ques-
"oorreponddncc:: slongpre@ media.mit.cdu

---

Published as a conference paper at ICLR 2026
FR
RU
110
Mookl MockSSoc
ZH
H1
NW
160
Malel Stee
32M
1M
100
Data Size (D)
Data Sizee (D)
Dora Size (D)
Figure I: Optimal Scaling Trajectories for English, French, Russian, Chinese, Hindi, and Swahili. The law for [monolingual vocabulary. monolingual training]=(-), the law for [multilingual vocab- ulary, monolingual training]=(--)) and the law for [multilingual vocabulary, unimax training]=(""). We find (1) per-language optimal scaling trajectories are similar, (2) there is compute effi- ciency tax for training with multilingual vocabularies or training sets (especially for English), and (3) Hindi and Swahili observe data repetition their curves slope upward from dimin- ishing returns.
tions, we develop new tools and estimates for multilingual practitioners, as well as the ADAPTIVE TRANSFER SCALING LAW, which significantly outperforms prior work across several dimensions of generalization. In total, our pretraining and finetuning experiments span 774 independent experi- ments MADLAD-400 (Kudugunta et al., 2024), for models sized 10M 2B, and evaluating 48 languages. Our main contributions include:
1. The ADAPTIVE TRANSFER SCALING LAW that offers better multilingual generalization larger/unseen N, D, C, and training mixes M, than prior work (Table 1). In Figure 1 also estimate a compute efficiency tax between multilingual and monolingual training.
2. A 38 x 38 CROSS-LINGUAL TRANSFER MATRIX, providing the largest resource for
empirically measured transfer beneeits/interffeencc between languages (Figure 2).
3. A scaling law for the curse of multilinguality, that informs practitioners how much to scale (N, D) to accommodate expansions in models' language coverage (Figure 5).
4. A general pretrain vs finetune formula, that informs practitioners whether is more ef- ficient to pretrain from scratch or begin from a multilingual Unimax checkpoint (Figure 7).
2 EXPERIMENTAL SETUP
Datasets and Evaluation We the MADLAD-400 dataset (Kudugunta al., 2024), popular CommonCrawl-based dataset with the most expansive coverage of languages, totaling over 400. The MADLAD-400 authors prioritized multilingual-speciff curation for this pretraining corpus, in- cluding language detection, filtering, and preprocessing. For 50 languages, chosen to represent range of language families, scripts, and degrees of resourcefulness, we partition random test set, to evaluate vocabulary -nnernsitieeee loss (Tao et al., 2024) for fairer cross- gross-lingual comparisons. Fol- lowing (Muennighoff et al., 2024; He al., 2024), we evaluate the fit of our parametric scaling laws using R² calculated on held-out test sets partitioned to assess specific dimensions of generalization: (N(N) for the largest model sizes, (D) for the largest token ranges, RC) for the largest com- pute runs, and R² (M) for unique training language mixtures not
training (more details in Appendix B.3). We separate these dimensions in response to prior work that has demonstrated the importance of rigorous test-set splits in scaling laws research (Li al., 2025a).
Model Training We pretrain models from 10M to 8B parameters, using hyperparameter choices similar those trained in Kudugunta et al. (2024) (with updates prescribed by recent work). In this work, we train both monolingual and multilingual models, varying the model size N, training tokens D. multilingual data proportions, and the total training compute C. In particular, we focus
2

---

Published as a conference paper at ICLR 2026
training monolingual models, bilingual models, and massively multilingual Unimax models (Chung et al., 2023). most experiments center around scale, token, and mixture variations on these lan- guages: English, French. Chinese, Hindi, Swahili, Russian, Spanish, Portuguese, Japanese, though certain experiments evaluate across 50 languages. We use a 64k Sentence Piece Model vocabulary (Kudo, 2018). Across experiments we pretrain 280 monolingual models, 240 bilingual models, 120 multilingual mixtures, and we finetune 130 monolingual models from Unimax checkpoints, total- ing over 750 independent training runs. Full experimental details, including the specific language choices, hyperparameters, and training mixtures are provided in full in Appendix B.
Table 1: The R² evaluation metrics for the fitted scaling laws, holding out separate dimensions of generalization: the largest model sizes N. most training tokens D. most compute C, and unique multilingual training mixtures M. In both monolingual and multilingual settings, average R² across languages=(EN, FR, RU, ZH, HI, SW], including (ES, DEJ for the multilingual setting. We find ADAPTIVE TRANSFER SCALING LAW outperforms prior work in Monolingual and Mul- tilingual settings. We ablate the use of the terms for Dother and transfer languages Ck, D.).
SCALING LAWS
R²
R2(N) R²(D) (C(C) (M
CHINCHILLA SCALING LAW (Hoffmann et al., 2022)
0.94
0.68
0.94
0.90
MONO
DATA-CONSTRAINED SCALING LAw (Muennighoff et al., 2024)
0.93
0.78
0,93
0.88
[Ours] ATLAS (D, ONLY)
0.92
0.88
0.91
0.88
CHINCHILLA SCALING LAw (Hoffmann et al., 2022)
0.64
-0.99
0.72
0.66
0.61
MULTI.
MULTILINGUAL SCALING LAW (He et al., 2024)
0.67
-0.65
0.73
0.67
0.70
[Ours] ATLAS (D, ONLY)
0.70
-0.75
0.80
0.72
0.64
[Ours] ATLAS (D, + Dother)
0.98
0.89
0.97
0.97
0.66
[Ours] ATLAS (D, + Dother Σk, D:)
0.98
0.89
0.96
0.98
0.82
3 ADAPTIVE SCALING LAWS FOR MONOLINGUAL. & MULTILINGUAL SETTINGS
Challenges with existing scaling laws for multilingual modeling. In this section, we discuss our attempts to precisely model different monolingual and multilingual pretraining experiments. Scal- ing laws for English are typically based on Chinchilla (Hoffmann et al., (922)-whichl we refer to as the CHINCHILLA SCALING LAW (CSL). Hoffmann et al. (2022)'s 's two power-law terms (Equa- tion (1)), allows us to separate how loss scales model size N and data size D. is the irreducible loss (representing the entropy of natural text), (A, B) are learned coefficients and and (a, B) are scaling exponents for for model size and data size respectively. Beyond English, languages may have different scaling laws (Arnett & Bergen, 2025), and language resources often limited, quiring a DATA DATA-CORe -CONSTRAINDD SCALING LAW (DCSL) (Muennighoff et al., 2024) to account for diminishing returns after multiple epochs. However, DCSL requires ample data both before and after one epoch to accommodate its two-stage fitting process. For high- languages (English, French, Chinese). it costly to collect ample data beyond one epoch, and for how/mid-resoure lan- guages (even Hindi, Hebrew, and Swahili) it can be very difficult to collect sufficient observations below one epoch.' 1 As such, even for monolingual modeling, we require a scaling law that is data repetition-aware and robust to different collection allowances for (N, D).
For modeling multilingual mixtures, monolingual scaling laws can be used (either with D repre- senting the total or target language data), or He et al. (2024) introduce MULTILINGUAL SCALING LAW (MSL), which expresses the loss for target language using N, and the sampling ratio for the target languages' language family in the training mixture. However, each of these solutions only accounts for one of: the sampled target language data D., the total data altogether D, + Dothers or set of the target and likely positive transfer languages D; + Σ, D,. We posit that a better model would separate and learn to weight these independent contributions. In summary, none of the existing multilingual solutions account for (a) multi-epoch data repetition, or (b) the cross-lingual transfer effects beyond the target's 's language family.
In MADIAD-400 these languages have < 0.3% English tokens. At standard batch sizes, even frequently evaluating (every 1k) isn't 't enough collect many (sometimes any) observations hefore epach is reached.
3

---

Published as a conference paper at ICLR 2026
A Na
C(N,Dett -E E +
(1)
D
Dest = Sx, +
Σ (i) +
Tother Sx Spather (Dother Uother
(2)
EE
Monolingual
Other Languages
Transfer Langungos
D. Sx(D:U)= -
D (< eppohh
1 U [1+ LexpX- T
(3)
D>U (> epoch
The ADAPTIVE TRANSFER SCALING LAW. To resolve these challenges, we introduce the ADAP- TIVE TRANSFER SCALING LAW (ATLAS), a simpler variant of DCSL that is repetition- aware, introduces fewer additional parameters (for the monolingual variant), is fit in one stage, and is adapt- able to an open-enden number of languages that would benefit from an explicit Equation (1) the core scaling law formula includes the standard parameters governing the irreducible loss and N, D scaling: E, A, B, ax B. Its effective data exposure term Detr (Equation (2)) unpacks data sources into three terms: Monolingual term for the target language data D,, an optional Transfer Language term for up to [K,] languages we wish to learn independent transfer coefficients for Th, and Other Languages remainder term that sums all training tokens not already accounted for in the first two terms D = Diot D; -x. D,. The latter two terms are optionally added for multilingual modeling, and IC₃ can be adapted any combination of the languages with pre- sumed highest positive transfer to the target language, or languages with the greatest representation in the experiment training mixture. We found the latter was most effective and selected the |K,] most highly co-sampled languages along target language across mixtures. For each term, we apply a saturation function (Equation (3)), ensuring a smooth decay on the effective data for each subse- quent epoch where the training tokens have surpassed that languages' unique tokens U. The new parameters introduced over standard scaling laws are: the repetition parameter X, which is shared across each data source, the transfer weights TÜ for each language in Ic, (which are initialized from language transfer scores derived in Section 4), and the transfer weight Tother for the remainder of language tokens.
loos-llngual term. In
Research Question: How do scaling laws differ by language, and by monolingual vs multilingual training mixtures?
Monolingual scaling behavior consistent in form, and multilingual variants exhibit a variable-sized compute-efficiency tax. Figure compares optimal scaling trajectories across six- variable resourced languages, under three regimes: monolingual vocabulary + monolingual train- ing, multilingual vocabulary monolingual training, and multilingual vocabulary + unimax train- ing. Across languages, the monolingual curves are nearly parallel, indicating similar exponents and comparable returns additional data and parameters (see full law parameters in Table C.I). Both multilingual vocabulary and Unimax training shift the frontier upwards, evidencing a compute- efficiency tax relative the nounced for English, indicating it benefits less from language transfer than other languages do from English. For Hindi and Swahili, the right tail bends upward, consistent with diminishing returns from severe data repetition after many epochs. These qualitative trends motivate single functional form that remains stable across languages while accounting for eerabularytraining-regin effects.
setting. This is most pro-
Research Question: How well can our scaling laws capture unique monolingual constraints, and complex multilingual cross-lingual transfer dynamics?
ATLAS outperforms prior scaling laws, with more robust fit across held-out axes in mono- lingual and multilingual settings. Table evaluates fitted laws by holding out the largest model sizes N. token ranges D, compute C' 6NN D. and held-out language mixtures M, reporting R² averaged over available languages. In the monolingual setting, all scaling laws perform comparably across most dimensions of generalization. However, when generalizing to the largest size of model, laws that model data repetition outperform those that don't 't. ATLAS provides the strongest gener- alization greater model sizes: R (N) 0.88 versus 0.68 (CSL) and 0.78 (DCSL), respectively.
In the multilingual setting, monolingual scaling laws can only observe their target data tokens, and as such perform adequately, but not well. In particular, they unable to generalize to the largest
4

---

Published as a conference paper at ICLR 2026
models (N) all. This is also the case for MSL, although it obtains better generalization to unseen language mixture in R (M) ) = 0.69 > 0.64 by virtue of modeling the whole language fam- ily. By separating the sources of data, ATLAS is able to significantly outperform the other laws across all dimensions, frequently achieving >> 0.9 fit. However, our ablation of the data terms shows that only by adding the Transfer Language term is the model able outperform MSL and achieve a better multilingual mixture generalization (t² = 0.82. In summary, ATLAS offers a sim- ple framework adaptively model cross-lingual transfer, and enables reliable extrapolation across out-of-sampll dimensions in both monolingual and multilingual settings addressing critical gap where existing approaches fall short.
HOW Do LANGUAGES BENEFIT OR INTERFERE WITH EACH OTHER?
Research Question: Which languages synergize or interfere most with one another's performance?
Baye
Benan
41
21
42
42
34
46
41
40
41
44
44
65
37
Source Language
Figure 2: The CROSS-LINGUAL TRANSFER MATRIX, depicting the measured Language Transfer Score across 30 30 language pairs. Positive scores indicate more positive transfer, negative scores more interference, during bilingual co co-training. The dashed boxes indicate the top-5 source lan- guages for each target language. Refer to Appendix B.6 for full details, and Figure C.2 for the larger 38 x 38 matrix. While English is the best source language for many of the languages, we find language similarity highly predictive of these scores.
When allocating multilingual training mixtures, practitioners need to know which source lan- guage(s) provide the most positive benefit, in co-training, to the targer language(s) they are opti- mizing for. Throughout this paper, the terms transfer interference to refer the positive
5

---

Published as a conference paper at ICLR 2026
Dill Seot thet Famby hame Sobe bancheen hhem Sengi
bamethrll Solp
0.0
Languge Trmmfer Scon
-0.
Taasfe Scoe
-3.0
Sacie Fasily.
DET Fanily Sempe Linguisic Similantt
Sunoo Fasilly Dut Fanilyy
-15
-18 103
8.5
Treafer Score (Lagsgge
Figure 3: Left: A language transfer scatter plot, comparing score symmetry: is language A as help- ful to B vice versa? Points cluster near the diagonal, indicating strong symmetry. The most syn- ergistic and symmetrie pairs almost exclusively share both language family and script. Greater linguistic distance correlates with increased asymmetry and reduced positive transfer. Right: The impact of linguistic similarity (via language family or script) on transfer scores. The box spans the inter-quartile range, with the median line the center. We find the differences between each group are statistically significant (p < .001), suggesting that sharing either language family or a script independently contribute greater positive cross-lingual transfer.
(or negative) inductive transfer between the training signal two or more tasks (Caruana, 1997). We construct, our knowledge, the most comprehensive matrix of language-to-language transfer scores, to assist practitioners in selecting training languages.
Language Transfer Score (s t) BTS Tbi (Ia 2d
dmono
To do this, measure how much training in source language affects the test loss target language f. We define a Bilingual Transfer Score (BTS) as the relative training efficiency of a bilingual model (s, t) (50%, 50%) compared the monolingual model reaching the same loss level.2 dmono denotes a pre-defined target step (42B tokens), (ddd) computes the monolingual models' loss step d, and ah finds the number of tokens for the bilingual model to reach that loss. Because BTS > 0 is the zero-centered, when is 0 there is no transfer, > there is positive transfer, and double the steps dmono the monolingual model took to reach the target loss (dmonn.) Refer to Appendix B.5 for full experimental details and methodology.
there is negative interference, leading the bilingual model taking more than
In Figure 2, we plot the normalized BTS scores between 30 30 language pairs (or the full 38 x 38 Figure C.2), spanning language families and scripts, on 2B parameter models. Where prior work has developed comprehensive resources for measuring syntactic/phonologica distances between languages (Khan et al., 2025), or measured transfer scores with smaller models specifically between high and low-resource languages (Protasov et al., 2024), our work offers among the most expansive and rigorous, sylly-symmetric transfer matrices, to our knowledge. contains many compelling observations. For instance, as might expect, notable positive transfer exists between related lan- guages, such as between Spanish, Catalan, and Portuguese, or Swedish and Norwegian, or Indone- sian and Malay. The top-5 most helpful source languages to co-train with per target highlighted, with English appearing the most (19 of 30 instances), followed by French (16 of 30), Spanish (13 of 30), and even Hebrew (11 of 30). In other cases, low-resource languages such Urdu Pashto have significant negative transfer with all other languages considered.
We measure BTS directly for 80 language pairs, then estimate it using other training signals (with high fidelity R² 0.85), detailed in Appendix B.

---

Published as a conference paper at ICLR 2026
Wobihe Sore
Belaive
6.1
angungee (
40
0.
0.0
28
Nom Langont
25B
50B
Madel Size (94)
5B
1000
125
Figure 4: We empirically measure the relative degradation in loss (y-axis), as compared mono- lingual model of the same (N, D). from adding pretraining languages (z-axis). Left: We fix D = B tokens, and vary the model size N. Right: We fix N 2B parameters, and vary the tokens D. The points are real empirical observations, averaged across languages. The mesh-grid a surface estimate, using a cubic spline. Target language loss most affected by the number of training languages, but this loss penalty declines for larger models with more capacity.
Language script, followed by language family are strongly correlated with positive transfer scores. In Figure 3, we correlate the transfer scores from the full language matrix Figure 2. Our analysis confirms a clear and statistically sigmificant link between transfer performance and the sim- ilarity of the source and target language. Specifically, we found sharing a script family introduced statistically significant shifts in the mean of the transfer score (always < .001). These findings corroborate (He al., 2024)'s scaling laws that grouped data by language family. This effeet is strongest for shared scripts. As shown in Figure 3, language pairs that share the same writing sys- tem (e.g. Latin) exhibit dramatically improved transfer scores compared to pairs with disparate scripts (e.g., Latin and Cyrillic), with a mean score of -0.23 versus 0033.. The larger effect size for script suggests that the ability to share surface-level representations and subword vocabularies primary mechanism for positive transfer, more than deeper grammatical or lexical similarities alone. Section 5 and Appendix C.2 we explore how N, D affect language transfer.
Language transfer scores are often symmetric within the same language family and script, but surprisingly, they cannot be assumed to be reciprocal otherwise. Next, we examine the extent to which language transfer is symmetric, that is, whether the benefit from language A to B is correlated to that from to A. As illustrated in Figure 3 (left), we find a surprising triangle pattern, with a -0.11. This implies two things. First, and perhaps most importantly,
Pearson correlation of across all language pairs, there is no significant correlation, and because we observe language A is helpful language B, we cannot assume the same in reverse. This corroborates findings in Li et al. (2025b) that altruistic languages don't 't always yield mutual benefits. Second, there does appear to be clear structure the scatter: language pairs that share family and script (blue) cluster mostly in the top right quadrant and more tightly around the identity line, whereas language pairs with differing script and family (red) are simultaneously less symmetric, and less synergistic. For instance, pairs (French, Spanish) and (Russian, Ukrainian) share highly symmetric transfer scores, whereas (Chinese, Farsi) and (Russian, Vietnamese) are highly asymmetric. We hope these findings, as well as the Figure 2 transfer matrix, will be directly applicable to practitioners selecting language mixtures based on empirical measurements, rather than intuition.
5 THE CURSE OF MULTILINGUALTY-MON MODEL CAPACITY CONSTRAINTS
Research Question: Is the "curse of multilinguality" measurable- phenomenon where adding languages the training mixture can degrade loss of each language, due to limited model capacity?
7

---

Published as a conference paper at ICLR 2026
A model's capacity can hinder its ability to learn multiple languages at once known the curse of multilinguality (Conneau et al., 2019 Chang et al., 2024). We can empirically measure this relationship, between N. D, and the number of training languages K, by running experiments with variable K. Full experimental details and scaling law derivations are provided in Appendix B.7.
Modeling the Relationship between K, N, and D. In Figure 4 examine the relative loss increase, over a monolingual baseline, from varying the number of training languages K, and either the model size N (left plot) total training tokens Diot (right plot). For simplicity, we don't 't explicitly model token repetition, and we sample tokens from each language uniformly-we believe this is a reasonable assumption for models designed serve all languages. We observe three phenomena. First, the number of training languages K has far more impact on the relative loss than N or D. Relative loss appears to grow monotonically as K increases. Second, both increasing model size N and total tokens Deot mitigate this penalty. However, computing the partial along the fitted surface shows |8S/0 log N > lag/alog D|; indicating N delivers larger marginal gains than Drot- Consequently, maintaining performance as K grows requires scaling both data and model size, but scaling the model size has a greater effect. To understand the relationship among these variables more precisely, we model per- per-target -angut-languge loss
K
K D
L(K,N, Lo Na B
where, under even sampling the total tokens across languages are Drot = K Dr. We tested several scaling law variations for this research question, but settled on this as (a) it retains Chinchilla- style power-law decay properties that cleanly separate model capacity from data (it also reduces to Chinchilla when K=1). (b) it makes the role of the number of languages explicit and interpretable, and (c) it achieved a robust R² 2 0.87, indicating a strong fit. The exponent captures how capacity requirements grow with the number of languages, while captures how data requirements change with multilinguality. tb < 0 indicates positive transfer (sublinear data needs per language as K increases), and > implies negative transfer/inerferennee.
We fit the scaling law for each of 8 differ- ent languages, as well as the combination of all, achieving similar coefficients, and > 0.8 R² for each. Using the scaling law fit on all language data find & 0.11 and curse
coling
K--0
Moddel multipie (N)
conpsiespiinl
-0.04, indicating a mild capacity- of multilinguality, tempered by positive transfer across languages. In other words, as languages added the training mixture, positive means the loss will decline from limited model capacity, however, a negative v means that less data per language is needed. (Though the total amount of data increases, as we sample from new languages too.) This provides clear, mea- surable evidence of positive cross- mooss-lnggull trans- fer.
O D Diya
V'aNon C'appiii-wwa
Taken milliplicr (D (D))
Figure 5: Iso-loss from expanding language coverage: When scaling the number of lan- guages a model serves from K + r-K, we can es- timate how much they need increase model size N'/N and/or training tokens D'oe// Drot= without degrading any of their languages' loss. We de- rive the compute optimal scaling equations. The iso-losses indicate that a practitioner should ex- pand their compute budget by C' . 0.99 to ex- pand their language coverage by r.
Estimating the Iso-Loss Frontier, and Compute-Optinal scaling of N, D. when adding languages: examine the practical case where a practitioner wants to retrain a new model, increasing the number of languages it serves from K to rK, without hindering the performance the original model achieved on the existing languages. To accommodate these new languages, they will need scale C' with some combination of N + N' and D + Diot- We can compute this lso-loss by equating the loss terms: L(K, Dtne) = AK* this can derive the closed-form expression for how scale (N, Drot) when increasing K to rK
+ rK. Next, we
Na + BK A(Kr)" " No + D
B(Kr) From
D2
8

---

Published as a conference paper at ICLR 2026
(as derived in Appendix B.7):
N (rK) (K)
DT(KK)
Dive(rK)
/
,140/8
(K)
Diot (K)
We can also derive the minimal increase in the compute budget:
C'
No
( N' (
(x))
p1t//at//8
C NDot
Dear
In Figure 5 we plot the iso-loss frontiers, and compute-optimal allocations, when scaling model from K to K languages. It shows in closed-form how a practitioner should scale Deot. N, and by extension their compute budget C' to accommodate additional languages, without degrading their loss. For instance, we find expanding to 4 K languages requires a practitioner to expand Dzor by 2.74, and the model size N by 1.4. Incidentally, evenly sampling across languages, this actually corresponds to using sampling
(2.74/4) 32% less data per language. Although there are fewer tokens in each target language (by 32%) the total tokens has risen by 2.74, and the positive transfer prevents any potential loss degradation.
8.10
La35
145B
-.00
0.5
46.35
0.3
RU
VI
E3
Z11
1.85
815
1.00
1898
2348
2308
2538
-1.0
-.00
2.55
3.00
-0.00
200
200
302
400
100
200
Tokema
Tolern
Tolere
Tolons
Figure 6: For eight languages, we plot the loss curves for 2B parameter models pretraining mono- lingually from scratch (r"), finetuning monolingually from the Unimax base model (- (-, and the difference between these losses (-). We annotate the number of tokens which the pretrained loss becomes better than the finetuning loss. The difference between the interpolated pretraining curve and finetuning curve. When the loss difference reaches zero, pretraining from scratch has surpassed finetuning using equal compute. Depending on the token (or compute) budget, it is usually more effective to finetune a multilingual checkpoint there are < 144B tokens, and pretrain from scratch if the budget accommodates > 283 tokens.
6 PRETRAIN OR FINETUNE?
Research Question: When training model for target language t. is it more effective to pretrain from scratch, or finetune a general-purpose massively multilingual checkpoint?
Prior scaling law work focuses on compute efficiency with respect pretraining from scratch. How- ever, in reality, practitioners who aim to optimize for a target language may have the option to start from multilingual public checkpoints or pretrain point for many downstream models. Consider a practitioner with a compute budget C. In Figure 6, plot the interpolated loss curves for the model pretrained from scratch, the model finetuned from a checkpoint, and the difference between these losses, for several languages. We find that while the warm- stamtet finetuned model performs better first, pretraining from scratch eventually surpasses its performance after 144B 283 B tokens, depending on the language. Note that we leave the rea- son for these convergence differences to future work, without clear hypothesis. Though English pretraining converges the fastest, it is also allocated 5% sampling rate within Unimax, whereas each of the other languages are allocated 1.4%
multilingual checkpoint to as the starting

---

Published as a conference paper at ICLR 2026
From the number of training tokens D we can also infer the inflection point which decides whether it is more computationally efficient (C = DND) to pretrain from scratch or fine- tune from the multilingual checkpoint. Using our range of model size experiments, we can es- timate best fit power law in Figure relating N to C. We find log(C) = 1113708 x N N1.05 to hold consistently across languages. For sim- plicity, we choose not to account for data repe- tition this model. Practitioners may use this to estimate whether pretraining will yield greater performance for their com- pute budget. Note one limitation to consider: not all multilingual base models are trained with the same mixture or for sufficiently long- and these factors would impact the pretrain vs finetuning intersection points. We choose widely used Unimax mixture trained for 1B to- kens (Chung et al., 2023). We believe this is a useful heuristic, with reasonable assumptions, to determine the optimal choice between pretraining and finetuning for a given compute budget.
10
English
10"
Rassian
Spunichh
10
C- 10285121 NTM
French
Portugues
Vicimmnes
10"
Chinoss
Aepuncse
Hindi
10
finetuning
Model Size (N) [x30"9]
Figure 7: In terms of N, we model the compute budget C required for model pretrained from scratch to outperform the model finetuned from the generic Unimax multilingual checkpoint. We estimate this relationship as C 10283128 x N1.65
CONCLUSION
In this work, we answer several scaling questions on training multilingual models, validating our findings on scales up to 8B model parameters. First, we introduce a novel scaling law formulation called Adaptive Transfer Scaling Law (ATLAS) that significantly improves scaling law by more than 0.3 R2 for multilingual mixtures. To aid our analysis, we also derive transfer scores for 38 lan- gauages. We find that English is often a strong positive transfer language, alongside languages with shared script and language families, though note that this transfer is often asymmetric. However, there is still a compute-efficiene tax when we train multilingual models. We quantify this "curse of multilinguality" and provide mathematical trade-off between model capacity and language cover- age: we find that the performance penalty from adding languages is mitigated more effectively by scaling model size (N) rather than training tokens (D). Finally, we quantitatively answer a critical practical question for practitioners: determining the compute crossover point between pretraining and finetuning. We find that depending on the language, for budget under 144-283B tokens, finetuning a general- purpose multilingual checkpoint is more efficient. Collectively, these contri- butions provide valuable guidelines for practitioners determining data mixtures, language coverage and computational requirements for creating multilingual foundation models.
8 ACKNOWLEDGEMENTS
We thank Luke Zettlemoyer, Catherine Arnett and Stella Biderman for helpful discussions on the paper. We thank Biao Zhang and Xavier Garcia for the technical discussions and feedback on early directions.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for gen- erative mixed-modal language models. In International Conference Machine Learning, pp. 265-279. PMLR, 2023.
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-22312, 2022.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey data selection for language models. arXiv preprint arXiv:2402.16827 2024.
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference on machine learning. pp. 173-182. PMLR, 2016.
Anthropic. System card: Claude opus 4 & claude sonnet 4.
Technical www.aaaic.hhccccomm cwgpaa/to..........oooo........
port, Anthropic, May 2025. URL https:/ /wwp///www//cc///w////////.//wss
com/
Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. Under- standing scaling laws for recommendation models. arXiv preprint arXiv:2208.0844 2022.
Catherine Arnett and Benjamin Bergen. Why do language models perform worse for morphologi- cally complex languages? In Proceedings of the 31st International Conference on Computational Lingnistics, pp. 6607-6623, 2025.
Michele Banko and Eric Brill. Scaling very very large corpora for natural language disambigua- tion. In Proceedings of the 39th annual meeting of the Association for Computational Linguistics, pp. 26-33, 2001.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In International Conference on Machine Learning, pp. 1482. PMLR, 2022.
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Meng- meng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexan- der Gutkin, Apurva Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. Building Machine Translation Systems for the Next Thousand Languages. arXiv e-prints, art. arXiv:2205.03990 033933, May 2022.
David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, and Sham Kakade. Loss-to-loss prediction: Scaling laws for all datasets. arXiv preprint arXiv:2411.12925, 2024.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Tyler Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. When is multilinguality curse? lan- guage modeling for 250 high-and low-resource languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4074-4096, 2024.
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang. Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in 1lms. arXiv preprint arxiv:2410.08527, 2024.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2819-2829, 2023.
contrastive

---

Published as a conference paper at ICLR 2026
Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining, arXiv preprint (rXxi:2334.091555 2023.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057- 4086. PMLR, 2022.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettlemoyer and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. arXiv preprint arXiv1911.022166, 2019.
DeepSeek-AL. Deepseek-v3 technical report, 2024. URL Utt.ttttee://aaxvv..oo/////////a 19437.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint urXiv:2407.21885 2024.
Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation. arXiv preprint 1iXiv:2302.096500 2023.
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely foundation models. arXiv preprint arXiv:2309.00 08520, 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. Bimix: A bivariate data mixing law for language model pretraining. arXiv preprint arXiv:2405. /40.0 2024a.
Ce Ge, Zhijian Ma. Daoyuan Chen, Yaliang Li, and Bolin Ding. Data mixing made efficient: A bivariate scaling law for language model pretraining. arXiv preprint arXiv:2405. /4908, 2024b.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740, 2021.
Google DeepMind. Gemini 2.5 deep think model card. Technical report, Google Deep- Mind, August 2025. URL https: ptttp::/sstorgee wwwglooooolloooooooo eapis com/ deepmind-media Model-Carde Geminim 2 S-eepp---- Maink--Mooell Card, pdf. Model card pub- lished/updated August 1, 2025.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5915-5922, 2021.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, San- jana Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores- 101 eval- uation benchmark for low-resource and multilingual machine translation. 2021.
Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and Zico Kolter. Scaling laws for data filtering data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22702-2222 22211, 2024.
Albert Gu and Tri Dao. Mamba: Ainear-tmm sequence modeling with selective state spaces. arXiv preprint arXiv:2312.007.52 2023.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv preprint arXiv:2405. 18392, 2024.
12

---

Published as a conference paper at ICLR 2026
Tatsunori Hashimoto. Model performance scaling with multiple data sources. In International Conference on Machine Learning, pp. 4107-4116. PMLR, 2021.
Yifei He, Alon Benhaim, Barun Patra, Praneetha Vaddamanu, Sanchit Ahuja, Parul Chopra, Vishrav Chaudhary, Han Zhao, and Xia Song. Scaling laws for multilingual language models. arXiv preprint wtt.:x:24100128850 2024.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 17 2017.
Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301.33442 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint wxXiv:2203.15550 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404. 06395, 2024.
Andy L Jones. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03113, 2021.
Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood. and Ruoxi Jia. Autoscale: Scale-aware data mixing for pre-training llms. arXiv preprint arXiv:2407.20177, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. 2020.
Aditya Armaan Khan, Mason Stephen Shipton, David Anugraha, Kaiyao Duan, Phuong H Hoang, Eric Khiu, A Seza Doğruöz, and Annie Lee. Uriel+: Enhancing linguistic inclusion and usabil- ity in typological and multilingual knowledge base. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 6937-6952, 2025.
Kudo. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing, arXiv preprint arXiv:/808.06220 2018.
Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. Madlad-400: A multilingual and document-level large audited dataset. Advances in Neural Information Processing Systems, 36, 2024.
Wen Lai, Mohsen Mesgar, and Alexander Fraser. Llms beyond english: Scaling the multilingual capability Iims with cross-lingual feedback. In Findings of the Association for Computational Linguistics ACL 2024, pp.
2024.
Margaret Li, Sneha Kudugunta, and Luke Zeutlemoyer. (mis) fitting: survey of scaling laws. arXiv preprint arXtv:2502.18900 2025a.
Zihao Li, Shaoxiong Ji, Hengyu Luo, and Jörg Tiedemann. Rethinking multilingual continual pretraining: Data mixing for adapting Ilms across languages and resources. arXiv preprint arXiv:2504.04152. 2025b.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01499) 2024.
13

---

Published as a conference paper at ICLR 2026
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's 's guide to training data: Measuring the effects of data age, domain coverage. quality, & toxicity. arXiv preprint arXiv:2305. 13169, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019. URL https: http::///ppeeeeeeeeeeeeeeee www.eeruueffii...o.ooom.om. Bkg6RicqY7.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Alcksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
OpenAl. Gpt-5 system card. Technical report, OpenAI, August 2025. URL https://
openai, wwm.com/////// gpt- 5- system
--pt-- waaeee--cadd..r card. pdf.
Vitaly Protasov, Elisei Stakovskii, Ekaterina Voloshina, Tatiana Shavrina, and Alexander Panchenko. Super donors and super recipients: Studying cross-lingual transfer between high- resource and low-resource languages. In Proceedings of the Seventh Workshop on Technologies for Machine Trunslation of Low-Resource Languages (LoResMT 2024), pp. 94-108, 2024.
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, al. D-ept law: Domain-specific continual pre-training scaling law for large language models. Advances in Neural Information Processing Systems, 37: 90318-90354, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05550, 2024.
Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, al. What language model train if you have one million gpu hours? arXiv preprint 10.::2220..444 2022.
Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. Multi- lingual instruction tuning with just pinch of multilinguality. In Findings of the Association for Computational Linguistics ACL 2024, pp. 2304-2317, 2024.
Luisa Shimabucoro, Ahmet Ustun, Marzich Fadace, and Sebastian Ruder. A post-trainer's guide to multilingual training data: Uncovering cross-lingual transfer dynamics. arXiv preprint arXiv:2504. 16677, 2025.
Mustafa Shukor, Louis Bethune, Dan Busbridge, David Grangier. Enrico Fini, Alaacldin El-Nouby, and Pierre Ablin. Scaling laws for optimal data mixtures. arXiv preprint urXiv:2507.09404 2025.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 35:19523-19536 2022.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint 2Xiv:2407.13625 2024.
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung. William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207. 10551, 2022.
Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
14

---

Published as a conference paper at ICLR 2026
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang. Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi. Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Py- atkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zet,lemyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. olmo 2 furious, 2024. URL https: /ttp///aaxai /arxivv....//////////a//////aav/ org/ abs/ /201. 00656.
Alexander Arno Weber, Klaudia Thellmann, Jan Ebert, Nicolas Flores-Herr, Jens Lehmann, Michael Fromm, and Mehdi Ali. Investigating multilingual instuuction-tnnin: Do polyglot models de- mand for multilingual instructions? In EMNLP, 2024.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024.
Jiasheng Ye. Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403. 16952, 2024.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 2104-12113, 2022.
15

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
A Extended Related Work
17
B Experimental Details
18
B.I Language Choice
18
B.2 Experiment Design
19
B.3 Experimental Details: Scaling Laws Evaluation
19
B.4 Experimental Details: Language Finetuming.
Experimental Details: Language Transfer ................................
20
B.6 Estimating Bilingual Transfer Score (BTS)
................................
24
B.7
Experimental Details: Multilingual Capacity
24
C Extended Results
27
C.I Extended Results: Scaling Laws
27
C.2 Extended Results: Language Transfer
28
16

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
Scaling Laws Scaling laws are used to study the behavior of deep learning systems on scaling training data and/or compute. Various researchers have observed scaling properties of generalization error with training data size and model capacity (Banko & Brill, 2001: Hestness et al., 2017; Amodei al., 2016), 6). Specifically, in the context of L.LMs, Kaplan al. (2020) explicitly propose power law for the relationship between the loss L, number of parameters in the language model N, and the number of dataset tokens D. Later, Hoffmann al. (2022) proposed a different formula:
A
B D8
L(N,)) E+ No
(4)
We build upon this form throughout the paper to fit our scaling laws. Other researchers, too, have built upon these scaling laws study various aspects of scaling. such as neural network architec- tures (Clark et al., 2022; Tay et al., 2022: Frantar et al., 2023; Gu & Dao, 2023; Scao et al., 2022) or transfer learning (Henighan et al., 2020). Researchers have also investigated scaling properties in different domains of deep learning such neural machine translation (Ghorbani et al., 2021;
Gordon et al., 2021), vision language models like CLIP (Cherti et al., 2023; Henighan et al., 2020), vision (Alabdulmohsin et al., 2022; Zhai et al., 2022), reinforcement learning (Hilton et al., 2023; Jones, 2021; Gao et al., 2023), and recommendation systems (Ardalani et al., 2022).
Scaling Laws for Data Mixtures Many prior works (Hoffmann et al., 2022) assume a fixed dataset distribution to study the effect of scaling on model performance. However, empirically, dataset composition can have a significant impact on quality (Longpre et al., 2023; Albalak et al., 2024; Sorscher et al., 2022). Hashimoto (2021) studies the relationship between dataset composi- tion and model loss, finding a simple scaling law quantifying this relationship. Bansal et al. (2022) study the impact of data quality and noise on architecture, finding that synthetic data has a different exponent. Fernandes et al. (2023) study scaling for multilingual neural translation (Bapna et al., 2022), but experiments are restricted to 2-3 language pairs. Aghajanyan et al. (2023). in a similar vein, propose bimodal scaling laws for multimodal foundation models (Reid et al., 2024). Other researchers have considered different aspects of dataset composition that can affect model scaling: Muennighoff et al. (2024) study the effect of data repetition on scaling, while Goyal et al. (2024) consider how the effect of mixing data pools of varying quality changes with scale.
Foundation models involve training models certain validation loss on various validation sets and downstream datasets of interest. Based on the observation that optimal data mixture changes with scale, several recent works have attempted to characterize the change in the scaling law equation when either the train or test distribution changes.
mixture of datasets with the aim of achieving
Some researchers predict the optimal data mixture by first ablating mixtures using smaller models, and second, training a larger scaled-up model using the found optimal mixture (Ye et al., 2024; Xie al., 2024; Liu et al., 2024). However, it is unclear if this optimal data mixture can extrapolate to much larger magnitude, with some evidence in computer vision that this is not the case (Goyal et al., 2024)- - i.c., the optimal data mixture changes with scale.
Some researchers have described scaling laws for different downstream evaluation sets for the same model (Dubey et al., 2024; Chen et al., 2024), while others (Brandfonbrener et al., 2024) have derived power law relationships between models trained on different distributions and the same model on different test sets.
Recently, several researchers have attempted to predict the final loss of model for a given data mixture with scaling laws. Ge et (2024b) use the observation of logarithmic shift in scaling pattern to predict loss on subsets of the Pile weighted in various proportions on fixed model size. Ye et al. (2024), too, fit a scaling law for data mixtures based on a single scale. Kang et al. (2024) empirically show that the optimal mixture for a model changes with scale, and propose a method that addresses this on GPT-2 scale models. Que et al. (2024) use continual pre-training as a tool to develop their scaling law. Shukor et al. (2025), too, develop scaling laws that account for the transfer between data mixtures changing at scale for English LLMs and multimodal models for less than 10 domains.
17

---

Published as a conference paper at ICLR 2026
Multilingual Scaling Laws Prior work has also investigated multilingual transfer effects specif- ically in pretraining (Chang et al., 2024) and post-training (Shimabucoro et al., 2025), though on smaller scale of models. He et al. (2024) is the work closest to ours, where the authors attempt to describe scaling laws for multilingual LLMs. A crucial distinction from our work is that we ac- count for individual- languggge transfer learning in our scaling laws, and achieve better resulting fit (Table 1).
B EXPERIMENTAL DETAILS
To understand multilingual scaling characteristics we run 774 separate experiments, across different model scales, and language mixtures. In Table B. we summarize the experiment segments, using the notation below, that defines sets of languages scales.
mono (7): {en, ru, zh, hi, sw, yo!
pairs (10): (en, fr, ru, zh, hi, de, es, pt. ja, vi)
Langet (15): en/X for X Lpuirs (es/pt, fr/zh, hi/ru, de/ja, hi/vi)
Leval (48): complete evaluation set (all 48 languages)
Cesp (12): x4 6, 8 2, 12, 16, 24, 50)
Stull (20): {0, 1, 2, 4, 6, 8, 12, 14, 16, 18, 20, 22, 24, 26, 30, 32, 32, 46)
Spart (11): (0, 4, 8, 12, 16, 20, 24, 28. 34, 42, 46)
Smin (7): {0, 8. 16, 24, 34, 42, 46)
To see the model sizes defined by the scales Sfull Sparts and Smin. see Table B.3. We select these scale ranges to empirically observe models from 10M to 8B parameters. Subsets of languages, or subsets of the full scale (Spart. or Smin) are chosen to ensure the experiment number is sufficiently tractable for each research question, but also computationally feasible given our resources.
Table B.I: An overview of experiment configurations in this work. We enumerate the experiment types: <Lang> for monolingual scaling, Unimax as a massively multilingual baseline, Language Pairs measure language-to-languagg transfer, Capacity measure the curse of multilinguality. or model capacity constraints on learning new languages, and Finetunes to understand how fine- tuning from massively multilingual model compares to pretraining from scratch. We use mix of Monolingual and Multilingual vocabularies, and training data. In the LANGUAGES and SCALES columns use parentheses show the number of language mixtures and number of scales run. Symbols such as Lmono (7) and Sjall (20) are defined above.
EXPERIMENT TAG
VOCAB.
TRAIN-DAR DATA
LANGUAGES
SCALES
# EXPS
Mono <Lang>
Monolingual
Monolingual Multilingual
Lavnv (7)
Stul (20)
7 x 20 140
Unimax
Multilingual
Lunimax
Stunl (20)
20
Multi <L.ang>
Multilingual
Monolingual
Laxoa (7)
Spant (11)
10 X 70
Language Pairs
Multilingual
Monolingual
Lecal (48)
34
50
Language Pairs
Multilingual
Bilingual
Epuirs (10)
34
Language Pairs
Multilingual
Bilingual
Larget (15)
Spnn (11)
10 X 15 = 150
Capacity
Multilingual
Uniform Sampling
Cosp (12)
Spart (11)
10
12 120
Finetunes
Multilingual
Monolingual
Leval (48)
34
50
Finetunes
Multilingual
Monolingual
Laxcn (7) (10)
Smin
7 12 84
Total
774
B. LANGUAGE CHOICE
The fifty languages we selected come from the intersection of languages covered by both Flores-200 and MADLAD-400. They were selected to cover wide range of language families, scripts, and resource levels. To get this collection, we sorted all languages in the Flores- MADEL- intersection by the number of characters, and then selected every six languages. We then went this list and
18

---

Published as a conference paper at ICLR 2026
perturbed the index up down if it too typologically similar to already-selected language. For example, Language 48 was Afrikaans, but there were already several Germanic languages on the list, so opted to take language 47 instead (Telugu).
B.2 EXPERIMENT DESIGN
Pretraining Hyperparameter Details We rely on the hyperparameter settings refined by prior work obtain reliable performance across model sizes. We also conducted initial experiments, varying batch sizes, learning rates, optimizer, and dropout to ensure our settings were relatively robust. table B.2 we enumerate and explain all our hyperparameter choices, citing related work.
Train & Test Data We use two test sets. For each we measure the vocabulary -intirsiiiinnniiiveee loss proposed in Tao et (2024) order fairly evaluate across languages.
1. MADLAD-400 Test We isolate test set from each of our 50 evaluation languages in MADLAD-400 (Kudugunta et al., 2024). In initial experiments we find stable metrics over random seeds requires sampling roughly (sequence length num instances) 2048 10000 20,480,000 tokens. For each language we randomly sample the minimum of either 20M tokens or 20% of the total tokens for low resource languages min(20M tokens, 20%).
2. Flores-101 101 We use Flores-101 (Goyal et al., 2021), a language-parallet set of 101 high- quality translations, test set, using those to compute sequence losses, comparable to MADLAD-400 Test.
comparable test set. We extract the monolingual sequences the
Vocabulary Details We use vocabulary trained by Kudugunta al., with 64000 sized vocab- ulary, T = 100 and 99.9995% character coverage.
B.3 EXPERIMENTAL DETAILS: SCALING LAWS EVALUATION
Typically, to evaluate fitted scaling laws, prior work has adopted the R² metric, for it's scale- invariant measure of laws' explanatory power over the observed data. However, the choice of what data to hold-out as the test set can have significant impact on the performance and it's in- terpretation (Li et al., 2025a). For this reason, and because multilingual scaling laws are used to measure more than just the predictive power of the law to larger scales, we use a set different R metrics to measure different objectives. These are:
1. R² Hold-out a randomly selected 20% of the data points from the data. This gives an im- pression of the models' ability to fit the variance, without measuring particular dimensions of fit.
2. R (D) Hold-out the training runs with training tokens with the top 20% of D tokens.
3. I2 (N) Hold-out couple of scales of model sizes, including the largest: N 660M
and N 8B parameters.
4. R2 (C') Hold-out the largest compute parts of the training runs, where C 6ND FLOPs.
This will include mainly a combination of the larger model sizes, and longer training runs.
5. R (M) Hold-out all mixtures that are not monolingual, bilingual, unimax. This allows the scaling law to get baseline sense of each interacting language, but it has to infer the rest of the mixture scaling properties.
These metric variants allow to unpack specifically where the scaling laws perform well, and for what purposes they are reliable.
B.4 EXPERIMENTAL DETAILS: LANGUAGE FINETUNING
In Section 6 experiment with continuous pretraining single language, starting from mas- sively multilingual model's checkpoint. For clarity, we refer to this as finetuning, to distinguish from pretraining from scratch.
We begin by training massively multilingual models, using UNIMAX (Chung et al., 2023) language sampling across all 420 languages in MADLAD-400. Chung al. (2023) demonstrate this is an
19

---

Published as a conference paper at ICLR 2026
Table B.2: The pretraining hy hyperparameters used across experiments. We detail the batch size scheduling, the vocabulary choice, as well as fine-gre inniinaieed hyperparameter choices. Each choice is justified and grounded in prior work, discussed in the Explanation column.
PARAMETER
VALUE
EXPLANATION
General Training Parameters
Learning Rate Scheduler
WSD
We adopt the Warmup Stable Decay (WSD) learning rate sched- ulc, designed to efficiently study data-model scaling law with- out extensive retraining (Hu al., 2024: Hägele et al., 2024) Commonly used in scaling law experiments (Loshchilov & Hut- ter, 2019; Hoffmann et al., 2022)
Optimizer
AdamW
Base Learning Rate
2c-4
Following (Muennighoff et al., 2024) and confirmation it works well in initial experiments.
Warmup Steps
1000
Standard practice, as in (Longpre et al., 2023)
Decay Period
10%
Following (Hu et al., 2024; Hägele et al., 2024) this is the min- imum well performing decay length.
Sequence Length
2048 As in (Longpre al., 2023; Muennighoft al., 2024).
Training Steps
30k+
We vary the steps according to experiments, to always ensure we are well above likely chinchilla compute optimal ranges. While English scaling laws work tends to use a dropout of 0.0, we see little notable difference, except for lower resource lan- guages, with repeating epochs.
Dropout
0.1
Batch Size Schedule
Batch Size (<150M params)
256
Hu et al. (2024); Muennighoff et al. (2024) both find the optimal batch size increases with model size. We adopt a slightly larger rising batch size than Muennighoff et al. (2024),
Batch Size (<I B params)
512
Batch Size (<2B params)
1024
Batch Size (2B+ params)
2048
following initial empirical results.
Vocabulary
Vocab Size
64k
Vocab (Unimax)
Vocabulary by Kudugunta et al., with T = 100 and 99.9995%
character coverage. Same settings above, but trained only sentences from the language being considered.
Vocab (Monolingual)
effective sampling strategy to perform well across languages, as is the objective with many large multilingual models. The UNIMAX sampling rate per language documented in Table B.4 Each of these models we train for 17 tokens, to reflect real-world practices.
Using these Unimax checkpoints, across model scales, we then conduct continuous monolingual pretraining (which we call finetuning here) on all 48 languages separately in Leval (48). While finetuning, we also observe the loss across all Leval (48)languages as well. We use the same hy- perparameters as discussed for pretraining, but reset the learning schedule, and train for at least another 30k+ steps. These empirical results allow us to compare (across a range of N, D, C) the loss on language é between model finetuned from Unimax checkpoint, and model pretrained monolingually from scratch on language I.
B.5 EXPERIMENTAL DETAILS: LANGUAGE TRANSFER
In Section 4 we measure how much training source language is beneficial for the test loss on the target language t. A language transfer score tells practitioner the extent to which training with each language would help or hinder their target language t. There are two ways in which we can measure this language transfer:
1. BILINGUAL TRANSFER SCORE. This score measures how many training tokens a 50/50 (s, t) bilingual model needs, relative to monolingual target language baseline, to reach the same validation loss L, A positive score indicates co-trai co-training with the source language has positive transfer, whereas negative score indicates it hinders convergence.
20

---

Published as a conference paper at ICLR 2026
Table B.3: The SCALE value mapped to the dimensions of the model, and the parameter count. The model dimensions, borrowed from Muennighoff et (2024), report the number of attention heads, number of layers, embed size, feedforward size, and key-value size. Our model may be slightly different sizes than prior work based on our vocabulary size (64000 + 512 special tokens).
SCALE
HEADS
LAYERS
EMBED
FFW
KV
PARAMETERS
3
128
512
9,044,352
4
224
896
32
17,662,848
288
1.152
24.847,776
448
1.792
45,763,200
512
2,048
66.588,672
576
2,304
84.939,840
10
10
640
2,360
106,830,000
10
16
640
2,560
64
146,155,520
12
12
768
3,072
162.800,6400
12
768
3,072
191,114.996
12
18
768
3,072
219,428,352
14
896
3,584
67,,666600 080
13
14
16
896
3,584
263. 163.337,900 984
15
16
16
1,024
4,096
64
134,512, 128
16
18
1,044
4,096
368.068, 608
16
1,024
4,096
401,625,08
10
18
1,280
5,120
128
554.457.600
1,280
5,120
633.104,64
20
1I
18
1,408
5,632
128
661.807,872
711.751.680
22
II
21
1,408
5,632
128
756.970.368
12
1,536
6,144
128
816,345,
1,408
5,632
128
852, 132.864
25
12
22
1,536
6,144
128
929,596.410
1,536
6,144
1,042,847,22 232
27
14
20
1,792
7,168
128
1,143,245 824
1,792
7,168
1,297.391.
29
14
26
1,792
7,168
128
1,451,537,920
16
2,048
8,192
128
1,608.
17
176
8,704
1,807, 137,536
32
16
25
2,048
8,192
128
1 1,809893,, 396
34
17
25
2,176
8,704
128
2,033,422,912
9,216
2,187,122
36
17
28
1,6
8,704
128
2.261 288
18
2,304
9.216
128
2.526.870.52.
18
2,304
9,216
2,866.61 8,
39
20
26
2,560
10,240
128
2 510
3.310.955,520
41
20
34
2,560
10,240
128
3,730,396, 160
4.335,303,
43
22
36
2,816
11,264
128
2.749.364,224
2,944
11,776
128
5,182.299,64 6418
3,072
12,288
5,634,109,444
46
28
40
3,584
14,336
128
8.452,190,208
11,538.702,336
48
32
47
4,352
17,408
128
14,514.315,520
128
15.245.973,55
50
32
47
4,608
18,432
128
15,8216655,52
51
32
4,864
19,456
128
17,402.920.
5,120
18.982.943.616
21

---

Published as a conference paper at ICLR 2026
Table B.4: The Unimax language sampling rates adapted from Chung et al. (2023), I Languages are listed in order of their percentage sampling rate, which sum to 100.
LANG
%
LANG
LANG
%
LANG
%
LANG
%
LANG
en
5.00c+000
af
1.4c++00
be
1.42=+00
fil
1.2c+000
g1
1.42e400
te
1.42e400
1,42++00
1.42c+00
fr
1,42c-00
144c+00
1.42c400
1.42c400
ml
1 1.4e++00
mr
1.42++00
n
1.42e-00
1.42++00
ta
1.42e+00
1.42++00
hi
1,42++00
1.42++00
1.42=-00
Iv
1,4c+000
ms
1.42c+00
m1 1,42c+00
1,42++00
1.42++00
5q
1,42o+00
1,42++00
tr
1.42c+00
1.42c+00 VL
ca 1.42++000
el
1.42++00
hu
1.42++00
Sw
1,42+000
TV
1.42++00
1.42e+00
th 1.42++00
zh 1.42e+00
1,42++00
1,42++00
fu
1,42e400
1,42e+00 fi
ja
142++00
ko 1.4c+000
It
1.42--00
sk
142c+00
uk
1.42c400
bg
1.4c+000
1.4c++00
1.42c+00
BO
1.42c-00
mk 1.38c+00
1.34c+00
1.34c+00 CU
ka
I. 1.19+000
mn
1.09c+00
bs
1.03c-00
UZ
1.02c+00
8. 19c-01
0.
6.87c-01
kea
6.76c-01
kn
6.72c-01
gu
6.43c-01
si
5.89c-01
cy
5.14c-01
5.06c-01
4.64e-01
hy
4.47c-01
4.37e-01
4.28c-01
ga
4.25e-01
ml 4.06e 01
my
3.95e 01
km
3.35e 01
3.14e 01
90
2.93e 01
ps
2.52e 01
2.50e-01
pa
2. 01
2.29c-01
2.06c-01
1.83c-01
1.78c-01
ckb
1.73c-01
fy
1 1.68c-01
lb
1.63c-01
mg
1.54c-01
1.52c-01
am
1.50c-01
gd
1.48-001
1.7e-01
gre
1.25c-01
jv
1.12c-01
1.09-011
hmn
1.09
sd
1.05e 01
mi
9.77e 02
ya
9.55e
ba
9.42e 02
[o
9.24e 02 cch
9.12 02
or
9.07e-02
8.12c-02
xh
7.21e-02
su
7.20c-02
ny
6.97c-02
sm
6.94c-02
6.68e-02
6.67e:02
pup
6.57e-02
6.46e-02
ig
02
yu
6.00e-02
st
5.70e-02
haw 5.38c
as
5.07e-02
OC
4.93c 02
CV
4.66e 02
lus
02
4.14c-02
gsw
4.04c-02
4.01c-02
3.29 3.29c-02
0.52c-02
sa
2.23c-02
bo
2.23c-02
om
2.22c-02
50
2.12c-02
CC
1.70c-02
eah
1.58c-02
ilo
1.49c-02
1.4e-42
1.39-002
08
1.26e-02
1g
1.21e-02
0.
vec
1.10k-02
ts
9.73e.03
tyy
9.66e 03
khd
9.23k 03
ce
8.25e 03
ihu
7.66e 03
av
7.57c 03
kha
7.57e-03
7.51e-03
In
7.33e-03
nso
7.08
7.02e-03
ZzA
6.60e-03
6.23e
aula
6.08c Q3
5.86e
dz
5.69e 033
bun
5,44e
cfin
0.4
In
5.39c-03
chm
$. S.36c-03
gn
5.23e-03
krc 5.21e-03
wa
1.
hif
4, 4.79
yua
4.32c-03
sm
4.26c-03
war
4.03c-03
3.99c-03
bik
3.94c-03
sg.
3.89c-03
lu
3.87e-03
ady
3.73c-03
kbp
3.68c-03
syr 3.51e-033
lig
3.49c-03
myv 3.48c-03
3.43e-03
3.43e-03
blu
03
ay
3.30e
3.10e
qu
0.
pa
3.02e-03
ngu
2.97c-03
ve
2.94
pak 0.88
zup
0.
tyz 2.83e-03
hui
2.73e-03
bbc 2.65e-03
Izu
2.65e-03
Liv 2.55
ksd
2.52e-03
gum 0.50e-03
min
2.47e.03
ang
2.4fe
nhe
2.45e-03
hyp 2.45e 03
nxi
0.
mnh
2.29c.03
nv
2.28c-03
bci
2.26c-03
kv
2.25c-03
new
le2
mps
2. 19c
alt
2, 18
meu
2.15c-03
2.13c-03
2.08c-03
2.08c-03
abt
2.07c-03
mgh
2.05c-03
tvl
2.02e-433
dav
2.00e-03
uh
1.96c-03
bo
1.96e-03
kw
1.92c-03
mrj
1.92e-03
1.89e.03
erh
1.89e 03
mbt
0.87e.03
emp
1.85e
ace
1.85e
ium
1.85e
mam
1.81c-03
gym
I. 1.74c-03
mai
1.72c-03
crs
1.70c-03
pon
1.69c-03
ubu
I 1.68c-03
que
1.62e
gv
1.57e 03
kj
1.49e.03
bix 1.48e 03
ape
1.46e 03
chk
1.45e 03
ref
1.44e-03
shn
1.42c-03
1.41e-03
mdf
1.39
rpk
1.38c-03
1.37e-03
gag
1.31c-03
cab 1.27c-03
kri
1.25c-03
sch
1.23c-03
ibb
1.23c-03
tbz
1.21e-03
bru
1.21e-03
enq
1.20<-03
ach
1.17c-03
cuk 1.16c-03
1.15c-03
wo
1.14c-03
kek
1.12e
qub
1.1le
tab
1.11e-03
bis
1.07e-03
03
kos
1.06e-03
TWO
1.05e
cak
1.05e-03
tuc
1.02c-03
bum
1.01e-03
gil
9.73e
sig
9.65e 04
1sg
9.47e-04
quh
9.39c-04
mak 9.37c-04
9.35c-04
ban
9.06c-04
jiv
8, 8.84c-04
8.49c-04
yap
8.33e 04
kcy
8.26c 04
Ioj
8.20c 04
twu
8.1 04
xal
8.08c 04
amu
8.05e: 04
me
8.04c-04
hus 7.83-004
7.77e-04
kjh
7.73e-04
bm
7.64c-04
guh
7.64c-04
mas
7.64c-04
acf
7.36c-04
dip
7.46--04
ksw
7.25c-04
bzj
7.22c-04
din
7.17c-04
zue
7.15e-4M
6.99c-04
mai
64
mag
6.60c-4
mkn
6.57c-04
kg
6.54c-04
thu
6.37e 04
ch
6.23e 04
qvi
5. 78% 04
mh
5.59c 04
djk
5.52c 04
sus
5.21e 04
mfe
5.20e-04
STm
5.126-04
dyu
-00
ctui
5.07c-04
gui
5.03e-04
pau
5.02e-04
inb
4.88c-04
bi 4.71c-04
mni
4 4,59c-04
guc
4,430-04
jam
4.39c-04
wal
4, 4,38c-04
jac
4.35e-04
has 4.30c-04
gor
4. 81
skr 4.15c-04
nyu
4.14c-04
mna
4.08e-04
sda
4.08c-04
gub 4.07c-04
nog
4.04c-04
too
4.01c-04
tdx
3.92c-04
SXn
3.81c-04
rki
3.76c-04
Itr
3.74e-04
fip
3.61c-04
alz 3.60c-04
taj
3.51e-04
Irc
3.50c-04
cce
3.22e-04
3.21e:04
jvn
3.14e-04
lvvn 3.08 44
nij
3.07c-04 04
dwz
2.99c-04
izz
2.79c-04
mym
2.78c-04
bus
2.73e-04
ktu 2.66c 04
chr
2.52e 04
maz
2.39c 04
trj
2.22e-04
S112
2.15c-04
knj
2.15e-04
1.99--04
gvl
1.98c-04
hge
1.98c-04
1.97c-04
pis
1.93c-04
laj
1.83c-04
qxi
1.82-004
niq
1.80--04
ahk
1. 1.80c-04
shp
1.78e-04
hne
1.75e-04
spp
1.71e-04
koi 1.67e-04
qui
0.
agr
1.39c-04
1.31c-04
may
1.276-04
gof
1.27c-04
gbm 1.25c-04
miq
1.220-04
djo
1.21c-04
awa
1.19c-04
qVZ
1.10--44
III
1.03c-04
ruj
1.02e-04
kjg
1.00e-04
quy
9,24e-05
cbk
8.52e.05
akh 8.48c.05
oj
0.46e.05
ily
8.39c
cac
hrx
7.64e. 05
gup
7.48c-05
ff
6.97c-05
ber
6.68c-05
tks
6.55e-05
6.46e-05
mnw
6.46c-05
adh
6.40-005
smit
6.16c-05
fim
5.93c-05
qvc
5.876-05
ann
5.40c-05
kaa Latn
5.26c-05
nut
4.62e-05
kwi
4,34e-05
msb
4.20c-05
Latn
4.09c-02
doi
3.40c-05
din
2.97c-05
Li_Lain 2.48e-05
cid Lain
1.03e
ru Latn
6.95e-06
te Lain
4.84e
ber. Lain
4.74c 06
az_ R_
3.23e-06
Latu
2.29c-06
tly IR
2.15c-06
nan Latn_TW TW
1.94c-06
ml Latn
1.80-006
zXXXxx_dynnois
2XX_xX diynoise
1.65c-06
gom_ Latn
1.28c-06
bg Laun 9.21e-07
kn Lain
6.18c-07
Latn
5.69c-07
cr Lain
2.59e-07
ba__
1.83e-07
Lu_L
1.69k-07
sat
Latn 1.51e-07
nde 7W
1.31e-07
kmt_Tatn
1.01e-07
ms. ms_Arab
6.00e-08
ms, A0_AAab__Ru
4.29e-08
22

---

Published as a conference paper at ICLR 2026
2. FINETUNING ADAPTATION SCORE. This score captures the effect on the target language loss Lt, when a massively multilingual model is finetuned only on the source language s.
B.5.1 BILINGUAL TRANSFER SCORE (BTS)
The Bilingual Transfer Score asks how data-efficient a bilingual learner is, relative to a purely mono- lingual one, on some target language t. To estimate this, we pretrain models, monolingual one on and a bilingual one that samples tokens equally (50/50) from the source and target languages (s. !). We then measure how many more training tokens it takes the bilingual model to attain the same validation loss L as the monolingual model. This "distance" between the learning curves measured throughout training. at different reference horizons- or, number of tokens trained on. In Figure 2 we use models with 2B parameters, and a reference horizon of d = 42B tokens, as it roughly equates to 10, 000 pretraining steps for our 2B parameter models, and learning curves have stabilized. In Figure C.1 we measure how the BTS varies by model size, and by number of training tokens seen.
We are able to compute the bilingual transfer scores, across every combination of pairings be- tween 10 languages: English, Russian, Spanish, French, German, Portuguese, Chinese, Vietnamese, chosen give distribution of high- and mid-resource languages from a variety of language familics. This totals 10 monolingual experiments (one for each language), and C(10, 2) = 45 bilingual experiments, which provide the results for a 10x10 grid of Bilingual Transfer Scores.
Japanese, and Hindi. These languages
In more detail, let dmone be the number of tokens trained language t by monolingual model. This model attains test loss of (d(dmoo) after dmonn tokens on language We record the number of training tokens doi which the bilingual model reaches the same loss, L (chs) Le (dmono) We define the Bilingual Transfer Score (BTS) as the signed step surplus
BTS, Obi (dm(oo(dmono) 2dmono
dmono
where dui (L Tbi maps loss value to the number of steps taken by the bilingual model reach that loss. In short, this score measures the relative training efficiency of the bilingual model compared to monolingual model at reaching the same loss level for language t. Subtracting 2dmono and dividing by dmono simply centers the value on O, and forces positive transfer to give a positive value. A value of O, implies neutral transfer, as the multilingual model requires exactly the same number of steps in the target language f as the monolingual model, a value below 0 implies negative transfer, the multilingual model is slower to reach the loss L, (dloooo) and value above 0 implies positive transfer, as the bilingual model reaches the target loss in < 2d Note that for Figure these scores anchored on 2B parameter models, trained for d 42B tokens, for consistency. But as model size increases, or tokens trained decreases, the language transfer scores improve monotonically.
B.5.2 FINETUNING ADAPTATION SCORE (FAS)
Complementary BTS, the Finetuning Adaptation Score measures the instantaneous impact of continued exposure to single language on all others while holding model capacity fixed. Specif- ically, it measures how finetuning a massively multilingual model on source language affects the performance of target language l. Because we only need to train on each source language once, and evaluate on all languages, is less computationally expensive than BTS, which requires training on every pair of languages. As such, are able to derive 38 * 38 1444 (s, t) comparisons from 38 finetuning experiments.
We start from shared multilingual checkpoint: a Unimax pretrained model, trained for 17 tokens. For each source language s, we continue pre-training (for simplicity we denote this as finetune) exclusively this language, and evaluate at regular intervals every target language's 's t E L vali- dation set. Let (d)(dd) be the validation loss on after d additional updates on s, and let L unimoo denote the baseline loss of the shared Unimax checkpoint, before finetuning has begun. The Finetun-
23

---

Published as a conference paper at ICLR 2026
ing Adaptation Score (FAS) aggregates the reduction in loss over common time window [0, dmax]:
FAS
Lunimaa
Law(dd) dd.
max
Positive values indicate that exposing the model to s accelerates learning or yields immediate gains on (Chelpful transfer"). whereas negative values capture negative interference. Normalising by dmax makes the score comparable across language pairs and training horizons.
cause unpredictable behavior in the validation loss of target language (it might immediately increase, or decrease then eventually increase), we start with de- riving series of features about the finetuning loss curve. We calculate the following features for finetuning loss curve:
Because finetuning language
Baseline Loss Deviation (6g-z). For each language pair s * t we quantify the deviation in validation loss between the baseline I. Tuunimaz and the loss Tu on the final step:
8 U L
unimar (dmax)
6-رر means tine-tuing on helps 85-+1 > 0 means hurts.
Adaptation Gain (gt). This measures the normalized reduction in loss language / from finetuning on language This allows us to capture the relative learning dynamics of both the source and target language.
L Lunimaz -ig-s((d) ds.
91 dmax
9t > indicates net improvement; large values imply that / benefits greatly from additional supervision.
For each language transfer pair (s, t), we compose the following feature vector Tz-st As languages differ intrinsic difficulty, we normalize each feature.
9: Pg
danst 145.t
(5)
F8,t
For adaptation gain, we use the global mean and standard deviation (lg. 0g), and for baseline loss deviation we compute the mean and standard deviation (Ma,e; across all source languages for each fixed target language L.
B.6 ESTIMATING BILINGUAL TRANSFER SCORE (BTS)
In the Bilingual Transfer Score experiments we empirically measured language transfer scores y for subset Poss C L L. We construct training set from these 90 experiments, using them as the gold labels.
x- [ğ., St. Jo gc. 8_ 8..1)"
true V--st 2-20
(6)
We fit random-forest regressor sPD R1 - R with 300 trees and unlimited depth. The predictive performance is assessed by ke-oold cross-validation (k (k=5), obtaining an R² = 0.85 and Spearman correlation of p 0.88. After training all observed pairs, is invoked on every unlabelled Ex, C, yielding predicted BTS values, that estimate the language transfer shown in Figure 2.
B.7 EXPERIMENTAL DETAILS: MULTILINGUAL CAPACITY
B.7.1 MULTILINGUAL CAPACITY SCALING LAWS
In Section 5 we aim to understand how a model's capacity hinders multilingual learning-alss termed the curse of multilinguality (Conneau et al., 2019; Chang al., 2024). To do this, we train models of various sizes, and with a range of language mixtures, shown in Table B.5. For simplic- ity, languages always evenly sampled within their mixtures, even if some languages have more available text than others. We train models on these mixtures, and evaluate the validation loss of all other languages, throughout training.
24

---

Published as a conference paper at ICLR 2026
Table B.5: The set of experiments designed to measure the curse of mulillingualiy--or how language performance is impacted by both the number of training languages and the model size. We defined VERSION mixtures with NUM languages, ranging between 4 and 50. For mixtures with 4 languages, we define few different versions oriented to languages that might be more likely to be grouped together. This set experiments are summarized the Capacity row of Table B.1.
VERSION
NUM LANGUAGES (ALPHABETICAL)
4v0
4
de, fr, pt de, es, fr
4v1
4
4v2
4
hi, ja, vi, zh en, hi, pt, de, en, es, fr, pl,
4v3
4
6v0
6
Sy0
8
de, en, es, fr, pl, ru, vi, zh de, en, hi, ja, ru, vi, de, en, es, fr. hi. it, ja. pl. pt. ru, vi, de, en, es, fr, hi, id, it, ja. nl, pl, pt, nu, sv, tr, vi, zh
8y1
8
12v0
12
16v0
16
24v0
24
cs, da, de, en, es, fa, fi, Ir, hi, hu, id, it, ja. nl. pl. pt, ro, ru, sv. Lh, ir. uk, vi, zh
32v0
32
ar, bg. ca, cs, da, de, cl, en, es, fa, fi, fr, hi, hu, id, it, ja, ko, It, nl, no, pl. pt. ro, ru, sk, sv, th, tr
uk. vi, zh
50v0
50
af, ar, ba. bm, bn, ca, ckb, de, ec, el. en, es, fa, fil, fr, gn, gu, ha. he, hi. hr, id, it, ja, jv. kk.
ko, mr, my, nl, no, om, pl, ps, ro, ru, sm, sw, te, th, uk, ur, vi, yo, zh
For a given target language we have empirical loss scores across 11 differently sized models (from 10M 8B), and across each mixture that contains the language training, including monolingual, bilingual, and the multilingual mixtures shown in Table B.5.
We model each language's loss as L(N, D, K), where N is the model size, D are the number of training tokens for the targer language, and K is the number of languages in the training mixture (evenly sampled). We define L(N, D, K) as follows:
AK
BK D8
DN,D,K) Loo
Na
This law preserves the well-validated power-law behavior of monolingual scaling laws, while intro- ducing K into both the capacity term (AK') /Na) (AK*/Na) captures how capacity requirements grow with the number of languages, while v captures how data requirements change with multilinguality: data needs as K increases), e = implies no net transfer in the data term, and i implics nega- tive transfer/iteeffrennce. Under sampling, the total tokens across languages are Drut K D, in which case the data term be rewritten B K KV+B /Di
and the data term (BK*/D²) The exponent
< 0 indicates positive transfer (sublinear per-language
B.7.2 ADDING LANGUAGES: K K
Practitioners may wish to change the number of languages supported by a model, from K to K r. If the model size and training tokens remain unchanged, will lead degradation in loss across languages, as the same compute budget is allocated across more languages. When adding languages, the practitioner may wish know how much they need scale the model (N, D) and the training compute C' to maintain the same performance per language, as before. We call this an iso-loss, as it approximately preserves the loss values, with a change in K.
Compute-Optimalte for Equation (7). As we increase K to K * we would like to understand the new N', D' that minimize training compute the iso-loss surface. Let K'/K. /KK Minimizing C" x D'D subject to
A(Kr) Nta
B(Kr)
AK*
BK
+
D
Na
D
yields the closed-form optimum:
N'
= r 4/a
(
111
D'
14018
N
Diot
25

---

Published as a conference paper at ICLR 2026
These expressions show clear separation of roles: ø/a govemns the parameter burden of adding lan- guages, whereas 2/B governs the data burden (with < 0 reducing the per- per-langu data require- ment due positive transfer). Since the compute budget is in terms of total tokens Diot K D, and C xx N Droc.
Adding languages: K to k Finding the compute multiplier. At the iso-loss optimum, the compute multiplier is
C' N' C NDoo
-x*)
= 1100/+4188
=
Dooc
Intuitively, a measures the extra parameter capacity per added language, while y measures the extra (or reduced, if < 0) per-language token budget. But since compute is defined with total tokens Doot then we consider the additional factor of from enlarging the language inventory.
Let N' and t P the multipliers for model size and (per-target-language data when we grow the language inventory from K K" rK. Define the baseline contributions
Adding languages: K to K
Finding the Iso-loss curve.
D' denote
A
AK AK*/Na BK*/DR
WD
1
The iso-loss condition equates the of the two error terms before and after the change in K:
A(Kr) _(Kr)²
AK
BK
(Ns)a
(Dt)a
No
D
Dividing both sides by AK*/No + BK /D yields the normalized iso-loss constraint
7 wN s + wp 8
(8)
Solving for given 8. Rearranging equation 8 gives
1/8
WD
r WN
The denominator must be positive, which implies the feasibility condition
WN.
As 5t r wy the denominator tends to 0 and 3o; as co, the denominator tends and t + (/wp)) WD/
Solving for given By symmetry,
1/a
WN
with feasibility t0 > r WD.
r
wp
t
Total-token form. Under even sampling, Diot
KD and D
K'DD' = r rKD
D'et
D'
r WD
1/B
D'D =
Diot
WN
Compute-optimal frontier (starting from a compute-optimal (K, N, D)). Suppose the baseline
(K, N. D) is compute-optimal for its loss level, i.e., it minimizes C x ND subject to AK* /No + BK D const. A standard Lagrange multiplier argument then yields
AK
BK D
8 wN a a WD
No
a a
After changing K rK, the compute along the iso-loss curve is /²/C= = st (or C²/C s if compute is defined with total tokens). Minimizing this product subject equation again L Lagrange multipliers gives the stationarity condition
D'
Days
8 uN s 3 2 wD -3
26

---

Published as a conference paper at ICLR 2026
Table C.I: A summary of the monolingual scaling laws for each language The columns are: the language, the language family, the language script, its number of unique tokens [D] in MADLAD- 400 (with the percentage of unique tokens as compared to English), and the derived scaling law.
LANGUAGE
FAMILY
SCRIPT
[D] (% ENj
SCALING LAW
Monolingual Vocabulary, Monsbiggal Training
English
Indo european
Latin
2.8T (100.0%)
0.67
NU
6.18
2..25
european
Latin
(13.01%)
400.11
N9.04
.13224
Russian
Indo-curopean
Cyriltic
7058 (25.26%)
0.82
.0.007
739.03
13.25
Chinewe
Sino libetan
Hans
125B (4.48%%)
1.08
N0.0
00:40
Hindi
Indo-european
Devanagari
7,98 (0.28%%
0.52
6,08
20:39
.8.03
Swahili
Niger-congo: atlantic congo
Latin
770M (0.03%)
0.00
27:22
20:22
5.51
Muhtilingual Vocabwlory, Monollnggal Training
English
Indo-curopean
Latin
2.8T (100.0%)
0.83
w0.63
9.91
213.00
130.03
French
Indo-curopean
Latin
363B (13.01%)
0.66
20:45
67.12
88,94
Russian
europeun
Cyrillic
(25.26%%
N0.40
.7.97
9.91
Chinese
Sino-tibetan
Hans
(4.48%)
1.18
Hindi
Indo-curopean
Devanaguri
7,9B (0.28%%
0.63
N0:3
Swahili
Niger-oongo: atlantic congo
Latin
770M (0.03%%)
0.00
00.30
6.43
Muhiilinuua Vocabulary, Unimax Training
English
Indo european
Latin
2.8T (100.0%)
0.00
N0.11
5:03
20:16
9.15
French
Indo european
Latin
363B (13.01%)
0.00
170
3.68
750.18
3.94
Russion
Indo-curopean
Cyrillic
705B (25.26%)
0.00
270.20
3,90
4.28
Chineve
tibetan
Hans
(4.48%5
N0:21
..58
5.44
Hindi
Indo-european
Devanaguri
7,98 (0.28%)
0.00
N0.8
Swahili
Niger-congo: atlantic congo
Latin
770M (0.03%)
0.00
N0.3
20.18
Combining with equation and substituting the compute-optimal weights above yields the unique minimum-compute point on the new frontier:
N' N
a/a
1/8 v/3
D Desc
1+0/8
At this point each error component is individually restored to its baseline magnitude: A(Kr) AK* / and B(Kr)*/D's BK D 8 so the weights (WN, wp) and hence the balance of capacity/data bottlenecks-remenn unchanged. Because the frontier is convex in (log 8, log f) and log C" log + log linear, this stationary point the global compute mini- mum along the iso-loss curve.
C EXTENDED RESULTS
C.I EXTENDED RESULTS: SCALING LAWS
In Table C. we illustrate the fitted scaling laws for each language. Specifically, we show the scaling law parameters for each language when trained with a monolingual vocabulary on monolingual data, when trained with multilingual vocabulary monolingual data, and when trained with multilingual vocabulary on the Unimax multilingual mixture. These results match up with those presented in Figure 1.
27

---

Published as a conference paper at ICLR 2026
0.5
0.0
Trazffr Soors
Trazafer Scors
00
-05
Scure 0.39 1.3-
Score 0.11 +0.08
6.6
Training Tokens (D) [x10!!]
Model Size (N) (x10*)
Figure C.I: Left: X-axis training tokens (D), Y-axis is language- transfer score. Right: X-axis is model size (N), Y-axis is amguage-transtie score.
C.2 EXTENDED RESULTS: LANGUAGE TRANSFER
Research Question: How does language transfer change with larger models, or when you train for longer?
We further investigated how language transfer evolves with training data (D) and model size (N) in Figure C.I. Our analysis of transfer scores over the course of training reveals that transfer dynamics are established early. The performance gap between synergistic pairs (e.g., es - pt) and interfering pairs (e.g., en zh) appears within the initial stages of training and remains largely consistent, suggesting that the fundamental compatibility of languages is not significantly altered by longer training. However, the impact model scale is more pronounced. As illustrated in Figure C.I, we observe clear trend in which larger models are more effective in facilitating cross-lingual transfer. For synergistic pairs, the positive transfer score increases modestly with model capacity. More importantly, for challenging pairs that exhibit interference in smaller models, larger models significantly better at mitigating this negative effect, bringing the score closer to zero. This is similar what has been observed while training larger and larger foundation models (Team et al., 2024). Given that transfer dynamics change at scale, it is an important factor to consider when designing multilingual foundation models that performant for all languages being trained on.
28

---

Published as a conference paper at ICLR 2026
Spenss
Sinte Ramana
Berse-Conge Gemaaiio
Stadi
South Cenrnl Doviidaa
40
A8
J03
44
82
44 AJ HUA JEL
AUL A1
48
Targe Langungg
43
Source
Language
Figure C.2: The language transfer scores measure benefit to the target language of (co-) (ro))trainin with the source language. We bold the top-5 source languages for each target language. The top left 9x9 are the Bilingual Transfer Score computed directly, whereas the rest are estimated from the Finetuning Adaptation Score. While English is the best source language for many of of languages, we find language similarity is highly predictive of these scores.
29


Reviews:
[{"review_id": "xlZ5IXbVL4", "reviewer": "Reviewer_gXoZ", "summary": "The paper proposes a new multilingual scaling framework that models how performance in multilingual language models scales with model size, data size, and the number of languages during pretraining and finetuning. To understand cross-lingual transfer, the work provides a large-scale empirical study quantifying the pairwise language transfer in a model-based manner. In addition, it models and quantifies the curse of multilinguality, providing scaling rules for maintaining performance as language coverage expands. Finally, the work analyzes when it is more efficient to pretrain from scratch compared with finetuning from a multilingual checkpoint.", "questions": "- In section 5, the study of the curse of multilinguality, the authors do not seem to explicitly model the effect of inter-language interaction. For instance, if the added languages are highly similar to the existing ones, the curse of multilinguality might not be as evident. On the other hand, if the added language is very different, the effect would be more pronounced. Does the current analysis take this language similarity into account?\n\n- The transfer between pairs of languages is clearly modeled in Figure 2. What about transfer in terms of subsets of languages? The effect of cross-lingual transfer for a subset may not merely manifest as the sum of transfer in each of the constituent languages as multiple languages combined could show redundancy or synergy. How would this be captured by the current scaling law?\n\n- For finetuning discussed in section 6, do the authors actually mean continual pretraining? If so, it might be a better term as finetuning could be confused with SFT in the context of LLM, causing confusion.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The problems the paper wants to tackle are important in the multilingual learning literature. Each section begins with a clear research question, which guides the reader through the narrative logically.\n\n- The work shows significant experimental efforts. Notably, the bilingual transfer table in Figure 2 is a valuable asset to the community of multilingual learning. \n\n- The findings offer actionable insights for multilingual model practitioners, especially the compute-optimal scaling frontier and pretrain-vs-finetune tradeoff. These results help translate empirical observations into practical guidance for scaling multilingual models efficiently.", "weaknesses": "- It is unclear which part of the proposed law is a unique contribution of the authors, and which is adapted. For instance, it is known that equation (1) essentially follows the Chinchilla scaling law, but why do equations (2) and (3) take the given specific form? Although the fitting accuracy is high, it would be helpful to provide some justification about why the proposed law organizes those terms in the optimal way. If those constructions are an improvement or combination of previous scaling laws, it would be better to first explicitly write their corresponding forms and name the improvement or generalization of the proposed law mathematically. \n\n- The practical use of the proposed scaling law for predicting model performance or guiding training design remains somewhat implicit. In section 5 and 6, how is the proposed law used? A clearer description of the parameter fitting process would also strengthen the work: for example, what data points are used to estimate the coefficients, what optimization method is applied, and how stable the fitted parameters are across runs. Furthermore, the computational cost of fitting the scaling law is a key practical factor. If estimating these parameters requires comparable or greater resources than training a target model directly, the law’s real-world usefulness would be limited.\n\n- The paper lacks a conclusion section that summarizes the findings or discusses future directions. The narrative ends abruptly after the analysis, making the work feel unfinished.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ArIbUDFMQ", "reviewer": "Reviewer_taDw", "summary": "This paper studies the scaling law of multi-lingual models w.r.t. model size, data size and computation budgets.", "questions": "1. Line 015, in what sense is this the largest multilingual scaling law study? \n\t2. Line 075, please define N, D, C before they used? I am guessing they are model size, data size, computational buduget? Especially the symbol N is difficult to bet against. \n\t3. Line 093-097, as R^2 is a key concept in this paper, it is worth explaining/re-iterating the intuition behind R^2 on model size, token numbers, computational costs and number of pairs of languages (M --- I am again guessing the meaning of this symbol).\n\t4. Line 104, will the vocab size affect the analysis results? Is 64K vocab size too small for 48 languages especially with glyphic scripted languages.\n\t5. Line 159 - Line 178, please expand the description to include formal definitions, intuitive explanation and other details which can help readers to understand the design of ATLAS. It would be more rigorous if this can be grounded on any statistical models that can draw relationship among the variables in ATLAS and draw relationships on these variables and statistical data collected from model training experiments regarding validity of the proposal.\n\t6. Line 197, how do you choose C=6ND? \n\t7. Line 1208, \"well-validated power-law behavior\", please provide references and evidence in terms of statistical data fitting for experical training tasks. As the LLM training behavior is highly complex, it needs substantial evicence to draw conclusion that simple linear or power-law equations can fit the training behaviors with stats over data size, model size, language number, and computation cost. \n\t8. Line 483-485, any conclusion or summary section?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "• Systematic and comprehensive study of scaling law for multilingual models is an important topic.\n\t• Significant number of experiments are conducted. A few important findings are drawn (the evidence to support the claims requires additional attention though).", "weaknesses": "• Key definitions are missing for a few key concepts and key equations. For example, the symbols in equation (1) are not defined. The grounding of these symbols and equations are not available in this paper. The readers will need to look up the citations with significant efforts of guessing to understand the key idea in an inaccurate manner. \n\t• The formal model of scaling laws are not well defined in this paper.\n\t• The writing and structure of this paper doesn't meet the scientific paper quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w7Hmgb6E2t", "reviewer": "Reviewer_Yyfb", "summary": "1. The paper presents scaling laws in the multilingual setup across different axis:\n\n1.1 For repeated epochs in the monolingual setup\n\n1.2 To account for cross lingual transfer in a language mixture setup\n\n1.3. Account for the curse of multilinguality by providing a closed form approximation to account for how much more data to train / how many more parameters to account for to keep the loss on a target language consistent, while adding new languages.\n\n1.4. Address the issue when is it better to pretrain a language model for a target language from scratch vs finetune a multilingual language model.\n\n2. The authors demonstrate that their scaling laws fit better for monolingual and multilingual scenarios compared to other monolingual, data constrained and multilingual scaling laws respectively. In order to account for scaling across different languages with different vocabularies, they fit scaling laws to the vocabulary insensitive loss.\n\n3. The authors also present a very large scale cross lingual transfer study. They propose a cross lingual transfer metric: the number of target language tokens taken by a bilingual model to reach the same loss as that taken by the target language monolingual model. By analysing the transfer matrix, the authors highlight the key factors associated with positive transfer: language script as well as language family. They also demonstrate that these factors additionaly seem predictive of whether a symmetric transfer might occur between the languages. \n\n4. The authors present their results over an impressive number of experiments. If the results (esp. details on number of parameters, tokens, loss convergence values, mixture weights, data subset etc.) are released, it can enable additional analysis especially w.r.t cross-lingual transfer.", "questions": "1. For leveraging the vocabulary agnostic loss, how did that compare against fitting without the load agonstic setup. Does this design choice have any implication on the results presented in the paper? Likewise, were the baseline MSL fit on the vocabulary agnostic loss or on general log-likelihood? \n\n2. Somewhat related to the previous question: what is your opinion on balancing out the vocabulary at a per language level [1], and leveraging that for loss computation (instead of doing a vocabulary agnostic loss) ?\n\n3. For the MSL evaluations, what was the language grouping that was used for computing the scaling laws for fitting ? Also as a meta point, it would be good to also add details on how the curve fitting was actually carried out.\n\n4. If my understanding of Eqn (3) is correct, the authors make the assumption that the languages decay at the same rate. Intuitively, the decay rate (\\lambda) should be a function of the dataset quality, which varies significantly across languages. So why would the assumption hold true ?\n\n5. For Figure 1, hi, seems like overtraining hurts both the monolingual and the unimax model, however the monolingual model with the multilingual vocabulary seems to be doing okay. Why do the authors think that might be the case?\n\n\n[1] Zheng, Bo, et al. \"Allocating large vocabulary capacity for cross-lingual language model pre-training.\" arXiv preprint arXiv:2109.07306 (2021).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a new functional form for modeling multilingual setups, accounting for repeated tokens as well as data mixtures. The consequent scaling law has better predictive power compared to other baselines.\n\n2. For me, the biggest contribution is the cross-lingual study for understanding transfer at scale: the significant number of pairwise experiments definitely help identify key factors for cross-lingual transfer, and additionally be an important resource for trying to further understand what factors might actually help influence the degree of the transfer.\n\n3. The paper's answers an important question on how should I scale compute (N,D) to achieve the same loss level while adding new languages. They also provide a good heuristic approximation on when it might be better to fine-tune a pre-trained multilingual language model vs train the lm from scratch. \n\n4. The authors present their results over an impressive number of experiments: 750 independant training runs. Managing experiments at this scale can be and is difficult, and the effort is appreciated.", "weaknesses": "While I think the paper make some really good contributions, I do have the following concerns:\n\n1. The scaling law's functional form doesn't explain why it was chosen to be the functional form in the first place, compared to other ways of formulating the interaction. Concretely [1,2], both demonstrate that L(N, D, p) ~ L(N, D)*p exp(\\gamma), which also requires that the term modeling the parameters should also depend on the proportion of language (a similar symmetrization argument was also made in [3]).  Some motivation on why the functional form was chosen would be good to have: both for the general purpose law as well as the target language specific loss (Page 7: L373).\n\n2. For the section on curse of multilinguality, the authors mention that they sample the tokens from each language uniformly: however that is counter to the sampling setup used on most other multilingual experiments (Unimax). For consistancy, it would be good to know if the laws also hold under the unimax distribution setup. Orthogonally, the authors give a recommendation on how to scale compute (N, D) for accommodating new languages. But without actual verification of the scaling law (i.e actually training with the additional compute on the new languages), it is hard to verify the accuracy of the proposed law. Concretely, it would be informative to have a setup to train with languages not used for the fitting, but belonging to either a group with positive transfer, neutral transfer or negative transfer, and then showing how accurate the law is at predicting the performance.\n\n3. The section on pre-train vs fine-tune seems very rushed, and is a bit hard to follow. Additionally, it is quite counter-intuitive that the proposed inflection point is not a function of the compute capacity spent on the pretrained unimax model. Overall, as a meta comment, I think this section requires a bit more of a rigorous setup to compare (eg: varying C for the pre-trained model). In addition to that, there are a number of observations from the cross-lingual setup that are difficult to explain: eg: for the target language of En, why do source languages id and ms have a high BTS score. Having a better understanding of that might be very informative.\n\n\n[1] He, Yifei, et al. \"Scaling laws for multilingual language models.\" arXiv preprint arXiv:2410.12883 (2024).\n\n[2] Akimoto, Kosuke, and Masafumi Oyamada. \"Optimizing Low-Resource Language Model Training: Comprehensive Analysis of Multi-Epoch, Multi-Lingual, and Two-Stage Approaches.\" arXiv preprint arXiv:2410.12325 (2024).\n\n[3] Muennighoff, Niklas, et al. \"Scaling data-constrained language models.\" Advances in Neural Information Processing Systems 36 (2023): 50358-50376.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)