Exploring Polyglot Harmony: On Multilingual Data Allocation for  Large Language Models Pretraining
Authors: Ping Guo
Yubing Ren
BINBINLIU
Fengze Liu
Haobin Lin
Yifan Zhang
Bingni Zhang
Taifeng Wang
Yin Zheng
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Pre-training
Scaling Law
Language Mix Ratio
Primary area: deep_learning

Abstract: Large language models (LLMs) have become integral to a wide range of applications worldwide, driving an unprecedented global demand for effective multilingual capabilities. Central to achieving robust multilingual performance is the strategic allocation of language proportions within training corpora. However, determining optimal language ratios is highly challenging due to intricate cross-lingual interactions and sensitivity to dataset scale. This paper introduces CLIMB (Cross-Lingual Interaction-aware Multilingual Balancing), a novel framework designed to systematically optimize multilingual data allocation. At its core, CLIMB introduces a cross-lingual interaction-aware language ratio, explicitly quantifying each language’s effective allocation by capturing inter-language dependencies. Leveraging this ratio, CLIMB proposes a principled two-step optimization procedure—first equalizing marginal benefits across languages, then maximizing the magnitude of the resulting language allocation vectors—significantly simplifying the inherently complex multilingual optimization problem. Extensive experiments confirm that CLIMB can accurately measure cross-lingual interactions across various multilingual settings. LLMs trained with CLIMB-derived proportions consistently achieve state-of-the-art multilingual performance, even achieve competitive performance with open-sourced LLMs trained with more tokens.

Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
Ping Guo
Fengze Liu Yin Zheng* Instittte of Information Engineering, Chinese Academy of Sciences
Yubing Ren'
Binbin Liu
Haobin Lin
Yifan Zhang Bingni Zhang* Taifeng Wang
ByteeDance
Abstract
Large language models (LLMs) have become integral to a wide range of applica- tions worldwide, driving an unprecedented global demand for effective multilingual capabilities. Central to achieving robust multilingual performance is the strategic allocation of language proportions within training corpora. However, determining optimal language ratios is highly challenging due to intricate cross- mosss-lingua interac- tions and sensitivity dataset scale. This paper introduces CLIMB (Cross-Lingual Interaction-aware Multilingual Balancing), a novel framework designed to sys- tematically optimize multilingual data allocation. At its core, CL.MB introduces a cross-lingual interaction-avee aware language ratio, explicitly quantifying each lan- guage's effective allocation by capturing inter-language dependencies. Leveraging this ratio, CLIMB proposes principled two-step optimization procedure- first equalizing marginal benefits across languages, then maximizing the magnitude of the resulting language allocation vectors significatty simplifying the inherently complex multilingual optimization problem. Extensive experiments confirm that CLIMB can accurately measure cross pross-lingual interactions across various multilin- gual settings. I.LMs trained with CLeMee-drrved proportions consistently achieve advanced multilingual performance, even achieve competitive performance with open-sourced L.LMs trained with more tokens.
Introduction
Large language models (LLMs), exemplified by the GPT series 42 43 LLaMA series 61
60
25..
Gemma series 20 19 21, Qwen series 48 58 and DeepSeek series [4]
13 have
reshaped various language- based applications worldwide, powering advanced chatbots 12) machine translation systems 770). and intelligent virtual assistants (62]. Such impressive capabilities emerge predominantly from extensive pretraining on enormous textual datasets, frequently spanning tens to hundreds of trillions of tokens, enabling the capture of rich and diverse linguistic knowledge. Driven by the growing global demand and the need for equitable language representation, there has been an accelerating shift toward multilingual pretraining, aiming to transcend linguistic boundaries and serve a broader range of linguistic communities effectively [69]. Central this shift lies a fundamental question: how should the proportions of different languages be optimally allocated within the training corpus to achieve balanced and superior model performance across all target languages?
However, determining an optimal multilingual mixture poses considerable challenges. The foremost difficulty arises from cross-lingual interactions: performance on one language can be significantly influenced by other languages trained concurrently (6. 71 As illustrated in Figure even when the training proportion of Arabic remains fixed to 10%, modifying the proportions of the other four
Yin Zheng (yzheng3xg gmail.com) is corresponding author and tech lead multilingual LLM pretrain project.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

languages (increasing one language to 60% proportion) in five-language LLM can substantially alter Arabic's performance. This interdependence prevents isolated optimization of individual languages and necessitates joint optimization of the entire language set. Additionally, optimal language ratios are sensitive to the scale of the training corpus 32 28
55 24 Specifically, language proportions identified as optimal at smaller scales (e.g.. I billion tokens) may no longer remain optimal when scaled to larger training sets (e.g., 4 trillion tokens), rendering simple extrapolations unreliable and incurring prohibitive experimental costs, Consequently, current multilingual LLMs often resort to heuristic trial-and-error approaches [1] or reuse language ratios derived from prior models without systematic justification [34], highlighting critical need for a principled and scalable solution to multilingual data allocation.
In pursuit of achieving the optimal language allocation, this paper explores whether it is possible to accurately predict model performance under various language allo- cations without explicitly training the models. Inspired by the concept of scaling laws, which characterize how model's validation loss systematically scales with model
'ar' Loss at Fixed 10% Proportion
3.5
R0% 10% Others 10%
3.44
10%
ko: 68%, un: 10%, Others: 10%
ot Arabis
3.3
Ja: 60%, ar: 19%, Others: 10%
3.2
3.1
size (N) and data volume (D) 33 30), we hypothesize that a similar predictive framework could be applied to multilingual settings by incorporating language propor- tions. Specifically, if we can formulate mathematical re- lationship that captures how validation performance varies with language proportions in the training corpus, then it becomes feasible infer optimal language ratios by identifying the allocations that minimize validation loss. However, due to the intricate cross-lingual interactions among languages, precisely modeling and predicting validation performance across different lan- guage compositions remains highly challenging.
2.9
2.8
20
25
30
35
10
15
50
Total
Size
Figure 1: Cross-lingual Interactions in a five-langauge LLM.
In this paper, we propose CLIMB Lenoss- Interaction-aware Multilingual Balancing), a novel framework designed to systematically optimize language proportions for multilingual LLM pre-training. Our approach consists of two interconnected components. First, we introduce the cross-lingual inferaction-avedt allocation of each language in the presence of cross- lross-lingual interactions, effectively reflecting the impact of other jointly trained languages. Second, leveraging these cross-lingual interaction-aware ratios, we can estimate the optimal multilingual balance by decomposing the optimization into two steps: initially, we determine the direction of optimal allocation by equalizing the marginal benefits across languages; subsequently, we obtain the estimated optimal proportions by maximizing the magnitude of the resulting cross-lingual interaction awaae language ratio vector. This principled two-step procedure enables efficient and accurate computation of multilingual data distributions, significantly reducing the complexity inherent in direct joint optimization.
language ratio, novel metric that explicitly quantifies the effective
To comprehensively evaluate the effectiveness of CLIMB, we conduct experiments in two primary aspects. First, validate the predictive accuracy of the proposed cross-lingual interaction-aware language ratio. By integrating this novel ratio into the multilingual scaling law framework, we observe substantial improvement in predictive accuracy compared to baseline scaling laws relying on independence assumptions among languages. Second, leveraging the optimal proportions computed via CLIMB, we train multilingual LLMs both 1.2B and 7B parameter scales. Experimental results demonstrate that models pretrained with CLIMB-derived ratios consistently achieve leading performance compared to various baselines with alternative language allocations. Remarkably, even compared to upen-sourced models pretrained on more tokens, our CLIMB- --MMM--ppiiizedd models exhibit highly competitive performance across multiple multilingual benchmarks.
2 CLIMB
Our approach is grounded in extensive multilingual experiments designed to disentangle how loss dynamics evolve with respect to language composition, total training tokens, and cross-lingual proportions. Building on these observations, our framework consists of two main components: the Cross-lingual Interaction-aware Language Ratio, which explicitly models effective language proportions by incorporating inter-language dependencies, and the Optimal Multilingual Balance,
2

---

(a) For as a function of Far with varying co-training languages
(b) For as function fur with varying language counts
(c) Pur as function for with varying training data scales
Figure 2: Illustration of cross -ongull interaction-awann aware language ratio (Far) and its dependency on original training proportions (Far).
which leverages these interaction-aware ratios to estimate the optimal allocation r* that minimizes multilingual validation loss.
2.1 Experimental Setup
To study how multilingual training dynamics depend on language composition, token scale, and inter-language proportions, we conduct series of controlled experiments across diverse language settings, organized along three dimensions:
(1) Number of Languages.
We explore multilingual configurations of increasing scope, including bilingual (les-ko, en-zh, de-ar, ko-ja)), trilingual ((es-de-ar, es-ko-zh, en-zh-ja]), five-language ((es-de-ar-koja)) and a sixteen-language setting covering (de, en, nl, es, pt, it, id, ja, ko, zh, ru, ar, th, vi, tr).
(2) Total Training Tokens. For each setting, models are trained under ten token budgets from 5B to 50B (step size 5B). To ensure comparability, all runs share the same learning rate schedule, decaying to 10% of the initial rate by training end.
(3) Language Proportion. To examine proportional effects, one language's 's share is fixed while others evenly split the remainder. For each language L,, its proportion is varied over {0.02, 0.025, 0.05, 0.1, 0.2, 0.25, 0.4, 0.5, 0.6, 0.75, 0.8, 0.9, 0.95, 0.975, 0.98) to observe loss trends.
Combining these factors yields over 500 multilingual runs. Section summarizzs empirical findings and fitting equations, while Section 3 exxamines their extrapolation and generalization performance.
2.2 Problem Formulation
Given a multilingual corpus consisting of training data from m distinet languages II...., Joms our objective is to determine the optimal language allocation for pretraining L.MMs. Formally, we define
the language proportion vector as r = [ri. r Tm] E Rm, where R m {r E 1mm mm 1, 2 0, Vij denotes the probability simplex.
Given total token budget D, each language L; contributes D; [r: DJ tokens to the training set. The model parameters 9 are trained via empirical risk minimization: (D, = arg ming L(0; D, r). where L(e; D, r) denotes the next-token prediction loss on the multilingual training set defined by proportions r and token budget D.
To evaluate the pretrained model, measure validation loss language-specific held-out set DT: (6" (D,r) = L( (D, r); D.))
Our goal is to identify the optimal language proportion vector r" that achieves balanced multilingual performance by minimizing a weighted sum of validation losses across all languages:
r' arg min PETR
C (0" (D,r)),
(1)

---

where hyperparameter 2 O specify the relative importance of each language Lg, set according to application -specific requirements or practical considerations.
This formulation defines a bi-level optimization problem, in which the outer optimization seeks optimal language proportions, and the inner optimization involves training L.MM given these language proportions. Due to the intrinsic complexity of cross-lingual interactions and the prohibitive computational cost of repeated model retraining, directly solving this optimization problem through standard gradient-based approaches is computationally infeasible.
2.3 Cross-Lingual Interaction-aware Language Ratio
Given a total token budget D and a language proportion vector r, we can obtain the validation loss (D, for language Li- Let D; be the number of tokens required to reach the validation loss 2 rD, from monolingual model solely language L,, then define the cross-lingual interaction-aware ratio r; the ratio of this equivalent monolingual token budget D, to the actual multilingual token budget D: r. = Dj. Formally, F; can be formally expressed as:
B.
1/8
(2)
D
CD
where parameters B,, 3r. and E, derived from the monolingual scaling law [30]. Specifically, the monolingual scaling law characterizes how the validation loss decreases as training data volume increases for single language L,, expressed as:
B. D
A (D; = =
E.,
(3)
where D; represents the token budget allocated exclusively to language L,. In the absence of cross- lingual transfer, the interaction-awee ratio F; equals the actual ratio r;, thus the difference n; - r; quantifies the magnitude of cross-lingual effects from other languages.
2.3.1 Empirical Observations and Insights
To systematically understand the behavior of the cross-lingual interaction-awart language ratio f; conducted over 300 experiments on Transformer-based models. Specifically, we varied the number of jointly trained languages (2, 3, and languages), total token budgets ranging from 5 billion 100 billion tokens, and explored a wide range of language proportion vectors r. For each configuration, computed the pairs (ra, examine how the effective language ratio deviates from the actual proportion due to cross-lingual transfer. These results are visualized in Figure] 2 from which we identify following key empirical insights:
Dependency on absolute language proportion. Cross-lingen transfer strength diminishes the actual language proportion (r:) increases, with the slope gradually decreasing and approaching linearity at higher proportions, as illustrated in Figures (a), (b), and (c).
Dependency on co-training languages. The specific set of co-training languages affects cross-lingual transfer primarily when the language proportion r3 is small, demonstrated in Figure 2 (a). This influence diminishes as r; grows.
Dependency on model language counts. Increasing the number of co-trained languages affects the intercept rather than the slope of the cross-lingual transfer relationship. This variation shifts the onset point at which transfer strength approaches linearity, shown in Figure 2 (b).
Dependency on data scale. Cross-lingual transfer consistently weakens with larger total token budgets (D), indicating that increased training data volume reduces dependency between languages, as depicted in Figure (c).
These patterns are consistent with prior findings the curse of multilinguality 3 10 report reduced transfer when auxiliary-language data dominates and when model capacity is spread across too many languages.
which similarly

---

2.3.2 Parametric Modeling of Cross-Lingual Interaction-aware Ratio
Motivated by the empirical insights described above, we propose parametric model to capture the relationship between the cross-lingual interaction-awe language ratio i; and the actual language ratio rj. Specifically, we model F; as:
=
Eaj-i(D) .ry (1 (1-e -::))
(4)
where the parameters are defined as follows:
(.j+s( (D) represents the transfer strength from language T.y to language I., Empirically, we find that this transfer effect diminishes linearly with increasing token budget D, which we model as: (D) = byz + 4. where bys indicates the initial strength of cross-lingual transfer from Lj. and kize quantifies the rate at which this transfer strength decays data volume increases. Details about (D) is in Appendix A
Ti captures the intrinsic data sufficiency language L, A larger value of yji indicates that language L, remains reliant cross-lingual transfer across wider range of proportions, exhibiting a pronounced curved signals that language L, quickly enters linear (self-dominated)) regime, reflecting sufficient self-contained data.
regime. Conversely, a smaller %
2.3.3 Complete Cross-
interaction-aware Scaling Law
By incorporating the parametric definition of the cross-lingual interaction-awan ratio into the monolingual scaling law, we obtain our final scaling law formulation:
B; E,
(D,
(5)
[D [D.A;] f.)
B, [D. (r:+ (Eogr(ss .r)) (1- - (I-e-mr))) )]
+ E,.
(6)
The complete set of parameters to estimate are: {Bi, Br, Edj (bye. Ays am 111=1, hm
Parameter Estimation Procedure. To fully determine these parameters, we perform targeted experiments involving each language individually. Specifically, for each language L,, we conduct three experiments with distinct proportions: one monolingual scenario (where r; 1) to estimate the baseline scaling law parameters B; Bi. and E,, and two additional multilingual experiments with randomly chosen language proportions r; and the remaining languages allocated equally as 1. Each of these experiments is repeated at two distinct training token budgets to ensure reliable parameter fitting across data scales. Following the experimental setup 30), we fit our scaling law parameters only using data points from the last 15% of training. Thus, for a setting with m languages, this structured approach requires a total of 3 x m x 2 experiments, enabling comprehensive and accurate estimation of the proposed scaling law parameters. The detailed fitting procedure is summarized in Algorithm
2.4 Estimating Optimal Multilingual Balanced Allocation
Directly minimizing the multilingual validation loss defined by Equation (6) is challenging, as forms non-convex optimization problem in language proportions r. While it may appear intuitive directly optimize the cross-lingual interaction-aware language ratios 7, under Equation (6. this objective is intractable in practice, as the total sum Σ, f remains unknown. To address this difficulty, we propose two-stage optimization procedure that decomposes the original complex problem into two simpler, sequential steps. Specifically, we first determine the optimal direction in the cross-lingual interaction-aware language ratios 7; space, ensuring balanced marginal benefits across languages. Subsequently, we optimize the magnitude along this determined direction to identify the final allocation that maximizes the overall cross-lingual interaction-awe language ratios 7, effectively minimizing the multilingual validation loss.
5

---

Algorithm CLIMB
Input: Languages [L,.... Lm token budgets (D D(2))
Output: Parameters [Be, Br, E, :=1: (bji, 1 1.j.jjz {y I optimal language proportions r".
Part I: Parameter Modeling of Cross-Lingusd Interaction-aware Language Ratio
1: for each language I, do
2: Conduct monolingual experiments (r; = 1) at D D
3: Fit monolingual scaling law 3 to estimate B; 3,. E,.
4:
for each proportion r; = c, E (0, 1). repeat twice do
5: Set other languages proportion r- m- L Vj *
6:
for each token budget D E {D(2), ( do
7:
Train model with proportions r and budget D. Record validation loss CV r).
8:
9:
Compute 7g from Eq. (6
10: end for
11: end for
12: Fit parameters byg. kyas TJ using (ra, P:) pairs.
13:
end for
Part II: Estimating Optimal Multilingual Balanced Allocation
14: Compute optimal direction components pi via Eq. 77)
Normalize direction: P: < pi/ Σ, Ps for all
16: Solve constrained optimization (Eq.
17: return parameters B, 3, Eg 1 (o kus M 11=11 (nd m and optimal proportions r".
2.4.1 Optimal Direction via Marginal- Bennfft Balancing.
In the first stage, we identify the optimal direction for the cross-lingual interaction-awant language ratios by balaneing the marginal benefits across all languages. Specifically, we derive the optimal proportional relationship between the interaction-ane -waare ratios by equalizing the marginal validation- loss reduction contributed by each language. The resulting optimal direction p; for each language L; is formally given by (see detailed derivation in Appendix B):
(w: 1/(,+ D Pe/(0.+1)
Pi
== (wr B.) 1 1/0x+1) D-
where B, and 83 are the monolingual scaling-law parameters of language L,, and wi represents the predefined importance weight for language 74. Intuitively, the direction p; indicates the ideal relative allocation of interaction-aware language ratios, balancing each language's 's data efficiency, validation-loss reduction rate, and relative importance. Identifying this optimal direction substantially reduces complexity in subsequent optimization steps by constraining the search space for the final language proportions.
2.4.2 Optimal Magnitude via Constrained Effective Allocation Maximization.
With the optimal direction p; identified, the second stage focuses on determining the optimal magni- tude along this direction. Nevertheless, due to the monotonicity of the scaling law function we find that a larger aggregate Σ; F; consistently implies lower overall training loss, thereby revealing an implicit preference for maximizing effective data contributions across languages (details in Appendix C.. Specifically, we recover the actual language proportions by solving a constrained optimization problem that maximizes the total cross- moss-lingual interaction-awee awaree language ratio while staying close to the previously determined direction p. Formally, this optimization objective is defined as:
min Σ Fr(x) + (m.())) p.)²
S.L. Er= rs r 2.0
(8)
where F; E, is the normalized interaction- ratios and direction components, respectively.
The first term of the objective function aims at maximizing the overall interaction-awant language ratio, corresponding directly to minimizing multilingual validation loss, while the second term (soft-
constraint) penalizes deviations from the optimal direction p. The hyperparameter 0 balances

---

Table 1: Fitting and Extrapolation Performance of Different Methods (Rt² and Huber Loss for Multilingual LLMs at 100B and IT Tokenn.
2 lang LLM
3-lang LLM
5-lang LLM
16-lang LLM
Huber + (x10 3)
Huber 4 (x10 3)
Huber 4 (x10
Huher J
Rº1
R²1
R
R
(x10 3)
Fitting Results (Total Training Tokens: 100B)
Isolated
0.649
7.95
0.743
5.35
0.734
5.34
0.768
5.26
MSL.
0.832
5.61
0.854
2.15
0.823
1.94
0,836
2.20)
CLIMB
0.978
0,518
0.986
0.301
0.992
0.205
0.981
0.274
Extrapolation Results (Total Training Tokens: ITT)
Isolated
0.648
8.21
0.741
5.38
0.732
5.36
0.767
5.30
MSL
0.830
5.79
0.852
2.24
0.822
1.98
0.834
2.24
CLIMB
0.964
0.525
0.947
0.310
0.948
0.208
0.936
0.278
3.4
4.7
09
0.2
04
03
Figure 3: Curve fitting Ara- bic data (5-lang LLM). Solid line: fitted results.
Figure 4: F vs. r across corpus
Figure 5: Validation loss fit- ting across language ratios (0.01-0.99).
scales 2 different languages (cs, ko).
these two objectives, with smaller values of emphasizing pure loss minimization and larger values enforcing adherence to the optimal direction.
This problem is inherently non-convex due to the interaction-aware ratio's nonlinearity. Thus, we adopt a Trust-Region Interior-Point Method efficiently handle this constrained optimization problem. This structured reformulation significantly reduces the complexity and dimensionality of the original allocation problem, accelcrating convergence and improving numerical stability.
3 Crosslingual Interaction-aware Language Ratio Evaluation
3.1 Experimental Setup
architecture with 1.2 billion parameters, training all models from scratch with randomly initialized weights. All experiments are condcuted on Nvidia H100 GPU cards. To ensure consistency with established scaling-law practices, we follow the Chinchilla configuration and set fixed number of training steps for each dataset size, adjusting the learning- rate decay schedule to cosine. Validation losses are calculated by averaging the results obtained from the final three training steps.
Model Architecture. We utilize the LLaMA-2 660
Datasets. All experiments are conducted using data sampled from the Fineweb-2 corpus 45 To
rigorously evaluate our Cross-Lingual Interaction-aware Language Ratio across diverse linguistic scenarios, conduct experiments involving models trained on 2, 3, 5, and 16 languages, respectively. For each multilingual setting, we vary the token budgets from 5 billion to 100 billion tokens. The specific language compositions and detailed training procedures are documented in the Appendix B
Evaluation Metrics & Baseline. We assess the accuracy of our validation-loss predictions primarily using two metrics: the coefficient of determination (R2) and the Huber loss. The R²2 score measures the proportion of variance explained by our fitted scaling-law model, with values closer to indicating

---

greater predictive accuracy. Additionally, we employ Huber loss, robust error metric combining properties of mean squared error and mean absolute error, which provides resilience against outliers; lower Huber loss values reflect more accurate predictions.
Baselines We compare our approach against two baselines: 1) an assumption of no cross- lins-ll transfer, where each language's 's validation loss depends solely on its own proportion, labeled as "isolated"; and 2) a recent multilingual scaling-lar study (28). referred as "MSL".
3.2 Scaling Law Fit Accuracy
We present the prediction errors of our proposed scaling law compared to the baseline (MSL) in Table[]] evaluating models trained across various multilingual scenarios (2-16 lang LLMs) with token budgets of 100 B and 1 tokens. Our cross-lingual interaction-aware approach consistently achieves lower prediction errors compared the baseline, effectively capturing validation-loss trends in both homogeneous (same tanguage-familyy and heterogeneous language settings. From 2-lang LLM to 16-lang LLM, CLIMB'S Huber loss remains consistently an order of magnitude lower than both baselines, highlighting the importance and prevalence of cross-lingudl transfer effects in multilingual models. Conversely, our scaling-lav formulation remains robust and delivers accurate loss predictions even in highly complex multilingual scenarios. We also tried different parametric models to fit and the results are shown Appendix
3.3 Scaling Law Applicability
To evaluate the robustness of our scaling law, we test its validity across varying training scales and language proportions (see Figure and Figure 4.. At smaller scales (below 25 B tokens). prediction accuracy is limited due to unstable cross-lingual transfer. As data increases beyond 25 B tokens, predictions align closely with empirical losses, and extrapolation up to T tokens remains consistent [T. Moreover, bilingual experiments (Figure confirm that our formulation accurately fits validation loss across the full range of language proportions (0.01-0.3 (..9), demonstrating its broad applicability to multilingual training.
4
Multilingual Balanced Allocation Performance
4.1 Experimental Setup
Model Architecture and Training Setup. We evaluate multilingual performance by training Transformer models based on the LLaMA-2 (0) architecture at two different scales: 1.2 B and 7 B parameters. All models are trained using the Fineweb-2 corpus, identical to the datasets employed in scaling-law experiments, with each model ingesting total of 1 T tokens. Peak learning rates are set to 4.3 x 10-5 for the 1.2 B model and 3.6 x 10 for the 7 B model, both following cosine-decay schedule that decays the learning rate down to 10% of its initial value.
Baselines. We compare our proposed CL.IB--drivee allccations against two categories of baselines. First, we evaluate against publicly available multilingual models, specifically, LLaMA-3.22 (25), GEMMA-3-IBB 1-ppt (21). -wwe-331..7--ble distributions are either open-sourced or reported in official documentation. These models serve as strong multilingual references trained on large
(58) and XGLM-1.7D (37), whose training data
dellmdocumented corpora.
Second, under identical model architecture and data volume constraints, we train models using several alternative language allocation strategies: (1) Uniform, which distributes tokens equally across all 16 languages; (2) Isolated, derived independently from individual monolingual scaling laws; (3) MSL, based on the existing multilingual scaling law formula assuming taguuaee-famil independence; (4) Natural, reflecting each language's 's original data frequency; (5) Temperature Sampling (Temp), which smooths token allocation via temperature-controllees reweighting of language proportions, we use = 0.3 as baseline; and (6) UniMax 8 which maximizes the minimum marginal gain across languages to improve balance under limited total tokens.
Evaluation Benchmarks. comprehensively evaluate CLIMB, we translate several English bench- marks into multilingual to further assess model performance; benchmarks translated by us marked
8

---

Table 2: Performance of CLIMB on IB models and baselines across 18 multilingual benchmarks. Benchmarks translated by us are marked with $. Bold numbers denote the best results among data allocation methods. Standard error is in Appendix !
Analytical Reasoning
Commonsense Reasoning
MGSM XARC-E'
XARC-C
XTQA
INCLUDE
XGPQA
XCOPA
XSC
XHS
Open Source Multilingual LLM
LLaMA-3.2-1B
3.89
47.19
29.55
37.73
28.48
24.91
57.07
58.13
41.52
Qwen3-1.7 1.7B
36,95
59.53
40.18
48.88
46.87
28.55
59.58
60.47
46.59
Gemma-331BBpt
1.78
49.29
29.69
40.07
25.63
27.32
55.58
54.84
41.59
XGLM-I.7B
L.89
40.17
24.60
38.42
25.96
23.31
56.73
36.18
Different Data Allocation Methods
Uniform
2.07
59.76
35.41
39.62
25.12
26.18
$9.49
$8.99
48.12
Isolated
2.11
34.78
39.71
24.82
24.58
59,42
Natural
2,05
56.32
33,54
40.63
25.20
26.37
56.54
57.38
45.68
MSL
2.10
57.60
34.17
39.49
25.00
36
$8.06
57.94
47.02
Temp
2.11
59.31
34,93
39.38
24.82
26.50
58.69
59.54
47.56
UniMax
2.07
59.39
35.78
39.67
25.12
27.10
59.35
58.99
48.22
CLIMB
2.40
60.45
36.56
40,94
25.92
59.98
48.75
Comprchension
Linguistic Competence
Knowledge
Translation
XNLI
Belebele
MuliBLiMP
XWinograd
GMMLU
CMMLU
JMMLU
VMLU
FLORES
Open Source Multilinuual
LLaMA-3,2-IB
41.29
30.69
77.66
76.08
28.74
30.30
29.84
29.09
44.14
Qwen3-1.72 1,7B
43,25
74.81
77.72
79.12
34.24
45.16
36.09
36.26
50.25
Gemma pt
XGLM-I.7BB
37.35
24.21
68.10
62.60
26.08
29.04
28.37
29.41
21.80
Different Data Allocation Methods
Uniform 40.08
23,30
62.24
73.34
29.05
34.79
32.47
31.44
47.51
Isolated
38.93
25.27
60.75
72.31
28.
33.78
31.85
30.91
47.58
Natural
39.05
24.11
63.18
74.98
30.23
32.10
30.94
31.12
48.54
MSL
38.54
24.55
61.94
73.09
29.00
33.24
31.49
30.25
47.75
Temp
41.03
25.27
60.75
74.76
29.04
33.03
31.98
30.27
47.46
UniMax
CLIMB
41.65
26.17
65,54
77.48
31.78
33.67
33.21
31.76
50.43
$ in Table 2 Specifically, we adopt the following tasks: analytical reasoning (MGSM 53 ARC-
Easy/ARC Challenge (XARC-E/C) (9. XTQA (Cross-lingual TruthfulQAj [36] INCLUDE (5],
XGPQA# (50), commonsense reasoning (XCOPA [46). XStoryCloze (XSC) 37). XHellaSwag
(XHS) 667 comprehension (XNLI [T Belebele (]). linguistic competence (MultiBLiMP [31,
XWinograd(XWG) 41)), and knowledge GMMLU (54), CMMLU (35), JMMLU VMLU and
translation (FLORES [57] Details are provided Appendix
4.2 Results
Table 2 reports the main results; per-language scores are in Appendix J With 1.2 B model trained on tokens, we remain competitive with LLaMA-3.22, Gemma-3, and Qwen-3. Against alternative allocations, CLIMB is consistently strong up to +2.60% on XNLI over isolated allocation and +1.85% on average across all tasks demonstrating the effectiveness of our multilingual allocation.
Generalization to Larger Models. Though our optimal language proportions were initially derived using a -araamemte model, the methodology generalizes effectively larger scales. We trained 7B- 1B-parameeter model using the same total token budget (IT tokens) and evaluated performance (Figure 6). CLIMe-derived allocations consistently outperform baselines by an average of 3.4%. Results for 7B models appear in Appendix G
Allocation Differences Across Methods. To illustrate differences among allocation strategies, Figure 7 compares language proportions of various methods. C1.MBB distinctly allocates higher proportions languages benefiting most from cross-lingual interactions, unlike baselines (Isolated, characteristics. This
MSL, Nature), which either allocate evenly or based solely focused allocation emphasizes the practical advantage of modeling cross-lingual transfer explicitly.
htttpe://githab..co//lll..p https://vmlu.cr
JMLLU

---

Taking
Figure 6: Different allocation re- Figure 7: Illustration of each lan- Figure 8: Illustration of how ra- sults on 7B model.
guage allocation methods.
tios varying with training data.
Table 3: Average cross-lingual transferability per language family. "Transfer-out" measures how much language benefits others, "Transfer-in" how much it benefits from others. Lang indicates the strongest transfer source.
Language
pt
Transfer-out 0.123
0.199
0.130
0.218
0.115
0.174
0.146
0. 1.181
0.124
0.146
0.121
0.139
0.144
0.108
0.066
0.144
Transfer-in
0.168
0.151
0.144
0.188
0.146
0.145
0.092
0. 139
0. 0.164
0.129
0.108
0.185
0.130
0.152
0.127
0.136
Lang
de
pt
Optimal Language Allocation Shifts with Data Scale. Figure presents how optimal language allocations evolve with increasing token budgets. smaller scales, simpler or less-resourced lan- guages initially receive higher allocations, quickly lowering validation loss. As data scale increases, allocations shift towards linguistically complex and diverse languages due to their sustained effective- ness at reducing loss. This dynamic trend highlights the necessity of adjusting language proportions according to looss-lingual transfer effects at varying training scales.
Per-Language and Per-Family Analysis. We analyze cross-lingual interaction ratios (transfer- out/in) derived from the learned scaling coefficients (Table to explain CLIMB'S 's allocations. Two Japanese-Korte Korean),
patterns are clear: strong -ntra-family transfer (e.g., and (ii) high-transfer languages (e.g.. English, Spanish. Indonesian) receive larger shares because their gains generalize broadly. These results improve the interpretability of CLIMB and support its data-driven balancing strategy.
5 Related Work
Data Allocation in Language Model Pretraining. Recent work optimizes pretraining mixtures at
the domain 32 17). point (63 (6), and token levels [38] (7). typically targeting validation loss. Early
approaches use GroupDRO-style reweighting (e.g., DoReMi (64)). while newer methods leverage
influence functions and surrogate models 33 65 gradient approximations (59 68), and loss-guided
heuristics [7], (6)] to refine mixtures under budget constraints. However, these techniques are largely monolingual (English), leaving multilingual data allocation comparatively underexplored.
Scaling Laws for Multilingual LMs. Scaling laws 33
40 47 reliably relate performance to model/data scale and guide efficient allocation. While early work monolingual, recent studies extend scaling to multilingual
30
maainng in NMT (3 22
[8
72
4
5 52 n -buut typically
under simplified bilingual assumptions and recoder--ecoder setups. Our formulation makes this
interaction explicit for multilingual pretraining.
6 Conclusion
This paper introduces CLIMB, a multilingual optimization framework that models cross-lingual interactions to predict optimal language allocations under scaling-law paradigm. Empirical results show that CLIMB attains strong predictive accuracy and consistently surpasses baselines at both 1.2B and 7B scales. Future work will extend its predictive capacity to languages unseen during training.
10

---

References
[1] Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar, editors. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749-775, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
[2] Gaëtan Caillaut, Mariam Nakhlé, Raheel Qader, Jingshu Liu, and Jean-Gabriel Barthélemy. Scaling laws of decoder-only models on the multilingual machine translation task. In Barry Haddow, Tom Koemi, Philipp Kochn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation, pages 1318-1331, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[3] Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. When multilinguality a curse? language modeling for 250 high- and Jow-resource languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074-4096, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[4] Liang Chen, Shuming Ma. Dongdong Zhang, Furu Wei, and Baobao Chang. On the pareto front of multilingual neural machine translation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
[5] William Chen, Jinchuan Tian, Yifan Peng, Brian Yan, Chao-Han Huck Yang, and Shinji Watanabe. Owls: Scaling laws for multilingual speech recognition and translation models, 2025.
[6] Zhixun Chen, Ping Guo, Wenhan Han, Yifan Zhang, Binbin Liu, Haobin Lin, Fengze Liu, Yan Zhao, Bingni Zhang, Taifeng Wang, Yin Zheng, and Meng Fang. Murating: A high quality data selecting approach multilingual large language model pretraining, 2025.
[7] Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. How do languages influence each other? studying cross-lingual data sharing during LM fine-tuning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 13244-13255 Singapore, December 2023. Association for Computational Linguistics.
[8] Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, and Noah Constant, Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. In The Eleventh International Conference on Learning Representations, 2023.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
[10] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online, July 2020. Association for Computational Linguistics.
[11] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium, October-Novembee 2018. Association for Computational Linguistics.
11

---

[12] Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang. A complete survey on Ilm-based ai chatbots, 2024.
[13] Dee.Seek-AI. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[14] DeepSeek-AI. Deepseek-v3 technical report, 2025.
[15] Jack W. Rae et al. Scaling language models: Methods, analysis & insights from training gopher, 2022.
[16] Fahim Faisal and Antonios Anastasopoulos. An efficient approach for studying cross-lingual transfer in multilingual language models. In Jonne Sälevä and Abraham Owodunni, editors, Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 45 42, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[17] Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: domain reweighting with generalization estimation. In Proceedings of the 41 International Conference Machine Learning, ICML'24. JMLR.org, 2024.
[18] Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. 2023.
[19] Team Gemma. Gemma 2: Improving open language models at a practical size, 2024.
[20] Team Gemma. Gemma: Open models based on gemini research and technology, 2024.
[21]
Team Gemma. Gemma 3 technical report, 2025.
[22] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In International Conference on Learning Representations, 2022.
[23] Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In MariiiFFanccnne Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 5915-5922, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[24] Sachin Goyal, Pratyush Maini, Zachary Chase Lipton, Aditi Raghunathan, and J Zico Kolter. The science of data filtering: Data curation cannot be compute agnostic. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
[25] Aaron Grattafiori and Abhimanyu Dubey et al. The Ilama 3 herd of models, 2024.
[26] Wenhan Han, Yifan Zhang. Zhixun Chen, Binbin Liu, Haobin Lin, Bingni Zhang, Taifeng Wang. Mykola Pechenizkiy, Meng Fang, and Yin Zheng. Mubench: Assessment of multilingual capabilities of large language models across 61 languages, 2025.
[27] Nan He, Weichen Xiong, Hanwen Liu, Yi Liao, Lei Ding, Kai Zhang, Guohua Tang, Xiao Han, and Yang Wei. SoftDedup: an efficient data reweighting method for speeding up language model pre-training. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 4011 Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[28) Yifei He, Alon Benhaim, Barun Patra, Praneetha Vaddamanu, Sanchit Ahuja, Parul Chopra, Vishrav Chaudhary, Han Zhao, and Xia Song. Scaling laws for multilingual language models, 2024.
[29] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, 2020.
12

---

[30] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Fric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
[31] Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. Multiblimp 1.0: A massively multilingual benchmark of linguistic minimal pairs, 2025.
[32] Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: Scale- data mixing for pre-training llms, 2025.
[33] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[34] Wen Lai, Mohsen Mesgar, and Alexander Fraser. LLMs beyond English: Scaling the multilin- gual capability of LLMs with cross-lingual feedback. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 8186-8213, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[35] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. In Lei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 11260-11283 Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[36] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 3214-3252 Dublin, Ireland, May 2022. Association for Computational Linguistics.
[37] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang. Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019-9052, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[38] Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Not all tokens are what you need for pretraining. In A. Globerson, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 29029-29063. Curran Associates, Inc., 2024.
[39] Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. In The Thirteenth International Conference on Learning Representations, 2025.
[40] Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michat Krutul, Szy- mon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóždú, Piotr Sankowski, Marek Cygan, and Sebastian Jaszezur. Scaling laws for fine-grained mixture of experts. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 33270-33288 PMLR, 21-27 Jul 2024.
13

---

[41] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf. Xiangru Tang. Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Wehson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetun- ing. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( pages 15991-16111, Toronto, Canada, July 2023, Association for Computational Linguistics.
(Volume 1: Long Papers),
[42] OpenAI. Gpt-4 technical report, 2024.
[43] OpenAl. Gpt-4o system card, 2024.
[44] Guilherme Penedo, Hynek Kydlńček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. Fineweb2: sparkling update with 1000s of languages, December 2024.
[45] Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi. Leandro Von Werra, and Thomas Wolf. Fineweb2: One pipeline to scale them all adapting pre-training data processing to every language, 2025.
[46] Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376, Online, November 2020. Association for Computational Linguistics.
[47] Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. Scaling laws of synthetic data for language models, 2025.
[48] Qwen. Qwen technical report, 2023.
[49] Qwen. Qwen2.5 technical report, 2025.
[50] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level mrogle-proor q&a benchmark. In First Conference on Language Modeling, 2024.
[51] Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed A. Haggag, Snegha A, Al- fonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemińssii, Gabriel Adriano de Mclo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia soltani moakhar, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, and Antoine Bosselut. INCLUDE: Evaluating multilingual lan- guage understanding with regional knowledge. In The Thirteenth International Conference on Learning Representations, 2025.
[52] Anton Schäfer, Shauli Ravfogel, Thomas Hofmann, Tiago Pimentel, and Imanol Schlag. The role of language imbalance in cross lonssllingual generalisation: Insights from cloned language experiments, 2024.
[53] Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay. Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual offin-of--hhughh reasoners. In The Eleventh International Conference Learning Representations, 2023.
14

---

[54] Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani. Jian Gang Ngui, Daniel Vila-Suero. Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei- Ko, Sebastian Ruder, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation, 2025.
[55] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
[56] Daouda Sow, Herbert Woisetschläger, Saikiran Bulusu, Shiqiang Wang. Hans Amo Jacobsen, and Yingbin Liang. Dynamic loss- based sample reweighting for improved large language model pretraining. In The Thirteenth International Conference on Learning Representations, 2025.
[57]
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safyyya Saleem, Holger Schwenk, and Jeff Wang. No language Scaling numan-centered machine translation, 2022.
left behind:
[58]
Qwen Team. Qwen3 technical report, 2025.
[59] Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2033 2045, Singapore, December 2023. Association for Computational Linguistics.
[60] Hugo Touvron and Louis Martin al. Llama 2: Open foundation and fine-tuned chat models, 2023.
[61] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[62] Dominik Wagner, Alexander Churchill, Siddharth Sigtia, and Erik Marchi. Selma: A speech- enabled language model for virtual assistant interactions. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2025.
[63] Jiachen T. Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for Ilm training in every iteration. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 131197-131223. Curran Associates, Inc., 2024.
[64] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le. Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36. pages 69798-69885 Curran Associates, Inc., 2023.
[65] Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. The Thirteenth International Conference Learning Representations, 2025.
15

---

[66] Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. In A. Globerson, L. Mackey, Belgrave. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 108735-108759, Curran Associates, Inc., 2024.
[67] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Marquez, edi- tors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791444999991 Florence, Italy, July 2019. Association for Computational Linguistics.
[68] Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Qiu Jiantao, Lei Cao, Ju Fan, Ye Yuan, Guoren Wang, and Conghui He. Harnessing diversity for important data selection in pretraining large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[69] Shaolin Zhu, Supryadi, Shaoyang Xu, Haoran Sun, Leiyu Pan, Menglong Cui, Jiangeun Du, Renren Jin, António Branco, and Deyi Xiong. Multilingual large language models: systematic survey, 2024.
[70] Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages Mexico, June 2024. Association for Computational Linguistics.
Mexico City,
[71] Xiaoxuan Zhu, Zhouhong Gu, Baiqian Wu, Suhang Zheng, Tao Wang, Tianyu Li, Hongwei Feng. and Yanghua Xiao. Toremi: Topic-aware data reweighting for dynamic pre-training data selection, 2025.
[72] Zhang Zhuocheng, Shuhao Gu, Min Zhang, and Yang Feng. Scaling law for document neural machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2290-8303, Singapore, December 2023. Association for Computational Linguistics.
16

---

A Details of Fitting Transfer Strength az-i
100
(a) Caar-ten function of D
(b) Oarr-kk function of
(c) Car-+sh function of D
Figure 9: Illustration of cross longull interaction- aware language ratio (Far) and its dependency
original training proportions (Far).
As introduced in Figure 2 (c) the main text, we observe that the curve relating i; and r; shifts vertically depending the total token budget D. Specifically, as D increases, the F; versus r; curve tends to move downward, while smaller D values correspond to upward shifts. According to Equation (4), the parameter (D) effectively acts an intercept controlling this vertical shift.
To accurately characterize the relationship between a' and the data budget D, we adopt a two-step procedure. First, individually fit the relationship between F; and at different values of D using Equation (4). This yields empirical estimates of a at various token budgets. Figure 9 iillustrates the computed values of aj-i for three representative language pairs across different scales of D.
Moreover, our empirical findings suggest two eritical properties for the a: relationship:
Non-monotonice aj-s does not continuously decrease with increasing D; rather, it converges towards a stable limiting value as D) becomes sufficiently large.
Sign variability: ay-i can be either positive or negative. Positive values indicate beneficial cross-lingual transfer, whereas negative values reflect interference effects, where additional data from language Ly eventually hinder the learning of language L,.
Considering these empirical insights, we propose modeling (D) with the following parametric form:
kjs D
a)-)(D) = bوs +
where by: represents the asymptotic transfer strength as D - xo, and ky controls the decay rate of this transfer effect as the data budget increases.
The fitting results using this parametric form, depicted by the green curves in Figure 9 demonstrate excellent agreement with the empirical aj-x-- relationships across various language pairs, validating our choice of functional form.
B Derivation of Optimal Direction for Cross-Lingual Interaction-Aware Ratios Pa
To compute the optimal direction of the Cross-Lingual Interaction-Aware Ratios (f,). we formulate and solve the following uncoupled optimization subproblem:
B
min
s.t.
Σ A M.
(10)
D
where M is fixed normalization constant, and B;, B,, D are known positive parameters.
Introducing a Lagrange multiplier X, we construct the Lagrangian:
B
J(r
Σ
+
Σ r. M
(1))
Di.
1=1
17

---

Taking derivatives with respect to each i; and setting them to zero, we obtain the first-order optimality conditions:
B,, D 8: r3
(B.+1)
A
(12)
B, 3
B
(13)
AD
Comparing the conditions for any two languages i, have:
B, B: D&5-8)
(14)
fj ByB, Bg
Thus, the optimal direction must satisfy:
is α (B. 1/(0+11)
(15)
Applying the normalization constraint
E, Fr M, we obtain the normalized optimal direction:
(B, (B, D D-B((B.-1)
P:
kk=1 (Br Br 1/(0+++++1) D-
(16)
Since each term (DF;)) is strictly convex in f; and the constraint is linear, the stationary solution derived above constitutes the unique global minimizer. This rigorous derivation justifies the Marginal- Benefit Balancing approach presented in the main text, providing the closed-fome solution for the optimal direction }
C Equivalence of Two-Stage Optimization with Direct Optimization
Here we provide rigorous justification demonstrating that our proposed two-stage optimization approach- first determining the optimal direction p: and subsequently maximizing the magnitude of effective data allocation-is equivalent to directly solving the original optimization problem.
(i) Necessity of Optimizing the Direction: Assume the direction of the cross-lingual interaction- aware ratios {F:} deviates from the optimal direction p;. Under any fixed effective data contribution
>; (D((D;;) the total validation loss will always be greater than or equal to that obtained using the optimal direction. Formally, the optimal direction condition is:
BB DR.
BBB, DA
Vi,j.
(17)
Any deviation from this balanced proportionality condition disrupts marginal equilibrium, causing certain languages to have unnecessarily higher marginal loss reductions, thus reducing overall efficiency. Hence, identifying the direction (rr} by balancing marginal benefits ensures minimal total loss given fixed effective data contribution.
(ii) Optimal Magnitude via Maximizing Effective Allocation: Once the optimal direction pa is fixed, we set F; = c. C.Ps. where e denotes the scaling magnitude of effective data allocation (with normalization Σ, F; = c). We then isolate the variable component of total loss as a function of
B
B
Logar (c) Σ (Depi)
Σ
(18)
D 3
Differentiating with respect to e, we have:
dLvar
- de
c ((.+1) < 0, Dotp?
3,B;
(19)
provided that all (3 > 0. This negative derivative demonstrates strictly monotonic decrease in loss as the magnitude e increases. Intuitively, larger e means greater effective data volumes Dř; for each
language, which consistently reduces loss due to the monotonicity of scaling laws. Therefore, to
18

---

Table Comparison of simplified models for fitting validation loss, Huber loss scaled by 10-3
Best values are in bold.
ID
Model Type
#Parameters
Huber Loss 4 (x10-x) -3
R²
1
f; a;T;;
1
15.32
0.236
2
T = airi + E, (,(Drr,,
+ 2 x (m 1)
1.44
0,563
3
ř; = a + D (D()) +b,
3+3 3 x (m 1)
0.474
0.978
4
CLIMB
3 x (m 1)
0.274
0.981
minimize the loss, we naturally aim to increase as much as feasible-maximizingg the total effective data contribution while maintaining the optimal relative proportions.
However, practical constraints limit the maximum achievable c. Given the normalization constraint Σr; 1 and the implicit mapping from {ra} to {f:). the magnitude has an upper bound e* corresponding to feasible allocations.
In summary, stage guarantees that adjusting the direction of ratios does not increase the loss, and stage optimally maximizes effective data volume along this direction, ensuring minimal achievable loss. Thus, the two-stage solution is proven equivalent to directly solving the original optimization problem. This result aligns with previous studies on multilingual scaling laws, demonstrating the consistency and optimality of the two-stage optimization procedure.
D Different Fitting A Attempts
To justify our parametric choice, we compare CLIMB with simpler surrogates that fit validation loss using progressively richer transfer terms (Table 4p. A linear variant with explicit cross-language sums (Model 2) reduces error versus a pure scaling baseline (Model 1), and adding exponents (Model 3) further helps. However, CLIMB achieves the lowest Huber loss and the highest R with fewer comparable parameters, indicating better balance of compactness and expressiveness.
E
Training Details
Dataset Description
All experiments utilize data sampled from the Fineweb-2 corpus [44]. further preprocess the dataset by training custom Byte-Pair Encoding (BPE) tokenizer using the BBPE method, resulting vocabulary 250k tokens for subsequent experiments.
Experimental Setup
We conduct multilingual experiments with various language combinations:
Bilingual Experiments: (es-ko, en-zh, de-ar, ko-ja)
Trilingual Experiments: (es-de-ar, es-ko-zh, en-zh-ja
Five-language Experiment: des-de-ar-kojjal
Sixteen-language Experiment: (de, en, nl, es, fr, it, id, ja, ko, zh, ru, ar, th, vi, tr)
As detailed in Algorithm for each multilingual setting, we first fix the proportion of language and evenly distribute the remaining proportion among the other languages. For each selected language L, we systematically vary its proportion across the set [0.02. 0.025, 0.05, 0.1, 0.2, 0.25, 0.4, 0.5. 0.6. 0.75, 0.8, 0.9, 0.95, 0.975, 0.98) to establish comprehensive fitting functions. In the sixteen-langueee experiment, follow Algorithm for extrapolation and validation.
Model Configuration
We adopt mansformer-based architecture inspired by the LLaMA-2 (60) model, specifically config- ured with approximately 1.2 billion parameters. The detailed architecture settings are:
Hidden size: 2048
19

---

Vocabulary embedding dimension: 2048
Intermediate layer dimension: 5504
Attention heads: 16
Layers: 24
Maximum positional embeddings: 4096
Layer normalization epsilon: 1.0 10
All models are randomly initialized.
Training Hyperparameters
Batch size: 3072
Sequence length: 4096
Optimizer: AdamW
Learning rate schedule: Cosine decay 10% of initial value
Training steps: Varied according to total token budget
Precision: bf16 (mixed--recission training)
Computational Resources and Runtime
Each experiment is conducted using 64 H100 GPUs, with an average runtime of approximately 10 hours per experiment.
Evaluation Methodology
The validation datasets for each language separately sampled from Fineweb-2, ensuring overlap with training samples. Validation loss is computed by averaging the loss across the final three training steps of cach run.
Detailed Evaluation Protocols for Benchmarks
To rigorously assess the capabilities of our proposed model, we select benchmarks that span diverse evaluation dimensions, including natural language inference, commonsense reasoning, question answering. multilingual multitask understanding, and translation tasks. Recognizing that several benchmarks were originally developed only in English, we manually translated these datasets into multilingual versions (marked as t; XHS*, XARC-E', XARC-C*, XGPQA3, XTQA*). Details about how we translate the benchmarks are listed in MuBench 26 benchmark grouped by task type.
Below, we detail each evaluation
Language Modeling and Natural Language Inference
XNLI (Cross-lingual Natural Language Inference) [1I]: Extended from MultiNLI, XNLI evaluates cross-lingual sentence representations across 15 languages, measuring models' inference capabilities.
XCOPA (Cross-lingual Choice of Plausible Alternatives) [6]] XCOPA tests models on causal commonsense reasoning across languages, providing insights into multilingual causal reasoning capabilities.
XStoryCloze assesses zero-shot and few-shot learning across 10 non-English languages, examining models' narrative understanding and inference skills.
XStoryCloze 37
Commonsense Reasoning
Originally English-only, HellaSwag involves selecting the most plausible sentence ending from multiple choices, thereby testing commonsense reasoning.
HellaSwag (XHS*
: As multilingual variant of the Winograd Schema Challenge, XWinograd evaluates pronoun resolution abilities in diverse linguistic contexts.
XWinograd 41:
Question Anserring
20

---

(XARC-C9) 1 ARC contains scientific multiple-chore choice questions designed for different complexity levels, evaluating reasoning from basic advanced.
ARC-Easy (XARC-EF)
GPQA (Graduate-Level Google-Proof Q&A, XGPQAi) [50] GPQA tests graduate-level understand- ing across domains like biology, physics, and chemistry, requiring deep comprehension beyond serecheengin--baeee answers.
TruthfulQA (XTQA*) [36]: This dataset assesses the factual accuracy and common misconception avoidance language models across diverse topics.
Multitask Language Understanding (MMLU Series)
CMMLU (Chinese Massive Multitask Language Understanding) B5
Evaluates Chinese language models' knowledge across multiple disciplines including natural sciences, engineering, and humani- ties.
JMMLU (Japanese Massive Multitask Language Understanding) f JMMLU assesses Japanese models on multitask language understanding, covering extensive topics.
VMLU (Vietnamese Massive Language Understanding) Focused on Vietnamese, VMLU evaluates broad academic and practical knowledge via a large set of multiple-choice questions.
GMMLU (Global Massive Multitask Language Understanding) [54]: GMMLU tests multilingual generalization capabilities across various languages and diverse tasks.
Translation Tasks
FLORES (Facebook Low Resource Languages Evaluation Suite) [57]: Supporting many-to-many translations, FLORES provides a high-quality benchmark suitable for assessing model performance on low-resource languages.
G Performance of CLIMB on 7B models
As shown in Table 5| CLIMB consistently achieves strong multilingual performance under 7B model architecture. Compared with heuristic allocation strategies such as Temperature Sampling and UniMax, CLIMB yields higher average scores across most benchmarks, particularly reasoning- oriented tasks (MGSM, MultiBLiMP) and commonsense datasets (XARC, XHS). While large open models like Qwen3-8B exhibit overall stronger results due to larger pretraining corpora, CLIMB narrows the gap despite being trained with comparable data volume, demonstrating the effectiveness of its interaction-aware data allocation in scaling multilingual models efficiently.
H Scaling to 300+ Languages.
To handle massive multilinguality, we extend CLIMBB from the 16-language setup to a family-level configuration (CLIMB-300+), where allocation is performed over language families rather than individual languages. This design allows us to maintain efficiency and stability when scaling to hundreds of languages.
We adopt the Fine corpus and filter out language families containing fewer than 100B tokens to reduce noise from extremely low-resouree groups, resulting in 58 retained families covering over 300 languages in total. As shown in Table the validation loss under CLIMB-300+ (3.12) is notably lower than that of temperature sampling (3.24), indicating better multilingual generalization. Results from the 1.2B model further show that CL.MB--30-- consistently outperforms heuristic temperature sampling across most benchmarks, even under this broader and more challenging setting. Moreover, despite the inherent disadvantage direct comparison with open-source multilingual models that are often fine-tuned on hunguage-speciffe data, CLIMB-300+ achieves competitive superior results on multiple benchmarks, highlighting the robustness and scalability our approach.
It is worth noting that these benchmark results serve primarily reference points, since not all benchmarks provide balanced coverage of the full 300+ languages included in our training. We plan
httt::://gichuub.com//l---- https://vmlu ai/
21

---

Table 5: Performance of CLIMB on 7B models and baselines across 18 multilingual benchmarks. Benchmarks translated by us are marked with $, Bold numbers denote the best results among data allocation methods.
Analytical Reasoning
Commonsense Reasoning
MGSM XARC-E
XARC-Ct
XTQA
Include
XGPOAT
XCOPA
XSC
XHS
Open Source Multilingual LLM
Qwenn-8BBBaan
60.51
74.13
54.42
43.90
59.45
30.23
67.02
66.97
59.98
XGL M-7.5B
0.69
48.23
28.79
32.13
25.58
24.64
61.40
60.97
43.86
Different Data Allocation Methods
Temp UniMax CLIMB
9.93
66.32
42.70
43,97
29.64
27.40
61.73
64.44
57.12
9.67
67.75
44.28
42.41
30.76
29.03
62.16
64.88
57.88
12.29
69.90
46.78
42.86
63.44
65.07
Comprehension
Linguistic Competence
Knowledge
Translation
XNLI
Belehele
MultBLiMP XWinograd
GMMLU
CMMLU
JMMLU
VLMU
FLORES
Open Source Multilingual LLM
Qwen3-8B-Base
46.85
87.99
71.18
86.49
42.70 27.78
56.38 32.26
47.15
47.59
56.32
XGLM-7. 7.5B
41.59
24.27
71.76
75.99
29.84
31.98
27.15
Different Data Allocation Methods
Temp UniMax
44.86
42.19
71.14
74.76
33.51
37.33
36,78
36.68
55.15
40.90
71.10
74.94
33.97
37.25
37.19
36.47
CLIMB
43.65
41.72
72.67
77.48
34.98
37.83
40.54
35.40
56.47
Table 6: Performance of CLIMB-300+ and baselines across representative multilingual benchmarks and validation loss. Validation loss (Val. Loss) reflects the average across all language families.
Model
Val. Loss
Include
MGSM
Belebele
MPP
XNI.I
XCOPA
XSC
Flores
GIMELII
CMMEIU
JMMII
VML.I
XWG
Temp-700+
3.24
24.82
2.11
25227
68.25
42.03
90.29
59.54
40
70.04
78,00
29.5
30.30
74.76
CL1Mn-1004
A12
25,35
2.22
27.32
73.21
42.91
08.04
60.12
$0.47
32,57
33,03
31,98
32,27
76.78
to extend our evaluation suite and continue exploring large-scale multilingual balancing in future work.
I Standard Errors and Significance Testing.
To quantify the reliability of our improvements, we report standard errors (stderr) computed from the evaluation harness in Table ??. The results show that the standard errors remain consistently small (mostly below 0.01), suggesting stable and reproducible performance across benchmarks. To further assess statistical significance, both 1B2 and 7B settings. Although the full significance table is omitted for brevity, we observe that all tests yield p-values greater than 0.05, indicating no spurious effects unstable improvements. Together, these results confirm that CLIMB'S s gains are statistically robust and not driven by random variance.
conducted paired t-tests between CLIMB and each baseline under
Detailed Per-Language Benchmark Results
This appendix presents detailed, per-language evaluation results corresponding the benchmarks summarized in Table 2. The following tables comprehensively report the performance of our CLIMB- derived multilingual allocation strategy across each evaluated language, facilitating an in-depth analysis and comparison against baseline methods.
K
Limitations and Future Work
While our experiments demonstrate strong performance using the proposed multilingual allocation strategy based on scaling laws, several limitations should be acknowledged. First, our parametric billion-parameter fitting and allocation strategies are primarily validated on 1.2 billion parameter (1.2B) model, and although Section 4.2 indicates robust performance at a larger scale (7B), explicitly incorporating model size (N) into the allocation optimization could potentially yield even more optimal data
22

---

Table 7: Standard errors (stderr) and significance testing results between CLIMB and baselines across multilingual benchmarks. l'indicates statistically significant improvement (p < 0.05) based on paired t-tests.
Table R3.3: Standard Errors (IB2 Models)
Model
MGSM
Belchele
MoliiBI iMP
XNLI
XCOPA
XSC
Flores
GMMLU
Uniform
2.11.10.003
26.17.10.003
62.24110.001
10.00
59.491.0.00
58.99.100
47.51
29.05-10.001
CLIMB
2.40:00003
26.17-+0.03
65.54
41.65=000033
59.98+0.007
60.54=0.0007
50.43-100066
31.78=0.001
Model
CMMLU
JMMLU
VLMU
XWG
XHS
XARC-F
XARC-C
XTQA
Uniform 0.004
73.34.0.007
59.76.10.00
35.41
39
CLIMB
3.067
33.21:10.005
31.76:10.015
77.48
48.75
60.45::0.002
36.56.10.003
40.94
Table 8: Detailed per-language performance on the XWinograd benchmark (5-shot accuracy). Bold numbers denote the best results among data allocation methods.
Model Method
EN
FR
JP
PT
RU
ZH
Open Source Multilingual LLMs
LLaMA-3.2
93.65
71.25
67.17
72.09
73.75
77.13
Qwen-3
92.54
76.61
78.49
77.12
69.51
80.69
Gemma-3
77.60
65.56
62.95
62.86
64.29
68.38
Different Data Allocation Methods
Uniform
82.93
72.45
71.39
73.80
67.89
71.58
Isolated
79.79
77.71
71.44
68.59
65.58
70.74
Natural
82.90
76.28
71.19
74.02
68.71
76.77
MSI. 82.14
73.84
69.90
71.56
67.04
74.06
CLIMB 90.57
78.14
74.27
74.98
73.66
73.25
distributions. Exploring how scaling laws evolve explicitly with both dataset size (D) and model scale (N) thus remains an open area for future research.
Secondly, our current methodology exclusively considers cross-lingual transfer between languages included within the training dataset. An important and intriguing direction for future work involves extending our approach to account for potential transfer effects to and from languages not directly represented in the training set. Such an extension would enable more comprehensive and strategically informed allocation decisions, optimizing not just for immediate languages but also for broader linguistic coverage and potential downstream adaptability.
Table 9: Detailed per-language performance on the IStoryCloze benchmark (0-shot accuracy). Bold numbers denote the best results among data allocation methods.
Model / Method
AR
EN
ES
EU
HI
ID
MY
RU
SW
TE
ZH
Open Source Multilingual LLMs
I.L.MA3.22
52.99
73.18
63.20
51.77
57.81
60.26
50.74
61.94
52.12
56.29
59.57
Qwen-3
56.96
74.71
65.52
53.32
58.07
62.47
53.32
63.26
51.65
60.33
66.00
Gemma-3
51.94
62.49
57.01
52.74
54.69
54.63
50.73
55.35
51.87
56.61
55.18
Different Data Allocation Methods
Uniform
60.45
70.35
66.44
53.01
50.31
65.22
49.98
65.25
51.19
54,91
61.76
Isolated
59.87
71.34
64.97
52.27
50.82
63.92
50.25
65.87
51.06
54.67
61.09
Natural
59.19
67.96
62.06
51.26
52.19
61.62
49.82
61.33
50.19
54.31
61.24
MSL
60.19
69.16
63.30
51.42
52.59
62.49
49.30
62.21
50.13
54.51
62.04
CLIMB
62.39
73.09
66.22
53.74
55.59
64.49
51.11
66.20
52.57
58.10
62.43
23

---

Table 10: Detailed per-language performance on the XCOPA benchmark (5-shot accuracy). Bold
numbers denote the best results among data allocation methods.
Model Method
ET
HT
ID
IT
QU
SW
TA
TH
TR
VI
ZH
Open Source Multilingual LLMs
LLaMA-3.2
52.09
52.31
62.69
62.49
51.51
51.31
55.08
55,90
55.90
64.68
64,47
Qwen-3 52.57
53.17
66.63
65.07
49.77
53.17
54.38
57.81
57.81
70.03
74.64
Gemma-3 51.99
52.77
60.18
56.59
52.20
55.21
55.62
54.19
55.62
59.79
57.99
Different Data Allocation Methods
Uniform
49.59
67.99
66.99
51.63
51.63
56.46
61.05
61.26
69.59
67.20
Isolated
49.86
70.62
64.80
50.60
50.21
56.60
61.78
61.78
69.94
65.77
Natural
50.76
51.48
64.31
59.09
49.97
51.85
54.44
58.76
58.49
62.85
59.93
MSL.
52.32
52.77
66.29
61.15
51.18
52.88
55.45
59.54
60.00
64.75
62.33
CLIMB
54.21
53.80
68.06
63.89
52.18
54.12
56.73
60.95
61.50
67.46
66.90
Table 11: Detailed per-language performance on the XNLI benchmark (5-shot accuracy). Bold
numbers denote the best results among data allocation methods.
M Method
AR
DE
EN
ES
FR
RU
TH
TR
VI
ZH
Open Source Multilingual L.MMs
LLaMA-3.2
34.05
42.16
46.15
40.41
42.20
40.48
38.41
39.90
39.90
39.86
Qwen-3
33.83
42.38
47.43
43.58
43.58
42.38
39.70
37.44
41.10
41.90
Gemma-3
38.94
41.35
44.81
41.53
41.92
41.92
39.74
40.18
42.28
41.03
Different Data Allocation Methods
Uniform
32.68
43.75
44.37
41.39
43.95
40.41
37.67
41.81
36.45
38.31
Isolated
31.15
40.95
42.76
40.16
42.84
40.22
36.53
41.60
35.64
37.46
Natural
34.26
40.61
43.19
40.23
41.53
39.40
37.50
39.58
35.74
38.46
MSI.
32.88
39.57
43.00
40.23
40.95
38.88
37.62
38.66
35.47
38.14
CLIMB
35.14
43.01
48.18
43.93
44.41
42.72
40.87
41.76
38.92
37.56
Table 12: Detailed per-language performance the Global MMLU (GMMLU) benchmark (5-shot accuracy). Bold numbers denote the best results among data allocation methods.
Model/ Method
FII.
FR
ID
II
JA
KO
TR
VI
ZH
Open Source Multilingual LLMs
1LMMA-322
25.88
29.12
35.30
29.31
28.05
28.84
28. 59
28.54
27 27.58
27. 27.90
28.33
28.11
29. 16
27.21
28.39
29.
3wen-3
29.62
34.79
43992
35.77
31.23
68
33.94
34.85
31.75
32.21
32.15
33.23
35.74
31.04
33.63
37.94
3
27.15
27.05
Different Data Allocation Methods
Uniform
27.56
29.78
31.30
29.81
25.64
29.85
29.76
29.37
28.45
28.77
28.51
28.88
30.18
28.75
28.99
29.20
Isolated
Natural
28.84
31.18
33.38
31.83
27.15
31.09
31.11
30.51
29.43
28.64
28.31
30.25
31.47
29.81
29.72
30.96
MSL
25.00
29.93
32.47
30.55
26. 46
30.13
29.30
28.84
27.51
27.38
26.69
29.01
30.47
28.30
28.58
29.60
CLIMB
30.53
32.91
36.26
33,85
28.86
33.95
33,04
32.11
30.70
30,33
29.68
31.48
33.92
30.79
31.16
28.91
24

---

Table 13: Detailed per-language performance the FLORES Translation benchmark (5-shot
chrF++ scores). Bold numbers denote the best results among data allocation methods.
Translation English (xx-en)
AR
1A
T1.
7H
1LMA-332
46.47
57.15
53.57
30.51
39.30
50.82
42.56
44.
Qwen-3
55.40
61.66
54.54
62. 48
58.90
56.20
48.68
48.82
58.09
53.48
64.10
55.65
49.75
51.98
51.39
54.26
52.12
Gemmar3
43.14
53.40
38.32
49.88
00.80
36.72
36.66
25.78
42.00
43.14
52.78
45.00
35.25
37.90
38.32
38,05
$0.29
Uniform
55.37
61.04
54.87
61.25
59.21
56.23
45
46.07
57.07
54.38
66.10
54,46
48.91
30.00
51.28
53.56
46.38
Isolated
55.57
60.91
54.34
61.77
59.62
55.73
45
56
46.43
57.82
$4.92
64.74
54.64
49.19
20.68
51.81
53.53
38
Natural
56.99
47.57
63:65
63.43
49.32
66.66
Model Method
Translation from English (en-xx)
AR
D0
FR
113
IT'
1A
KO
MS
NL.
PT
RU
TH
T1.
TR
V1
73
LLaMA-3.2
23.93
49.67
47.56
55.84
52.44
45.92
19.61
17.24
47.10
46.11
57.57
42.05
25.85
30.45
33.82
45.60
20.53
Owen-3
36,82
54,06
50.36
61.53
59.39
50.22
26.09
23.64
52.19
46.81
62. 44
47.53
35.09
37.32
39:47
53.24
30.23
Gemma-3
24.68
37.67
34.94
49.05
43.39
33.38
16. 16.18
14.72
38.58
32.05
49.51
32.01
24.07
25.38
28.95
36.45
19.52
Uniform
42.48
31.98
67.30
57.92
60.45
67.63
23.59
24.38
54.57
59.52
44.10
35.14
8.21
43.24
52,00
20.95
Isllund
41.99
52.49
47.94
24.12
24.0%
59.31
34.49
9.38
21.38
59.13
61.07
24.50
21.81
42.71
52.35
59.67
34.61
53.91
11.78
43.36
22.01
CLIMS
45.03
55.30
50.28
61.43
63.18
50.64
26.59
26.66
36.94
51.92
62.14
46.58
37.75
13:45
46.16
54.82
22.65
Table 14: Detailed per-language performance on the ARC-Challenge benchmark (25-shot accuracy). Bold numbers denote the best results among data allocation methods.
Model/ Method
DE
ER
MS
PT
RU
TH
VI
Z11
Source Multilingual LLMss
3.M
30.93
35.42
30.76
30.45
Owen
42.03
$4.39
41.31
37.58
32.77
37.52
Gomma-3
Different
Uniform
35.60
35.34
31.71
34.71
Natuaa
30.05
33.45
Table 15: Detailed per-language performance on the ARC-Easy benchmark (25-shot accuracy). Bold numbers denote the best results among data allocation methods.
Model Method
AR
DE
TIN
E5
FR
1D
πT
JA
KD
MS
NL
PT
RU
TA
TH1
TR
V1
2H
Open Source Multiinguua LLMSs
LLaMA 3.2
39.15
50.09
70.21
54.98
52.33
15.35
51.18
40.29
41.26
45.43
$3.09
52.19
18 19.56
35.74
37.63
42.37
46.95
49.40
Qwen:3
49.25
62.56
80.14
67.58
66.09
60.14
62.73
$3.92
52.27
$1.49
$4.70
65.11
60.84
41.24
46.35
49.35
$7.51
69.64
Gomma-3
41.08
49.67
7080
55.19
53.55
50.50
54.01
48.37
47.44
43.70
43.02
$3.49
47.02
39.15
39.61
44.08
46.15
55.31
Different Data Allocation Methods
Unifeam
56.70
62.68
70.03
67.27
63.81
64.28
64.07
58.97
55.01
57.31
62.59
66.34
60.24
20.64
49 50
60.20
58.76
63.86
Iodnee
55.12
62.07
69:03
65.59
63.45
63.74
61.73
56.77
56.98
56.93
61.73
63.21
58.62
30.29
48.31
59.46
56.56
63.08
Naturil
$3.85
59.50
67.85
63.25
61.68
61.15
61.30
33.99
$3.45
52.40
57.84
62.65
57.09
29.54
47.44
56.40
55.10
60.12
MST.
55.21
60:91
68.99
64.61
63.06
62.34
61.63
55 14
$4.64
53.74
97
64.14
58.49
30.41
48
57.73
56.42
61.55
CLIMB
57.75
63.84
72.47
67.74
66.25
65.45
64.96
58.
$7.80
$7.00
62.03
67.11
61.45
32.24
50.96
60.64
99.12
63.02
Table 16: Detailed per-language performance on the GPQA benchmark (0-shot accuracy). Bold numbers denote the best results among data allocation methods.
Methoo
MS
Open Source Multiingual L.I.Ms
HaMA-33
35.67
26 19.19
26
12.8
72.81
23.79
23.
Owm-3
27.73
87
Gemma
24.58
23.30
25.45
23.03
26.44
23.09
25.25
23.49
22.76
26.91
23.09
21.91
22.05
24.58
21.25
22.76
26.50
25.45
Different Data Allocation Methods
27.28
27.18
24.00
36.72
36
26.07
25.71
Isouted
23.51
22.77
26.27
21.76
26.24
25.03
23.51
24.51
25.24
24:24
23.03
23.27
28.64
25.03
23.51
24,80
25.03
Natual
24,97
26.40
28.24
26.19
27.77
26.54
25.17
25.00
25.60
26.78
27.15
36.
25.66
26.
26.17
25.70
37.16
MSL
24.25
25.43
27.06
25.54
26.61
13.08
25.64
24
24.00
24.77
25.89
26.27
25.
24.44
25.5
25.30
24.53
36.20
CLINB
25.96
27.09
28,00
27.24
28.46
27.71
27.44
25.88
25.71
26.41
27.59
28.00
26.67
26.13
27.28
27.18
26.20
26.99
25

---

Table 17: Detailed per-language performance on the HellaSwag benchmark (10-shot accuracy). Bold numbers denote the best results among data allocation methods.
Model Method
DE
EN
FR
πI
JA
KD
MS
NL
PT
RU
7A
T11
VI
Open Source Multilingual
LLMss
3.2
42.69
67.10
47,45
46
42.14
45.06
36.78
37,02
40.66
43.35
46.23
35.30
35.21
36.46
42.35
45.48
Qwen
45.56
65.36
50.93
66.08
25
39.52
42.49
43.30
51.26
45.53
35.81
36.79
44.79
52.39
Gomna-3
44.27
39.03
34,94
Different Data Allecation Methods
Unifoem
19.72
58.01
54.03
54.83
45.08
47.15
53.67
30,04
47.84
Iodnne
58.02
54.35
47.23
48.32
Natural
51.95
Table 18: Detailed per-language performance on the TruthfulQA benchmark (0-shot accuracy). Bold numbers denote the best results among data allocation methods.
Model/ / Method
AR
DE
EN
ES
FR
II
JA
KD
MS
NL
PT
RU
TH
TL
TR
VI
ZH
Open Source Muttilagual
L.L.S
LLMMA-3.2
33.66
37.41
$4.59
57.20
36.56
53.92
34,16
36.91
30.59
36,09)
S7.20
36.76
40.04
36,76
36. 36.32
57.90
42.73
41 16.36
Owen3
19.16
50.27
47.94
50.01
51.54
43.24
50.01
51 51.56
47.18
47.70
46.15
$2.96
51.37
47.35
42.46
46.15
$2.03
48.96
Gemma-3
39.20
41.50
69
39.83
36.73
42.47
40.93
37 39.39
41.14
37.83
16.28
42.03
44.24
40.05
34.51
39.38
43.80
42.25
Different Data Allecation Metheds
Uniteem
41.42
38.07
38.28
41.63
41.42
19:09
40.74
35. 97
41.42
29.09
39:73
39.94
39.54
38.07
31.00
38.28
41.42
41.63
bellted
36.55
42.71
37.18
30.93
40.55
39.49
40.13
37.
40.78
36.5
30.73
41.43
38.
38.26
38.26
40.13
45.00
41.61
Natmil
41.62
40.17
40.95
42.12
41.83
40.91
40.76
38 60
40.09
39. 40
60.42
41
40 09
59.06
37.51
39.74
43.35
42.91
MSL
40.55
39.03
39.91
41.02
40.76
39:76
39.57
37.50
38.93
38.26
39:27
40.70
39.75
37.84
36.35
35.74
41.18
41.70
CLIKE
42.05
40.57
41.35
42.57
42.32
41.36
41.15
38.99
40.49
39.72
40.80
42.17
41.09
39.31
37.78
40.38
42.62
42.00
Social Impact
CLIMB contributes positively by systematically enhancing multilingual performance in large language models (LLMs), thereby significantly improving global accessibility to advanced Al capabilities across diverse linguistic communities. Such improvements have the potential reduce linguistic biases, bridge language gaps, and enhance equitable information access globally. However, there remain potential risks, including inadvertent reinforcement of cultural linguistic biases inherent in training data and the possibility of over-reliance on optimized multilingual models leading to reduced human oversight and critical evaluation. It is crucial responsibly deploy CLIMBBOptimized models with ongoing evaluation and monitoring, actively addressing ethical considerations and biases to ensure equitable and inclusive benefits.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims in the abstract and introduction describe the work clearly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix G.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
All theoretical results are provided with proof, either in Section 2 or Appendix
B, and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided all training details and hyperparameters in Experiment part and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We used open-source data sets and benchmarks as discussed in Experiments. And we will release our translated benchmarks.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccpppblligggddeeeeeeeeeeee Coaee///ddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided all training details and hyperparameters in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the cost of training large models, we do not have multiple runs. However, the clear & predictable trends suggest the noise is very small.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All experiments are ran on H100 and this stated in the Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have thoroughly read the NeurIPS Code of Ethics and ensured compliance in all aspects of our research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts of our work in Appendix H.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Not involved in misusing.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide proper citations for all baseline methods and datasets used in this study.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We document the new benchmark in Section 4.1 and Appendix E. We will release the new benchmark alongside the submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
The LLMs used in this study are described in Section and 4.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ htappp:////urips.... http:////ccccooeeeeee/////////// for what should or should be described.
33


Reviews:
[{"review_id": "4KGAtNMWDJ", "reviewer": "Reviewer_1T2b", "summary": "This paper proposes CLIMB, a new method to optimally allocate different languages in the training data for large multilingual language models. It introduces a way to measure how languages interact and influence each other during training, then uses this to find the best mix of languages to improve overall model performance. Experiments show that models trained with CLIMB’s language proportions perform better or as well the baselines in the paper.", "questions": "The paper is clear. I have mentioned the weaknesses in the weaknesses section. The authors can address these weaknesses by responding to them as questions.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The approach builds on scaling laws, an influential concept in LLM research, and innovatively extends it with cross-lingual interaction metrics.\n- Most of the benchmarks are multiple-choice questions; however, they include translations as well, which is a plus since they do not limit themselves to non-generative tasks.\n\nWeaknesses:\n\n- Weak baselines [Major reason \"weak evaluation\" for score 2]: The paper did not compare with more effective sampling methods such as UniMax (https://arxiv.org/pdf/2304.09151) or even simple temperature-based sampling (see the same paper).\n- Most benchmarks are automated machine-translated. We definitely have benchmarks that are not, such as Belebele, covering more than 100 languages (https://arxiv.org/abs/2308.16884).\n- The number of tokens is treated as a variable in the paper, but the model size (number of parameters) should also be considered as a variable (they have two sizes: 1.2B and 7B). Otherwise, conclusions might be confounded by this factor and may not hold true across different model sizes.\n- The order of operation is quite high (I did not calculate the exact value; this is based on the number of nested loops). Would you mind correcting me if I am wrong and calculating the number of times training occurs for a given number of languages n?\n\nSuggestions:\n\n- Update the FineWeb2 citation for the camera-ready version, as it now has an arXiv paper (https://arxiv.org/abs/2506.20920).\n- Missing highly related work: This paper https://arxiv.org/abs/2404.07982 shows the role of language imbalance (which you are optimizing) in cross-lingual generalization (which you are measuring).", "was_revised": true, "final_justification": "The authors were unfamiliar with some fundamental methods, and I essentially guided them to at least include basic comparisons.\nI increased my score from 2 to 3 given their effort in the rebuttal.", "rebuttal": "Thank you for your thoughtful review and valuable suggestions. **We are eager to further address any additional concerns you may have. We sincerely hope this response helps clarify the key points and merits of our work, and would greatly appreciate the opportunity to improve the evaluation of our work.**\n\n---\n\n### Q1.1: Weak Baselines\n\nThank you for raising this point. We have conducted additional experiments to include two stronger baselines: **UniMax** and **temperature-based sampling (Temp)**. The results below show that our proposed method, **CLIMB**, consistently outperforms both baselines across multiple benchmarks and model sizes (1.2B and 7B):\n\n**Table R1.1** 1B2 model Performance on new baselines and new benchmarks.\n| 1B2 | include | mgsm | belebele | multiblimp | XNLI  | XCOPA | XSC   | Flores | GMMLU | CMMLU | JMMLU | VLMU  | XWG   | XHS   | XARC-E | XARC-C | XTQA  |\n|--------|---------|------|----------|-------------|-------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|--------|\n| Temp   | 24.82   | 2.11 | 25.27    | 60.75       | 41.03 | 58.69  | 59.54  | 47.46  | 29.04 | 33.03 | 31.98 | 30.27 | 74.76 | 47.56 | 59.31  | 34.93  | 39.38 |\n| UniMax | 25.12   | 2.07 | 23.30    | 62.24       | 40.88 | 59.35  | 58.99  | 49.12  | 31.27 | **33.75** | 32.26 | 31.16 | 74.94 | 48.22 | 59.39  | 35.78  | 39.67 |\n| CLIMB  | **25.92**   | **2.40** | **26.17**    | **65.54**       | **41.65** | **59.98**  | **60.54**  | **50.43**  | **31.78** | 33.67 | **33.21** | **31.76** | **77.48** | **48.75** | **60.45**  | **36.56**  | **40.94** |\n\n**Table R1.2** 7B model Performance on new baselines and new benchmarks.\n| 7B | include | mgsm | belebele | multiblimp | XNLI  | XCOPA | XSC   | Flores | GMMLU | CMMLU | JMMLU | VLMU  | XWG   | XHS   | XARC-E | XARC-C | XTQA  |\n|--------|---------|------|----------|-------------|-------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|--------|\n| Temp   | 29.64   | 9.93 | **42.19**    | 71.14       | **44.86** | 61.73  | 64.44  | 55.15  | 33.51 | 37.33 | 36.78 | **36.68** | 82.72 | 57.12 | 66.32  | 42.70  | **43.97** |\n| UniMax | 30.76   | 9.67 | 40.90    | 71.10       | 44.85 | 62.16  | 64.88  | 55.57  | 33.97 | 37.25 | 37.19 | 36.47 | 82.40 | 57.88 | 67.75  | 44.28  | 42.41 |\n| CLIMB  | **31.76**   | 12.29| 41.72    | **72.67**       | 43.65 | **63.44**  | **65.07**  | **56.47**  | **34.98** | **37.83** | **40.54** | 35.40 | **84.02** | **59.90** | **69.90**  | **46.78**  | 42.86 |\n\nIn addition, we conducted statistical significance testing. Results show that the improvements of CLIMB over UniMax and temperature sampling are statistically significant (*p* < 0.05) on the majority of benchmarks. Detailed significance tests are provided in our response to Reviewer iYcL. These results confirm that CLIMB is a strong and reliable method for multilingual data allocation.\n\n---\n\n### Q1.2: Benchmark Quality: Human vs. Machine Translated\n\nWe appreciate this feedback. To address the concern, we have included results on additional human-translated benchmarks such as **Belebele**, **Include**, **MultiBLiMP** and **MGSM** (suggested by Reviewer iYcL), as shown in Table R1.1 and R1.2. The consistent advantage of **CLIMB** on both human-translated and machine-translated benchmarks demonstrates its robustness and generalizability. We will include these results in the camera-ready version.\n\n---\n\n### Q1.3: Controlling for Model Size in Scaling Law\n\nThank you for this thoughtful suggestion. Our current work primarily focuses on understanding how **training data scale** influences optimal language allocation. This is the central variable in our formulation and analysis.\n\nWe also evaluated CLIMB on models of different sizes, including **530M**, **1.2B**, and **7B**, and applied the same scaling law fitting procedure. We found that the fitted parameters remain **highly consistent** across model sizes, with only minor differences—primarily in the scaling constant **$E$**, which does **not affect the final computed ratios**.\n\nWe further quantified this consistency by computing the **relative standard deviation (RSD)** across all parameters, which was only **1.62%**, indicating strong robustness to model scale. In addition, as shown in **Tables R1.1/R1.2** and **Figure 6**, CLIMB performs well on both **1.2B and 7B models**, demonstrating its practical effectiveness across scales.\n\nWe agree that explicitly modeling **parameter count** as an additional input to the scaling law could further improve allocation quality, and we have included this as a future direction in **Appendix G**.\n\n\n---\n\n### Q1.4: Complexity of CLIMB Training Procedure\n\nThank you for raising this concern. We clarify that the overall training complexity of CLIMB is **linear in the number of languages $N$**, not exponential.\n\nAs reported in Line 171, CLIMB performs the following steps:\n\n- **Monolingual Scaling Fitting**: For each language, we train **2 monolingual models** with different data scales → total: **$2 × N$**\n- **Cross-lingual Interaction Measurement**: For each language, we sample **2 random language ratios**, each evaluated under **2 different data scales** → total: **$4 × N$**\n\n**Total training runs $= 2 × N + 4 × N = 6 × N$**\n\nThis **$O(N)$** complexity is manageable in practice, and we believe it is justified by the substantial improvements in convergence and downstream performance.\n\n---\n\n### Q1.5: Related Work on Language Imbalance and Generalization\n\nThank you for pointing out this relevant work on language imbalance and its effect on cross-lingual generalization. We acknowledge its relevance and will include a discussion and proper citation in the camera-ready version."}, {"review_id": "OTwLrO10te", "reviewer": "Reviewer_4qz9", "summary": "This paper proposes a multilingual optimization framework, called CLIMB, to address the challenges to determine the optimal language ratios in the multilingual LLM pre-training. Experimental results demonstrate that the proposed approach achieves better accuracy and consistently outperforms baselines on a variety of multilingual tasks at different size of model parameters. \n\n- The paper is well motivated and mostly easy to follow. However, each figure is too small to read. Please enlarge them in the camera-ready. \n- The authors conducted extensive experiments across different languages in a variety of multilingual tasks. The experimental results with both 1.2B and 7B models support scalability of the proposed approach. Overall, the proposed approach is technically sound.\n- Have you increased the language coverage up to > 100 languages to test the massively multilingual settings? Temperature sampling is a well-known technique to balance language data during multilingual training, and I was wondering if you gave simple comparison.\n- Any insights on which languages (e.g., low-resource languages) mostly benefit from this approach?", "questions": "see the summary.", "limitations": "n/a", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths\n- The paper is well motivated and mostly easy to follow. However, each figure is too small to read. Please enlarge them in the camera-ready. \n- Extensive experimental results. It shows that the proposed approach is technically sound.\n\nweakness\n- while the authors explores multilingual experimental settings, it'd be interesting to conduct an experiment in a massively multilingual settings with nearly 100 languages. The language coverage up to 16 languages or so look relatively less complexity. \n- Along with that, it'd be nice to analyze the effectiveness of the proposed approach per language or language family", "was_revised": true, "final_justification": "Thank you for sharing the new results in response to the questions. These results should enrich the current manuscript. I am keeping my score of 4 to reflect a still learning toward acceptance, though not a strong accept (<5). As pointed out by the other reviewers, the initial manuscript lacked the critical baselines such as temperature sampling. This inclusion of these new results could be considered as major changes.", "rebuttal": "Thank you for your thoughtful review and valuable suggestions. \n\n---\n### Q2.1: Figures are hard to read—can they be enlarged?\n\nThank you for pointing this out. We will enlarge all figures and improve their readability in the camera-ready version.\n\n---\n### Q2.2: Did you compare CLIMB with temperature sampling?\n\nThank you for the suggestion. We have added two strong baselines—**temperature sampling** (Temp) and **UniMax** (suggested by Reviewer 1T2b)—to our updated experiments. We also included **non-translated benchmarks** (Belebele, MGSM, MultiBLiMP, Include, suggested by Reviewer iYcL) to better assess multilingual capabilities.\n\nAs shown in **Table R2.1** and **R2.2**, CLIMB consistently outperforms both baselines across benchmarks and model sizes, confirming its advantage over heuristic sampling methods.\n\n**Table R2.1** 1B2 model Performance on new baselines and new benchmarks.\n| 1B2 | include | mgsm | belebele | multiblimp | XNLI  | XCOPA | XSC   | Flores | GMMLU | CMMLU | JMMLU | VLMU  | XWG   | XHS   | XARC-E | XARC-C | XTQA  |\n|--------|---------|------|----------|-------------|-------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|--------|\n| Temp   | 24.82   | 2.11 | 25.27    | 60.75       | 41.03 | 58.69  | 59.54  | 47.46  | 29.04 | 33.03 | 31.98 | 30.27 | 74.76 | 47.56 | 59.31  | 34.93  | 39.38 |\n| UniMax | 25.12   | 2.07 | 23.30    | 62.24       | 40.88 | 59.35  | 58.99  | 49.12  | 31.27 | **33.75** | 32.26 | 31.16 | 74.94 | 48.22 | 59.39  | 35.78  | 39.67 |\n| CLIMB  | **25.92**   | **2.40** | **26.17**    | **65.54**       | **41.65** | **59.98**  | **60.54**  | **50.43**  | **31.78** | 33.67 | **33.21** | **31.76** | **77.48** | **48.75** | **60.45**  | **36.56**  | **40.94** |\n\n**Table R2.2** 7B model Performance on new baselines and new benchmarks.\n| 7B | include | mgsm | belebele | multiblimp | XNLI  | XCOPA | XSC   | Flores | GMMLU | CMMLU | JMMLU | VLMU  | XWG   | XHS   | XARC-E | XARC-C | XTQA  |\n|--------|---------|------|----------|-------------|-------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|--------|\n| Temp   | 29.64   | 9.93 | **42.19**    | 71.14       | **44.86** | 61.73  | 64.44  | 55.15  | 33.51 | 37.33 | 36.78 | **36.68** | 82.72 | 57.12 | 66.32  | 42.70  | **43.97** |\n| UniMax | 30.76   | 9.67 | 40.90    | 71.10       | 44.85 | 62.16  | 64.88  | 55.57  | 33.97 | 37.25 | 37.19 | 36.47 | 82.40 | 57.88 | 67.75  | 44.28  | 42.41 |\n| CLIMB  | **31.76**   | 12.29| 41.72    | **72.67**       | 43.65 | **63.44**  | **65.07**  | **56.47**  | **34.98** | **37.83** | **40.54** | 35.40 | **84.02** | **59.90** | **69.90**  | **46.78**  | 42.86 |\n\n---\n### Q2.3: Why only 16 languages? Can CLIMB scale to 100+?\n\nThis is a great question. To handle massive multilinguality, we applied CLIMB at the **language family level** instead of individual languages.\n\nWe filtered out families with fewer than **100B tokens** in FineWeb-2 and retained **58 language families**, covering **300+ languages** in total. Results from **Table R2.3** on the 1.2B model show that CLIMB still outperforms temperature sampling under this broader setting. What's more, CLIMB outperform open-source models on many benchmarks. Given the disadvantage of this comparison (details are listed in Rebuttal to iYcL Q3.2), this further claims the advantage over heuristic sampling methods.\n\n**Table R2.3** 1B model Performance on CLIMB-300+.\n\n| Model         | Include | MGSM  | Belebele | MultiBLiMP | XNLI  | XCOPA | XSC   | Flores | GMMLU | CMMLU | JMMLU | VLMU  | XWG   |\n|---------------|---------|-------|----------|-------------|-------|--------|--------|--------|--------|--------|--------|--------|--------|\n| Llama-3.2-1B   | 28.48   | 3.89  | 30.69    | 77.66       | 41.29 | 57.07  | 58.13  | 44.14  | 28.74 | 30.30 | 29.84 | 29.09 | 76.08 |\n| Qwen3-1.7B (distilled from model trained with 26T token)     | **46.87**   | **36.95** | **74.81**    | 77.72       | **43.25** | 59.58  | 60.47  | 50.25  | **34.24** | **45.16** | **36.09** | **36.26** | **79.12** |\n| gemma-3-1b-pt  | 25.63   | 1.78  | 28.13    | **80.88**       | 36.33 | 55.58  | 54.84  | 46.55  | 27.12 | 28.59 | 28.48 | 30.05 | 65.79 |\n| xglm-1.7B      | 25.96   | 1.89  | 24.21    | 68.10       | 37.35 | 56.73  | 56.99  | 21.81  | 26.08 | 29.04 | 28.37 | 29.41 | 62.60 |\n| Temp-300+      | 24.82   | 2.11  | 25.27    | 68.28       | 42.03 | 59.29  | 59.54  | 49.46  | 29.04 | 28.90 | 29.54 | 30.30 | 74.76 |\n| CLIMB-300+     | 25.35   | 2.22  | 27.32    | 73.21       | 42.91 | **60.04**  | 60.12  | **50.47**  | 32.57 | 33.03 | 31.98 | 32.27 | 76.78 |\n| CLIMB-16       | 25.92   | 2.40  | 26.17    | 65.54       | 41.65 | 59.98  | **60.54**  | 50.43  | 31.78 | 33.67 | 33.21 | 31.76 | 77.48 |\n\n\nWe plan to further extend CLIMB to scale efficiently in even larger language scenarios.\n\n---\n\n### Q2.4: Can you provide per-language or per-family analysis?\n\nThank you for the suggestion. To better understand CLIMB’s behavior across languages, we analyzed **cross-lingual interaction ratios** among the 16 languages in our setting. This helps explain how CLIMB assigns language proportions in a data-driven manner.\n\n**Table R2.3** average transferability for each language and the top-1 transfer-out language.\n\n| Language          | de    | en    | nl    | es    | pt    | fr    | it    | id    | ja    | ko    | zh    | ru    | ar    | th    | vi    | tr    |\n|--------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| Transfer-out Score | 0.123 | 0.199 | 0.130 | 0.218 | 0.113 | 0.174 | 0.146 | 0.181 | 0.124 | 0.146 | 0.121 | 0.139 | 0.144 | 0.108 | 0.066 | 0.144 |\n| Transfer-in Score  | 0.168 | 0.151 | 0.144 | 0.188 | 0.146 | 0.145 | 0.192 | 0.139 | 0.164 | 0.129 | 0.108 | 0.185 | 0.130 | 0.152 | 0.127 | 0.136 |\n| Top1_Lang     | nl     | it     | de     | pt     | es     | es     | pt     | de     | zh     | ja     | ja     | tr     | it     | vi     | th     | ru     |\n\n\n**Key observations:**\n1. **Intra-family transfer is strong.** Languages within the same family (e.g., Spanish–Portuguese, Japanese–Korean) tend to benefit from one another, which supports CLIMB’s use of family-level grouping in large-scale allocations.\n2. **High-transfer languages** like English, Spanish, and Indonesian receive larger allocations, as their benefits generalize well to other languages (see also Figure 7).\n\nThis analysis provides interpretability for CLIMB’s language decisions and further validates its allocation strategy.\n\n\n---\n\n### Q2.5: Which languages benefit the most from CLIMB?\n\nThank you for the question. To understand CLIMB’s allocation behavior, we analyze **transfer-out** and **transfer-in** scores for each language (see Table R2.3). \n\n- A **higher transfer-out score** means the language contributes more to improving other languages.  \n- A **lower transfer-in score** means the language gains less from others and must rely more on its own data.\n\nWe observe that **English, Spanish, and Indonesian** have the highest transfer-out scores. Adding these languages benefits many others, so CLIMB tends to allocate more to them to maximize overall utility.\n\nIn contrast, **Chinese (zh)** has the **lowest transfer-in score**, indicating it gains little from cross-lingual transfer. As a result, CLIMB assigns it a higher share of training data to preserve its performance—this may also explain its relatively smaller improvements on downstream tasks.\n\nThese findings are consistent with CLIMB’s final allocation in **Figure 7**, confirming that the learned language proportions are grounded in measurable cross-lingual dynamics rather than heuristics."}, {"review_id": "0JBLx7j7M2", "reviewer": "Reviewer_iYcL", "summary": "The paper proposes a method for estimating training data ratios for different languages, taking into account language-specific interactions. The authors train several 1B and 7B models using different methods to determine language data proportions and argue that their proposed method offers the strongest results.", "questions": "* Appendix D (L70) implies that all models were trained using the same custom tokenizer. Is that correct? If so, how much of the tokenizer represents the different languages used in these experiments? How does compression differ across the languages? How might this impact the results of these experiments?\n* For GlobalMMLU, did you use the culturally specific or culturally agnostic subset (or both)?\n* Regarding the results in Appendix F, why do you think CLIMB underperforms existing models so consistently in GMMLU, GPQA, and TruthfulQA? Qwen beats it on almost every language. These are the more recently developed benchmarks and the ones that are most relevant in current research. \n\nMisc.\n* The lack of bolding in Table 8 makes it hard to readNo bolding on Table 8? Makes it hard to read. But I think this way of organizing the models together is best. \n* Weird spacing in “Evaluation Benchmarks” (L275-281)", "limitations": "No, this paper does not discuss the limitations of the evaluation method, in particular the potential impact of using machine-translated benchmarks. The authors state that they plan to release these benchmarks, however, without human validation, they may be adding low-quality benchmarks into the ecosystem, which may negatively impact work on the languages this paper works on.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n* The paper tackles an interesting problem, and a solution to it could help train more powerful and equitable models.\n* The paper is generally well written and clear.\n* The cross-lingual interaction-aware language ratio method seems generally sound and could be useful for other research. The results in Table 1 are compelling. \n\n\n**Weaknesses**:\n* The motivation for these experiments is circular. The authors state that the empirical observations in Section 2.2.1 motivate the following experiments (L144). But are these the same models that are used in the subsequent experiments? Either way, the authors could motivate the four points in Section 2.2.1 with evidence from previous work, with which the authors did not engage very deeply.\n[1] shows that the efficacy of crosslingual transfer begins to decrease if the amount of added data from other languages too greatly exceeds the amount of data in the target language (“Dependency on absolute language proportion”, L130-132). [1] also shows that the similarity (specifically in terms of syntactic typology) between the target language and the languages for which there is added data significantly impacts crosslingual transfer (“Dependency on co-training languages”, L133-135). “Dependency on model language counts” (L136-139) is predicted by [2] (e.g. blue bars in Fig. 4). As the number of languages a model is trained on increases, the model will hit capacity limitations. \nAll of this work falls under the topic of “the curse of multilinguality”, which is not mentioned in the paper. \n* The existing multilingual models that the authors compare their models to are all closed-data models. None of the models’ technical reports even state what proportion of the training data is in each of the different languages. Comparing against a model that is either open data or reports training data proportions, e.g. XGLM [3], would make for more meaningful comparisons. XGLM is available in both 1B and 7B sizes.\n* The reporting of results in Table 2 is somewhat misleading. The bolded values do not represent the highest values, but only the highest values among the models trained by the authors. The same is true for the appendices (with the exception of Table 8, which allows for easier comparison, though does not contain any bolding). The authors should indicate overall best performance for each task and/or clarify the bolding strategy in the caption for the table.\n* Relatedly, the differences across the different models are extremely small in some cases, and it is unclear whether differences would be statistically significant. The authors did not include any standard error. The justification for this decision (L700-701) is insufficient, as it is not necessary to train different random seeds to get standard error. Running evaluations through tools like the LM Evaluation Harness [4] automatically generate standard error, which helps the reader to identify meaningful differences.  \nThis seems especially critical for comparisons between Uniform and CLIMB models, where scores are often very similar. Whether or not these differences are significant impacts the fundamental argument in this paper. \n* The results are presented differently for the 7B (Fig. 6) and 1B (Table 2) models. The 7B models aren’t compared to any existing multilingual models. I found unclear which model the results in Table 2 were for. The appendices fail to report any evaluations for the 7B models as far as I can tell. The only comparisons that can be made are with the models trained by the authors (though, again, no standard error is reported, so it is unclear whether differences in performance are meaningful). The 7B results are not as convincing, therefore, as the 1B models. If CLIMB is not as effective at the 7B scale, this would significantly impact the argument of the paper. \n* The use of ‘state-of-the-art’ (L287) seems inappropriate since the authors do not show that CLIMB significantly outperforms the existing models. In several cases, Qwen numerically outperforms CLIMB (1B). There are no comparisons at all on the 7B scale.\n* The paper is missing some critical details about the translated benchmarks. The only information about how the benchmarks were translated is one sentence in Appendix E, stating that the authors use “GPT-based translation” (L115-116). The authors should at the bare minimum report what kind of translation system was used to translate the benchmarks. Additionally, the authors should add discussion about how well those systems perform on the different languages that you translate the benchmarks into and whether any additional checking was done to verify translation quality. \nThe authors do not explain why they translate benchmarks instead of using existing multilingual translations of those benchmarks (HellaSwag [5, 6]; GPQA [7, 8]; TruthfulQA) [9]) or use existing (not machine translated) benchmarks (Belebele for reading comprehension in 122 languages [10], MultiBLiMP for grammatical generalization for 101 languages [11]). There are at least two multilingual mathematical reasoning benchmarks MGSM [12] and MCLM [13]. \nThe evaluation results would be more convincing if the authors remove their machine translated benchmarks and use existing (ideally, at least human-validated machine translated) benchmarks.\n\n[1] https://aclanthology.org/2024.emnlp-main.236/ \n\n[2] https://aclanthology.org/2020.acl-main.747\n\n[3] https://arxiv.org/pdf/2112.10668 \n\n[4] https://github.com/EleutherAI/lm-evaluation-harness/tree/main \n\n[5] https://arxiv.org/abs/2410.08928 \n\n[6] https://arxiv.org/abs/2307.16039 \n\n[7] https://arxiv.org/abs/2504.11833? \n\n[8] https://arxiv.org/abs/2502.07346 \n\n[9] https://arxiv.org/abs/2406.14434? \n\n[10] https://aclanthology.org/2024.acl-long.44/ \n\n[11] https://arxiv.org/abs/2504.02768? \n\n[12] https://openreview.net/forum?id=fR3wGCk-IXp \n\n[13] https://arxiv.org/abs/2502.17407?", "was_revised": true, "final_justification": "The authors have mostly addressed my questions and concerns. I still feel that using the same set of models to motivate the experiments as to conduct the experiments is somewhat circular. However, I understand that training so many models from scratch is expensive and time-consuming. As long as the authors clarify this in the final version, I think this is acceptable.", "rebuttal": "Thank you for your detailed and thoughtful review. We have carefully addressed your comments and made concrete improvements accordingly. **If there are any remaining concerns, we would be more than willing to engage further and are genuinely eager for a reconsideration of the score.**\n\n---\n### Q3.1: Are the experiments in Section 2.2.1 circular in motivation?\n\nThank you for raising this important concern. We confirm that there is **no circularity** in our experimental design. Our methodology follows the standard approach in scaling law research. Specifically, we trained over **500 different 1B2 models** with diverse, randomly sampled language ratios and data scales across 16 languages. These controlled experiments allow us to **quantitatively measure cross-lingual interactions**, rather than relying on assumptions. The patterns shown in **Figure 2** are derived purely from these randomized runs.\n\nWe then fit a **scaling law function (CLIMB)** to these empirical results, capturing how validation loss depends on both data scale and multilingual composition. As shown in **Table 1**, the fitted function generalizes well to unseen settings, validating its accuracy. Based on this fitted function, we compute the **optimal language ratios**, and use them to train new models for the downstream evaluations reported in **Table 2**.\n\nIn this workflow, the formulation of CLIMB is based entirely on randomized experiments, while its effectiveness is verified through extrapolation and downstream performance—there is **no overlap between hypothesis generation and evaluation**, and thus no circular reasoning.\n\nWe appreciate the reminder that some qualitative observations in Figure 2 (e.g., diminishing returns, influence of language similarity) have been discussed in prior work. These works indeed support our empirical findings, and we will revise the paper to cite them accordingly.\n\n---\n### Q3.2: Why not compare with open multilingual models like XGLM?\n\nThank you for the suggestion. We have added **XGLM-1.7B** and **XGLM-7.5B** results to Table R3.1 and Table R3.2.\n\n**Direct comparisons with open models like XGLM are not entirely fair**, due to differences in:\n\n- **Training data**: Our models use only **FineWeb-2**, while open models always include curated data such as **QA, code, math and synthetic corpora**.  \n- **Training scale**: Open models are trained on **more tokens** (e.g., 26T for QWEN3), compared to **1T** for our models.  \n- **Language coverage**: Open models are trained on **more languages** than CLIMB.  \n- **Initialization**: Some open models benefit from **distillation from larger proprietary models**.\n\nDespite these disadvantages, CLIMB-trained models achieve **comparable or stronger results** on several multilingual benchmarks.\n\n**Table R3.1** Comparison of CLIMB with open-source baselines on 7B\n|7B|Include|MGSM|Belebele|MultiBLiMP|XNLI|XCOPA|XSC|Flores|GMMLU|CMMLU|JMMLU|VLMU|XWG|XHS|XARC-E|XARC-C|XTQA|\n|--|-------|----|--------|-----------|----|-----|----|------|-----|-----|-----|-----|----|----|-------|-------|-----|\n|Qwen3-8B-Base|**59.45**|**60.51**|**87.99**|71.18|**46.85**|**67.02**|**66.97**|56.32|**42.70**|**56.38**|**47.15**|**47.59**|**86.49**|**59.98**|**74.13**|**54.42**|**43.90**|\n|XGLM-7.5B|25.58|0.69|24.27|71.76|41.59|61.40|60.97|27.15|27.78|32.26|29.84|31.98|75.99|43.86|48.23|28.79|32.13|\n|Temp|29.64|9.93|42.19|71.14|44.86|61.73|64.44|55.15|33.51|37.33|36.78|36.68|82.72|57.12|66.32|42.70|43.97|\n|UniMax|30.76|9.67|40.90|71.10|44.85|62.16|64.88|55.57|33.97|37.25|37.19|36.47|82.40|57.88|67.75|44.28|42.41|\n|CLIMB|31.76|12.29|41.72|**72.67**|43.65|63.44|65.07|**56.47**|34.98|37.83|40.54|35.40|84.02|59.90|69.90|46.78|42.86|\n\n\n**Table R3.2** Comparison of CLIMB with open-source baselines on 1B\n|1B|Include|MGSM|Belebele|MultiBLiMP|XNLI|XCOPA|XSC|Flores|GMMLU|CMMLU|JMMLU|VLMU|XWG|XHS|XARC-E|XARC-C|XTQA|\n|--|-------|----|--------|-----------|----|-----|----|------|-----|-----|-----|-----|----|----|-------|-------|-----|\n|Llama-3.2-1B|28.48|3.89|30.69|77.66|41.29|57.07|58.13|44.14|28.74|30.30|29.84|29.09|76.08|41.52|47.19|29.55|34.91|\n|Qwen3-1.7B|**46.87**|**36.95**|**74.81**|77.72|**43.25**|59.58|60.47|50.25|**34.24**|**45.16**|**36.09**|**36.26**|**79.12**|46.59|59.53|**40.18**|38.55|\n|Gemma-3-1b-pt|25.63|1.78|28.13|**80.88**|36.33|55.58|54.84|46.55|27.12|28.59|28.48|30.05|65.79|41.59|49.29|29.69|37.32|\n|XGLM-1.7B|25.96|1.89|24.21|68.10|37.35|56.73|56.99|21.80|26.08|29.04|28.37|29.41|62.60|36.18|40.17|24.60|33.31|\n|UniMax|25.12|2.07|23.30|62.24|40.88|59.35|58.99|49.12|31.27|33.75|32.26|31.16|74.94|48.22|59.39|35.78|39.67|\n|Temp|24.82|2.11|25.27|60.75|41.03|58.69|59.54|47.46|29.04|33.03|31.98|30.27|74.76|47.56|59.31|34.93|39.38|\n|CLIMB|25.92|2.40|26.17|65.54|41.65|**59.98**|**60.54**|**50.43**|31.78|33.67|33.21|31.76|77.48|**48.75**|**60.45**|36.56|**40.94**|\n\n---\n\n### Q3.3: Why are the bold values in Table 2 not the best overall?\n\nThank you for the suggestion. As noted in Line 269 and Q2, **direct comparisons with open-source models are not fully fair** due to differences in data, scale, and setup.\n\nTherefore, in Table 2 we **bold only the best results among our sampling methods** to ensure fair intra-group comparison. We will clarify this bolding strategy in the table caption in the camera-ready version.\n\n---\n### Q3.4: Why are standard errors and significance testing missing?\n\nThanks for the question. We report **standard errors** (from the evaluation harness) in Table R4. However, they are consistently small across benchmarks and fail to reflect meaningful differences.\n\nTo better assess significance, we conducted **paired $t$-tests** between CLIMB and each baseline. As shown in Table R5, **CLIMB shows statistically significant improvements on 13–15 out of 17 benchmarks** under both 1B2 and 7B settings (*p* < 0.05).\n\n**Table R3.3** Standard errors (stderr) from Harness for 1B2 models.\n\n|1B2 stderr|Include|MGSM|Belebele|MultiBLiMP|XNLI|XCOPA|XSC|Flores|GMMLU|CMMLU|JMMLU|VLMU|XWG|XHS|XARC-E|XARC-C|XTQA|\n|----------|-------|----|--------|-----------|----|-----|----|------|-----|-----|-----|-----|----|----|-------|-------|-----|\n|Uniform|25.19±0.003|2.11±0.003|26.17±0.003|62.24±0.001|40.08±0.003|59.49±0.007|58.99±0.002|47.51±0.067|29.05±0.001|34.79±0.004|32.47±0.005|31.44±0.015|73.34±0.007|48.12±0.001|59.76±0.002|35.41±0.003|39.62±0.003|\n|CLIMB|25.92±0.003|2.40±0.003|26.17±0.003|65.54±0.001|41.65±0.003|59.98±0.007|60.54±0.002|50.43±0.066|31.78±0.001|33.67±0.004|33.21±0.005|31.76±0.015|77.48±0.006|48.75±0.001|60.45±0.002|36.56±0.003|40.94±0.003|\n\n**Table R3.4** Significance test on 7B and 1B2 models (✅ means p < 0.05)\n\n| CLIMB v.s.|include|MGSM|Belebele|MultiBLiMP|XNLI|XCOPA|XSC|Flores|GMMLU|CMMLU|JMMLU|VLMU|XWG|XHS|XARC-E|XARC-C|XTQA|#significant (p✓)|\n|-----------|-------|----|--------|-----------|-----|------|----|-------|------|------|------|------|----|----|--------|--------|-----|-------------------|\n|**7B Models**|||||||||||||||||||||\n|Uniform|✅|0.24|✅|✅|✅|0.28|✅|✅|✅|✅|0.11|✅|✅|✅|✅|✅|✅|14/17|\nTemp|✅|✅|✅|✅|✅|✅|✅|✅|✅|✅|0.34|✅|0.92|✅|✅|✅|✅|15/17|\n|UniMax|✅|✅|0.11|✅|✅|✅|✅|✅|✅|✅|0.39|✅|✅|✅|✅|✅|✅|15/17|\n|**1B2 Models**|||||||||||||||||||||\n|Uniform|0.43|0.94|✅|✅|✅|✅|✅|✅|✅|✅|✅|✅|✅|✅|✅|✅|✅|15/17|\n|Temp|✅|0.41|✅|✅|✅|✅|✅|✅|✅|✅|0.33|0.20|✅|✅|✅|✅|✅|14/17|\n|UniMax|0.44|0.92|✅|✅|✅|✅|✅|✅|✅|✅|0.11|✅|✅|✅|0.18|✅|✅|13/17|\n\n---\n\n### Q3.5: Are 7B results fairly compared to other models?\n\nThank you for highlighting this. 7B results are in Table R3 and the significance results are in Table R5. \n\n---\n\n### Q3.6: Is “state-of-the-art” appropriate?\n\n\nThank you for pointing this out. We agree that the term “state-of-the-art” may be misleading in this context. We will revise this wording in the camera-ready version.\n\n---\n\n### Q3.7: Missing Details about Translated Benchmarks\n\nThank you for the detailed feedback. We clarify that our translated benchmark was first generated using **GPT-4o**, and then evaluated via **human review**. Translations were rated on a 5-point scale, with **GPT outputs scoring 4.3 on average**, and **final human-reviewed versions scoring even higher**.\n\nHuman evaluators were selected based on **CEFR standard**: C1 level in the target language and B2 in English. In total, human evaluation cost **$31,212**, ensuring quality and reliability.\n\nWe also applied the same evaluation process to datasets from [5–9] and found our translations to be **comparable or stronger**. As a result, we prioritized **our own translated benchmark** to better align with the 16-language setup in our study.\n\nTo address concerns about translation artifacts, we further added **non-translated benchmarks** (Belebele, MGSM, MultiBLiMP) in Table R3/R4. Since our model is not trained on code/math data, we excluded MCLM (competition-level math) to reflect our domain coverage.\n\nWe will clarify these details in the revised version.\n\n---\n\n### Q3.8: Does the tokenizer design affect results?\n\nThank you for the insightful question. We used a **custom tokenizer** to better study the link between **tokenizer design and cross-lingual transfer**. Public tokenizers (e.g., XGLM, mT5) often have opaque training distributions, making such analysis difficult.\n\nTo ensure transparency, we trained our tokenizer on **6B tokens** from the 16 target languages, with **50% English** and the rest evenly split. This resulted in:\n- **English compression rate**: 21.3%\n- **Non-English average**: 29.4%\n\nWe found that CLIMB remains **robust across different tokenizers**, though the fitted parameters do vary. This suggests that tokenizer design influences cross-lingual interactions, which we plan to explore further in future work.\n\n---\n\n### Q3.9: Which GlobalMMLU subset is used?\n\nThank you for your question. We used **both the culturally specific and culturally agnostic subsets** of GlobalMMLU in our evaluation.\n\n---\n### Q3.10: Why does CLIMB underperform QWEN on GMMLU, GPQA, and TruthfulQA?\n\nThank you for the question. This is addressed in Q3.2."}, {"review_id": "ALG6JJn263", "reviewer": "Reviewer_ybjW", "summary": "This paper proposed a new method to optimize the proportion of training data allocated for different languages in a multilingual language model. They propose a cross-lingual interaction-aware ratio which, based on LLM scaling laws, computes the effective language proportions considering interactions with other languages. They then propose a parametric model of this ratio, which can be fitted based on LLM experiments with varied language proportions. The last stage of their method is to determine, based on the aforementioned fitted model, the optimal ratios for a set of languages and a specified token budget. This non-convex optimization problem is solved through a 2-step process that first finds the direction of the language ratio vector and then finds its magnitude by maximizing a constrained effective allocation problem. The authors test their framework on validation loss prediction, where it outperforms baselines. Using their method to compute optimal ratios, they also train a language model and evaluate it on downstream tasks, where it outperforms models trained with baseline ratio allocators and performs competitively with existing LLMs trained on much larger datasets.", "questions": "1. Would you be able to address Weakness #1 by including additional loss prediction models in the final version of your paper? I do think it would strengthen your claims.\n\n2. For a long time the standard approach to multilingual data allocation was upsampling under-represented languages (e.g. as in XLM-R). Is this included in any of your baselines? It’s not explicitly stated, but perhaps it’s used by one of the baselines you compare against.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n1. The contributions of the paper are original and interesting. Well done to the authors! It’s always refreshing to find a rigorous solution to a problem, in this cases language ratio allocation, which is usually approached with simple heuristics. The authors frame the task as an optimization problem, so their contributions align well with NeurIPS.\n\n2. The proposed method is well motivated and clearly formulated. By drawing on existing work on scaling laws, the authors present a logical framework for thinking about language ratio allocation as an optimization problem. Each stage of their framework is presented clearly (in section 2). \n\n3. The validation loss prediction results are strong. The proposed method comfortably outperforms baselines, highlighting the predictive power of the framework. Similarly, the downstream performance is strong, highlighting the practical value of the framework. \n\n**Weaknesses:**\n\n1. The validation loss prediction results would be more convincing if more baselines were included. The biggest question I have after reading the paper is: is it really necessary to go through this complicated, multi-stage model to obtain this kind of predictive power? Aren’t there simpler parametric models that we can fit with language ratios as input and validation loss as the output to be predicted? I understand that there is limited existing work along this line, but the authors would be able to create simple baselines themselves by fitting, e.g. regression models, on the data collected in their experiments. \n\n2. The paper would be improved by having more details about the different models trained by the authors for their experiments. For example, in 3.1 they mention “we vary the token budgets from 5 billion to 100 billion tokens”, but don’t specify in what intervals or how many models are trained. I know much of these details are included in the appendix, but it would be better to include in the main paper body if possible, since the claims of the paper depend heavily on these experiments.\n\n3. This is a small note, but I don’t think the authors should claim “state-of-the-art” performance (as they do in the abstract, introduction), since their models are not state-of-the-art - they are better than comparable baselines, but not better than the best LLMs available.", "was_revised": true, "final_justification": "The authors responded to my main queries and included new results to address my concerns. I was happy to recommend accept initially and I stand by this recommendation.", "rebuttal": "Thank you for your detailed and positive review. \n\n---\n\n### Q4.1: Can more experimental details be moved from the appendix?\n\nThank you for the suggestion. We will revise the camera-ready version to **move key experimental details from the appendix into the main text**, particularly information on the **number of models trained**, **token budget intervals**, and **training configurations**. We agree that this will improve clarity and self-containment.\n\n---\n\n### Q4.2: Is it appropriate to claim “state-of-the-art” performance?\n\nThank you for pointing this out. We agree that the term may be misleading given the limited training budget and data quality. We will **revise or remove** the phrase “state-of-the-art” from the abstract and main text. Instead, we will emphasize that CLIMB achieves **competitive results under constrained training and compute conditions**.\n\n---\n### Q4.3: Why not use simpler models to fit validation loss?\n\nThank you for this valuable suggestion. We fully agree that it is important to justify the complexity of our parametric model. To this end, we implemented and compared several **simpler baselines**, as summarized below:\n\n| ID | Model Type | #Parameters | Huber Loss ↓ $(\\times 10^{-3})$ | R² ↑ |\n|----|------------|-------------|--------------|------|\n| 1  | $\\tilde{r}_i = \\alpha_i r_i$| 1 | 15.32 | 0.236 |\n| 2  | $\\tilde{r}_i =\\alpha_i r_i + \\sum \\alpha_j(D) r_j + b_i$ | $2 + 2 \\times (m - 1)$ | 1.44 | 0.563 |\n| 3  | $\\tilde{r}_i =\\alpha_i r_i^{\\eta_i} + \\sum \\alpha_j(D) r_j^{\\eta_j} + b_i$ | $3 + 3 \\times (m - 1)$ | 0.474 | 0.978 |\n| 4  | CLIMB | $3 \\times (m - 1)$ | **0.274** | **0.981** |\n\n- **Model 2** is a linear regression model with explicit transfer terms. It serves as the simplest meaningful baseline beyond uniform allocation.\n- Despite being lightweight, **simpler models (ID 1–3)** yield significantly higher error or lower R² scores.\n- Even **Model 3**, which introduces exponentiation and has **more parameters than our model**, still underperforms our proposed formulation (Model 4).\n\nThis demonstrates that our final parametric form is not only compact but also more expressive and better aligned with the empirical patterns in the data. We will include this comparison in the appendix of the final version to make the justification more transparent.\n\n\n---\n\n### Q4.4: Did you include upsampling-style baselines like XLM-R?\n\nThank you for raising this point. We included two strong and widely-used upsampling-style baselines:\n\n- **Temperature sampling (Temp)**, which smooths the token distribution to favor low-resource languages.\n- **UniMax** (suggested by Reviewer 1T2b), which boosts the contribution of the most frequent language in each batch.\n\nWe evaluated both baselines under **identical 1B2 and 7B training setups**, and report results in Tables R4.2 and R4.3. Across all multilingual benchmarks, **CLIMB consistently outperforms both Temp and UniMax**, demonstrating its superior effectiveness under comparable conditions.\n\nWe will include these results in the revised version for completeness.\n\n\n**Table R4.2** 1B2 model Performance on new baselines and new benchmarks.\n| 1B2 | include | mgsm | belebele | multiblimp | XNLI  | XCOPA | XSC   | Flores | GMMLU | CMMLU | JMMLU | VLMU  | XWG   | XHS   | XARC-E | XARC-C | XTQA  |\n|--------|---------|------|----------|-------------|-------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|--------|\n| Temp   | 24.82   | 2.11 | 25.27    | 60.75       | 41.03 | 58.69  | 59.54  | 47.46  | 29.04 | 33.03 | 31.98 | 30.27 | 74.76 | 47.56 | 59.31  | 34.93  | 39.38 |\n| UniMax | 25.12   | 2.07 | 23.30    | 62.24       | 40.88 | 59.35  | 58.99  | 49.12  | 31.27 | **33.75** | 32.26 | 31.16 | 74.94 | 48.22 | 59.39  | 35.78  | 39.67 |\n| CLIMB  | **25.92**   | **2.40** | **26.17**    | **65.54**       | **41.65** | **59.98**  | **60.54**  | **50.43**  | **31.78** | 33.67 | **33.21** | **31.76** | **77.48** | **48.75** | **60.45**  | **36.56**  | **40.94** |\n\n**Table R4.3** 7B model Performance on new baselines and new benchmarks.\n| 7B | include | mgsm | belebele | multiblimp | XNLI  | XCOPA | XSC   | Flores | GMMLU | CMMLU | JMMLU | VLMU  | XWG   | XHS   | XARC-E | XARC-C | XTQA  |\n|--------|---------|------|----------|-------------|-------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|--------|\n| Temp   | 29.64   | 9.93 | **42.19**    | 71.14       | **44.86** | 61.73  | 64.44  | 55.15  | 33.51 | 37.33 | 36.78 | **36.68** | 82.72 | 57.12 | 66.32  | 42.70  | **43.97** |\n| UniMax | 30.76   | 9.67 | 40.90    | 71.10       | 44.85 | 62.16  | 64.88  | 55.57  | 33.97 | 37.25 | 37.19 | 36.47 | 82.40 | 57.88 | 67.75  | 44.28  | 42.41 |\n| CLIMB  | **31.76**   | 12.29| 41.72    | **72.67**       | 43.65 | **63.44**  | **65.07**  | **56.47**  | **34.98** | **37.83** | **40.54** | 35.40 | **84.02** | **59.90** | **69.90**  | **46.78**  | 42.86 |"}]

Author rebuttal:

Meta-review: This paper introduces CLIMB, a cross-lingual interaction-aware framework for optimizing multilingual data allocation in LLM pretraining. Extensive experiments across multiple scales show that CLIMB outperforms heuristic baselines. The work makes a strong claim of moving beyond heuristic sampling toward a principled, quantitatively grounded method for multilingual data allocation. However, weaknesses include the initial omission of strong baselines (later added during rebuttal), limited exploration of massively multilingual settings in the main paper, and presentation issues such as small figures and potentially misleading claims (around "SOTA" models).

During rebuttal, the authors addressed most major concerns, and also committed to revising misleading claims and citations. Most reviewers raised their scores after rebuttal, recognizing that the additional results significantly strengthened the paper. Weighing the novelty and technical rigor, I support acceptance of this paper, though the authors should polish the paper / fix the presentation issues as promised.
Decision: Accept (poster)