Title: Exploring Polyglot Harmony: On Multilingual Data Allocation for  Large Language Models Pretraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Pre-training

Scaling Law

Language Mix Ratio
Abstract: Large language models (LLMs) have become integral to a wide range of applications worldwide, driving an unprecedented global demand for effective multilingual capabilities. Central to achieving robust multilingual performance is the strategic allocation of language proportions within training corpora. However, determining optimal language ratios is highly challenging due to intricate cross-lingual interactions and sensitivity to dataset scale. This paper introduces CLIMB (Cross-Lingual Interaction-aware Multilingual Balancing), a novel framework designed to systematically optimize multilingual data allocation. At its core, CLIMB introduces a cross-lingual interaction-aware language ratio, explicitly quantifying each language’s effective allocation by capturing inter-language dependencies. Leveraging this ratio, CLIMB proposes a principled two-step optimization procedure—first equalizing marginal benefits across languages, then maximizing the magnitude of the resulting language allocation vectors—significantly simplifying the inherently complex multilingual optimization problem. Extensive experiments confirm that CLIMB can accurately measure cross-lingual interactions across various multilingual settings. LLMs trained with CLIMB-derived proportions consistently achieve state-of-the-art multilingual performance, even achieve competitive performance with open-sourced LLMs trained with more tokens.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new method to optimize the proportion of training data allocated for different languages in a multilingual language model. They propose a cross-lingual interaction-aware ratio which, based on LLM scaling laws, computes the effective language proportions considering interactions with other languages. They then propose a parametric model of this ratio, which can be fitted based on LLM experiments with varied language proportions. The last stage of their method is to determine, based on the aforementioned fitted model, the optimal ratios for a set of languages and a specified token budget. This non-convex optimization problem is solved through a 2-step process that first finds the direction of the language ratio vector and then finds its magnitude by maximizing a constrained effective allocation problem. The authors test their framework on validation loss prediction, where it outperforms baselines. Using their method to compute optimal ratios, they also train a language model and evaluate it on downstream tasks, where it outperforms models trained with baseline ratio allocators and performs competitively with existing LLMs trained on much larger datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The contributions of the paper are original and interesting. Well done to the authors! It’s always refreshing to find a rigorous solution to a problem, in this cases language ratio allocation, which is usually approached with simple heuristics. The authors frame the task as an optimization problem, so their contributions align well with NeurIPS.\n\n2. The proposed method is well motivated and clearly formulated. By drawing on existing work on scaling laws, the authors present a logical framework for thinking about language ratio allocation as an optimization problem. Each stage of their framework is presented clearly (in section 2). \n\n3. The validation loss prediction results are strong. The proposed method comfortably outperforms baselines, highlighting the predictive power of the framework. Similarly, the downstream performance is strong, highlighting the practical value of the framework. \n\n**Weaknesses:**\n\n1. The validation loss prediction results would be more convincing if more baselines were included. The biggest question I have after reading the paper is: is it really necessary to go through this complicated, multi-stage model to obtain this kind of predictive power? Aren’t there simpler parametric models that we can fit with language ratios as input and validation loss as the output to be predicted? I understand that there is limited existing work along this line, but the authors would be able to create simple baselines themselves by fitting, e.g. regression models, on the data collected in their experiments. \n\n2. The paper would be improved by having more details about the different models trained by the authors for their experiments. For example, in 3.1 they mention “we vary the token budgets from 5 billion to 100 billion tokens”, but don’t specify in what intervals or how many models are trained. I know much of these details are included in the appendix, but it would be better to include in the main paper body if possible, since the claims of the paper depend heavily on these experiments.\n\n3. This is a small note, but I don’t think the authors should claim “state-of-the-art” performance (as they do in the abstract, introduction), since their models are not state-of-the-art - they are better than comparable baselines, but not better than the best LLMs available.\n\nQUESTIONS:\n1. Would you be able to address Weakness #1 by including additional loss prediction models in the final version of your paper? I do think it would strengthen your claims.\n\n2. For a long time the standard approach to multilingual data allocation was upsampling under-represented languages (e.g. as in XLM-R). Is this included in any of your baselines? It’s not explicitly stated, but perhaps it’s used by one of the baselines you compare against.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for estimating training data ratios for different languages, taking into account language-specific interactions. The authors train several 1B and 7B models using different methods to determine language data proportions and argue that their proposed method offers the strongest results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n* The paper tackles an interesting problem, and a solution to it could help train more powerful and equitable models.\n* The paper is generally well written and clear.\n* The cross-lingual interaction-aware language ratio method seems generally sound and could be useful for other research. The results in Table 1 are compelling. \n\n\n**Weaknesses**:\n* The motivation for these experiments is circular. The authors state that the empirical observations in Section 2.2.1 motivate the following experiments (L144). But are these the same models that are used in the subsequent experiments? Either way, the authors could motivate the four points in Section 2.2.1 with evidence from previous work, with which the authors did not engage very deeply.\n[1] shows that the efficacy of crosslingual transfer begins to decrease if the amount of added data from other languages too greatly exceeds the amount of data in the target language (“Dependency on absolute language proportion”, L130-132). [1] also shows that the similarity (specifically in terms of syntactic typology) between the target language and the languages for which there is added data significantly impacts crosslingual transfer (“Dependency on co-training languages”, L133-135). “Dependency on model language counts” (L136-139) is predicted by [2] (e.g. blue bars in Fig. 4). As the number of languages a model is trained on increases, the model will hit capacity limitations. \nAll of this work falls under the topic of “the curse of multilinguality”, which is not mentioned in the paper. \n* The existing multilingual models that the authors compare their models to are all closed-data models. None of the models’ technical reports even state what proportion of the training data is in each of the different languages. Comparing against a model that is either open data or reports training data proportions, e.g. XGLM [3], would make for more meaningful comparisons. XGLM is available in both 1B and 7B sizes.\n* The reporting of results in Table 2 is somewhat misleading. The bolded values do not represent the highest values, but only the highest values among the models trained by the authors. The same is true for the appendices (with the exception of Table 8, which allows for easier comparison, though does not contain any bolding). The authors should indicate overall best performance for each task and/or clarify the bolding strategy in the caption for the table.\n* Relatedly, the differences across the different models are extremely small in some cases, and it is unclear whether differences would be statistically significant. The authors did not include any standard error. The justification for this decision (L700-701) is insufficient, as it is not necessary to train different random seeds to get standard error. Running evaluations through tools like the LM Evaluation Harness [4] automatically generate standard error, which helps the reader to identify meaningful differences.  \nThis seems especially critical for comparisons between Uniform and CLIMB models, where scores are often very similar. Whether or not these differences are significant impacts the fundamental argument in this paper. \n* The results are presented differently for the 7B (Fig. 6) and 1B (Table 2) models. The 7B models aren’t compared to any existing multilingual models. I found unclear which model the results in Table 2 were for. The appendices fail to report any evaluations for the 7B models as far as I can tell. The only comparisons that can be made are with the models trained by the authors (though, again, no standard error is reported, so it is unclear whether differences in performance are meaningful). The 7B results are not as convincing, therefore, as the 1B models. If CLIMB is not as effective at the 7B scale, this would significantly impact the argument of the paper. \n* The use of ‘state-of-the-art’ (L287) seems inappropriate since the authors do not show that CLIMB significantly outperforms the existing models. In several cases, Qwen numerically outperforms CLIMB (1B). There are no comparisons at all on the 7B scale.\n* The paper is missing some critical details about the translated benchmarks. The only information about how the benchmarks were translated is one sentence in Appendix E, stating that the authors use “GPT-based translation” (L115-116). The authors should at the bare minimum report what kind of translation system was used to translate the benchmarks. Additionally, the authors should add discussion about how well those systems perform on the different languages that you translate the benchmarks into and whether any additional checking was done to verify translation quality. \nThe authors do not explain why they translate benchmarks instead of using existing multilingual translations of those benchmarks (HellaSwag [5, 6]; GPQA [7, 8]; TruthfulQA) [9]) or use existing (not machine translated) benchmarks (Belebele for reading comprehension in 122 languages [10], MultiBLiMP for grammatical generalization for 101 languages [11]). There are at least two multilingual mathematical reasoning benchmarks MGSM [12] and MCLM [13]. \nThe evaluation results would be more convincing if the authors remove their machine translated benchmarks and use existing (ideally, at least human-validated machine translated) benchmarks.\n\n[1] https://aclanthology.org/2024.emnlp-main.236/ \n\n[2] https://aclanthology.org/2020.acl-main.747\n\n[3] https://arxiv.org/pdf/2112.10668 \n\n[4] https://github.com/EleutherAI/lm-evaluation-harness/tree/main \n\n[5] https://arxiv.org/abs/2410.08928 \n\n[6] https://arxiv.org/abs/2307.16039 \n\n[7] https://arxiv.org/abs/2504.11833? \n\n[8] https://arxiv.org/abs/2502.07346 \n\n[9] https://arxiv.org/abs/2406.14434? \n\n[10] https://aclanthology.org/2024.acl-long.44/ \n\n[11] https://arxiv.org/abs/2504.02768? \n\n[12] https://openreview.net/forum?id=fR3wGCk-IXp \n\n[13] https://arxiv.org/abs/2502.17407?\n\nQUESTIONS:\n* Appendix D (L70) implies that all models were trained using the same custom tokenizer. Is that correct? If so, how much of the tokenizer represents the different languages used in these experiments? How does compression differ across the languages? How might this impact the results of these experiments?\n* For GlobalMMLU, did you use the culturally specific or culturally agnostic subset (or both)?\n* Regarding the results in Appendix F, why do you think CLIMB underperforms existing models so consistently in GMMLU, GPQA, and TruthfulQA? Qwen beats it on almost every language. These are the more recently developed benchmarks and the ones that are most relevant in current research. \n\nMisc.\n* The lack of bolding in Table 8 makes it hard to readNo bolding on Table 8? Makes it hard to read. But I think this way of organizing the models together is best. \n* Weird spacing in “Evaluation Benchmarks” (L275-281)\n\nLIMITATIONS:\nNo, this paper does not discuss the limitations of the evaluation method, in particular the potential impact of using machine-translated benchmarks. The authors state that they plan to release these benchmarks, however, without human validation, they may be adding low-quality benchmarks into the ecosystem, which may negatively impact work on the languages this paper works on.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multilingual optimization framework, called CLIMB, to address the challenges to determine the optimal language ratios in the multilingual LLM pre-training. Experimental results demonstrate that the proposed approach achieves better accuracy and consistently outperforms baselines on a variety of multilingual tasks at different size of model parameters. \n\n- The paper is well motivated and mostly easy to follow. However, each figure is too small to read. Please enlarge them in the camera-ready. \n- The authors conducted extensive experiments across different languages in a variety of multilingual tasks. The experimental results with both 1.2B and 7B models support scalability of the proposed approach. Overall, the proposed approach is technically sound.\n- Have you increased the language coverage up to > 100 languages to test the massively multilingual settings? Temperature sampling is a well-known technique to balance language data during multilingual training, and I was wondering if you gave simple comparison.\n- Any insights on which languages (e.g., low-resource languages) mostly benefit from this approach?\n\nSTRENGTHS AND WEAKNESSES:\nstrengths\n- The paper is well motivated and mostly easy to follow. However, each figure is too small to read. Please enlarge them in the camera-ready. \n- Extensive experimental results. It shows that the proposed approach is technically sound.\n\nweakness\n- while the authors explores multilingual experimental settings, it'd be interesting to conduct an experiment in a massively multilingual settings with nearly 100 languages. The language coverage up to 16 languages or so look relatively less complexity. \n- Along with that, it'd be nice to analyze the effectiveness of the proposed approach per language or language family\n\nQUESTIONS:\nsee the summary.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CLIMB, a new method to optimally allocate different languages in the training data for large multilingual language models. It introduces a way to measure how languages interact and influence each other during training, then uses this to find the best mix of languages to improve overall model performance. Experiments show that models trained with CLIMB’s language proportions perform better or as well the baselines in the paper.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The approach builds on scaling laws, an influential concept in LLM research, and innovatively extends it with cross-lingual interaction metrics.\n- Most of the benchmarks are multiple-choice questions; however, they include translations as well, which is a plus since they do not limit themselves to non-generative tasks.\n\nWeaknesses:\n\n- Weak baselines [Major reason \"weak evaluation\" for score 2]: The paper did not compare with more effective sampling methods such as UniMax (https://arxiv.org/pdf/2304.09151) or even simple temperature-based sampling (see the same paper).\n- Most benchmarks are automated machine-translated. We definitely have benchmarks that are not, such as Belebele, covering more than 100 languages (https://arxiv.org/abs/2308.16884).\n- The number of tokens is treated as a variable in the paper, but the model size (number of parameters) should also be considered as a variable (they have two sizes: 1.2B and 7B). Otherwise, conclusions might be confounded by this factor and may not hold true across different model sizes.\n- The order of operation is quite high (I did not calculate the exact value; this is based on the number of nested loops). Would you mind correcting me if I am wrong and calculating the number of times training occurs for a given number of languages n?\n\nSuggestions:\n\n- Update the FineWeb2 citation for the camera-ready version, as it now has an arXiv paper (https://arxiv.org/abs/2506.20920).\n- Missing highly related work: This paper https://arxiv.org/abs/2404.07982 shows the role of language imbalance (which you are optimizing) in cross-lingual generalization (which you are measuring).\n\nQUESTIONS:\nThe paper is clear. I have mentioned the weaknesses in the weaknesses section. The authors can address these weaknesses by responding to them as questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}