Title: Scaling Laws for Optimal Data Mixtures
Venue: NeurIPS 2025
Accepted: yes
Keywords: Scaling laws

large models

data mixture
Abstract: Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance. The standard approach to selecting this mixture relies on trial and error, which becomes impractical for large-scale pretraining. We propose a systematic method to determine the optimal data mixture for any target domain using scaling laws. Our approach accurately predicts the loss of a model of size $N$ trained with $D$ tokens and a specific domain weight vector $h$. We validate the universality of these scaling laws by demonstrating their predictive power in three distinct and large-scale settings: large language model (LLM), native multimodal model (NMM), and large vision models (LVM) pretraining. We further show that these scaling laws can extrapolate to new data mixtures and across scales: their parameters can be accurately estimated using a few small-scale training runs, and used to estimate the performance at larger scales and unseen domain weights. The scaling laws allow to derive the optimal domain weights for any target domain under a given training budget ($N$,$D$), providing a principled alternative to costly trial-and-error methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the use of scaling laws to select data mixtures for pretraining language and multimodal models. They adapt the Chinchilla scaling laws to account for a vector of domain weights (= proportion of each domain/source in the data mixture, that sums to 1). They propose the following method to select optimal data mixtures: \n\n1. Train a set of small models (evenly spaced N and D) with different domain weight vectors (also evenly spaced on a grid).\n2. Minimize the selected loss function (on source or target domain) to fit scaling laws.\n3. Select the data mixture with the best predicted (extrapolated) loss value.\n\nThey demonstrate that their method holds for new data mixtures and domain weights.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Very clearly articulated problem, very relevant for pretraining. The experimental setup is quite thorough - including multimodal as well as language models. Results are also very well organized.\n2. I particularly appreciate that the authors have explicitly listed the GPU hours used for training the models.\n\nWeaknesses:\n\n1. The largest model size that was validated is 8B. While I appreciate that these experiments are expensive to run, language models in particular exhibit “lift-off” on certain capabilities beyond that scale, which can cause a shift in the fitted scaling laws, depending on what’s used as the target loss function. It will be helpful to include some discussion of this in the paper.\n\nQUESTIONS:\n1. How are the hyperparameters (eg. LR, batch size) set? Are they the same for all model sizes, or do you spend compute finding optimal HPs for each model size? Will the results hold with different HPs?\n2. You mention that the model sizes are varied by changing only the width dimension. Did you try varying num_layers at all? I’ll be curious to know if that significantly impacts the error rates.\n3. The GPU hours that you’ve listed - do they include the target model sizes on which the method is validated?\n4. You mention that only 10-20 runs are needed for fitting scaling laws. How much compute is that in terms of % of target model? It will be helpful to highlight that as well.\n5. It looks like the number of runs required tends to scale with the number of domains. Is that depended on the size of each domain, as well? Eg. if the domain is too small / fine-grained, do the results still hold? More generally, is there a minimum threshold for the domain size?\n6. Re: cosine LR scheduler, it’s very interesting that the optimal domain weights seem to be similar to those estimated with constant LR. Do you think it’s possible to fit scaling laws based on models trained with constant LR scheduler, and directly extrapolate the perf of a target model trained with cosine LR?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the prediction of model loss as a function of data size, model size, and domain weights. The paper proposes several parametric functions to model this relationship and evaluates their prediction accuracy in the pre-training of LLMs and multimodal models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper examines an important and challenging topic, particularly given the substantial computational costs involved in performing the experiments.\n2. The paper evaluates both LLMs and multi-modal models, a scope that is rarely explored in the literature.\n\nWeaknesses:\n1. As discussed in the related work section, there are several papers that share a similar goal of modeling the model loss and/or identifying the optimal mixture (such as DoReMi, Regmix, Data mixing law, and Doge). However, this paper lacks an empirical comparison with these papers. In particular, the paper overlooks a comparison with AutoScale [1], which addresses the same problem and appears to be more computationally efficient in scaling law fitting since their scaling law is more parameter-efficient.\n2. Related to the first weakness, the experiment's empirical evaluation is limited. The authors do not clearly state the evaluation metrics and appear to rely solely on the convergence of training loss to assess a mixture's effectiveness. A comprehensive evaluation would include validation loss and performance on relevant downstream benchmarks, which are of greater interest to practitioners.\n3. The paper's proposed scaling law as a function of mixing ratio lacks theoretical justification or intuition for its design.\n\n[1] Kang et al. AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs. 2024.\n\nQUESTIONS:\n1. What is the computational cost required for fitting the parameters of the scaling law? How does it compare to the cost of training the model at the target scale? \n\n2. Is there any intuition why cosine learning rate and constant learning rate yield similar domain weights?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates pre-training hyperparameters for Neural Multimodal Models (NMM) and Large Language Models (LLM) to explore scaling laws. It details the specific configurations and optimization strategies used, including model sizes, learning rates, optimizers, batch sizes, and data augmentations. A key aspect of the work is the comparison of their proposed scaling laws, which account for model scale (N), data (D), and data mixture (h), against existing laws that primarily consider data mixture alone. The authors demonstrate that their additive scaling law consistently achieves lower Mean Relative Error (MRE) on both training and testing datasets compared to alternative models. Furthermore, the paper explores the concept of optimal domain weights, revealing that training on a pure target domain often leads to optimal weights concentrated in that domain's corner of the probability simplex. The analysis of Jensen-Shannon distance suggests that, in general, training on a mixture different from the target mixture can be more effective for minimizing loss.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. The paper integrates Scaling Laws into Data Mixing. The paper introduces a principled method to determine optimal data mixtures. These laws predict model loss based on model size ($N$), training tokens ($D$), and data mixture ($h$), a comprehensive approach that surpasses prior work considering only mixture for fixed $N$, $D$. Fitting these laws on small-scale runs allows accurate extrapolation of performance and optimal mixture identification for larger, unseen models.\n2. Consideration of Multi-modality in Data Mixing. This paper uniquely extends data mixture optimization to multimodal models. It applies scaling laws to Neural Multimodal Models (NMMs) that process interleaved text and image tokens.\n\n**Weakness**\n1. The difference between single-modality and multi-modality in the data mixing problem is not very clear. I don't get how to treat these two cases differently in the formulation. Or they share the same method while just can apply to two different settings in applications. Could you please elaborate on this more?\n2. One domain can only include one type of data in multi-modality setting. In my understanding, one domain in the paper can only include one type of data like text or paired. How to process it if one domain includes multiple types of data like both text and paired?\n3. Missing the analysis of computational cost. While the authors claim that _\"Only 10-20 runs are needed for determining the scaling laws.\"_ in line 229. It's unclear what sizes of models it requires and how much computational cost it uses. \n4. It would be better if compare with other relative data mixing baselines like Data Mixing Laws [1] and RegMix [2]. These two papers also use multiple models with different sizes to predict domain weights for larger models. Could you please clarify the difference compared with them, including the prediction accuracy and computational cost? \n5. Missing downstream task evaluation. The paper only uses loss as a metric while the performance on downstream tasks is closer to reality.\n\n[1] Ye, Jiasheng, et al. \"Data mixing laws: Optimizing data mixtures by predicting language modeling performance.\" _arXiv preprint arXiv:2403.16952_ (2024).\n\n[2] Liu, Qian, et al. \"Regmix: Data mixture as regression for language model pre-training.\" _arXiv preprint arXiv:2407.01492_ (2024).\n\nQUESTIONS:\n1. How to treat the data changes? Does this method have to retrain multiple models if new data/domain is added or some domains are deleted? \n2. Does this method allow for different model architectures? For example, using multiple small models to fit the scaling law while using another larger model with a different architecture to train. Would it still get a benefit?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers a new scaling law study for data mixtures (simplex vector h) as a function of model size N and data size D. They experiment with two different types of scaling curves, additive and multiplicative (depending on the dependency between params/data and datamix weights). They first show an efficient way to fit these parameters with limited training runs, and then show that the estimation extrapolates sufficiently well to large-scale training. They provide ablations using pre-training MLLMs and LLMs with different scale of domain mixtures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n--------------\n\n1. The paper considers a very important problem of robustly estimating the datamix weights for large-scale pre-training using small-scale runs with impressive results.\n2. The actual modifications made to the standard scaling law is minimal (easy to adopt and optimize) yet effective (fits most training regimes well) validating the strengths of the hypothesis. \n3. Extensive empirical study (using ~300k GPU hours) on multimodal as well as text LLMs with several important conclusions and observations which go beyond current scaling study (like importance of text data for Large LLMs in fig 5). \n4. The paper provides insights into more efficient optimization strategies as the #domains scale up, which might be important for post-training stages which involve several different mixtures and domains. \n5. The paper honestly states several limitations, which is important to drive future work in this area.\n\nWeaknesses\n-----------------\n\nPlease refer to the questions below.\n\nQUESTIONS:\n- The paper does not investigate the case where there are lot of domain mixtures - like on the order of 100. It is very common in post-training to have diverse set of mixtures (unlike pre-training with homogenous mixtures). They authors can add their thoughts on this. \n\n- I am curious why did the authors choose inverse scaling law for h in eq 4? For N,D - it is intuitive because as we scale models and data, the loss should go down. But what might explain the fact that the coefficients are inversely related to the loss (in Eq 4 and 5)? \n\n- Minor: Looks like D and N have interchanged coeffiencts between Eq 3 and 4?\n\n- Minor: Having grid lines in Fig 3 might make it more clear to understand the slope of observed vs. predicted loss curves.\n\nLIMITATIONS:\nThe paper adequately discusses several limitations in sec 8, several of which I concur with.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-04"}