Scaling Laws for Optimal Data Mixtures
Authors: Mustafa Shukor
Louis Béthune
Dan Busbridge
David Grangier
Enrico Fini
Alaaeldin El-Nouby
Pierre Ablin
Venue: NeurIPS 2025 poster ()
Keywords: Scaling laws
large models
data mixture
Primary area: deep_learning
TL;DR: We propose scaling laws that predict the loss of models when trained on a mixture of source domains.

Abstract: Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance. The standard approach to selecting this mixture relies on trial and error, which becomes impractical for large-scale pretraining. We propose a systematic method to determine the optimal data mixture for any target domain using scaling laws. Our approach accurately predicts the loss of a model of size $N$ trained with $D$ tokens and a specific domain weight vector $h$. We validate the universality of these scaling laws by demonstrating their predictive power in three distinct and large-scale settings: large language model (LLM), native multimodal model (NMM), and large vision models (LVM) pretraining. We further show that these scaling laws can extrapolate to new data mixtures and across scales: their parameters can be accurately estimated using a few small-scale training runs, and used to estimate the performance at larger scales and unseen domain weights. The scaling laws allow to derive the optimal domain weights for any target domain under a given training budget ($N$,$D$), providing a principled alternative to costly trial-and-error methods.

Scaling Laws for Optimal Data Mixtures
Mustafa Shukor
Louis Bethune Apple
Dan Busbridge Apple
David Grangier Apple
Sorbonne University
Enrico Fini
Alaaeldin EI-Nouby
Pierre Ablin
Apple
Apple
Apple
Abstract
Large foundation models are typically trained on data from multiple domains, with the data mixture-the the proportion of each domain used-playine critical role in model performance. The standard approach to selecting this mixture relies on trial and error, which becomes impractical for large-scale pretraining. We propose systematic method to determine the optimal data mixture for any target domain using scaling laws. Our approach accurately predicts the loss of model of size trained with D tokens and specific domain weight vector h. We validate the universality of these scaling laws by demonstrating their predictive power in three distinct and large-scale settings: large language model (LLL)), native multimodal model (NMM), and large vision models (LVM) pretraining. We further show that these scaling laws can extrapolate to new data mixtures and across scales: their parameters can be accurately estimated using few small-scale training runs, and used to estimate the performance at larger scales and unseen domain weights. The scaling laws allow derive the optimal domain weights for any target domain under a given training budget (N,D), providing principled alternative to costly trial-and-enee methods.
Model
106M
616M
2.38
Training runs
3.6
Mass Muuse
Evaluation
Scaling prodiction
3.1
3.4
Loss
Loss
3.2
Extrapolation
Image- Caption
3.0
Text
Interlavedd
2.8
20
10
1020
10
FLOPs
FLOPS
Figure 1: Scaling Laws for Optimal Data Mixtures. Left: We derive scaling laws that predict the loss of a model as function of model size number of training tokens D, and the domain weights used to train the model (represented by the color of each point). The scaling law is fitted with small-scale runs with different domain weights, and used to predict accurately the loss large-scale models trained with new, wnseen domain weights. Right: We find the data mixture scaling law based on small-scale experiments (e.g., below 1B parameters) and use it to predict the optimal data mixture at larger scales (e.g., 8B parameters). Both our additive (eq. (4)) and joint (eq. (5)) laws lead to similar performance, and better than other mixtures (in the gray area). FLOPs are computed as 6ND.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Modern machine learning models [8, 16, 21] are pre-trained on diverse data domains, such as text for LLMs, images for vision models, and mixed modalities for multimodal models. For LLMs, these domains encompass general knowledge, code, reasoning, multilingual content, and more [5, 16, 26, 59, 60]. Multimodal models [1, 35, 44, 56, 70] are trained on mix of text, paired, and interleaved multimodal data, and finally, large vision models are trained on image domains of different qualities, containing or not images paired with text [17, 21, 48].
The domain weights determine the proportion of each domain used during training, significantly impacting model performance. However, these weights are typically chosen through ad-hoc trial and error, involving training with different domain weights and selecting what works best [16, 44, 56]. Despite their critical role, a principled method for selecting domain weights is largely absent.
Scaling laws provide a theoretical framework to predict model performance. Initially developed for LLMs [29, 31, 33), these laws model the loss of a model as a function of the number of parameters N and training tokens D. This framework has been extended to other domains and modalities [3, 56] and to account for factors such as the number of experts in mixture-of-expent models [34], sparsity [2], data repetitions [47], fine-tuning tokens [7, 69], and learning rate schedules [42].
In this work, we extend scaling laws to model the effect of domain weights on model performance. We show that the model loss depends in predictable way the domain weights, interacting with the number of training tokens and model parameters. We extensively validate our scaling laws in three large-scale settings: large language models (LLMs), native multimodal models (NMMs), and large vision models (LVMs) pretraining. We train large models up to 7B parameters and 150B tokens for LLMs, 8B parameters and 160B tokens for NMMs, and 1B parameters with 330B tokens for LVMs. across multiple text, multimodal, and image domains. The key takeaways from our work are:
Scaling laws that extrapolate. We demonstrate that our scaling laws can be fitted using small-scale runs, and then provide an accurate prediction of the loss of large-scale models trained with new, unseen domain weights. This is illustrated by Fig. 1, left, where we accurately predict the loss of models trained with an order of magnitude more compute than the small-scale runs.
Optimal domain weights estimation. Once fitted, these scaling laws give an accurate estimation of the loss function of the domain weights, which is minimized to give optimal domain weights. This provides a principled alternative to the costly practice of trying different domain weights and selecting the best one. This is illustrated by Fig. 1, right, where we report the average loss of NMMs.
This paper is organized as follows. In Sec. 2, we introduce the problem of domain weight selection and describe our scaling law formulations. In Sec. 3, we detail the model architectures and data domains for I.MM, NMM, and LVM pretraining. Sec. 4 demonstrates that our scaling laws accurately extrapolate to new domain weights, larger model sizes and number of tokens. Sec. 5 shows how the fitted laws can be used to estimate the optimal domain weights from few small-scale runs. Finally, Sec. explores various aspects of our scaling laws, including quantifying the number of runs needed to get satisfying estimation, how the best domain weights change when scaling flops, as well as alternative scaling laws formulations. Finally, we discuss related works in Sec. 7.
2 Data mixture scaling laws
2.1 Problem setup
We consider training models with data coming from data domains Di...., D&: we can query random samples x from any domain D, Consequently, we can sample from the mixture mix(h) = Σ h, for any domain weights h. following the law p(x)mmi(hh) = =E= hsp(x/D;). Here, h is a dimeensiio vector of positive entries that sum to one, that is, an element of the simplex AR- In plain words, data is sampled from the domain with probability h, We have target domain Dr, which may - but need not - be one of the training domains. We consider a model with N parameters, represented with the vector of parameters 0 E N Finally, we have loss function ((z, f) defined for any in the data domains D, or the target domain Dr. This defines the loss for any domain weights as well as the target loss, as the expectations
Li (0) Ex [e(z, and (0) ELNDY [((z,0)]
(I)

---

The training of the model, with fixed domain weights h, is done by running an optimization algorithm such as Adam to approximately minimize Ln. In the course of its execution, the optimization algorithm processes D tokens and outputs trained parameters 0" (h. D) of size N. The goal of this paper to predict the loss on the target domain Dr after training a model of size N with D tokens with domain weights h; quantity denoted as L(N, D, h) defined as I- (h. D)). In practice, we can, of course, have several target domains that capture different aspects of a model's capabilities, In that case, we estimate the target loss on all of the target domains by fitting multiple scaling laws.
This framework is sufficiently general to encompass various model architectures and modalities. In this work, we consider different domains to be either various text domain datasets, various image domains, different modalities (e.g., image and text), or different data types (e.g., paired or interleaved).
2.2 Scaling laws derivation
In their original form, scaling laws predict the training loss of model of size N trained with D tokens [33]. The Chinchilla scaling law models training loss as an additive power law [31]:
A
B
=
No
D
where E, A, and 3 are parameters that depend on the training set, model's architecture, and optimization algorithm. We depart from these original scaling laws in two ways: i) we consider the loss of model on targer domain that need not be the training domain, and more importantly, ii) we quantify the impact of the domain weights h on the loss. Regarding i), as already been shown in several works [24, 29, 46, 56], the loss target domain can still be modeled by scaling law of the form eq. (2). Hence, for every domain weights h used for training, we expect the loss on the target domain to follow Chinchilla power law, where the coefficients depend on h. In other words, the loss on the target domain can be expressed as:
A Bh Na +
C(N, D,h) F. +
D
The question now is, how do the parameters ER A a², Bhh and 3hh depend on h? We propose two different formulas that use simple formulas for these parameters. We first study the additive scaling law, in which E is modeled as a function of h, while the other parameters are taken as constants:
1
A
B D8
E
+
C.h,7
No
The parameters of the scaling law are (F, A,B,A, a, 3, (1))) which depend on the model architecture, the target and the source domains. This scaling law has 5 2k parameters. Since this scaling law is additive, the optimal domain weights h² that minimize it are independent of the model size N and the number of tokens D. In order to capture the interaction between scale and mixture, we also propose the joint scaling law:
A
B with A
& E CA and Bh = CC hh
k
L- E C.P
+
+
(5)
Na
D8
In that scaling law, we consider the same dependency in h for the bias term É as in the eq. (4), and we additionally model the terms A and 8h as simple functions of h. The parameters of the law are 74, law, there is an interaction between N. D and h, in the sense that aNOR # 0 and 8²c aDaA # 0. while these partial derivatives are 0 for the additive scaling law. This law predicts that the contribution of N and D to the loss depends on the domain weights, and as such, the optimal domain weights are compute-dependee
Z = (E, a, B, (C.) (C,4)) )
(C, ) )), which gives 5 + 4k parameters. In this
8ºc
The joint scaling law is more expressive than the additive scaling law, since we can recover eq. (4) by taking 7 = 7 1 and C.A = A, C B for all domains i. As such, if the scaling laws are fitted properly, the error on the training runs is always lower for the joint scaling law than for the additive scaling law. The joint scaling law still models the terms a and 3h as constants. We tried modeling these terms using the same simple parametric form depending on hi but it only yields minor improvements (see Sec. 6). On the other hand, going from the additive to the joint law often decreased the estimation error. Hence, we restrict the bulk of our study to these two laws.
3

---

2.3 Fitting the scaling laws
In order to fit the scaling laws, we launch several training runs with different domain weights h. model sizes N, and number of tokens D, and record the loss on the target domain Lr. We train model sizes and number of tokens that evenly spaced. We chose the training domain weights by taking grid of evenly spaced points in the simplex, where each domain weight is above a minimal value (i.e., 0.1).
We have input target pairs (N², D h²"), Ly for = I.....P where p is the number of training runs.
The optimal parameters Z obtained by minimizing the standard Huber loss:
Scaling law fitting loss
10-5
Basin hopping
1 H(Z)
Σ P j=1
P
Huber (14--(()) (ONN,,,, D2,12 (6)
L- BBGS
where Huber(x) = if < 5 and -1121-3] otherwise, where 5 is a hyperparameter which we take as 5 = 1e-3.
10- 10a
10
10
# Calls L-BFGS
The standard technique to fit scaling laws consists of us- ing L-BFGS [39] to minimize the loss starting from an evenly-spaced grid of initial parameters Z and then retain- ing the smallest local minimum. Contrary to most scaling laws that involve only 5 parameters, our scaling laws have + 24 or 5 + 4k parameters, where is the number of domains. In our experiments, we up k domains, which gives 37 parameters to fit. This increased dimen- sionality makes the standard technique cumbersome. We propose two changes that lead to good fit. Firstly, we use random search, sample the initial parameters Z. Secondly, we use the Basin-hopping algo- rithm [63] instead of L-BFGS to explore the minimizers of the loss function. The Basin-hopping algorithm itself uses L-BFGS an inner routine to minimize the loss function. but it also uses a random walk to explore the space of local minima. Fig. 2 gives an example of the performance of the algorithm: to reach low fitting loss, the Basin-hopping algorithm requires far fewer calls to L-BFGS than doing random search over the L-BFGS initializations.
Figure 2: Value of the Huber loss (6) as function of the number of L-BFGS calls fit eq. (5) on the Interleaved do- main from the multimodal experiment (p = 1062 umput-targe' pairs, h domains). We repeat 100 random tri- als, the bold line the median, and the shaded regions are the 25-75% quan- tiles. The Basin-hopping method with BFGS subroutine converges faster than repeated calls to L-BFGS.
fit scaling laws
In order evaluate the scaling law, take new set of runs that give different values of (N, D, h). and compare the loss on those runs predicted by the scaling law against the actual loss achieved by the model. We quantify this with the Mean Relative Error (MRE), computed as lprediction observation| / /observation, and we report it as a percentage.
Experimental setup
We give overview of the models and domains used in our experiments. Detailed architectures and hyperparameters are given in appendix A.
3.1 Pretraining of large language models (LLMs)
Models. We use transformers [62] for autoregressive language modelling. We use the same setup as Ilama [61], with rotary positional embeddings, SwiGLU activations, and RMSNorm. The models are scaled by changing the latent dimension, with model sizes ranging from 186M 7B parameters.
For some smaller-scale analyses, we also use GPT2-style transformers [51] to perform autoregressive language modeling with model sizes ranging from 90M to 3B parameters.
Datasets. For the main experiments, we use the k: = 7 domains from slimpajama [57]. We use these domains as distributed by the authors, without any additional data filtering. For some smaller-scale analyses, we use up
domains coming from the Pile dataset [22]: Wikipedia, StackFxchange, GitHub, 19, arxiy, free law, openwebtext, and PubMed Central.

---

3.2 Pretraining of native multimodal models (NMMs)
Models. We pretrain native multimodal models (NMMs), based on an early-fusion architecture [6, 56] and follow the design and implementation proposed in [56]. The model consists of a single transformer [62] without separate vision encoder, resulting in the same architecture used for LLMss The model processes sequence of interleaved text and image tokens. Text tokens are obtained using a standard LLM tokenizer, while image tokens are obtained by patchifying the image and applying a linear projection. Images are resized to 224x224 resolution with a 14x 14 patch size. The overall model architecture is aligned with [36], incorporating SwiGLU FFNs [54] and QK-Norm [15].
Datasets. Following previous works [35, 38, 56] we train on mixture of multimodal datasets, covering & = 3 data types: (1) text-only data from DCLM [36], (2) interleaved multimodal documents from Obelics [35]. and (3) paired image-caption datasets from DFN [20], COYO [10], and a private collection of High-Quality Image-Text Pairs (HQITP).
3.3 Pretraining of large vision models (LVMs)
Models. We pretrain large vision models with multimodal objective, following the AIMv2 recipe [21]. Unlike traditional language modeling or multimodal models described above that focus on text decoding. AIMv2 trains vision encoder using an autoregressive objective both image and text tokens. The model architecture is composed of a vision encoder and a multimodal decoder stitched together in late-fusion fashion.
Datasets. We train on a mixture of paired of image-caption drawn from four domains (A: = 4): (1) noisy alt-text including COYO-700M [10] and DFN2B [20], which provide large-scale real- world image-text pairs with varying levels of noise and quality; (2) HQ-ITP-I, a high-quality dataset containing 134 million samples; (3) HQ-ITP-2, another high-quality dataset comprising 400 million
samples; and (4) synthetic data, consisting of recaptioned versions of DFN2B and HQ-ITP-2.
3.4 Implementation details
In order to scale models, change the hidden dimension size in the transformers d, keeping fixed number of layers. To reduce the experimental cost, most of the experiments are done with a constant learning rate scheduler. This allows us to collect many points with varying numbers of tokens D for each run, rather than just one per run. We also validate our findings when using cosine learning scheduler in appendix C.I, where we show that the scaling laws also extrapolate from small-scale to large-scale behavior in that case.
4 Predicting large-scale performance from small-scale experiments
In this section, we demonstrate that (a) our scaling laws accurately capture the training data, and (b) generalize effectively to larger scales with significantly increased values N and D. To this end, we fit the laws using small models trained with a small number of tokens, and we validate them on larger models with a large number of tokens. We experiment with LLMs, trained with a mixture of text domains, NMMs, trained with a mixture of multimodal domains, and LVMs, trained with images of different qualities and paired or not with text. For LLMs, we consider the k domains from slimpajama, which are different text domains. For multimodal pretraining, and similar to previous works, [35, 56, 70], the data mixture spans = different domains: text, paired (image-captions), and interleaved multimodal data. For large vision model pretraining, we use k = 4 domains. Tab. 1 displays the different model sizes, number of training tokens, and number of different domains weights that use to train and evaluate the scaling laws.
Results. Fig. presents a comparison between the actual loss achieved by our trained models and the loss predicted by our scaling laws. We summarize the results on each domain by showing the average predicted loss (full results appendix B). Remarkably, the predicted losses align closely with the observed values for both the joint and additive laws. In addition, the laws show good extrapolation to larger model sizes. To further quantify this alignment, we report the mean relative error (MRE% in Tab. 2, which reveals a consistently low MRE% for both laws, with an improvement of the joint law over the additive one. These results demonstrate that we can the scaling laws on small scales and extrapolate to larger scales. We remark that the MREs have some degree of variability across
5

---

N:B (train)
3B (eval)
N:. IB (train)
N: 2B (eval)
N: SB (oval)
NS530M (train)
IB (eval)
LLM, Additive Law
NMM, Additive Law
LVM, Additive Law
3.8
Predicted Loss
2.75
3.1
2.50
2.25
3.0
2.5
2.00
2.0
2.00
2.50
Observed Loss
Observed Loss
Observed Loss
LLM, Joint Law
NMM, Joint Law
LVM, Joint Law
3.6
Predicted Loss
2.75
2.50
3.2
2.25
3.0
2.00
2.0
2.00
2.25
2.50
2.75
2.8
3.0
3.2
3.4
3.6
2.0
2.5
3.0
Observed Loss
Observed Loss
Observed Loss
Figure 3: Observed vs predicted loss for LLM pretraining on domains from the slimpajama dataset, NMM pretraining with multimodal domains, and LVM pretraining with image-caption domains. The scaling laws are fitted on small-scale models (blue points in the figure) and extrapolated larger models. We display here the average loss over all domains for each modality. The MRE% for each domain is reported Tab. 2.
Table 1: Experimental setup for the extrapolation experiment.
Table 2: Scaling laws MRE
# domaii weights
PLOPs
412M
4-20B
60
Modality
Target domain
MRES (Additive / Joint law)
K34M
7-268
10
Arxiv
0.50/0.39
2.09/ 1.62
LLM
1.48
13-468
20
book
0.29/0.24
0.80/1.19
C4
0.29/0.24
0.31 /0.34
20-1008
10
Language
Github
0.65/
1.17/.51
7B
150B
6.3 x 10
Commoncrawi
/..9/0.24
0.58/0.90
106M
20-1008
32
Stackexchange
0.51/0.35 /0.38
0.36/0.2
275M
20-1006
3.7 x 2
Wikipedia
NMM
616M
932M
20-100B
100B
34
Image-Caption
0.47/0.43
0.95 / 0.95
Multimodal
Interleaved
0.42
100-160B
10P2 3.1 x 1022
Test
0.12/0.10
0.44 0.37
88
100-1608
Noisy image-
0.35/0.23
1.20/063
LVM Train
306M
89M
17-1008
17-50B
17-508
15
x 10
Vision
High-quality
High-quality
Synthetic
0.64/0.31
1.89 1.89/0883
1.19/0.34
2.02/ 1.18
0.79 1.06
6.19 5,94
531M
17-1708
16
x
domains; for instance, on the LLM experiment, we get at the same time an extremely low MRE of 0.31% the C4 dataset and high MRE of 4.45% on Wikipedia.
FLOPs count. We report the approximate computational cost of running the small-scale experiments to fit the scaling laws, and compare to the cost of large-scale runs in Tab. 1. We compute the FLOPs required for one run as 6N D. The cost of large-scale runs is comparable to that of small-scale runs, and the large-scale scale runs could be trained with far more tokens, which would increase their FLOPs.
Optimal data mixtures
Optimal domain weights estimation Once the scaling law is fitted, can derive the optimal domain weights h" that minimize using mirror descent, i.e. iterating h++1 Σ, where h h² exp(-nV,,CN, D,h)), with n
under the simplex constraint E Ap. We solve it

---

Average Loss on the training domains
OpenHermes Loss
Optimal for Average
Optimal for Average
2.1
Optimal for OpenHermes
Optimal for Openieemess
Unifora distribution
Unifoms distribution
2,0
slimpajama disarbuttion
1.4
slimpajama distribution
Loss
Loss
1.3
1.8
1.2
1.7
20
40
60
100
20 40
60
Billion Training Tokens
Billion Training Tokens
Figure 4: Losses of the 7B models. After fitting the scaling laws on the small scale runs, we estimate the optimal domain weights havg that minimize the average loss over the training domains (left), and how that minimizes the loss on the OpenHermes dataset (right). We then train 7B models with these optimal weights, and compare them to two baselines: one with uniform weights, and with the standard distribution of slimpajama. The losses are averaged over all training domains, and also reported on the OpenHermes dataset. As expected, the model trained with b performs best on OpenHermes, while the model trained with havg performs best on the training domains.
small step size. In practice, we may want to obtain a model that works well on several tasks at once, with weights w. that case, we have m different target domains D...... DD. We estimate the scaling law for each target domain Di, and obtain different scaling laws (N,, D,h). The optimal domain weights h" that are good on average are given by:
m Σ D,N.
(N,,, D) E arg min
(7)
hEAL
The behavior of h² depends on the scaling law that we consider. Since eq. (4) assumes an additive relationship. the minimizer of eq. (7) is independent of N, D: in other words, it does not depend on scale. On the other hand, eq. (5) takes into account a multiplicative interaction between N, D and h. Therefore, the optimal h is scale-dependent. If one task more important than another, we can incorporate importance weights in the sum in eq. (7).
The main practical takeaway of our paper is this simplified approach to optimal mixture estimation. Indeed, as demonstrated in Sec. 4, we can accurately fit our scaling laws with small-scale runs. Using these scaling laws, we can then solve eq. (7) for various targets (N, D), which gives principled way of choosing domain weights, rather than using ad-hoe methods as usually done in practice. To demonstrate our point, do this for different modalities considered in the paper
LLM results. Since the additive scaling law gave us the lowest MREs, we use it to estimate the optimal data mixture that minimizes the average loss over the k = 7 training domains, which denote Nang We then train 7B model with 150B tokens with that optimal data mixture.
For all the runs, we also monitor the loss on the OpenHermes dataset, which is small figh-quality dataset used for model alignment. We fit the scaling laws for that domain as well, even though this domain is not part of the pre-training domains. The rationale is that we want to estimate weights that lead to the best performance on this high quality dataset, which should be a proxy of performance on downstream tasks. We then find the optimal domain weights for that scaling law, which we denote hon- and train another 7B model with 150B tokens. As baselines, we train two more 7B models with that many tokens, of tokens in each domain, and one with a uniform distribution over domains.
with the standard distribution of slimpajama, proportional to the number
Since we want the best models possible, we use a cosine learning rate schedule, which makes the scaling law extrapolation impossible to conduct. We report the average loss of these models on the training domains, on the OpenHermes dataset in Figure 4. We also evaluate them on many downstream tasks and report the results in Table The model trained with weights lỘ is overal better than the other models in terms of evaluations. We believe that the pipeline demonstrated in this paper - estimate the scaling law for the loss on a high quality domain with small scale runs, find the minimizer, train large scale model with it promising avenue to get better models.
7

---

Table 3: Evaluations of the 7B models. We report the median score on the CORE tasks [36] as well as other accuracies on standard benchmarks. The model trained with weights that attempt to minimize the loss on OpenHermes performs best.
Weights
CORE
MMLU
Arc-casy
Are-challenge
Boolq
Piqa
Siqa
Hellaswag
Winogrande
hang
56
32
71
40
60
79
52
72
65
koy
37
67
72
65
Uniform
30
46
70
65
Base
51
79
71
65
Multimodal. 3 domains
LLM. 4 domains
LLM, 6 domains
LLM, 8 domains
Eval MRE%%
10
Addiitive Tow
Addirive law
Additive law
Additive law
Joint law
Juint law
Juint low
Juint law
16a
10a
10"
100
10a
100
15
10
1.5
0
40
20 40
# training domain weights training domain weights # training domain weights # training domain weight
Figure 5: Evaluation of the scaling law as a function of the number of training runs. We randomly select q different domain weights htrain = [ ./...... hgl. and only the runs that use these histograms to fit the scaling laws. We then evaluate the MRE on all the domain weights heest that are not part of hurain For the multimodal and LLM with 4 domains, we compute the eval MRE on the large-scale (resp. IB and 8B) models. For the LLM with 6 and 8 domains, we compute the eval MRE on same size models.
NMM results. We fit both the additive and joint scaling laws on three multimodal data domains, using only small models. For the joint scaling laws, we predict the best training mixture h² that minimizes the average of the domain losses for each model size while fixing the number of tokens 100B for practicality. We then train models with these optimized mixtures. Fig. compares the performance of these best mixtures against uniform mixtures, those used in prior works [44, 56], and randomly sampled mixtures that cover an important area of mixture grid. Models trained with our estimated mixtures consistently outperform alternatives. Notably, both additive and joint laws perform similarly well, making additive laws a strong and more practical baseline, since takes the same optimal mixture for all runs. Remarkably, the optimized mixtures generalize effectively to larger model sizes, which validates the possibility of choosing the optimal mixture based on small-scale experiments, and then extrapolating to larger scales.
LVM results. We fit the scaling laws on the AIMv2 data mixture, which consists of domains, and we estimate the optimal domain weights that minimize the average loss over these domains. We then train IB model with these optimal weights, and compare it model trained with uniform weights. We find that the model trained with the optimal weights performs better than the one trained with uniform weights, which validates our approach once again.
6 Scaling laws analysis
In that section, we conduct LLM experiments in a different setting, using the Pile dataset [22], which allows us have a variable number of domains, between ls 4 and k 8.
Only 10-20 runs are needed to fit the scaling laws. We investigate the number of runs needed for an accurate scaling law fitting. We randomly partition the domain weights into heruin [/1,-- ... hgg. of size q, and put the other domain weights into heset. We fit the scaling law on Atrai and report the MRE on the test domain weights. Since the number of parameters of the scaling law depends linearly on the number of domains k, we expect the number of runs necessary to fit the scaling law to increase when we consider more domains. To verify this hypothesis, we consider the NMM pretraining experiments, with &- = 3 domains and I.L pretraining with k: 4, 6, 8 domains. For the NMM with 3 domains and L.MM with 4 domains as considered so far, we fit the laws on small-scale models and compute the MRE on large-scale models as in Sec. 4. For the LLM training with k 6, 8 domains,
8

---

because of the very large search space with a high number of domains, we take single model size and skip the dependency on N in the scaling law, only considering the dependency on h and the number of training tokens D. We report the MRE as function of the number of training histograms q in Fig. 5. We observe that we need about 10 runs for the NMM and L.M with domains to get to an optimal MRE, while we need about 20 for LLM with 6 and 8 domains. Interestingly, we observe that when the number of training runs is very low, the additive law has a slightly lower eval MRE, due to its lower number of parameters.
Optimal domain weights behavior when scaling FLOPs. We study how the optimal mixture h" (N, D) for the average loss evolves as function the compute- budget D) on the multimodal models, predicted by the joint scaling law. We report results in Fig. 6. We see that interleaved data gets less important increase D, whereas bigger models tend to rely more on text. The additive law captures the average behavior across all scales.
Image- Caaage-Caption
Model size. N
30M
1013
308
1013
Number of tolcers
300B
Other scaling laws formulas. We investigate alter- native scaling laws, and validate our proposed scaling laws, by evaluating other formulas in the same setup as in Sec. 4. First, we want to understand whether we could simpler form for the dependency the do- main weights h. To do so, we use the "simple additive" scaling law
10tt
Additive scaling
Text
Interleaved
Figure 6: Evolution of optimal domain weights h with compute budget (N, D) on the multimodal data, as predicted by the joint scaling law (eq. (5)).
A
B N
C C= E
Chh
88)
D
where the dependency in h is simpler compared the additive and joint scaling laws. This law has k - fewer parameters than the additive scaling law. The joint scaling law models the terms A and Bh as functions of domain weights. We want to understand whether also taking dependency of a and 3 on h helps capture more information about models' behavior. To do so, we consider the "full" scaling law:
A
Ba
E+
Σ
C.
Na
+
Daw
with
(9)
k c h, t a
Ah = cAli)" Bh
h and 8r = Σ hh))
(10)
i=1
This law is more expressive than the joint scal- ing law, and adds 2k + parameters. We give the full results of those laws in appendix B, and report the average MRE in Tab, 4. We see that the additive law reduces the MRE lot pared to the simple law, especially in the LLM experiment. Despite having a slightly smaller train MRE, the full law does not extrapolate as well as the joint law. For the LVM experiment, the picture is different: the simple law works better than the additive law, and the full scaling law brings some improvement over the joint law.
Table 4: Other Scaling laws average MRES %
Simple
Additive
Joint
Full
NMM LLM
0,70
0.62
0.58
0.60
1.70
1.39
1.30
1.21
LVM
2.31
2.21
2.04
7 Related works
Scaling laws. Scaling laws research investigates how model performance varies with training compute. Foundational studies [30, 31, 33] established that language models follow predictable power-law relationship between performance and compute, allowing for the optimal allocation of parameters and training tokens within specified budget. Scaling behavior has since been explored

---

across a wide range of domains, including vision models [21, 53], diffusion transformers [37], and other fields [12, 49]. While typical scaling laws consider the number of total parameters, other studies have examined the influence of both width and depth [45], or the number of parameters allocated to the teacher and student in ese of model distillation [9]. Sparse Mixture of Experts (MoF) models have been another focus, with investigations into how factors like sparsity, the number of experts, and routing strategies affect scaling [2. 14, 34, 64]. For multimodal models, scaling laws have been explored in studies such as [3, 56]. Of particular relevance is [56]. which examines native multimodal models. However, their analysis is constrained by a fixed pretraining mixture.
Scaling laws for data mixtures Optimizing data mixtures for model training is critical chal- lenge, often requiring extensive experimentation. Recent studies have begun exploring systematic approaches to identify optimal mixtures more efficiently. For instance, Goyal et al. [25] investigated scaling laws for data filtering in CLIP models, emphasizing data quality and repetition. Gu et al. [28] examined scaling laws for continual pretraining of language models, predicting the optimal balance between pretraining and domain-specific data, while Bethune al. [7] followed the same approach, focused on forgetting in finetuning. Similarly, [11] derived scaling laws that account for data quality factors such as diversity. Closer to our work, Ye et al. [68] and Ge et al. [23] propose scaling laws that model the loss as function of h for fixed (N, D). but they do not consider joint law for (N, D, h) as we do here. We also find that, in our experiments, for a fixed (N, D), our scaling laws extrapolate generally constrained to
better to unseen mixtures (see appendix B). Further, these approaches single modality, and they consider relatively small models, below IB parameters.
Data mixture selection. The standard approach to selecting training data mixtures relies on trial and error, where different combinations are tested to determine the best-performing mixture [21, 35, 38, 44, 55, 58, 70]. However, this method is costly, leading to recent efforts exploring alternative strategies. Some studies adopt heuristic methods, adjusting mixture ratios based on data sizes for each domain [13, 27, 52] or to match a target task's distribution [26]. Others predict model performance using small models that take the mixture as input [4, 19, 40, 67]. third approach employs auxiliary models to rank and select high-quality training data, which has been popularized recently by large foundation models [8, 16, 50, 65, 66].
8 Discussion
Limitations. Our current study is focused pretraining, but continual pre-training and finetuning are also scenarios in which the mixture is important. Our scaling law predicts a generic target loss [33]. which is known to correlate with downstream task performance [31, 433. Future work may involve predicting this performance directly, like [32]. Furthermore. assuming no data repetition (i.c.. an infinite stream of data from each domain), as typical for LLM pretraining, training with very scarce, high-quality sources. Finally, we assume that the mixture is fixed throughout training, but future works may consider dynamic evolution of the weights (e.g., curriculum learning).
unrealistic when
Broader impact. Mixture coefficients have a tremenduous impact on the performance on downstream tasks. Modern training corpora are typically a combination of dozens of sub-domains, striking balance between diversity and quality. Giving the cost of pre-training, finding the optimal mixture through extensive trials and errors can be prohibitively expensive. Our scaling law only require a few runs at small scale to yield meaningful coefficients for larger models. Our work also has environmental benefits, as it significantly reduces the cost of pre-training, including the amount of CO2 emission and the energy needed. Moreover, it may yield better models in the long run.
Conclusion. We propose scaling laws that predict the loss on an arbitrary target domain, from both mixture coefficients and the compute budget (N, D). Our laws hold for language, multimodal, and vision pretraining. The optimal mixture coefficients found from a small scale can be used for much larger models and training budgets, demonstrating significant improvement over domain weights found by naive grid search. This work paves the way principled theory of data mixture selection.
Aknowledgment
We thank Victor Guilherme Turrisi da Costa for technical support. We thank Marco Cuturi, Jason Ramapuram, Denise Hui, Samy Bengio, and the MIR team at Apple for their support throughout the project.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint 1XXv:2444.142199 2024. 2
[2] Samira Abnar, Harshay Shah, Dan Busbridge, Alaacldin Mohamed Elnouby Ali, Josh Susskind, and Vimal Thilak. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXiv:2501. /2370, 2025. 2, 10, 18
[3] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang. Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference Machine Learning, pages 265-279. PMLR, 2023. 2, 10, 18
[4] Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.0244 2023. 10
[5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16600 2023. 2
[6] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşıllar. Introducing our multimodal models, 2023. URL https://wwl adept. ai hwwbb//ggguuuaal 5
[7] Louis Bethune, David Grangier, Dan Busbridge, Fleonora Gualdoni, Marco Cuturi, and Pierre Ablin. Scaling laws for forgetting during finetuning with pretraining data injection. arXiv preprint arXiv:2502.06042, 2025. 2, 10
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020. 2,
10, 18
[9] Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws. arXiv preprint arXiv:2502.08866 2025. 10
[10] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. attps://githir com kwt//kkaaabiinn///ommmaaaaaaaaaa
coyo- dataset,,set, 2022. 5
[11] Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, and Vikas Chandra. Scaling parameter-constrained language models with quality data. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing: Industry Track, pages 80-97, 2024. 10
[12] Xingyi Cheng, Bo Chen, Pan Li, Jing Gong, Jie Tang, and Le Song. Training compute- optimal protein language models. bioRxiv, 2024. doi: 10.1101/0044.06.06.555 56.7997. URL https: /ttps:://wwww //www/ bioxxiv.. org/ content/ httr//aallyyl// http////222/////2/2///2222
06. .6.066666..66666666666...0..0. 597716. 10
[13] Hyung Won Chung. Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining, arXiv preprint arXiv:2304.09151 2023. 10
[14] Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pages 4057-4086. PMLR, 2022. 10
[15] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to billion parameters. International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.

---

[16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024. 2, 10
[17] Alazeldin El-Nouby. Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. arXiv preprini urXiv:2401.08555 2024. 2
[18) Alaaeldin El-Nouby Victor Guilherme Turrisi da Costa Enrico Fini*, Michal Klein*, Mustafa Shukor*, Jason Ramapuram, Jesse Allardic, Roman Bachmann, David Mizrahi, Vishnu Banna, Chun-Liang Li, Samira Abnar, Vimal Thilak, and Joshua M Susskind. Large multi-modal models (13m) pre-training, 2025. URL https: : //g gig.t...bbb... com/ htt///ppllmmlll m33m. 18
[19] Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation, arXiv preprint arXiv:2310. 2023. 10
[20] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. 17425,, 2023. 5
[21] Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju. Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, Alexander T Toshev, Marcin Eichner. Moin Nabi, Yinfei Yang. Joshua M. Susskind, and Alaaeldin El-Nouby. Multimodal autoregressive pre-training of large vision encoders, 2024. 2, 10
[22] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002) 2020. 4, 8
[23] Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. Bimix: Bivariate data mixing law for language model pretraining. arXiv preprint arXiv:2405. 14900 2024. 10, 22, 23
[24] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Gareia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.077400
2021. 3
[25] Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering-data curation cannot be compute agnostic. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 22702-22711, 2024. 10
[26] David Grangier, Simin Fan, Skyler Seto, and Pierre Ablin. Task-adaptive pretrained language models via elustered-importance sampling. arXiv preprint arXiv:2410.03733, 2024. 2, 10
[27] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.008838 2024. 10
[28] Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, and Fei Tan. Cmr scaling law: Predicting critical mixture ratios for continual pre-training of language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16143-16162, 2024.
10
[29] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.0129: 2021. 2,3
[30] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: /7 1712000099 2017. 9
[31] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 30016-30030, 2022. 2, 3, 9, 10, 18
12

---

[32] Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii. and Sanmi Koyejo. Scaling laws for downstream task performance machine translation. In The Thirteenth International Conference on Learning Representations, 2025. 10
[33] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wrXv:2001.08361, 2020. 2, 3, 9, 10
[34] Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóšdi, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.0787 078771 2024. 2, 10
[35] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelies: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024. 2, 5, 10
[36] Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794 2024. 5, 8
[37] Zhengyang Liang, Hao He, Ceyuan Yang, and Bo Dai. Scaling laws for diffusion transformers. arXiv preprini arXiv:2410.0814. 2024. 10
[38] Lin, Hongxu Yin, Wei Ping. Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 26689-26699, 2024. 5, 10
[39] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503-528 1989, 4
[40] Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang. Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01192 2024. 10, 18, 19, 22
[41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017. 18, 19
[42] Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun, Zhiyuan Liu, Maosong Sun, Kaifeng Lyu, and Wenguang Chen. A multi-power law for loss curve prediction across learning rate schedules. arXiv preprint arXiv:2503. 12811, 2025.
[43] Prasanna Mayilvahanan, Thaddaus Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Brendel. Lims on the line: Data determines loss-to-loss scaling laws. arXiv preprint 12X1:25502.12120, 2025. 10
[44] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mml: methods, analysis and insights from multimodal I1m pre-training. In European Conference on Computer Vision, pages 304-323. Springer, 2024. 2, 8, 10
[45] Sean McLeish, John Kirchenbauer, David Yu Miller, Siddharth Singh, Abhinav Bhatele, Micah Goldblum, Ashwinee Panda, and Tom Goldstein. Gemstones: A model suite for multi-faceted scaling laws. arXiv preprint
2025. 10
[46] Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi Maeda, and Kohei Hayashi. A scaling law for syn2real transfer: How much is your pre-training effective? In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 477-492 Springer, 2022. 3
[47] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, Advances in Neural Information Processing Systems, 36:50358-5000 50376, 2023. 2
13

---

[48] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077/1 07/93, 2023. 2
[49] Tim Pearce, Tabish Rashid, Dave Bignell, Raluca Georgescu, Sam Devlin, and Katja Hofmann. Scaling laws for pre-training agents and world models. arXiv preprint urXiv:2411. 14.4444,, 2024. 10
[50] Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:308 1-30849, 2024. 10
[51] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 4
[52] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. 2021. 10
[53] Jathushan Rajasegaran, Ilija Radosavovic, Rahul Ravishankar, Yossi Gandelsman, Christoph Feichtenhofer, and Jitendra Malik. An empirical study of autoregressive pre-training from videos. arXiv preprint ar arXiv:2501.060 005453, 2025. 10
[54] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 5
[55] Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. UnIVAL: Unified model for image, video, audio and language tasks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: hatttss:/openreviiii
10
[56] Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. Scaling laws for native multimodal models. arXiv preprint arXiv:2504.07951, 2025. 2, 5, 8, 10, 18
[57] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel SlimPajama: A 627B token cleaned and
Hestness, and Nolan Dey. deduplicated version of RedPajama. wew.pppiiieaeedeeeennooo-oooo of slimpaj ama-a-maaa-----aa- 2023. URL atttps://huggingfface httph//gggggggaa.coooo/eeeeeeeee co//attssttt//eeeerre////iaaaa aaaanaaaaaaamaaaa 627B. 4
https: www.:cerebba... net/blog/ www.eba...eeebbbrrr....eee.e
//t:://wwww
dnd-deuuplicatedd-
[58] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint 2rXiv:24022.0019 2024. 10
[59] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Caan-Bapistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai. Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 1xXi:22312.11003 2023.
[60] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. 2
[61] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023. 4
[62] Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017. 5,5
[63] David J Wales and Jonathan PK Doye. Global optimization by basing and the lowest energy structures of lennard-jones clusters containing up to 1 110 atoms. The Journal of Physical Chemistry A, 101(28)5511-5111 1997. 4
14

---

[64] Siqi Wang, Zhengyu Chen, Bei Li. Keqing He, Min Zhang, and Jingang Wang. Scaling laws across model architectures: A comparative analysis of dense and MoE models in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5583-5595, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 0:65//1/222..cmmmmmaam 319/. 10
URL https: hatp::///aclathology org/2024. mmnlp-main.
[65] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv: 2019. 10
[66] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.097399 2024. 10
[67] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36: 69798-69818, 18, 2023. 10
[68] Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952, 2024. 10, 18, 19, 22
[69] Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets 11m finetuning: The effect of data, model and finetuning method. arXiv preprint arX11:2402.17193, 2024. 2
[70] Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mml. 5: Methods, analysis & insights from multimodal IIm fine-tuning. arXiv preprint arXiv:2409.20555 2024. 2, 5, 10
[71] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. 1277, 2023. 18
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sec. 4, Sec. 5, Sec. 6
2. Limitations
Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Question:
Justification: see Sec. 8.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We have no theorem or proof in the paper.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detail how to reproduce results in Sec. 3 and Appendix.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The majority of the data that we use is public, with the exception of HQITP. We will release code.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: see Appendix.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Our laws and findings are supported by large numbre experiments, and validated evaluation data points. We report error bars on the errors of the scaling laws. Due computational constraints, we are unable to obtain error bars on the model's losses.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix.
16

---

9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: we use standard community datasets, and no unethical impacts are anticipated from fitting scaling laws.
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: see Sec.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No model nor datasets are released.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are cited properly in the paper.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new asset.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: N/A
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: N/A
17

---

Params
412M
834M
1.1B
1.4B
width
1024
1536
1792
2048
3072
4864
depth
24
Learning rate
6e-4
4e-3
3.4e-4
3e-4
2e-4
1.2e-4
Optimizer
Fully decoupled Adam [41]
Optimizer Momentum
81 = 0.9, B2 = 0.95 0
Minimum Learning rate
Warm up iterations
2k
Weight decay
lc-4
Batch size
2M
Table Pre-training hyperparameters used for pre-training of LLM to conduct the main scaling laws study.
A Implementation details
A.1 LLM pretraining
For the main experiments, we LLAMA style architectures. For the analyses with the Pile domains, we borrow architectures from [8]. We use a fixed depth of 24 and change the latent dimension the network to obtain different model scales. All hyperparameters are described in Tab, and Tab. 7.
A.2 Multimodal pretraining
Implementation details. We closely follow the implementation of [56] and present in Tab. 6 the pre-training hyperparameters for the model configurations used in our scaling laws study. Training conducted in the L3M code base [18]. Our models range from 100M to 8B parameters, with width scaling accordingly while maintaining a constant depth of 24 layers. We use causal attention for text tokens and bidirectional attention for image tokens. Learning rates are adjusted based on model size, generally decreasing for larger models. These values were determined through empirical testing. Optimization is handled using fully decoupled AdamW optimizer with momentum values set to 8_1 = 0.9, 3_2 0.95, and weight decay of 10-4 Each training batch consists of 2,000 samples, totaling 2 million tokens with a 1,024-token context length. Gradients are clipped at 1.0, and training begins with a warmup phase of 1,000 iterations, followed by constant learning rate schedule to reduce the number of experiments.
For vision inputs, we process images as (14, patches with augmentations including Random Resized Crop (224px, scale range of [0.4, 1.0]) and Random Horizontal Flip with 50% chance. Model training benefits from efficiency techniques such as bfloat16 precision, Fully Sharded Data Parallel (FSDP) [71], activation checkpointing, gradient accumulation, and sequence packing to minimize padding in the image-captioning dataset.
We assess model performance on three held-out data subsets: interleaved data (Obelics), image- caption data (HQITP), and text-only data (DCLM), following prior works [2, 31J. This setup provides a robust evaluation of model generalization across diverse data types.
B Detailed extrapolation results
We report the detailed per-domain and per-model size MRE corresponding to each experiment and scaling law in the paper.
B.1 Comparison the laws of Ye al. [68] and Liu al. [40]
Ye et al. [68] propose four laws to model the behavior of the loss on domain function of only, that for fixed N, budget. They propose the following formulas, rewritten with notations consistent with our notation:
C(h) E
C exp(1chs)
(Ye MI)
18

---

Params
275M
468M
932M
1.63B
2.28B
3,35B
8,13B
width
800
2208
3232
5120
depth
24
Learning rate
1,5c-3
1.5c-
5c-4
4.2c-4
4c-4
3.5e-4
2.4c-4
Training tokens
2.5B-600B
Optimizer
Fully decoupled AdamW [41]
Optimizer Momentum
81 0.9, 82 0.95
Minimum Learning rale
Weight decay
Batch size
2k
Patch size
(14, 14)
Gradient clipping
Warmup iterations
Augmenlations:
KandomFealzedCrrp
size
224px [0.4, 1.0]
scale
Raanomtiorizontat1F7
0.5
Table 6: Pre-training hyperparameters used for pre-training of NMM to conduct the scaling laws study.
Params
90M
200M 768
350M
700m 1536
1.3B 2048
3B
Width
512
1024
3072
Depth
24
Learning rate
constant after warmup: 1c-4
Training tokens
8.4B
8.4B
8.4B
16.8B
33.6B
67.2B
Optimizer
Adam W [41]
Optimizer Momentum
B1 = 0.9, 3g 0.95
Batch size
128
Sequence length
1024
Gradient clipping
1.0
Warmup iterations
1k
Table 7: Pre-training hyperparameters used for the pre-training of LLM with PILE dataset to conduct the analyses
((h) = E C &
exp(1ali
(Ye M2)
&
((h) = E + C exp II
7hh4)
(Ye M3)
1=1
k
((h) = E expl
7hh4)
(Ye M4)
where the parameters of the law the E, C, Ci, 7x. Liu al. [40] propose model the loss linear function of h. We compare this with the form of our scaling law eq. (4) when N and D are fixed:
1
[(h) = E Σ C.P.
(Additive, fixed (N, D))
We fit all those scaling laws the LLM training data, keeping only one value for N, D (we take N 200m, D = 8B tokens). We only keep 25 training mixtures to fit the laws, and report the MRE on the 84 25 59 remaining Tab. 13. Note that we had trouble fitting the M3 law, which also reported to underperform in [68]. Overall, our formula gives systematically better training errors, and it most of the time translates to better testing errors on unseen mixtures. We stress once again that one of the core contributions of our work is to explain how scale interacts with data mixtures, by proposing scaling laws that take N, D. and h as inputs. [68] only consider scaling laws with respect to h.
19

---

Scaling Law
Domain
Train (RE(())
3B MRE (%)
Simple
Arxiv
0.55
2.40
Additive
Arxiv
0.50
2.09
Joint
Arxiv
0.39
1.62
Full
Arxiv
0.39
1.83
Simple
Book
0.38
1.11
Additive
Book
0.29
0.80
Joint
Book
0.24
1.19
Full
Book
0.24
1.14
Simple
C4
0.35
0.47
Additive
C4
0.29
0.31
Joint
C4
0.24
0.34
Full
C4
0.23
0.41
Simple
GitHub
0.81
2.05
Additive
GitHub
0.65
1.17
Joint
GitHub
0.54
2.51
Full
GitHuub
0.52
1.97
Simple
Commoncrawl
0.34
0.65
Additive
Commoncrawl
0.29
0.58
Joint
Commonerawl
0.24
0.90
Full
Commonerawl
0.23
0.74
Simple
Stackexchange
0.57
0.68
Additive
Stackexchange
0.51
0.36
Joint
Stackexchange
0.38
0.47
Full
Stackexchange
0.36
0.42
Simple
Wikipedia
0.97
4.53
Additive
Wikipedia
0.92
4.45
Joint
Wikipedia
0.57
2.09
Full
Wikipedia
0.56
1.98
Table 8: Full results of experiments in Sec, 4 for the LLM experiment.
Scaling Law
Domain
Train MRE(%)
2B MRE(%)
8B MRE(%)
Simple
Text
0.15
0.44
0.50
Additive
Text
0.12
0.40
0.51
Joint
Text
0.10
0.39
0.32
Full
Text
0.09
0.38
0.33
Simple
Image-Caption
0.52
0.89
1,36
Additive
0.47
0.83
1.23
Joint
Iage-CCaptions
0.43
0.85
1.17
Full
Image-Captions
0.43
0.90
1.33
Simple
Interleaved
0.22
0.65
0.80
Additive
Interleaved
0.14
0.44
0.58
Joint
Interleaved
0.10
0.41
0.45
Full
Interleaved
0.10
0.40
0.45
Table 9: Full results of experiments in Sec. 4 for the multimodal experiment.
20

---

Scaling Law
Domain
Train MRE(%)
IB MRE(%)
Simple
Noisy image-text
0.34
1.05
Additive
Noisy image-text
0.35
1.20
Joint
Noisy image-text
0.23
0.63
Full
Noisy image-text
0.21
0.58
Simple
Synthetic
1.85
5,96
Additive
Synthetic
1.89
6.19
Joint
Synthetic
0.83
5,94
Full
Synthetic
0.70
5.56
Simple
High quality
0.70
0.99
Additive
High quality
1.19
2.02
Joint
High quality
0.34
1.18
Full
High quality
0.32
1.08
Simple
High quality
0.64
1.22
Additive
High quality 2
0.64
0.79
Joint
High quality 2
0.31
1.06
Full
High quality 2
0.31
0.93
Table 10: Full results of experiments Sec. for the LVM experiment.
Scaling Law
Domain
Train MRE(%)
700m MRE(% (6)
IB MRE(%)
Simple
Wikipedia
0.28
0.75
1.13
Additive
Wikipedia
0.24
0.77
1.11
Joint
Wikipedia
0.13
0.24
0.39
Full
Wikipedia
0.12
0.23
0.39
Simple
GitHub
0.60
1.38
3.28
Additive
GitHub
0.42
1.10
1.69
Joint
GitHub
0.23
0.38
1.46
Full
GitHub
0.22
0.49
1.91
Simple
StackExchange
0.40
0.90
1.50
Additive
StackExchange
0.33
0.88
1.31
Joint
StackFxchange
0.17
0.26
1.05
Full
StackExchange
0.16
0.30
1.17
Simple
PG-19
0.21
0.53
0.91
Additive
PG-19
0.16
0.55
0.94
Joint
PG-19
0.15
0.40
0.71
Full
PG-19
0.14
0.35
0.54
Table I1: Full results of experiments Sec. 6 for the LLM experiment.
Scaling Law
Domain
Train MRE(%)
700m MRE( %))
IB MRE( (%)
Additive
Wikipedia Wikipedia
0.45
0.53
0.53
Joint
0.22
0.18
0.19
Additive
GitHub
0.70
0.88
1.49
Joint
GitHub
0.39
0.31
1.09
Additive
StackExchange
0.50
0.55
0.50
Joint
StackExchange
0.29
0.23
0.44
Additive
PG-19
0.24
0.37
0.53
Joint
PG-19
0.18
0.25
0.44
Table 12: Full results of experiments in Sec. 4 for the cosine schedule experiment.
21

---

Scaling law
Domain
Train (MRE%)
Test (MRE%)
Additive, fixed (N, D)
Wikipedia
0.07
0.18
Ye MI
Wikipedia
0.08
0.14
Ye M2
Wikipedia
0.09
0.17
Ye M3
Wikipedia
2.54
4.20
Ye M4
Wikipedia
0.17
0.31
Regmix
Wikipedia
0.92
1.31
Additive. fixed (N, D)
GitHub
0.10
0.19
Ye M1
GitHub
0.20
0.61
Ye M2
GitHub
0.22
0.40
Ye M3
GitHub
5,45
5.26
Ye M4
GitHub
0.36
0.44
Regmix
GitHub
0.65
1.35
Additive, fixed (N, D)
StackExchange
0.07
0.18
Ye MI
StackExchange
0.14
0.32
Ye M2
StackExchange
0.14
0.21
Ye M3
StackExchange
4.11
3,20
Ye M4
StackExchange
0.22
0.34
Regmix
StackExchange
0.78
0.92
Additive, fixed (N, D)
PG-19
0.08
0.12
M1
PG-19
0.09
0.17
Ye M2
PG-19
0.13
0.18
Ye M3
PG-19
2.21
3.14
Ye M4
PG-19
0.16
0.21
Regmix
PG-19
0.64
0.89
Table 13: Comparison of our scaling laws for a fixed (N, D) budget with those of [68] and [40] on the
LLM data.
B.2
Comparison Ge al. [23]
Ge [23] propose scaling law evaluate the loss the i th training domains, function of both / and number of tokens D:
B D3 + E
C h
D)
(Ge 24)
where the coefficients B, 3, E, C' and have to be fitted. Here, the loss must be on domain i, and only involves the proportion that domain h; not that of the other domains. our view, this is caveat since it implies that all other domains would have the same impact on the loss for domain i, while one other training domain might be very useful for that task, and another might be useless.
Since this law does not take into account model scale, we compare to our additive law for fixed model scale:
B Σk C. D
D), E+
+
(Additive, fixed N)
We consider fixed size of model (N=200M) on the LLM training experiment, take only 25 mixtures to fit the scaling laws, and report the MRE on the remaining testing set in Tab. 14. We see that our proposed scaling law achieves significantly lower MRE on both train and testing data, highlighting the importance of taking all other domains into account.
22

---

Scaling law
Domain
Train (MRE%
Test (MRE%
Additive, fixed N Ge 24
Wikipedia Wikipedia
0.13
0.17
0.51
0.62
Additive, fixed N Ge 24 Additive, fixed N Ge 24
GitHub
0.20
0.26
GitHub
1.99
2.26
StackExchange StackExchange
0.14
0.19
0.94
1.21
Additive, fixed N
PG-19
0.13
0.15
Ge 24
PG-19
0.49
0.54
Table 14: Comparison our scaling laws for a fixed model size N with that of [23] on the LLM data.
C
Additional analysis
C.1
Cosine learning rate scheduler.
The bulk of our experiments use a constant learning rate, which helps us gather many D points for each run, but this departs from what is done in practice when training competitive models, where cosine learning rate is typically used. In order to validate that our scaling laws are still valid when training with cosine learning rate, repeat the LLM experiments with k: pile domains with cosine learning rate decays, with fewer runs, training for 5 different values of D, with 25 different domain weights. We use 90M, 200M,, 350M and 700M models for training, and extrapolate 1.3B. We observe that our scaling law fit is similar to those in the main experiments: on the 1B model, get average MRE of 0. 0.76% for the additive law and 0.54% for the joint law. We report detailed results in appendix B. Interestingly, the estimated optimal domain weights for the average loss are very similar those estimated with the constant learning rate: for the additive law, we have hios = [0.35, 0.18, 0.30, 0.17] and hinnst [0.34, 0.17, 0.32, 0.17].
C.2 Asymptotic behavior
We can get an information-theorriic explanation of the bias term in the scaling laws. Let be the true data distribution the target domain. Let g(h) be the output distribution of model of size N + trained for D oo tokens with domain weights h. Let h- be the optimal domain weights, which minimizes the scaling law L(+00, +oc, E' (E* C,h?")-1 i. the cross- entropy term C CE,, {(h)) = (p) + K (p)) with (-) the Shannon entropy and K L-Il-) the Kullback-Leibler divergence. We have the following decomposition:
CE(p.q(h)) H(p) K LL(plg(h"))
KL(pplg(h)) - KL(pl((hh))
constant, indopendant of
2.0 by hypothesis on ht*
(11)
E C
Ch C.
We can identify both terms since they are of the form "constant plus non- non-negative function that cancels". see that (>x C captures both the intrinsic entropy (p) of the target distribution, and the shift K LT(pllg(h")) induced by training on the optimal mixture lt", while the right-hand term is the expected log- likelihood ratio Ep g(h"), which measures how far the model trained on h is from the optimal one. If p is one of the training domains D., for disjoint domains we can assume hg R (see next section for justification) and simply bound entropy H(p) S E E+C. C.
capturrs
g(b)
C.3 Optimal domain weights
Optimal weights for the additive scaling law. We report the optimal domain weights h. for the additive scaling law in Tab. 15.
Recall that Sec. 5 defines the optimal domain weights (+) as function of compute budget (N, D). In the main text, assumed uniform weights of the target domains D.. However, can also
23

---

Table 15: Optimal domain weights h" for LLMs. Note that for the OpenHermes optimal weights, the law predicted weight of O for wikipedia, which we artificially set to 1 1%. Model
Arxiv
Book
C4
GitHub
Commoncrawi
Stackexchange
Wikipedia
For average loss has
9.6
9.5
25.1
8.0
12.1
17.9
17.0
For OpenHermes NoM
9.4
4.8
27.8
6.6
36.9
13.5
1.0
Base
4.6
4.2
26.7
5.2
52.2
3.3
3.8
Additive law
Joint law
Stack Exoltage
Stack Rackegg
Wikipelia
Oarlab
P019
Wikipota
Github
R019
100
Wilepedia
Targe Tongeddomnnng domains
GeHHub
DLOSS
50.4%
9.095
SL0SS
0.00S
96.1%
3.9%
0,00%
StockBschange
98.61%
PG19
5.4%
0.0%
0.0%
94.6%
1.9%
0. 1.95%
1.0%
97. .7%
-20
Training demmins
Thining domoinns
Figure 7: Optimal domain weights for a single (pure) domain, typically lies at the corner of the probability simplex. Scaling law predictions for 1.3B model trained on 10B tokens.
consider a weighted average scenario, with an arbitrary weight vector wt.
m
h" h"(w, N, D) E arg min hE
Σ (N,
(12)
Once again, this objective is seamlessly optimizable with mirror descent. When training domains and target domains are the same, w and ht are probability distribution over the same simplex. Therefore, we can study the mapping w + h² (w, D).
Behavior at the corners We predict the optimal domain weights for both laws, chosing each training domain j as the target, that is, putting w. if = and w; = 0 otherwise. The results are given in Fig. 7.
We see that the data mixture law predicts that the optimal domain weights are typically located in the corresponding corner of the simplex- which is not too surprising when there is little domain overlap. This justifies the rule of thumb h x when the target domain is D:.
JS(w, A" (w)) with 25% of PG19 Stuckexchange
0.22
Jonsen Shincr-SSammoe Shammon distance
0.20
0.14
Wikipedia
Github
Figure 8: The optimal domain weights are always different from the target mixture, except at the corners of the simplex 1.3B model with 10B tokens, on 4 domains of The Pile.
24

---

Jensen-Shannor Shannon metric [S0 (w))
PG19
0.20
Stackexchange
0.15
Github
0.10
Wikipedia
Figure 9: Jensen-Shannon distance between target mixture / and its optimum training mixture ht" (wr on the 4D simplex. 1.3B model with 10B tokens, on domains of the Pile. No fixed-point are visible, except at the corners. This suggests that in general, it is better not to train on the mixture you want to be good on.
Fixed-points For given (N, pair, the optimization problem of eq. (12) defines function w + (w) that maps the simplex onto itself. The fact that (w) + w indicates the surprising phenomenon that, in order minimize the loss CERM Σ (,), it is faster instead minimize - Σm h² (O), rather than minimizing directly CERM
Fixed points the map w + ht" (w) correspond to target mixtures w that are minimized by training on itself: this is the empirical risk minimizer.
To find these points, we compute the / LI/2(KL(w [m) + [[(h+]]m) with m = (w))2))/22 and we look for near-zero values, in Fig. and Fig. 9.
distance, defined as JS(w, h²)
Accumulation point asymptotes. In mirror of Fig. we can monitore how the optimal mixture scale parameters N and data D. For example. can keep D constant and scale N, or the opposite, or scale both simultaneously with x N. Results are highlighted Figure 10. These asymptotes reach accumulation point when D > N + They depend on the speed at which N and D grow.
h" evolves
25

---

Image-Caption
Optimum weights (w)
Small model, few tokens
N O(1), O(t)
N =Ot)) /) = O(I)
N 0()) = O(vI)
N O(Vi), D O(1)
N O(I), O(logi)
N O)logt), O(t)
Isocurve: N D- O())
x Accumulation point + 00
Text
Interleaved
Figure 10: Asymptotes of optimal mixture when increasing N and D at different speeds on multimodal data. Surprisingly, there little diffenrence between proportional scaling O and square-root scaling O(vi): both are fast enough. However, when one quantity is held constant, or only grow at logarithmic speed, the accumulation point changes.
26


Reviews:
[{"review_id": "fQEZeM0qAQ", "reviewer": "Reviewer_G7h4", "summary": "This paper offers a new scaling law study for data mixtures (simplex vector h) as a function of model size N and data size D. They experiment with two different types of scaling curves, additive and multiplicative (depending on the dependency between params/data and datamix weights). They first show an efficient way to fit these parameters with limited training runs, and then show that the estimation extrapolates sufficiently well to large-scale training. They provide ablations using pre-training MLLMs and LLMs with different scale of domain mixtures.", "questions": "- The paper does not investigate the case where there are lot of domain mixtures - like on the order of 100. It is very common in post-training to have diverse set of mixtures (unlike pre-training with homogenous mixtures). They authors can add their thoughts on this. \n\n- I am curious why did the authors choose inverse scaling law for h in eq 4? For N,D - it is intuitive because as we scale models and data, the loss should go down. But what might explain the fact that the coefficients are inversely related to the loss (in Eq 4 and 5)? \n\n- Minor: Looks like D and N have interchanged coeffiencts between Eq 3 and 4?\n\n- Minor: Having grid lines in Fig 3 might make it more clear to understand the slope of observed vs. predicted loss curves.", "limitations": "The paper adequately discusses several limitations in sec 8, several of which I concur with.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n--------------\n\n1. The paper considers a very important problem of robustly estimating the datamix weights for large-scale pre-training using small-scale runs with impressive results.\n2. The actual modifications made to the standard scaling law is minimal (easy to adopt and optimize) yet effective (fits most training regimes well) validating the strengths of the hypothesis. \n3. Extensive empirical study (using ~300k GPU hours) on multimodal as well as text LLMs with several important conclusions and observations which go beyond current scaling study (like importance of text data for Large LLMs in fig 5). \n4. The paper provides insights into more efficient optimization strategies as the #domains scale up, which might be important for post-training stages which involve several different mixtures and domains. \n5. The paper honestly states several limitations, which is important to drive future work in this area.\n\nWeaknesses\n-----------------\n\nPlease refer to the questions below.", "was_revised": true, "final_justification": "I am fairly convinced that this paper addresses a very non-trivial problem (scaling laws for pre-training datamixes) with effective observations (validation if the proposed law + importance of text<->image data). I'd stay with my rating.", "rebuttal": "Dear reviewer, \n\nWe thank the reviewer for the very positive feedback, thoughtful questions and helpful suggestions.\n\n\n### Scalability to 100s of domains\n\nAs you mention, our current method does not scale efficiently to such large numbers of domains, since, as a rule of thumb, the number of experimental runs required scales approximately linearly with the number of mixture components. However, it is  significantly more efficient than existing alternatives based on exhaustive trial-and-error, which becomes prohibitively expensive at scale. We agree it’s important to be upfront about this limitation, and we will clarify it in the final version. On the practical side, in order to find a middle ground and still use our laws, one can group similar domains to reduce the size of the search spaces (e.g. Coding, Math, General Text …). Identifying a more efficient algorithm that can scale to 100s of microdomains is a great question for future research.\n\n\n### Choice of Inverse Scaling in Eq. 4\n\nOur choice was motivated by a natural extension of classical scaling laws, which model loss as a power-law function of data size and model size. While it’s intuitive for loss to decrease with increasing N or D, the situation with h is more nuanced. We would like to clarify that in our formulation, the exponents are free to take on negative values, so the coefficients themselves are not necessarily inversely proportional to the loss. We selected this formulation because it yielded the best empirical performance among several plausible extensions, and we will further clarify this rationale in the revised text. For an empirical comparison with other formulations please refer to Sec 6./Tab. 3 and Appendix B. We also observed that the relative ratios had more impact than the absolute ones (e.g, 10% VS 5% has more effects than 30% VS 25%), which is well captured by a value of |\\gamma|<1.\n\n\n### Minor: Coefficient Swap Between Eq. 3 and 4\n\nThank you. We will fix this.\n\n\n### Minor: Grid Lines in Figure 3\n\nWe agree, adding grid lines will improve clarity in interpreting the slopes. We'll update the figure accordingly.\n\n\nWe thank you again for your review, and we hope that we have clarified any grey areas."}, {"review_id": "MU9oawjX8h", "reviewer": "Reviewer_4cfG", "summary": "This paper investigates pre-training hyperparameters for Neural Multimodal Models (NMM) and Large Language Models (LLM) to explore scaling laws. It details the specific configurations and optimization strategies used, including model sizes, learning rates, optimizers, batch sizes, and data augmentations. A key aspect of the work is the comparison of their proposed scaling laws, which account for model scale (N), data (D), and data mixture (h), against existing laws that primarily consider data mixture alone. The authors demonstrate that their additive scaling law consistently achieves lower Mean Relative Error (MRE) on both training and testing datasets compared to alternative models. Furthermore, the paper explores the concept of optimal domain weights, revealing that training on a pure target domain often leads to optimal weights concentrated in that domain's corner of the probability simplex. The analysis of Jensen-Shannon distance suggests that, in general, training on a mixture different from the target mixture can be more effective for minimizing loss.", "questions": "1. How to treat the data changes? Does this method have to retrain multiple models if new data/domain is added or some domains are deleted? \n2. Does this method allow for different model architectures? For example, using multiple small models to fit the scaling law while using another larger model with a different architecture to train. Would it still get a benefit?", "limitations": "See weakness.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. The paper integrates Scaling Laws into Data Mixing. The paper introduces a principled method to determine optimal data mixtures. These laws predict model loss based on model size ($N$), training tokens ($D$), and data mixture ($h$), a comprehensive approach that surpasses prior work considering only mixture for fixed $N$, $D$. Fitting these laws on small-scale runs allows accurate extrapolation of performance and optimal mixture identification for larger, unseen models.\n2. Consideration of Multi-modality in Data Mixing. This paper uniquely extends data mixture optimization to multimodal models. It applies scaling laws to Neural Multimodal Models (NMMs) that process interleaved text and image tokens.\n\n**Weakness**\n1. The difference between single-modality and multi-modality in the data mixing problem is not very clear. I don't get how to treat these two cases differently in the formulation. Or they share the same method while just can apply to two different settings in applications. Could you please elaborate on this more?\n2. One domain can only include one type of data in multi-modality setting. In my understanding, one domain in the paper can only include one type of data like text or paired. How to process it if one domain includes multiple types of data like both text and paired?\n3. Missing the analysis of computational cost. While the authors claim that _\"Only 10-20 runs are needed for determining the scaling laws.\"_ in line 229. It's unclear what sizes of models it requires and how much computational cost it uses. \n4. It would be better if compare with other relative data mixing baselines like Data Mixing Laws [1] and RegMix [2]. These two papers also use multiple models with different sizes to predict domain weights for larger models. Could you please clarify the difference compared with them, including the prediction accuracy and computational cost? \n5. Missing downstream task evaluation. The paper only uses loss as a metric while the performance on downstream tasks is closer to reality.\n\n[1] Ye, Jiasheng, et al. \"Data mixing laws: Optimizing data mixtures by predicting language modeling performance.\" _arXiv preprint arXiv:2403.16952_ (2024).\n\n[2] Liu, Qian, et al. \"Regmix: Data mixture as regression for language model pre-training.\" _arXiv preprint arXiv:2407.01492_ (2024).", "was_revised": true, "final_justification": "I'm not sure if it's okay to ignore the modality difference in such problem. So I decrease my confidence.", "rebuttal": "We thank the reviewer for these thoughtful and constructive comments, which will improve the final version of the paper. Below we address each point in detail:\n\n### Clarification regarding “The authors demonstrate that their additive scaling law consistently achieves lower Mean Relative Error (MRE) on both training and testing datasets compared to alternative models”\n\nWe would like to clarify that we are the first to propose a joint (N, D, h) law, and the alternative methods are other baselines that we propose. We have a discussion in the “Related Work” regarding the limitations of other works.\n \n### Single-Modality vs. Multi-Modality Settings\n\nOur framework, described in Section 2.1, defines training as sampling from a mixture of data sources (called domains), with mixture weights denoted by h. This formulation is agnostic to the type of data in each domain. In our LLM experiments, all domains are text, while for multi-modal experiments (e.g., NMM training), and similar to how multimodal models are are commonly trained, each domain happens to correspond to a different modality (e.g., text,  paired image-text, interleaved image-text). To strengthen this more, we also validate our scaling laws for large vision encoder training, where domains come from different image-caption datasets.\n\nFor that experiment, we use AIMv2 style models / scaling (Fini 2024), with models of size 90M to 531M for fitting the laws, and up to 1.1B for evaluation. The four domains are paired image-caption datasets: (1) noisy alt-text sourced from the Internet (COYO-700M and DFN2B), which provide large-scale real-world image-text pairs with varying levels of noise and quality; (2) HQ-ITP-1, a high-quality dataset containing 134 million samples; (3) HQ-ITP-2, another high-quality dataset comprising 400 million samples; and (4) synthetic data, consisting of  recaptioned versions of DFN2B and HQ-ITP-2.\nWe obtain the following low MRE % for the four domains:\n\n|Domain | Eval MRE % (Additive / joint)|\n| -------- | ------- |\n|1 | 1.20 / 0.63|\n|2|  6.19 / 5.94|\n|3|  2.02 / 1.18|\n|4|  0.79 / 1.06|\n\nWhich gives another different setup where our scaling laws work, with another definition of a \"domain\". \n\nOverall, our method does not require separate handling of single- vs. multi-modal settings; both are unified under the same scaling law formulation.\n\n### Domains with Mixed Modalities\n\nIn the paper we consider each domain to have the same modalities, either only text, or only paired image-text or interleaved. \n\nUpon your remark, we have also run an experiment in the NMM setting where a domain is a mixture of text and image captions. In that experiment, the third domain consists of data sampled randomly from either the text or the image captions domains, with equal probability for each. The first and second domains are like in the main experiments, they are respectively image-caption and interleaved data. We then fit the scaling laws using these new three domains, where the goal is to predict the loss on the image caption, interleaved, and text domains. The evaluations MREs we obtain are as follows:\n\n\n|Domain | Eval MRE % (Additive / joint)|\n| -------- | ------- |\n|Image caption: | 0.88 / 0.96|\n|Interleaved|   0.53 / 0.40|\n|Text | 0.43 / 0.34|\n\n\nWhich are very similar to the results in table 2 with the original domains. We will add this important discussion to the text; this further strengthens our argument that these scaling laws are universal, regardless of the definition one uses for a “domain”.\n\n\n### Computational Cost Analysis:\n\nThis is an interesting point. We use the following assumptions: Flops are computed as 6ND, and we use a number of training runs that is enough to get to a low MRE, as per Fig.4. We report the training costs for the model sizes shown in table 1. \nNMM (using 6 runs per sizes): small scale runs cost: 6.9e21 flops. Cost of one 8B run: 7.8e21 flops (Note that we could have trained this model for longer, linearly increasing the computational costs. Also, reducing the number of training tokens to 60B for the small scale runs does not significantly impact the eval MRE, while it reduces the small scale flops to 4.1e21).\nLLM (using 10 runs per size): small scale runs cost: 2.9e20 flops. Cost of one 3B run: 1.1e21 flops, which is approximately five times higher.\nWe will add these important numbers in the final version of the paper.\n\n\n### Comparison with Related Work ([1] and [2]):\n\nWe already compare against Data Mixing Laws [1] extensively in Appendix B. Their approach models performance for fixed (N,D), while ours introduces a joint law over (N,D,h), enabling extrapolation across mixture compositions and scales. In comparable settings where we fix N and D, our method achieves lower MRE than theirs.\n\nRegMix [2], similarly, proposes a linear law over h for fixed (N,D), without modeling scaling behavior. We compared against RegMix in the same setup as in appendix B.\nWe get the following test MREs for the 4 domains:\n\n\n| Scaling law formula | Wikipedia | GitHub|Stackexchange|Pg-19|\n|------|------|------|------|------|\n| Additive, fixed (N, D) | **0.18**| **0.19**|**0.18**|**0.12**|\n| Regmix | 1.31|1.35|0.92|0.89|\n\nThe test MREs for regmix are very far from the ones we get with the proposed scaling law. We will add these results to the paper.\n\n\nWe already compare our laws to that of Bimix in Table 10 in Appendix B. The Bimix scaling law does not predict how the loss evolves with N while our law does, and even in the fixed N setting, our laws have a far lower test MRE.\n\n\n### Downstream Task Evaluation:\n\nWe agree that downstream performance is ultimately what matters. To address this, we added a new study with larger-scale LLMs (up to 7B parameters), where we show that the model trained with our predicted optimal mixture, achieves the best results.\nThe study is as follows:\nWe train llms on 7 domains from slimpajama, and fit the scaling laws with small scale runs (<1B parameters). We then train large 7b models with 160B tokens with the optimal domain weights to get the best loss on average for the 7 domains (model A) , or to get the best loss on OpenHermes, a high quality domain (model B). We compare these models to models trained on the base distribution (Model C) for slimpajama and the uniform distribution (Model D) across the 7 domains. To get the best performing models, we train them with a cosine schedule, while the small scale runs were done with constant lrs. In terms of loss, Model A achieves the best loss on average over the domains, and model B achieves the best loss on openhermes. This shows that the optimal weights found by our laws still yield superior performances at large scales. We then score those four 7B models on standard benchmarks. Model B, targeting a high quality domain, yields the best scores, followed by model A, while D and C are comparable. This demonstrates that the optimal weights found at smaller scale to optimize the next token prediction loss also yield good performances on downstream tasks. The downstream tasks' results are as follows (CORE is the median of low variance benchmarks (Li 2024)):\n\n|Model | CORE median score| MMLU|\n| -------- | ------- |--------|\n|Optimal weights for avg| 56% | 32% |\n|Optimal weights for OpenHermes|  **58%** |**37%**|\n|Base | 52% | 25% |\n|Uniform | 53% | 30% |\n\nThat said, we acknowledge that the precise relationship between pretraining loss and downstream task success is still an open question in the scaling law literature, see Schaeffer (2024); Mizrahi (2025); Gadre (2025); Bhagia (2024) for recent attempts.  \n\n*Li, Jeffrey, et al. \"Datacomp-lm: In search of the next generation of training sets for language models.\" NeurIPS (2024): 14200-14282.*\n\n*Schaeffer, R., Schoelkopf, H., Miranda, B., Mukobi, G., Madan, V., Ibrahim, A., Bradley, H., Biderman, S. and Koyejo, S., 2024. Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?. arXiv preprint arXiv:2406.04391.*\n\n*Mizrahi, D., Larsen, A.B.L., Allardice, J., Petryk, S., Gorokhov, Y., Li, J., Fang, A., Gardner, J., Gunter, T. and Dehghan, A., 2025. Language Models Improve When Pretraining Data Matches Target Tasks. arXiv preprint arXiv:2507.12466.*\n\n*Gadre, S.Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S. and Xin, R., Language models scale reliably with over-training and on downstream tasks. ICLR 2025.*\n\n*Bhagia, A., Liu, J., Wettig, A., Heineman, D., Tafjord, O., Jha, A.H., Soldaini, L., Smith, N.A., Groeneveld, D., Koh, P.W. and Dodge, J., 2024. Establishing task scaling laws via compute-efficient model ladders. arXiv preprint arXiv:2412.04403.*\n\n### Q1.\n\nWe follow standard pretraining practice and assume a fixed set of domains. That said, our method is flexible to removing domains--this is handled by setting the corresponding weight in h to zero. To add a new domain, additional training runs involving that domain would be needed, but all previous runs remain valid datapoints (with the new domain weight set to zero).\n\n### Q2.\n\nThis is an interesting question. To our knowledge, even in standard scaling law literature (e.g., Chinchilla, Llama), scaling is typically studied within a fixed architecture family, varying only model width/depth. We follow the same approach here.\nEmpirically, changes in architecture should only shift the scaling law coefficients (i.e., constants A,B,E), but leave the overall functional form unchanged. Formalizing this is an interesting direction for future work. This phenomenon has been documented in the past (Mayilvahanan et al, 2025): the data mixture has a stronger impact than the model architecture on scaling.\n\n*Mayilvahanan, P., Wiedemer, T., Mallick, S., Bethge, M. and Brendel, W., LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws. In Forty-second International Conference on Machine Learning (2025).*\n\n\nWe thank you again for your review, and we hope that we have clarified any grey areas."}, {"review_id": "ZnXkDobd2C", "reviewer": "Reviewer_mQkL", "summary": "This paper examines the prediction of model loss as a function of data size, model size, and domain weights. The paper proposes several parametric functions to model this relationship and evaluates their prediction accuracy in the pre-training of LLMs and multimodal models.", "questions": "1. What is the computational cost required for fitting the parameters of the scaling law? How does it compare to the cost of training the model at the target scale? \n\n2. Is there any intuition why cosine learning rate and constant learning rate yield similar domain weights?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper examines an important and challenging topic, particularly given the substantial computational costs involved in performing the experiments.\n2. The paper evaluates both LLMs and multi-modal models, a scope that is rarely explored in the literature.\n\nWeaknesses:\n1. As discussed in the related work section, there are several papers that share a similar goal of modeling the model loss and/or identifying the optimal mixture (such as DoReMi, Regmix, Data mixing law, and Doge). However, this paper lacks an empirical comparison with these papers. In particular, the paper overlooks a comparison with AutoScale [1], which addresses the same problem and appears to be more computationally efficient in scaling law fitting since their scaling law is more parameter-efficient.\n2. Related to the first weakness, the experiment's empirical evaluation is limited. The authors do not clearly state the evaluation metrics and appear to rely solely on the convergence of training loss to assess a mixture's effectiveness. A comprehensive evaluation would include validation loss and performance on relevant downstream benchmarks, which are of greater interest to practitioners.\n3. The paper's proposed scaling law as a function of mixing ratio lacks theoretical justification or intuition for its design.\n\n[1] Kang et al. AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs. 2024.", "was_revised": true, "final_justification": "The paper, motivated by the problem of data mixing, has not yet been evaluated comprehensively in terms of identifying an effective data mixture. Specifically, many key baselines are missing. While the authors insist that their contributions also include finding a scaling law of loss as a function of data, compute, and mixing ratio, it is only a building block towards identifying an optimal mixture, which is the main motivation as stated in the abstract. Overall, the paper needs to rigorously evaluate against these baselines using already established metrics, such as downstream performance yielded by the data mixture, wihtout which the contribution is not convincing enough.", "rebuttal": "Dear reviewer, \n\nWe thank you for your helpful feedback, which will help us improve the paper. The paper Autoscale that you mention is indeed relevant, and we will make sure to include a comparison in the final version. We now reply to your questions.\n\n### Comparison with Related Work (AutoScale, DoReMi, RegMix, Doge)\n\nWe want to highlight that the primary goal of our paper is to give a way to model the loss as a function of (N, D, h). None of the works that you mention do this: Autoscale models the loss as a function of (D, h) but not N, and Regmix only as a function of (h). DoreMi and Doge **do not** model the loss.\n\nOne of the applications - but not the only contribution - of our work is to estimate the optimal data mixture as a function of (N, D), which is indeed a goal shared by all the works you mention. However, DoreMi and Doge respectively rely on costly Distributionally robust optimization and bilevel optimization to estimate the optimal mixture, which incurs a significant change to the training loop and a very large compute/memory overhead. On the other hand, our work, Autoscale and Regmix only observe the loss as a function of N, D, h, which does not require any change to the training loop. We therefore defer a comparison to Doremi and Doge to future work.\n\nUpon your remark, we have added a comparison to Regmix and Autoscale, in the same setup as in appendix B.\n\nFor regmix, in the fixed (N, D) case, we get the following test MREs % :\n\n\n| Scaling law formula | Wikipedia | GitHub|Stackexchange|Pg-19|\n|------|------|------|------|------|\n| Additive, fixed (N, D) | **0.18**| **0.19**|**0.18**|**0.12**|\n| Regmix | 1.31|1.35|0.92|0.89|\n\n\nWhile for autoscale, in the fixed (N) case, we get the following test MREs %:\n\n| Scaling law formula | Wikipedia | GitHub|Stackexchange|Pg-19|\n|------|------|------|------|------|\n| Additive, fixed (N) | **0.17**| **0.26**|**0.19**|**0.15**|\n| Autoscale | 1.94 |2.03|1.31|1.18|\n\nThe MREs obtained with our proposed scaling laws are far lower than those of Regmix or Autoscale.\n\nWe will add this insightful experiment to the text. This comparatively higher MRE for AutoScale means that the estimated optimal domain weights with that method can be far from optimality.\nThe second part of AutoScale consists of a method to extrapolate from optimal domain weights at a small scale to a large scale. Since our scaling laws directly predict the loss as a function of (N, D, h), we can directly estimate h^*(N, D), as explained in Sec. 5, while AutoScale requires a two-stage procedure.\n\nWe thank you again for recommending the inclusion of AutoScale as a comparison. This will help better situate our contribution in comparison to prior work and provide an improved sense of the utility of our proposed method.\n\n### Evaluation Metrics and Empirical Scope:\n\n\nOur primary evaluation metric is Mean Relative Error (MRE) on held-out loss predictions, which is a standard measure for assessing the accuracy of scaling law fits.\n\nWe want to clarify a misunderstanding here, which we will specifically highlight in the final version of the paper: **we never report training losses** (since the training loss of a run depends on the training mixture). Instead, all the losses that we report in the paper are validation losses, computed on a fixed held-out subset of the domain's data.\n \n\nAdditionally, Figure 1 includes validation losses, which show that standard uniform mixtures perform worse than those selected via our method. \n\n\nWe agree that downstream performance is ultimately what matters. To address this, upon your remarks, we added a new study with larger-scale LLMs (up to 7B parameters), where we show that the model trained with our predicted optimal mixture achieves the best results.\nThe study is as follows:\nWe train llms on 7 domains from slimpajama, and fit the scaling laws with small scale runs (<1B parameters). We then train large 7b models with 160B tokens with the optimal domain weights to get the best loss on average for the 7 domains (model A) , or to get the best loss on OpenHermes, a high quality domain (model B). We compare these models to models trained on the base distribution (Model C) for slimpajama and the uniform distribution (Model D) across the 7 domains. To get the best performing models, we train them with a cosine schedule, while the small scale runs were done with constant lrs. In terms of loss, Model A achieves the best loss on average over the domains, and model B achieves the best loss on openhermes. This shows that the optimal weights found by our laws still yield superior performances at large scales. We then score those four 7B models on standard benchmarks. Model B, targeting a high quality domain, yields the best scores, followed by model A, while D and C are comparable. This demonstrates that the optimal weights found at smaller scale to optimize the next token prediction loss also yield good performances on downstream tasks. The downstream tasks' results are as follows (CORE is the median of low variance benchmarks (Li et al 2024)):\n\n|Model | CORE median score| MMLU|\n| -------- | ------- |--------|\n|Optimal weights for avg| 56% | 32% |\n|Optimal weights for OpenHermes|  **58%** |**37%**|\n|Base | 52% | 25% |\n|Uniform | 53% | 30% |\n\nTherefore, the optimal weights found with our method also yield models with superior downstream performances.\n\n*Li, Jeffrey, et al. \"Datacomp-lm: In search of the next generation of training sets for language models.\" Advances in Neural Information Processing Systems 37 (2024): 14200-14282.*\n\n\n### Lack of Theoretical Justification for the Scaling Law Form\n\nWhile we do not include a thorough theoretical justification, we tried to explain our motivations and insights to obtain these laws (Sec. 2.2). Our work is mostly empirically driven: we experimented with several candidate formulations and selected the one that consistently minimized MRE across diverse settings. A theoretical characterization would be extremely interesting and help give a clearer interpretation for the components of the proposed scaling laws, and would be valuable future work. However, this is not necessary; demonstrable empirical performance is sufficient for establishing practical pretraining pipelines.\n\n\n\n### Question 1\n\nThis is an interesting point. We use the following assumptions: Flops are computed as 6ND, and we use a number of training runs that is enough to get to a low MRE, as per Fig.4. We report the training costs for the model sizes shown in table 1. NMM (using 6 runs per sizes): small scale runs cost: 6.9e21 flops. Cost of one 8B run: 7.8e21 flops (Note that we could have trained this model for longer, linearly increasing the computational costs. Also, reducing the number of training tokens to 60B for the small scale runs does not significantly impact the eval MRE, while it reduces the small scale flops to 4.1e21). LLM (using 10 runs per size): small scale runs cost: 2.9e20 flops. Cost of one 3B run: 1.1e21 flops, which is approximately five times higher. We will add these important numbers in the final version of the paper.\n\n\n### Question 2\n\nWe experimented with different learning rates schedulers to investigate if they have an effect on the laws. The results suggest that the lr schedulers are orthogonal to data mixture selection. The intuition is that the optimal domain weights are more about data quality / similarity, which is an intrinsic property of the data domains only, not of the architectural hyperparameters of the models that are trained.\n\n\nWe thank you again for your review, and we hope that we have clarified any grey areas."}, {"review_id": "WT94BgXKwr", "reviewer": "Reviewer_QVnK", "summary": "The paper proposes the use of scaling laws to select data mixtures for pretraining language and multimodal models. They adapt the Chinchilla scaling laws to account for a vector of domain weights (= proportion of each domain/source in the data mixture, that sums to 1). They propose the following method to select optimal data mixtures: \n\n1. Train a set of small models (evenly spaced N and D) with different domain weight vectors (also evenly spaced on a grid).\n2. Minimize the selected loss function (on source or target domain) to fit scaling laws.\n3. Select the data mixture with the best predicted (extrapolated) loss value.\n\nThey demonstrate that their method holds for new data mixtures and domain weights.", "questions": "1. How are the hyperparameters (eg. LR, batch size) set? Are they the same for all model sizes, or do you spend compute finding optimal HPs for each model size? Will the results hold with different HPs?\n2. You mention that the model sizes are varied by changing only the width dimension. Did you try varying num_layers at all? I’ll be curious to know if that significantly impacts the error rates.\n3. The GPU hours that you’ve listed - do they include the target model sizes on which the method is validated?\n4. You mention that only 10-20 runs are needed for fitting scaling laws. How much compute is that in terms of % of target model? It will be helpful to highlight that as well.\n5. It looks like the number of runs required tends to scale with the number of domains. Is that depended on the size of each domain, as well? Eg. if the domain is too small / fine-grained, do the results still hold? More generally, is there a minimum threshold for the domain size?\n6. Re: cosine LR scheduler, it’s very interesting that the optimal domain weights seem to be similar to those estimated with constant LR. Do you think it’s possible to fit scaling laws based on models trained with constant LR scheduler, and directly extrapolate the perf of a target model trained with cosine LR?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Very clearly articulated problem, very relevant for pretraining. The experimental setup is quite thorough - including multimodal as well as language models. Results are also very well organized.\n2. I particularly appreciate that the authors have explicitly listed the GPU hours used for training the models.\n\nWeaknesses:\n\n1. The largest model size that was validated is 8B. While I appreciate that these experiments are expensive to run, language models in particular exhibit “lift-off” on certain capabilities beyond that scale, which can cause a shift in the fitted scaling laws, depending on what’s used as the target loss function. It will be helpful to include some discussion of this in the paper.", "was_revised": true, "final_justification": "No change to my positive rating. The paper presents a method that is practically useful for determining optimal data mixtures. My one comment on using 8B model has been addressed.", "rebuttal": "We thank the reviewer for these excellent points and questions. Below we address each in turn:\n\n\n### On Extrapolation Beyond 8B and Emergent Capabilities\n\nWe agree that some language model capabilities emerge beyond certain parameter thresholds, which could cause a shift in performance trajectories. However, these emergent behaviors typically manifest in downstream task metrics, while the training loss--which we use as the target for our scaling laws--remains smooth and well-behaved. That said, we acknowledge that extrapolation to extreme large model scales  remains challenging. We will expand the discussion in the paper to make this limitation explicit.\n\n\n### Hyperparameter Selection Across Model Sizes\n\nOur hyperparameters are consistent across scales and follow smooth trends with model size. For example, learning rate scales as 1/dim for LLMs and follows a power law found empirically for multimodal models. These choices are found based on our previous experience training these models and are detailed in Appendix A.\n\n\n### Width vs. Depth Scaling\n\nAs noted, we vary the model size by changing width, holding the number of layers constant. We did not experiment with varying depth in this study, but agree that it would be a worthwhile direction for future work. \n\n\n### GPU Hours and Target Model Inclusion\n\nYes, the GPU hours we report include both the runs used to fit the scaling law and the larger target model runs used for validation. We will clarify this in the revised version.\n\n\n\n### Compute Overhead of Scaling Law Fitting:\n\nAs noted, 10–20 runs are typically sufficient for fitting the law. In terms of compute, we will also improve the paper by disentangling the flops required to train models to fit the law and those required for the large scale runs. \n\nWe use the following assumptions: Flops are computed as 6ND, and we use a number of training runs that is enough to get to a low MRE, as per Fig.4. We report the training costs for the model sizes shown in table 1. NMM (using 6 runs per sizes): small scale runs cost: 6.9e21 flops. Cost of one 8B run: 7.8e21 flops (Note that we could have trained this model for longer, linearly increasing the computational costs. Also, reducing the number of training tokens to 60B for the small scale runs does not significantly impact the eval MRE, while it reduces the small scale flops to 4.1e21). LLM (using 10 runs per size): small scale runs cost: 2.9e20 flops. Cost of one 3B run: 1.1e21 flops, which is approximately five times higher. We will add these important numbers in the final version of the paper.\n\n### Impact of Number and Size of Domains\n\nThe number of required runs increases with the number of domains, since the number of coefficients to fit grows accordingly. This is because our formulation treats each domain as a separate component in the mixture vector h. In this work, we assume non-repeating data, where each training step samples from the domains without reuse. We’ll make this assumption clearer in the paper. \nThe case of repeating data is a very interesting future research direction, which becomes more and more pressing as data is being repeated more often in standard pre-training pipelines.\n\n\n### Transferability Across Learning Rate Schedulers\n\nYes, we tested this. We have run a new, large scale LLM experiment. \nThe study is as follows:\nWe train llms on 7 domains from slimpajama, and fit the scaling laws with small scale runs (<1B parameters). We then train large 7b models with 160B tokens with the optimal domain weights to get the best loss on average for the 7 domains (model A) , or to get the best loss on OpenHermes, a high quality domain (model B). We compare these models to models trained on the base distribution (Model C) for slimpajama and the uniform distribution (Model D) across the 7 domains. To get the best performing models, we train them with a cosine schedule, while the small scale runs were done with constant lrs. In terms of loss, Model A achieves the best loss on average over the domains, and model B achieves the best loss on openhermes. This shows that the optimal weights found by our laws still yield superior performances at large scales. We then score those four 7B models on standard benchmarks. Model B, targeting a high quality domain, yields the best scores, followed by model A, while D and C are comparable. This demonstrates that the optimal weights found at smaller scale to optimize the next token prediction loss also yield good performances on downstream tasks. The downstream tasks' results are as follows (CORE is the median of low variance benchmarks (Li et al. 2024)):\n\n|Model | CORE median score| MMLU|\n| -------- | ------- |--------|\n|Optimal weights for avg| 56% | 32% |\n|Optimal weights for OpenHermes|  **58%** |**37%**|\n|Base | 52% | 25% |\n|Uniform | 53% | 30% |\n\n\nIn that large scale experiment, we empirically found that the optimal domain weights estimated using models trained with a constant learning rate transfer well to models trained with a cosine schedule.\n\nHowever, we do not believe it is possible to *extrapolate* the loss of the model trained with the cosine schedule directly, since this model would typically have a much lower loss than the same model trained with a constant learning rate. We believe that extrapolating the loss would require having at least a few small scale runs with a cosine schedule. How to best incorporate the fixed lr runs to these small scale cosine runs to get a robust scaling law estimation is a very interesting research direction for future work.\n\n\nWe thank you again for your review, and we hope that we have clarified any grey areas.\n\n\n*Li, Jeffrey, et al. \"Datacomp-lm: In search of the next generation of training sets for language models.\" Advances in Neural Information Processing Systems 37 (2024): 14200-14282.*"}]

Author rebuttal:

Meta-review: This paper propose a systematic method to determine the optimal data mixture for any target domain using scaling laws. The reviewer ratings are mixed. Reviewers acknowledge the strengths of the paper, such as it is tackling a very important problem of robustly estimating the datamix weights, the method is easy to adopt and optimize yet effective, the empirical study is extensive and thorough, and multi-modality is considered in data mixing. At the same time, concerns are also raised, such as missing comparison with related work (e.g., Data Mixing Laws, RegMix, Autoscale, Doremi, Doge), a lack of clarity regarding modalities, missing downstream evaluations. The authors have provided comprehensive response to address the concerns. Although the reviewers have not confirmed whether the concerns have been fully addressed or not and still maintain negative ratings, I thought they are answered to some extent. Considering this, we recommend an acceptance to this paper.
Decision: Accept (poster)