Loss-to-Loss Prediction: Scaling Laws for All Datasets
Authors: David Brandfonbrener
Nikhil Anand
Nikhil Vyas
Eran Malach
Sham M. Kakade
Venue: Accepted by TMLR ()

Abstract: While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should change as we change the distribution. In this paper, we derive a strategy for predicting one loss from another and apply it to predict across different pre-training datasets and from pre-training data to downstream task data. Our predictions extrapolate well even at 20x the largest FLOP budget used to fit the curves. More precisely, we find that there are simple shifted power law relationships between (1) the train losses of two models trained on two separate datasets when the models are paired by training compute (train-to-train), (2) the train loss and the test loss on any downstream distribution for a single model (train-to-test), and (3) the test losses of two models trained on two separate train datasets (test-to-test). The results hold up for pre-training datasets that differ substantially (some are entirely code and others have no code at all) and across a variety of downstream tasks. Finally, we find that in some settings these shifted power law relationships can yield more accurate predictions than extrapolating single-dataset scaling laws.

Published in Transactions on Machine Learning Research (02/2025)
Loss-to-Loss Prediction: Scaling Laws for All Datasets
David Brandfonbrener
Kempner Institute, Harvard University
Nikhil Anand
nikhil www.ddnaaaaadddd.oomm edu
Kempner Institute, Harvard University
Nikhil Vyas
nibhhl@@g.harvord.coo edu
SEAS, Harvard University
Eran Malach
www..malachh@gmaail..co com
Kempner Institute, Harvard University
Sham Kakade
sham@seas.. www.:aassss.hawaa.d.oo edu
Kempner Institute and SEAS, Harvard University
Reviewed OpenReview: https: /// // opwneevie......ee.e net/ forum? id= 1Aub4JY3Lb
Abstract
While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should change as we change the distribution. In this paper, we derive a strategy for predicting one loss from another and apply to predict across different pre-training datasets and from pre-training data downstream task data. Our predictions extrapolate well even at 20x the largest FLOP budget used to fit the curves. More preciscly, find that there simple shifted power law relationships between (1) the train losses of two models trained on two separate datasets when the models are paired by training compute (train-to-train), (2) the train loss and the test loss on any downstream distribution for a single model (train-to-test), and (3) the test losses two models trained on two separate train datasets (test-to-test). The results hold up for pre-training datasets that differ substantially (some are entirely code and others have no code at all) and across a variety of downstream tasks. Finally, we find that in some settings these shifted power law ronstions can yield more accurate predictions than extrapolating single-dataset scaling laws.
1 Introduction
Scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) have become a reliable tool for extrapolating model performance (as measured through, c.g., cross-entropy loss on held-out data), as well as a way determine optimal model size given FLOP budget (Llama 3 Team, 2024). In their standard form, scaling laws essentially predict the training loss for given model size and dataset size. However, these scaling laws are distribution--eeenddee Relatively little is known about how they change across different pre-training distributions, and how to use scaling laws to predict transfer performance on downstream Lest. distributions.
and only apply to the training distribution that is used to fit the scaling law.
In this paper, we take a first step towards understanding how scaling laws change as change either the training distribution or the testing distribution. To do this, we propose loss-to-loss prediction, methodology for predicting the loss on one data distribution from the loss on another. This is useful since once we have a function that predicts one loss from another, we can take scaling law fit on the first loss and immediately
Notcbooks: http::////gttiiooo/////e/eeeeeea Training code: http://s//....cecee/////eeeeeets Models:

---

Published in Transactions on Machine Learning Research (02/2025)
Train-to-Tras -rain (Join)
Test-to-Test (Join)
Train-to-Tee Test (Transfer)
4.0
Loss on Hellaswag [Trained on data
5.0
Loss on dala (Traimed on dala
3.5
4.5
3.0
4.0
Downstream loss
2.5
3.5
3
2.0
1.5
2.5
1.0
2.0
2.0
2.5
3.0
3.5
2.5
3.0
3.5
2.0
2.5
3.0
3.5
Loss on FineWWeb-Ead Edu (Trained on FineWeb-Edu)
Loss on Hellaswag (Trained on FineWeb-Eak
Loss on FineWeb Edu (Trained on FineWeb (nneWe--Edu)
Data
Downstream Data
FineWeb
ProofPile 2
SmolLM Corpus
Hellaewag ARC-Easy
MML UMLU--umenities
FineWeb-Edu
SimPaajame
StarCodar
MMLU-STEM
Figure 1: (Left) Train-to-train prediction from WibeWeb to all 6 training sets. Each datapoint represents pair of models that are "joined" model size N and dataset size D. Dashed lines represent extrapolation and stars represent 3.3B models trained with 20x compute of the largest dot. These large models are not used to fit the curves. (Center) Test-to-test prediction of Hellaswag cross entropy loss between models trained on FineWeb-edu and models trained on the other datasets. Again each datapoint represents two models joined on model and datasct size. The downstream loss the cross entropy loss of the correct answer to the multiple choice problem when phrased as cloze task. (Right) Train-to-tess test prediction from FineWeb-edu to four downstream tasks. Fach datapoint represents single model and its "transfer" performance on the val data.
translate to a scaling law for the second loss. Further, if we have a suite of models trained on one dataset. and want to predict the performance we would get from training on new dataset, we can apply loss-to-loss prediction. Moreover, there is an independent scientific question of how scaling laws change across datasets.
Our main results are the observations of three types of loss-to-loss relationships shown in Figure 1. First, consider train-to-train, comparing training loss across models trained two different datasets. When models are paired by training compute we find that there is shifted power law that relates the two losses. This has implications for how scaling laws vary across datasets and for being able to predict new scaling laws from smaller samples by translating existing scaling laws from other datasets.
Second, we consider train-to-test transfer where a model trained on one dataset is evaluated on different dataset. Again, we find that a shifted power law is predictive (although with a slightly different shift). These results are less useful for prediction, since they do not predict performance on new train sets. However, they have implications for understanding how pre-training transfers downstream tasks.
Third, we consider test-to-test prediction where we compare downstream test loss across models trained on two different datasets. Like train-to-train prediction, we find a shifted power law when pairing models by model and dataset size. These results are noisier than the others, but have implications for selecting data improve performance on downstream tasks.
Finally, we consider applications of these relationships to practical setting. In this setting, a scaling law has already been fit on one dataset and we wish to make some prediction about what will happen on new dataset given only a very small number of training runs on the new dataset. Explicitly, we look at two types of prediction in this setting. First, we consider a setting where want to fit scaling law on new training set and show that leveraging train-to-train predictions can yield substantially better predictions with as few as eight models trained on the new dataset. Second, we consider predicting the test performance of a larger model trained on the new dataset and find that test-to-test prediction can yield better predictions than extrapolating from runs on the new datasct alone.
To summarize, our main contributions are:
We derive methodology for loss-to-loss prediction that translates scaling laws between datasets.
2

---

Published in Transactions on Machine Learning Research (02/2025)
We illustrate train-to-train, train-to-test, and test-to-test prediction across pre-training datasets 6 diverse pre-training datasets and 11 downstream tasks. We discuss implications for understanding scaling laws, transfer learning, and generalization to downstream tasks.
We show that leveraging data from multiple pre-training datasets can yield better predictions about what will happen when training on new datascts than fitting independent scaling laws.
2 Related work
2.1
Scaling laws
Standard approaches to scaling laws attempt to fit a curve to the optimal number of model parameters and training tokens D minimize the pre-training loss under given budget of FLOPPs (Hestness al., 2017; Kaplan et al., 2020; Hoffmaan et al., 2022; Porian et al., 2024; Abnar et al., 2021; Maloney et al., 2022; Bordelon et al., 2024a).
To fit these curves, it is useful to specify a parametric form of the loss in terms of N and D. Hoffmann et al. (2022) assumes this curve takes the following form:
B D5
= N
(1)
This formula inspired by classical upper bounds on a loss decomposition that attributes error to Bayes risk (entropy), approximation error (from having finite parameters), and estimation error (from having finite data) (Bottou and Bousquet, 2007).
On the other hand Kaplan et al. (2020) instead assumes that:
L(N,D) = ((1 a/9
B
(2)
D
Below, we will advocate for this formulation, but with an added irreducible entropy term E:
L(N,D) =+ a/9
(
(3)
D
Regardless the functional form, scaling laws have been an integral part of the success of modern neural language models. Our work builds on the ideas originated in this line work and extends them to consider how to translate scaling laws across data distributions.
2.2 Scaling laws for transfer and downstream tasks
Scaling laws for pro-training loss are useful a proxy to guide pre-training, but we ultimately care about downstream task performance. Prior work atlempting to lackle this issue has found that directly computing hard metrics like accuracy can lead to the appearance of emergent behaviors and suggests using softer metrics like cross entropy loss instead (Schaeffer et al., 2024a;b). This is corroborated by Du et al. (2024) which notes that while downstream accuracy can vary smoothly with training loss at some points in the curve, the hardness the accuracy metric means that. no progress in accuracy above random chance will be observed until some "emergent" loss level.
On the other hand, Gadre et al. (2024) claims that downstream accuracy can be predicted as a function of training loss with a similar exponential curve to the one we propose for predicting downstream loss. However, they only claim this predictable when averaging over many tasks and carefully selecting which tasks to use. In this paper when considering downstream tasks we focus on single downstream tasks and find loss be more stable metric than accuracy. A detailed discussion of loss versus accuracy is in Appendix A.
Another related line of work comes from the distributional robustness literature on "accuracy on the line" (Miller et al., 2021; Tripuraneni al., 2021; Awadalla et al., 2022). This phenomena focuses on the relationship
3

---

Published in Transactions on Machine Learning Research (02/2025)
between the accuracy of a single model across two closely related tasks, like different versions of imagenet, and finds that accuracy on one will predict accuracy on the other. We consider loss rather than accuracy, language modeling rather than vision, and find non-linear fits.
Another line of work uses linear data models to predict performance for fixed model sizes when training on subsets of some larger set of data (Ilyas et al., 2022; Li et al., 2023a; Engstrom et al., 2024). In contrast, our work focuses on predicting losses across scales and across distinct distributions rather than subsets. Moreover, we find that we need simple non-linear models in our setting.
Note, in this work we focus on zcro shot transfer where there is no finetuning on the target task. Prior work on "transfer scaling laws" focuses instead on finetuning setting (Hernandes et al., 2021; Abnar et al., 2021; Isik et al., 2024), which is interesting, but beyond the scope of this work.
3 Setting
3.1 Notation
We are interested in studying transfer across different training distributions. To formalize this, we will define two distributions: Po and Pt. We will consider Po as the "source" and P1 as the larget. The goal is use a function of the loss Po to predict the loss P1. As example, Po could be Web and Pi could be Starcoder Hellaswag. We use L; to indicate the loss calculated Pi represents a multiple choice task, we will let L1 be the loss of correct answer when the question is phrased as cloze task (following (Schaelfer et. al., 2024b; Madaan et al., 2024))-
distribution P. (averaged per-token). If
Given a pre-training distribution P. we let f.N.D denote an N parameter model trained on D tokens sampled from P.. Our results present comparisons across losses Lo, L1 for models fo ANDD S1 ,N.D when sweeping across different choices of Po, PI, as well as N, D)
When we refer to a scaling law fit from Equation (3) on distribution P., we will append a subscript to the corresponding paramcters. For example, the irreducible entropy of the scaling law fit on Po denoted by Eg.
3.2 Experimental methodology
To facilitate our analysis, we pre-train models of varying size with varying flop budgets on 6 pre-training datasets: FineWeeb (Penedo et al., 2024), FineWeb-edu (Penedo et al., 2024), Proof Pile 2 (Azerbayev et al., 2023; Computer, 2023; Paster et al., 2023), SlimPajama (Soboleva et al., 2023), SmoILM Corpus (Ben Allal et al., 2024), and Starcoder vl (Li et al., 2023b). We train all models using OLMo (Groeneveld et al., 2024) and generally follow hyperparameter settings from Wortsman al. (2023); Zhao et. al. (2024). Importantly, use linear warmup and cosine decay schedule for every run and only report the final performance (Porian et al., 2024). For downstream evaluations, we directly evaluate the models zcro-shot on the downstream tasks as cloze lasks (Schaelfer et al., 2024a). Full hyperparameters and details can be found in Appendix D.
FLOP budgets for our sweep range from 2c17 to 4.84e19 and model sizes range from 20M to 1.7B. The optimal model at the largest FLOP budget is roughly 750M (it varies per dataset). The total grid contains 528 models, or 88 models per training dataset. For our extrapolation experiments, we train 6 larger models (one for each dataset. at a FLOP budget of 1e21 each of size 3.3B. Full scaling law fits and illustrations of the full grid of model sizes are in Appendix C.
4 The hypothesis of loss-to-loss prediction
The main idea we propose in this paper is to use one loss to predict another. In this setup, each loss can be either train loss or test loss and the two losses can be for models trained on different datasets. Formally, we want to predict (FN.N))) the loss on distribution P; for a model trained on Pj, when given (fN. where k, l can be different from j.j. Essentially, we want to parameterize a function g such that
L.(FN 9 (Iu(EN-)).
(4)
4

---

Published in Transactions on Machine Learning Research (02/2025)
Hypothesis. Our main hypothesis is to posit the following functional form for g:
T.() K. (Ia(JTTP)) - -Fesee Eay-
(5)
where K, k are free parameters and Egg is the irreducible error in the scaling law of the loss L, (computed on the distribution P.) for models trained on data from the distribution Pj.
Motivation. The functional form is inspired by a simple observation: both L. and (.. obey scaling laws N and D. So g needs be a valid mapping between scaling laws, meaning that if pass in loss that obeys scaling law as imput to 9 we get out valid scaling law as output.
To see how our functional form respects this observation, we can start from two independent scaling laws:
L.U, Ei +
A1 N
01/2
B + D
$1
L = Es +
((#))
A2 N
a2/B2
+
B2
(6)
,
D
Then, we can see that Equation (5) is a valid translation. In particular, we get the following relationships between parameters of the scaling laws under our parameterization of g:
= K02, 3 K82, A1 = Kon A2, B2 = K RP2 B2.
(7)
Note that there is not similar way to transform the (Hoffmann al., 2022) scaling laws of Equation (1) since cannot modify the exponents without introducing cross terms that depend on both N and D. We should caveat that while the (Hoffmann et al., 2022) formulation does not allow for valid translations, it does empirically give good and very similar fits to the (Kaplan et al., 2020) version. We think it is an interesting open question to precisely pin down the correct formulation for scaling laws and view our work as one data point supporting (Kaplan et al., 2020) with an addod irreducible entropy term.
Caveat. We should caveat that while the hypothesis is stated for any four distributions (P,, Pj. Pk, Pt), we do believe that there could be adversarial distributions constructed so that the relationship no longer holds. In this work we focus on natural distributions of text and we find that the relationship does hold across surprisingly diverse data (e.g. from data that is entirely code to data that does not contain code at all), though there are likely limits to how far these relationships hold.
Comment on theory. In appendix G, we examine how existing theoretical models of scaling laws connect to our empirical findings. Using simplified linear model with power-law feature spectra, show that the model exhibits scaling behavior qualitatively ively similar to our proposed form. While this theoretical framework successfully captures some aspects of in-domain transfer, we note that it does not yet explain the full richness of empirical scaling behaviors, particularly for our out-of-distribution transfer results. This analysis suggests that while existing linear models provide useful insights, additional theoretical development may be needed to fully explain the observed phenomena real datasets.
Summary. So far, this is a very general hypothesis. It says that given any pair losses (which can be train losses or test losses) trained on any pair of data distributions we can fit two free parameters K, x translate between the losses (given estimates of the irredueible entropy of each loss). In the following sections make this more concrete and show how it can be used for train-to-train, train-to-test, and test-to-test. prediction and how these predictions can be practically useful.
5 Empirical evidence for loss-to-loss prediction
In this section, we present the loss-to-loss relationships that for the core observation of the paper. In turn we will present train-to-train, train-to-test, and test-to-test relationships.
5

---

Published in Transactions on Machine Learning Research (02/2025)
Data FineWeb
Data O: FineWeb-Edu
Data ProofPile
Loss 1 (Trained on data 1)
4
Loss 1 (Trained on data 1)
3
3
(Trained on 9
2
1.5
2.0
2.5
Data 1
Loss (Trained on data 0)
Loss (Trained on data 0)
Loss 0 (Trained on data 0)
FineWeb
Data O: SlimPajama
Data D: SmolLM Corpus
Data O: StarCoder
FineWeb-
Loss (Trained on data 1)
4
Loss (Trainod on data 1)
ProofPile
(Trained data
SlimPajama
SmolLM Corpus
3
StarCoder
2
2
2
1.0
1.5
2.0
Loss 0 (Trained on data 0)
Loss 0 (Trained on data 0)
Loss 0 (Trained on data 0)
Figure 2: Train-to-train fits. Each point on the plot represents the final loss of two models: fo 2N.D which is trained dataset 0 and fN.D which is trained on dataset 1. The models are paired when they use the same number of parameters N and tokens D. Starred points indicate a large model trained for the purpose of testing the extrapolation of the curves, which are only fit on the dotted points.
5.1 Train-to to-tatn prediction
Our first main result is observe a consistent scaling relationship between train losses across datasets. Explicitly, we find that by fitting just two parameters K and we can capture and extrapolate the scaling relationship between pairs of training losses as follows:
LA K- (Lo(fo' Ea) + E1
Note, this is comparing different losses and different models, but the models are paired when they each have N parameters trained on D tokens. Also, recall that Eg, Eg are the irreducible errors from independent scaling law fits on Po and P1 respectively. Finally, note that since we are only fitting a slope and exponent, each curve linear on a shifted log-log scale. However, since we are plotting 6 curves in one plot, each with different E,, we cannot display them all consistently log-log plot and opt for a lincar scale for clarity. Further details the curve-fitting process in Appendix F. Results for fitting these curves can be seen in Figure 2. We also include comparisons to linear fits rather than power laws in Appendix B.1.
Compute optimal models. Under the parameterization in Equation (8) for translating between losses, the size of the compute optimal is invariant. To see this, note that the optimal model size for given flop aA=/s budget (C) can be expressed as (GC) ye for a = atg and G = 88 under the assumption that C' = 6N Coupled with the relationships described in Equation (7), this implies that under the transformations induced by Equation (8) the function N (C') is invariant.
This implics that for given FLOP budget, the optimal model size is the same for any data distribution where this translation relationship holds. This seems like a strong conclusion, but does fit in with common empirical practice after Hoffmann et al. (2022) where practitioners often train on approximately 20x more tokens than parameters in a model across datascts. Of course, anything changes in the model architecture or training algorithm, then this translation and this invariance would not hold anymore, but under Equation (8) the compute optimal model size is invariant to changes in the data distribution. It is an interesting open

---

Published in Transactions on Machine Learning Research (02/2025)
Train: FineWeb
Train: FineWeb- Edu
Train: ProofPile 2
6
Tost Loes
Test Loss
*
4
Test
28
2
2
3.0
3.5
2.5
3.0
3.5
1.5
Test data
Train
Train
Train Loss
FineWeb
Train: SlimPajama
Train: SmolLM Corpus
Train: StarCoder
FineWeb-Edl
ProofPile 2
SlimPajama
6
SmolLM Corpus
Test Loss
Test Loss
StarCodor
Test
2
2
2
3 Train Loss
1.0
1.5
2.0
Train Loss
Train Loss
Figure 3: Train-to to- test fits. Each datapoint represents single model trained on the dataset in the subplot title and then evaluated different dataset as indicated by the color.
question to Lest. how generally this invariance holds. roughly holds across the datasets we test which differ substantially (some are all code, others all English), but it may break down for some other datasot pairs.
Parameter values. Note that the exponents x tend to be close to K: for a pair of datasets, this means that they have the same scaling exponents. Across all pairs of datascts the minimum is 0.88 and maximum is 1.13, which occur between Starcoder and SlimPajama depending on the direction of prediction. While these are close to 1, these are sufficiently far enough from 1 that trying make a linear fit will lead substantially worse extrapolation predictions.
On the other hand, K tends to be further from 1. There the largest differences come between SmoILM Corpus and ProofPile (either 0.55 or 1.72 depending the direction of prediction). This suggests that the differences in returns to scale between datasets are clearly seen in differences in the numerators of the scaling laws. Further, note that it is interesting and not obvious priori that we can fit just a single multiplicative constant K which modifies both and in Equation (3).
Implications. In summary, the train-to-train prediction results have few implications:
Since K, k are not near 1, different datasets can indeed lead to substantially different returns to scale in terms of reductions in loss. However, under our translations the compute optimal model size is invariant to the training distribution.
Equation (3) is the only formulation of the underlying scaling law that is compatible with the train-to-train fit given by Equation (8). If we instead used eq. (1), then the transformed scaling law after applying Equation (8) would no longer satisfy the same functional form.
5.2 Train-to-tes prediction
Next, we want go beyond the train loss and consider translating the train loss to test loss for the same model under different distribution. We now hypothesize that the functional form of the relationship is
7

---

Published in Transactions on Machine Learning Research (02/2025)
Train: FineWeb
Train: FineWeb-Edu
Train: ProofPile
7
7
6
6
6
Test Loss
Test Loss
Test Loss
5
4
3
2
2.5
3.0
3.5
4.0
2.0
2.5
3.0
3.5
1.5
2.0
2.5
Test data
Train Loss
Train Loss
Train Loss
Hellaswag
Train: SlimPajama
Train: SmolLM Corpus
Train: StarCoder
ARC-Easy
7
MML U-Humanities
MMLU-STEM
6
Test Loss
Test Loss
Loss Loss
5
5
5
Test
4
4
4
3
3
3
2
2
2
3
1.0
1.5
2.0
Train Loss
Train Loss
Train Loss
Figure 4: Train-to-test transfer for downstream tasks. On the test set we evaluate the CE loss of the correct multiple choice answer as a cloze task.
follows:
L = K. (DoL(N-D)
- -Po)* Fijo
(9)
Note, this is comparing different losses, but the same model. Further, recall that we define Eijo to be the irredueible error of La for the optimal function on Po with infinite model and data sizes:
Eijo Li(i)
(10)
We can estimate this quantity by fitting a scaling law to L1 under data from Po- In practice we take the 88 models trained on Po and evaluate each of them on the OOD test set for IT. This gives a dataset of (n, d, I) tuples that we can use to fit a scaling law and Enj is the entropy term of that scaling law. Note that this assumes the existence of an underlying scaling law for the test loss that takes the same form as Equation (3).
Results in Figure 3 show predictions to validation sets from the pre-training distributions. Results in Figure 4 translate from train-to-downstream test sets where we use downstream multiple choice questions. Following (Schaeffer et al., 2024b; Madaan et al., 2024), we evaluate the downstream tasks by the cross entropy loss on the correct answer when the question is phrased et al., 2019), ARC-Fasy (Clark et al., 2018), and subset of MMLU (Hendrycks et al., 2020), further results for ARC-Challenge, Openbook QA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017), Winogrande (Sakaguchi et al., 2021), and the rest of MMLU are in Appendix B.2.
cloze task. Here we show results for Hellaswag (Zellers
Note that Kaplan et al. (2020) points out a similar trend to Figure 3 in their Section 3.2.2, but they only consider transfer to wikipedia and books and assume the relationship to be linear. By considering a broader array of datasets, we are able to see a more nuanced picture of transfer relationships.
Looking at the train-to-test curves on validation sets in Figure 3, we again see that many of the curves are close to linear (N near 1). However, now there are some notable exceptions when trying to transfer from datasets with little to no code (e.g- FineWeb) to datasets that are entirely code (e.g. StarCoder). These convex curves illustrate diminishing returns to pushing down the Fin loss for transfer performance to StarCoder, suggesting that even we Iearn very good model for english it does not improve much on code.
The lines in each plot extend left until we reach the predicted irreducible entropy. Using this fact, another takeaway from Figure 3 is that the asymptotic transfer performance on test sets can be substantially worse
8

---

Published in Transactions on Machine Learning Research (02/2025)
than the performance from training on that dataset directly. This is intuitive, but does imply that including broader training data that includes the test domains we care about is quite important. This is even true for seemingly similar datamixes like SlimPajama and SmolLM. Getting good performance by training on one of the datasets does not imply optimal performance on the other for a given budget.
Turning to downstream tasks in Figure we see substantially higher curvature than we do across pre-training distributions. Moreover, the curves are often concave rather than convex (i.e. & < 1). This is interesting the training loss. We hypothesize
since here we are actually seeing increasing returns to improvements that this may occur when due to training dynamics, the target task (like ARC-Easy) lives in some tail of the pre-training distribution that only gets fit by larger models later in training. Despite this increasing return to scale, we see the improvements in a smooth way because we mcasure loss rather than accuracy. A detailed discussion of accuracy vs. loss is in Appendix A.
Implications. In summary, train-to-test, prediction has several implications:
The predictions across pre-training datasots indicate the importance of data selection. Even we extrapolate the curves to their ends (where they reach the irreducible error), the loss on transfer datasets do not reach close to the irreducible error for the task, i.. Eijo does not approach Egg
Downstream loss is predictable and does not illustrate emergent properties. Tracking this downstream loss gives a smooth proxy to extrapolate performance on tasks of interest.
Some tasks have convex relationships (x > 1) with pro-training loss where decrcases in pro-training loss have diminishing returns, while others have concave relationships (x < 1) where decreases in pre-training loss have increasing returns. Downstream tasks typically have concave relationships.
5.3 Test-to-test prediction
Next, can move to test-to-test prediction which can be seen as composition of the prior two rules. This now involves three different data distributions: Po the initial training distribution, P1 the target training distribution, and Ps the test distribution that we use to measure loss. Explicitly, we consider:
L2 K
K (La(fo Eajo) + E2j1
(11)
Like train-to-traiis these predictions compare the same loss on different models, but now we are using test loss rather than train loss. In this way, test-to-test can be seen as a generalization of train-to-train. Models are paired when they use the same number of parameters N and number of training tokens D.
Results on four downstream losses are shown in Figure 5. Note that now that we are combining three distributions rather than two, there are many more possible combinations. Here we focus on a fixed Po as FineWeb-edu and show results across training data P1 and test distributions Pz. Further results on other sweeps and combinations can be found in Appendix B.3.
Again the fits are usually good and able to extrapolate to models trained with 20x the FLOP budget of the largest one used to fit the curves. The fits are especially good Hellaswag, but before the other downstream datasets tend to be substantially noisier. This is magnified now since this evaluation noisc affects both the x and axes when they are both measuring test loss (unlike in train-to-test when only one axis depends on test loss). In the next section we will discuss a practical use case for test-to-test prediction.
6 Practical usefulness of loss-to-loss prediction
Consider the following situation that a practitioner could encounter: after having fit scaling law and performed a large run on one dataset, they want to know what would happen if they trained on different dataset. They could fit an independent scaling law on the new dataset, but that would not be leveraging the computation that has already been done. Instead, we can use loss-to-loss prediction. This can allow us to get good predictions of the scaling laws and test performance with only few model runs on the new data distribution since we leverage information already have from the original training distribution.
In this section we consider two variants of this situation, one where we fit scaling law on the new distribution and where we predict the test loss of training large model the new distribution.
9

---

Published in Transactions on Machine Learning Research (02/2025)
Hellaswag
ARC-Easy
MMLU-Humanties
MMLU-STEM
5.0
6.0
6.5
Test loss (Trined data
4.5
Test loss (I [Taiieed on data
Teat loss (Tained on data
Text lass, (Tiined dita
15.0
4.0
3.5
4.0
3.0
3.5
to
2.5
2.0
2.5
2.5
3.5
Tost loss (Trained on FineWeb-edu)
Test loss (Trained on FineWeb edu)
Test loss (Trained on FineWab-cdu)
Test lass (Trained on FineWeb-edu
Train
FineWebb
FireWeb-Edu
ProofPiee
SlimPajame
SmolLM Corpue
StarCoder
Figure 5: Test-to-test predictions for downstream tasks. Each subplot illustrates a different downstream task. The x-axis always reports the test loss for models trained on FineWeb-edu, and the y-axis shows test. loss for all 6 of the different training distributions. Fach point represents two models, joined when they share the same model size and training dataset size.
FineWet-Edu Train
Train-to-Train
ProofPile Train
3.8
10
2.2
16
2.2
ProofPie 2 Train
2.1
Parameters
Paramelers
+
2.0
2.0
3.0
1.9
10
1.8
17
10'
10'
10"
10"
10
2.5
3.0
3.5
10"
10"
10
FLOPs
FiuWWeb-Edu Train
FLOPs
Figure 6: An illustration of using train-to-train prediction to leverage a full set of training runs on FineWeb- edu plus training runs on ProofPile 2 to yield a full scaling law fit on ProofPile
6.1 Translating a scaling law
For the scaling law setting, we consider the following scenario. There are two pre-training distributions Po and P1. Assume that we have already fit a set of 88 small models on Po so as to fit a scaling law. Then, we fit only small models on new distribution P1. Note that with our grid, training the full 88 models takes a cumulative 1e21 FLOPs, while only fitting models (one at each FLOP budget) takes only 9e19 FLOPs. We want to get scaling law Pt.
We will consider two approaches illustrated in Figure and Figure 7:
(Ours) Train-to-train translation. We fit train-to-train curve using the models on P.. From this we can translate the scaling law from Po to P..
(Baseline) Independent scaling laws. Here fit an independent scaling law on Pt from only the 8 models we have that trained on that dataset.
The point of this experiment is to illustrate how train-to-train fits can unlock an efficient way to fit a new scaling law on new dataset. Note that as we said above, we should caution that under train-to-train translation the size of the compute optimal model invariant.
Note: while previous sections, we use Ej Eajz from the scaling law fits here fit any entropy that depend as free parameters the loss-to-loss fits. This is because from small datasets, the sealing law fits are not reliable.
10

---

Published in Transactions on Machine Learning Research (02/2025)
Baseline (ProofPile 2 Train)
Skyline (ProofPile 2 Train)
2.3
2.2
1o"
10
2.1
Paremelers
Train Looe
2.0
2.0
1.9
1.8
1.8
1.7
10
1.8
10
10"
FLOPs
FLOPs
Figure 7: (Left) The baseline just fits the full scaling law on the small dataset of 8 runs on ProofPile 2. (Right) The skyline uses a full suite of models trained on ProofPile 2 to a gold-standard scaling law.
We also consider skyline of fitting a scaling law on P with access to all 88 models trained on P1 Then we compute the 1² of each of the three scaling law models (skyline, ours, and baseline) the entire set of 88 models trained P1 to assess the goodness of fit. Results reported in Table 1.
Target Dataset
Skyline
Ours (mean)
Baseline
FineWeb SineWeb-EEde
0.992
0.990
0.961
0.992
0.990
0.953
ProofPile 2
0.988
0.988
0.928
SlimPajama 0.992
0.991
0.975
SmolLM Corpus
0.992
0.991
0.947
StarCoder
0.987
0.986
0.450
Table 1: R2 values for scaling laws fit with different methods. For our train-lo-train translation we report the mean H² averaged over the possible values for Po for each target distribution Pt. With only 8 runs from the new dataset, our method can nearly match the skyline which has access to 88 runs from the target dataset. In contrast, the baseline of fitting independent scaling law fails badly in this limited data regime since it does not leverage prior computation.
We find that loss-to-loss prediction yields substantially better scaling law fits than the baseline. In fact, even with only 8 models on P1, using train-to-train prediction to tranlate the original scaling law nearly matches the R² of the skyline that has access to all 88 models on P1, up to about 0.001. In contrast, fitting a new scaling law on only this data very ineffective. This experiment shows that leveraging the existing models from Po can yield more efficient scaling law fits on new distribution P1 when using loss-to-loss prediction.
6.2 Predicting test loss large model
For the test loss setting, we consider the following scenario. There are two pre-training distributions Po and P1. Assume that we have already fit a set of small models and one larger model (3.3B parameters and 1c21 FLOPs)) Po- Then, consider new dataset P1 and fit only 8 small models with various budgets Pt- We want to predict what would happen if train a large model on P..
We will consider the approaches illustrated in Figure plus one additional baseline:
(Ours) General train-to-test prediction. We fit train-to-test using the paired small models. Explicitly, we predict LD() from LoN...d) This allows extrapolate using the train loss of the large model trained on Po an input.
across different training sets
(Ours) Test-to-test prediction. We fit test-to-test curve using the 8 paired small models. Explicitly, we predict L2(f.N.D) from Le(soN This allows extrapolate using the loss of the large model trained Po input.
11

---

Published in Transactions on Machine Learning Research (02/2025)
General Train-to-Tess
Test-to-Test
Baseline: FLOPs-to-loss
Baseline: independent scaling
Teat loes (Trained on Proo/Plle 2)
4.50
4.5
Teet (Trained ProoPlla
4.5
4.5
4.25
10"
Lose
4.0
Paramaters
4.00
3.75
3.5
3.5
3.25
3.0
3.0
3.00
10'
2.5
3.0
3.5
2.5
3.0
3.5
10
10"
10"
10
10"
10"
10"
Train loss (Trained on FineWab--Edu)
Test loss (Trained on FineWabb-Edu
FLOPs
FLOPs
Figure 8: An illustration of four of the methods we consider for making extrapolative predictions of Hellaswag test loss. (Left) General train-ton tost prediction uses train loss of models trained on Po (in this case FineWeb- Pi (in this case ProofPile 2). (Center-left) Test-to-test prediction
edu) to predict test. loss models trained uses test loss of models trained on Po to predict test loss models trained on Pt. (Center-right) Predicting the Lest loss only using runs from Pi by fitting the relationship bet ween FLOPs and lest loss. (Right) Fitting full scaling law to P1 using only the limited data from P1.
(Baseline) FLOPssto-test. As first. baseline that does not information from Po, we fit. curve from FLOPs to test loss. Since each of the models is near the chinchilla-optimal model size for the FLOP budget, it is reasonable to fit curve and extrapolate it here.
(Baseline) Independent scaling law. As before, we can fit fulll scaling law to the set of small models on Pi and extrapolate the predictions.
(Bascline) Identity. As an even simpler baseline, we can just predict that the test loss when training on Pi exactly the same as training on Po-
Target Loss
General Train-to- Test
Test-to-Test
FLOPs-to-loss
Scaling law
Identity
Hellaswag
1.6%
1.2%
1.7%
2.1%
9.2%
ARC-Easy
10.2%
17.6%
14.3%
16.8%
24.8%
2.8%
23.1%
4.4%
4.7%
11.0%
MMLU-STEM
6.4%
6.4%
5.9%
7.6%
11.5%
Table Relative error, i.e. |pred-actual] of various methods for predicting the test. loss of the the extrapolation actual
run trained on new dataset. All assume that we have already run a of pre-training runs on Fibe edu as Po. All values are averaged across the 5 possible target pre-training datasets Pt. Loss-to-loss predictions are usually the most accurate.
We report results in terms of the relative error (Ireedactuaally of the prediction of the test loss for various test sets in Table 2. We find that the loss-to-loss methods tend to perform the best. This makes sense because are able to leverage extra information, especially the loss of the large model on Po to improve the predictions. The basclines have way to incorporate this information that we know from already having trained models on Po. Note that train-lo-test tends to out-perform lest-to-test on the noisier eval datasets (i.e. those other than Hellaswag). This makes sense because using a noisy x variable to regress onto a noisy y variable going to be higher variance than using a lower variance ø variable. Espocially since standard train-to-test prediction suggests that there no more information in the test loss on Po compared the train loss. An interesting direction for future work is to figure out how to leverage this type of prediction to perform data selection.
12

---

Published in Transactions on Machine Learning Research (02/2025)
7 Discussion
Here we discuss the takeaways of our findings, some limitations, and directions for future work.
Takcaways.
Loss-to-loss fits with shifted power laws provide a good description of empirical trends across a variety of pro-training datasets and to downstream tasks. These fits can effectively extrapolate well beyond the scale they were trained on.
Loss-to-loss prediction of scientific interest since it provides several insights into the nature of how training data affects models and how transfer performance scales predictably.
Loss-to-loss predictions can be practically valuable for translating scaling laws and predicting test loss of large models trained on new data.
Limitations and caveats.
Our fits rely on estimating the asymptotic entropy of various scaling laws, This is a fundamentally difficult quantity estimate and we hypothesize that where our fits fail is often due to poor estimates of this quantity. Moreover, we hypothesize that when our fits fail to extrapolate beyond the 20x results reported in the paper, is likely due to errors in estimating these irreducible loss terms.
Note that many of the train-to-test and test-to-test fits have noisier trends, especially at high losses. It is not totally clear if this is pure noise or may be indicative that the power law trend does not hold as globally as we hypothesize. Future work could dive into this issue more directly.
We only test on a relatively small set of rlownstream tasks compared to all possible choices. We also focus on multiple choice tasks instead of generative tasks since they have been more extensively studied in prior work and have casier to compute proxy loss metrics.
Our results hold for specific choices of hyperparameters and may not hold under some other choices. In particular, we would be interested in checking robustness to pre-training hyperparameters like sequence length, batch size, and learning rate.
Future work.
One exciting direction is to take the implications of the loss-to-loss relationships further so as to directly inform data mixing and filtering. Once we have reliable predictions, we can use those inform choices about which data to train on. Perhaps this could use the scaling laws derived here in combination with recent relating scaling laws and data mixtures (Jiang al., 2024; Ye et al., 2024).
We hope to gain tighter theoretical understanding as to why the loss-to-loss relationships are so clean by studying simplified models. In Appendix G we attempt to connect some of the existing theory literature to our results, and show that a prototypical version of train-to-train transfer emerges in a class of previously studied linear models. would be interesting have a better theoretical understanding of train-to-test transfer as well as richer model that could capture the full extent of the phenomena that. we observe in practice.
Our results connect surprisingly disparate datasets. We able to predict performance code data from data that contains no code and vera-versa. It would be nice to have better mechanistic understanding of how this works. It is possible that all the models converge to "features" that share some high level distributional properties (e.g. similar eigenvalue decay of the covariance). Or at a different level of granularity, it possible that there the data more similar than we think and there large enough amount of English in code and visa versa that losses are predictive. Or perhaps there are particular shared structures that emerge, e.g. in context learning.
13

---

Published in Transactions on Machine Learning Research (02/2025)
References
Samira Abnar, Mostafa Debghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXio:2110. 020090 2021.
Alexander Atanasov, Jacob A. Seratone-Vethhe and Cengiz Pehlevan. Scaling and renormalization in high- dimensional regression, 2024. URL https://arxiv,, httpp/////aaiiooo///a/////////// httbs//245500555
Anas Awadalla, Mitchell Wortsman, Gabriel Ilbarco, Sewon Min, lan Magmusson, Hannaneh Hajishirzi, and Ludwig Schmidt, Exploring the landscape of distributional robustness for question answering models. arXiv preprint ar Xiv:22200. 12517, 2022.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27), June 2024. ISSN 1091-6490. doi: 10.1073 141177/ppas.23111 231187811 URL
wtt.:di........1.00/. 10w3/pas...............m.m pnas. 2311878121.
Loubna. Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro Werra. Smollm-corpus, 2024. URL https: / //ugggigga/g
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint 1011022444.10002, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, el al. Piqa: Reasoning about physical commonseuse in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432 7439, 2020.
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 18-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1034-1034. PMLR, 2020. URL http::/pproceedings mlr html.
Blake Bordelon, Alexander Atanasov, and Congiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01002 2024a.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws, 2024b. URL https://arxiv... org/ abs//499..99././///// 17858.
Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning, Advances in neural information processing systems, 20, 2007.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communications, 12(1), May 2021. ISSN 2041-1723. doi: 101103//s41467.022222211103 URL http://áx. doi .r.//0.. s41467-021 23103-1.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiu:1803.0 100:000535557, 2018.
Together Computer. Redpajama: An open source recipe reproduce Ilama training dataset, 2023. URL bttpss////t wtw.thh...ccccceette http://////oggeeercoopttt///eaaa
-ata.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless noisy regime. In Marc' Aurclio Ranzato, Alina Beygehimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, oditors, Advances in Neural Information Processing Systems 34: Avauua Conference on Neural Information Processing Systems 2021, NeurIPS 2021,
14

---

Published in Transactions on Machine Learning Research (02/2025)
December 6-14, 2021, virtual, pages 10131-10143, 2021. URL http:://ppooeetngg.oorr........ 2021
Elvis Dohmatob, Yunzhen Feng, Pu Yang, François Charton, and Julia Kempe. A tale of tails: Model collapse a change of scaling laws. In Forly-first International Conference on Machine Learving, TCMT, 2024, Vienna, Austria, July 21-27, 2024. OpenReview.nct, 2024. URL https://openreviex wtwpreewwwii.eettooourrrem KVrku47ahW.
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiu:2408.15796, 2024.
Logan Engstrom, Axel Feldmann, and Aleksander Madry, Dsdi: Model-aware dataset selection with datamodels. arXiv preprint PxXx:2201.12226, 2024.
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercal, Alex Fang, Jeflrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540 2024.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Tvison, Tan Magmusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2/02.00833 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXic preprint arXiv:2009:03300, 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint XrXiv:2102.0 01298, 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empiri- cally. arXiv preprint arXiv:1712. 112.0090,000499, 2017.
Jordan Hoffmann, Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.155:: 2022.
Marcus Hutter. Learning curve theory, 2021. URL https:///axt:: org/ wttp///////..//////////// 04074.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint 2rXiv:2202.00622, 2022.
Berivan Isik, Natalia Ponomareva, Husscin Hazimch, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. ar preprint or aiv:22402.00 177, 2024.
Ayush Jain, Andrea Montanari, and Eren Sasoglu. Scaling laws for learning with real and surrogate data, 2024. URL https://arxiv..or// httg//bbs/4422 /202..2222220 04376.
Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J Zico Kolter. Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arX1::4400111820, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0894 2020.
Dongyue Li, Huy Nguyen, and Hongyang Zhang. Identification of negative transfers in multitask learning using surrogate models. arXiv preprint 2iv:2303..4482, 2023a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Mucnnighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint ar xrXi:22305.06111 2023b.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Licong Lin, Jingfeng Wu, Sham M. Kakade, Peter L. Bartlett, and Jason D. Lee. Scaling laws in linear regression: Compute, parameters, and data, 2024. URL http::///aart...orr//////...... 08466.
Llama Team. The Ilama herd of models, 2024. URL httpss//arxxxv.og///b/22/77 221883.
Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. arXiv preprint arXiv:2406.10222 2024.
Alexander Maloney, Daniel Roberts, and James Sully. solvable model of neural scaling laws. arXiv preprint XrXi::2210. 16859, 2022.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, edi- tors, Advances in Neural Information Processing Systems, volume 36, pages 28699-28722. Cur- ran Associates, Inc., 2023. URL https: /appa//////ccegngs... wtwpieeeedinnnnnuuriiie neurips. cctppppr..illlepppp//////////1 3b6644a65.55744cc2111999933
Paper- Conf Conference....eeeeeocee pdf.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint Xivi: 1809 020900 2018
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International conference on machine learning, pages 7721-7735. PMLR, 2021.
Yoonsoo Nam, Nayara Fonseca, Scok Hyeong Lee, Chris Mingard, and Ard A. Touis. An exactly solvable model for emergence and scaling laws, 2024. URL https://axxiv. wwtp://.rgg/////////////oo/ss/// http//bb/2440/// 17563.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+3 phases of compute-optimal neural scaling laws, 2024. URL https: //araiv.//xiivv///a hegg//b//2405.... 15074.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Guilherme Penedo, Hynek Kydlfček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint Xiv::24406. 1557, 2024.
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancics in compute-optinal scaling of language models. arXie preprinl #xXi:2246.19144 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Commenications of the ACM, 64(9):99-106, 2021.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024a.
Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradlcy, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilitios of frontier ai models with scale remained elusive? arXiv preprinl Xrxiv22406 04390,,,.4.1.. 2024b.
Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9):1-34, 2022. URL http://jmm
arg/paperrs/23/20-1 11111 html.
Daria Soboleva, Faisal AI- Khatebb Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token clcaned and deduplicated version of RedPajama. https://www. cerebras. cerebras.net//loog. slimpajame ama- wor7d-token-clae aaw.-aandddd---ddddddiiaatteeddd wersion-or- oorsion-of-rrdpajaana 2023. URL https:
/ //huggingface http:/fac...ooataaaateeeee // 62277.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Stefano Spigler, Mario Geiger, and Matthicu Wyart. Asymptotic learning curves of kernel methods: empirical data v.8. teacher-student paradigm. CoRR, abs/1905.10843, 2019. URL http://arxivv.rrg http://..rga/bbb////b/abaae//// 1905. 10843.
Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Covariate shift in high-dimensional random feature regression. arXiv preprint arXiu:2111.0823, 2021.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17 23 Jul 2022. URL https://proceedingg.
PMLR,
Johannes Welbl, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprinl vxXi::177.00220 2017.
Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309. 21X222209.14922, 2023.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint (r31v:243..16952 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXic:1905.078: 07830, 2019.
Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes good optimizer for language models. arXiv preprinl 007.07447 2024.
17

---

Published in Transactions on Machine Learning Research (02/2025)
A From loss to accuracy
A.1 Train-to-error
We focus on loss-to-loss prediction, but of course would be useful to be able to predict accuracy. Prior work (Schaeffer al., 2024a;b; Du et al., 2024) indicates that predicting accuracy from loss can be difficult, and we generally agree. However, other work (Gadre et al., 2024) claims that downstream accuracy can be predictable in some cases and want to consider here whether accuracy is predictable in our data with methods similar to those presented in the main text.
Challenge
Easy
Hellswag
Openbook
0.8
Cheeffaatin Ent
01 Checs Eise
0r Cless Ente
0.7
Steestratton Erne
0.7
0.8
0.4
0.5
0.66
0.8
0.4
0.5
0.3
0.5
0.4
Train
Train Loss
Train
FnulWib-Edu
FroofFilo
Sciq
Winogmande
OLMo Sutte
SlmFajamo
0.5
BrrolMM Copus
0.52
0.8
SbrCoder
0.4
0.4
01 Olsssootton E30
0.50
on Dasthootton Eiror
0.5
9n Clealoalion
0.3
0.3
0.48
0.4
0.2
0.45
0.1
0.44
Train Loss
Troin
Figure Fitting training loss accuracy on the OLMo tasks individually (first subplots), and then in aggregate (bottom right). Unlike the plots in the main paper where each line only fits paramcters K, n. here we also fit a third parameter in place of Eijo-
Mmlu Humanities
Social Sciences
0.750
0.75
0.75
0.775
0.728
CiMoicatoon Eire
0.70
0.750
0.700
0.65
0.725
0.675
0.80
0.60
0.700
FineWeo
FineWee Edu
0.850
0.50
0.675
Prooffhi
StinPajeeme
0.825
0.800
0.50
0.58
0.660
Sta Coder
SmalLM Corpui
0.625
0.50
2
2
Loas
Truin Losa
Train Loss
Lowa
Figure 10: Fitting training loss to accuracy on MMLU splits.
In particular, (Gadre et al., 2024) specifically finds that when they select a subset of 17 particularly easy benchmarks (where performance is better than chance for small models), then they can get good prodictions for the average accuracy by fitting shifted power laws with a methodology similar to the one that we use for loss-to-loss prediction (but where Eijo is treated free parameter). We are able to reproduce similar result on our suite of 7 tasks from OLMo, see Figure 9. Explicitly, we fit the following relationship to and let the multiple choice error & (i.e. 1 accuracy):
E, C.N.N. ~ K- (Lo(fo.D)- Eo)) -Eo) + M
(12)
18

---

Published in Transactions on Machine Learning Research (02/2025)
where Err is the error and unlike in the main text are now fitting 3 parameters K, M instead of just K, K.
The fits are fairly good for the aggregate, but it is clear that some of the fits (e.g. Hellaswag and ARC challenge) are systematically wrong. They end up overostimating the error because power law fits fundamentally cannot handle the fact that bad models will perform at random chance. The asymplotics of a power law mean that as I - do we get Err- + 0c, which is not possible. This is fundamentally related to the loss perspective on emergence (Du et al., 2024) where for multiple choice tasks there some value of loss where the models start performing better than random chance. This is also perhaps even more clear for MMLU in Figure 10. In general, we would not expect this technique to work on individual tasks and especially not on more challenging tasks.
One potential remedy for this issue would be to introduce a fourth parameter the fits that can handle the transition from predicting at chance making progress. Explicitly, can let the curve be the soft-min (softmin(2r, y) = log(exp(-az)) + exp(-oy)) for = 10) between a constant representing the chance error rate and the shifted power law from before. Explicitly:
E.(for.) x softmin(c, K. (Lolfo.p) -Eo) + +)
(13)
Challenge
Easy
Hellaswag
Openbook
0.80
Cassoolin Ermr
01 CaasTeatton Enror
0n flastoatton Ener
0.6
01 Castiiatoon Ener
0.75
0.70
0.4
0.4
0.85
0.2
0.2
0.80
0.60
2
Train Loxs
Towin Loo0
Tran
Towin Look
FineWho
FinuWao-Edd Edu
FroofPile 2
Piga
Sciq
Winogrande
OLMo Suite
PlinPajemaa
0.5
0.5
0.52
SmalL Corpu
Ela/Coder
Clusfiicabbn E2t
0.4
Cleeitaab Elrce
07 Carmiralion
Clasoitcatonn
0.48
0.4
0.
0.1
0.44
0.1
Loso
Loso
Train Loss
Troin Loss
Mml Humanties
Mmiu Other
Mmnu Social Sciences
Mmiu Stem
0.76
0.75
0.70
0.75
Chexibattinn Ettck
Clasittation Error
Casshattion Error
0.68
0.8
0.70
Tran date
0.50
0.65
FineWoo
FineWeo Edu
0.70
0.55
0.5
0.60
ProufPia 2
0.50
SkmiPajam
0.88
0.4
Corpus
0.45
SturCoder
0.40
0.3
0.50
Loss
Train Loss
Figure 11: The relationship between downstream CE loss and classification error shows unified trends across pre-training distributions, i.e. it seems that all points roughly fit onto one trend line regardless of their color.
Results for this approach with 4 learned parameters per curve are shown in Figure 11. In general, we find that this seems to help (c.g. on Hellaswag and MMLU), but may introduce bias on others (c.g. on Openbook or SciQ). We think this is a promising approach and does seem to yield more robust predictions than the prior approach on harder tasks like MMLU. But, we are not certain that these fits are quite right
19

---

Published in Transactions on Machine Learning Research (02/2025)
universal as the simple shifted power laws relating losses. As seen in prior work, computing accuracies is nuanced since it is hard metric that also depends on the wrong answers (Schaeffer et al., 2024b). As such, we focus the main paper on losses which we find to more consistently obey shifted power law relationships.
A.2 Test-to-erron
For similar reasons, we also found it difficult to fit loss-to-error maps from the downstream CE loss to the classification error. However, while the exact functional form of the dependence is unclear, there is useful information in the loss-to-crror plots Figure 12. Importantly, there convergence across pre-training distributions where irrespective of the pre-training distribution there is a relatively consistent relationship between downstream CE loss and classification error. This is markedly different from the patter we see when looking at train loss where each pre- pre-training dataset yields different relationship between train and any test loss or crror.
Arc Challenge
Arc Easy
Hellaswag
Operbbook Qa
Cheollottt Erer
07 Dlas/fattonnn Enorr
0.7
Olsiffaatio Entr
0.7
01 Cassfictton Ener
0.75
0.8
0.70
0.70
0.65
0.4
0.5
0.65
0.80
0.4
0.60
Tran dade
Tret
FineWWo
ProofPlic
Piga
Sciq
Winagrande
OLMo Sulto
SlimPajjemm
BmooL Corus
0.45
Olassitcaton Erot
011 Carzotcaton Ertor
0f Consaccaion Ertt
0.00
Sta/Coder
011 Clanffiaioon
0.65
0.40
0.3
0.48
0.50
0.35
0.2
0.48
0.45
0.30
0.44
0.40
3
4
Test Loss
Tost Loss
Tost Loco
Test Losa
Mmlu Humanities
Mmla Other
Mmnu Social Sciences
Mmlu Stem
0.750
0.78
0.750
0.725
Cluiiaatton Eitor
01 Caasmhattonn EnTOr
0.78
0.72
0.675
0.700
0.74
FiseWoo
0.675
FincWeo Edu
0.850
0.72
ProofFhl 2
0.70
0.626
0.600
Bli
0.825
SmaLLM Corpus
0.600
StarCoder
0.800
0.775
Lost
Tost Loss
Tost Loos
Tost Loss
Figure 12: The relationship between downstream CE loss and classification error shows unified trends across pre-training distributions, i.e. it seems that all points roughly fit. onto one trend line regardless of their color.
The fact that the trend from test loss to error is unified across pre-training data suggests that this test loss is good proxy measure for the downstream task and supports using it as our main endpoint in the paper. Tn particular, if we consider the causal relationships between different variables, we are suggesting that the train loss only causes the downstream accuracy through a mediating variable that is the downstream CE loss on the correct answer. As a result, once we compute the downstream CE loss, we break the causal relationship between pre-training data and downstream accuracy. This seems to be generally true, but may not be strictly true at high loss values (e.g, on SciQ or Hellaswag). But, this does suggest that the CE error is a useful proxy since it mediates the pre-training-specifi effects from the test accuracy.
20

---

Published in Transactions on Machine Learning Research (02/2025)
B Additional loss-to-loss results
B.1 R and extrapolation error for train-to-train
For the train-to-train fits Figure 2 we provide tables here with the R value of the fits on the training data and the relative error of the predictions on the extrapolation point. Note that the gap between our method and linear fits much larger for the train-to-test and test-to-test, so we focus on this case where the comparison is closest.
Both methods fit two parameters: K, x for ours and a slope and intercept for the linear method. The results show that the R² values quite close, but the extrapolations is because method respects the differences in asymptotic entropy between different datasets, while the linear fits cannot do this. The gap is especially large on the most different dataset, StarCoder, where the nonlinearity of the relationship most clear.
generally better for method. This
Target data
Ours
Linear
FineWeb
0.9998
0.9998
ProofPile
0.9990
0.9989
SlimPajama 0.9997
0.9996
SmolLM Corpus
0.9999
0.9999
StarCoder
0.9979
0.9972
Table 3: Train-to-train R when the source data is FineWeb-odu (other datascts similar).
Target data
Ours
Linear
Feb
0.141 %
0.300%
ProofPile 2
0.086%
1.429%
Slim SlimPajama
1.339%
0.643%
SmolLM Corpus
0.649%
0.822%
StarCoder
1.957%
5.795%
Table 4: Train-to-train relative error of the predictions at the extrapolation point when the source data is Fine Web-edu (other datasets are similar).
21

---

Published in Transactions on Machine Learning Research (02/2025)
B.2
Train-to-test
Train: FineWeb
Train: FineWeb-Eon
Train: ProofPile 2
7
Test Loss
Test Loss
5
5
Test
3
2
2
2.5
3.5
2.0
3.0
Test data
Train Loss
Train Loss
Train Loss
MMLU-Other
Train: SlimPajama
Train: SmolLM Corpus
Train: StarCoder
MMLU-Social Sciences
7
7
MMLU-STEM
Test Loss
Test Loss
5
5
Test
4
4
3
2
2
2
1.0
1.5
Train
Train Loss
Train Loss
Train: FineWeb
Train: FineWeb-Edd
Train: ProofPile 2
8
8
Test Loss
6
Test Loss
2
2
2.5
3.0
3.5
4.0
2.0
2.5
3.0
3.5
1.5
2.0
2.5
Test data
Train Loss
Train Loss
Train Loss
Hellaswag
Train: SlimPajama
Train: SmolLM Corpus
Train: StarCoder
ARC-Easy
8
ARC-Challenge
Openbook Qa
Piça
Test Loss
6
Test Loss
Sciq
Test
Winogrande
2
3
1.0
1.5
2.0
Train Loss
Train Loss
Train Loss
Figure 13: Train-to-test predictions across all individual downstream tasks.
22

---

Published in Transactions on Machine Learning Research (02/2025)
B.3
Test-to-test
Test data: SmolLM Corpus
Test data: FineWeb-Edu
Test data: SlimPajama
Teat lasss (Trained on data 1)
Test loss (Trained an deta 1)
(Trained on data 1)
6h
4
Train data
Test loss (Trained on FineWeb-Edu)
Test loss (Trained on FineWeb-Edu)
Test loss (Trained on FineWeb-Edu)
FineWebb
FineWebb Edu
Test data: FineWeb
Test data: ProofPile
Test data: StarCoder
ProofPlle
Test loss (Traimed on dala 1)
Test loss (Traiiedd on dala 1)
6
Pajama
SmolLM Corpus
5
StarCoder
Teat (Trained
12
2.5
Test (Trained on FineWeb-EEuu)
(Trained on FineWWeb-Edu Edu)
(Trained FineWeb.Edu)
Figure 14: Test-to-test prediction using the validation sets from pre-training data the targets. Each subplot shows different test loss. Within each subplot, the training data Po is always FineWeb- Edu and the curves illustrate all of the 6 possible options for P1. Each point corresponds to two models.
23

---

Published in Transactions on Machine Learning Research (02/2025)
Hellaswag
ARC-Easy
MMLU-Humanilies
MMLU-STEM
Teal loes (Trained on data 1)
Teal loon (Trairned on datte
10 02 9 9
Test cos (Tramoc
loes
2.5
3.0
3.5
(Trathed FineWobb
Tost (Trained FineWab)
(Trained FineWoa)
Testloss [Trained on FineWab)
Hellaswag
ARC-Easy
MMLU-Humantiee
MMLU-STEM
Tent tose (Trained on diaae
Test looe (Trained on dista
4
(Trained
on loze
2.5
(Theined on FineWWeb-Edi Edii)
loss (Trained FineWWib-Edu)
(Trained FineWab-EEd))
loss (Trained on FinWebbEElu
Hellaswag
ARC-Easy
MMLU-Humanities
MMLU-STEM
Teal bosn (Trairacc on data 13
Teat bas (Trained on data
t0 5n
(Tratned on
2
Tost (Trained on ProdtPile 2)
Test (Trained on Prooftile 2)
Test loss (Trained on ProctPile 2)
Testloss (Trained on ProotFilc 2)
Hellaswag
ARC-Easy
MMLU-Humanttles
MMLU-STEM
(Trained datt 1)
Test losa (Trained on dlete 1)
Test lose (Trained on diste 1)
(Trained data
>
Test loss (Trained on SimPajame)
Test loss (Trained on SinPajams)
Test loss (Trained on SimPajama)
Test loss (Trained on SimPajams)
Hellaswag
ARC-Easy
MMLL-Humanilies
MMLU-STEM
Test loes (Trained data 1)
6
Teal bosn (Traineed on data 1)
Teat loos (Trained on dala 1)
Test (Trained dala
3.0
3.5
2
3
Testloss (Trained on SmolLM Corpus)
lost (Trained on SMoILM Corpus) Tost (Trained on SmoLLM Corpus) Test (Trained on SmolLM Corpu
Hellaswag
ARC-Easy
MMLU-Humanilies
MMLU-STEM
(Trained 1)
Tost lose (Trained on divle 1)
Test loss (Trained on dalts 1)
TTst (Trained data
2
Test los (Trained on StarCoder)
Test loss (Triined StarCoder)
Test loss (Trained on StarCoder)
Test loss (Trained StarCoder)
FineWebb
FineWWeb-Eck
ProofPle 2
SlinPajanes
SmoLLM Corpus
StarCoder
Figure 15: Test-to-test prediction on the four losses from the main text. Each row shows different training loss Po on the x-axis. Each point corresponds to two models.
24

---

Published in Transactions on Machine Learning Research (02/2025)
Test data: MMLU-Other
Test data: MMLU-Social Sciences
Test data: MMLU-
Test data: MMLU-STEW
Test loss (Trained on data 1)
Test loss (Trained on data 1)
Test lops (Trained on data
Test loss (Trained data
5
6
6
3
3
10
2
2
3
5
2
3
4
3
Test loss (Trained on FineWeb-Edu)
Test loss (Trained on FineWeb-Edu)
Test loss (Trained FineWeb-Edu)
Test loss (Trained on FineWeb-EEu
Test data: Hellaswag
Test data: ARC-Easy
Test data: ARC-Challenge
Test data: OpenBookOA
Test loss (Trained on data 1)
Test loss (Trained on data
Test loss (Trained on data
Test loss (Trained on data
6
3
2
2.5
3.0
3.5
2
2
5
4.5
5.0
5.5
6.0
Test loss (Trained on FineWab-Edu)
Test (Trained FineWeb-Edu)
Test (Trained FineWeb-Edu)
Test loss (Trained FineWab-Edu
Test data: PIQA
Test data: SciQ
Test data: Winogrande
Test (Trained on data 1)
Test (Trained on data 1)
Test loss loss (Trained on data 1)
Train
5
FineWeb
FineWeb-Edu
SlimPajama
SmolLM Corpus
StarCoder
2
4.0
(Trained on FineWeb-Edu)
(Trained FineWebb-Ed))
Test (Trained FineWeb-Edu)
Figure 16: Test-to-test prediction all 11 downstream losses consider. The training data Po is fixed to FeneWeb-edu in all subplots. Each point corresponds to two models.
25

---

Published in Transactions on Machine Learning Research (02/2025)
C
Scaling law fits
We follow the methodology from Hoffmann et al. (2022); Besiroglu et al. (2024) for fitting scaling law curves and illustrate fits for both Equation (3) and Equation (1). In particular, have average 88 datapoints for each training dataset, each consisting of the model size, number of training tokens, and final loss. We use the estimate that FLOPs is 6ND when reporting FLOPs. We then fit the curves using L-BFGS in log space with Huber loss We plot the results and give the parameter values.
SmolLM Corpus
FineWaa-Edu
Siimajama
-32
10"
-30
-34
Poramaters
23
Panmmeles
32
.*...
10
to
-.8
10*
2.0
-28
26
10"
22
22
10"
10"
10"
10"
10"
10
10"
10"
10"
10"
10"
10"
10" FLOPS
10"
10"
FLLOPs
FLOPs
FineWWeb
ProofPile
StarCoder
2.4
-38
10
-38
-22
10"
1.8
Parameters
Las
Parmmerrs
2.0
Train Looe
1.8
32
1.4
2.8
1.2
288
1.0
10'
10'
10"
10!"
20
20
10
10"
10
10"2
20
FLOPs
FLOPs
FLOPs
Figure 17: Contour plots for the curves fit with Equation (3) (our version of the scaling law parameterization).
Red line indicates the optimal model size. The star point is not used for fitting the curves.
Data
A
B
E
α
3
a
SmolLM Corpus
7.79e+007
1.06e+009
1.53
0.42
0.45
0.52
FideWeb-Edu
6.68e+007
8.90e+008
1.97
0.41
0.46
0.52
SlimPajama
7.47e+077
1.06e+009
1.97
0.40
0.43
0.52
FineWeb
6.79e++07
9.31e+08
2.17
0.41
0.45
0.52
ProofPile 2.14e+007
3.29e-+08
1.32
0.45
0.46
0.50
StarCoder
2.23c++07
3.78c+008
0.85
0.45
0.47
0.51
Table 5: Parameters for the curves fit with Equation (3) (our version of the scaling law parameterization).
3 is the exponent of the optimal model size relative to FLOPs.
26

---

Published in Transactions on Machine Learning Research (02/2025)
SmolLM Carpus
FireWeb-Edu
SinPajama
338
38
4... es b 68e
10
10"
-34
Parametens
28
Irain Loke
Permmelo
32
tran m0
32
2.6
3.0
3.0
10
2.4
22
2.6
28
2.0
-24
24
10'
10'
16'
10"
10"
10
10"
10"
10
10"
10"
10" FLOPs
10"
FLOPss
FLOPs
FineWab
ProofFile
StarCoder
-24
20
338
1.8
m02
Paramoters
Parameters
1.6
Train
10"
3.0
10"
1.4
2.
1.6
1.0
10"
10"
16
16
10
10
16
10
10
10
10
10"
10
1
Figure 18: Contour plots for the curves fit with Equation (1) (the chinchilla version of the scaling law parameterization). Red line indicates the optimal model size. The star point is not used for fitting the curves.
Data
A
B
E
a
SmolLM Corpus
2.44e+03
6.92e+03
1.55
0.45
0.44
0.50
FineWeb-Edu
2.52c+03
7.16e+03 03
2.00
0.45
0.45
0.50
SlimPajama
2.05e+033
6.022+03
2.01
0.44
0.44
0.50
FibeWeb
1.64e++03
4.20e+003
2.15
0.43
0.42
0.50
ProofPile 2
3.77e++03
3.59c+03
1.33
0.51
0.43
0.46
StarCoder
7.75e+003
4.
0.86
0.55
0.44
0.45
Table 6: Parameters for the curves fit. with Equation (1) (the chinchilla version of the scaling law parameteri- zation). a a+3
B is the exponent of the optimal model size relative to FLOPs.
27

---

Published in Transactions on Machine Learning Research (02/2025)
D Hyperparameters
Here we list all the hyperparameters for our model training, We always report the final losses at the end of training and use the approximation that FLOPs 6ND compute flops from model size and data size. Each training run conducted using the OLMo (Groeneveld al., 2024) codebase with the listed hyperparamters. We construct grids of various model sizes for each FLOP budget, as shown in the previous section.
Table 7: Model parameters (Groeneveld et al., 2024; Wortsman et al., 2023; Zhao et al., 2024)
Parameter
Value
6-24 for small models, 40 for the 3.3B model
Number of heads
n
Head dimension
64
MLP hidden multiplier Depth
4
n
Context length
512
Activation
GeLU
Positional encoding
RoPE
Biases
False
Normalization
PyTorch Layernorm
QK normalization
True
Precision
Mixed, bfloat16
Tokenizer
Llama2
Table 8: Training parameters (Groeneveld et al., 2024; Wortsman et al., 2023; Zhao et al., 2024)
Parameter
Value
Optimizer
Adam
Batch size
1024
Learning rate
1e-3
Schedule
Linear warmup, cosine decay
Warmup steps
20% of total steps
y-loss coefficient
le-4
Weight decay
0.0
81
0.9
82
0.95
1c-15
E Full loss-to-loss parameter fits from Figure 1
Table 9: Train-to-train fits
Data
Data
K
Eg
E
Webe
Fine Web
1.00
1.01
1.97
2.17
FineWeb- Edu
Fine Web-Edn
1.00
1.00
1.97
1.97
FineWeb-
ProofPile 2
1.07
0.60
1.97
1.32
FineWeb-Edu
SlimPajama
0.97
1.05
1.97
1.97
FineWeb-
SmolLM Corpus
1.01
1.07
1.97
1.53
Fine
SeeeWebEdde
StarCoder
1.10
0.63
1.97
0.85
28

---

Published in Transactions on Machine Learning Research (02/2025)
Table 10: Test-to-test fits
Train data 0
Train data 1
6
E2jo
E211
FineWeb- Edu
FineWeb
1.05
0.98
2.12
2.08
Fine Edu
Fine WebeWeb-ddu
1.00
1.00
2.12
2.12
FineWeb-Edu
ProofPile 2
0.74
1.60
2.12
2.39
FineWeb-Bae
Slim BijmPajaaa
0.95
1.11
2.12
2.08
FineWeb-Fddu
SmolLM Corpus
0.99
1.01
2.12
2.10
FineWeb-Edn
StarCoder
0.74
1.64
2.12
2.48
Table 11: Train-to-test fits
Train data
Test data
K
K
Eijo
FineWeb-Edd
Hellaswag
1.08
0.93
1.97
2.12
FineWeb-Eie Edu
ARC-Fasy
0.36
4.68
1.97
0.07
FineWeb-BBe
MMLU-
0.96
1.14
1.97
2.79
FineWeb-Bie
MMLU-STEM
0.53
2.35
1.97
1.41
F Loss-to-loss procedure
Here summarize the procedure used to produce the loss-to-loss curves in the paper. We present this in the general form, assuming we want to predict L:(f,N.) given (ff.N.))
We independently fit. scaling laws to [a(, and La (fP.. on the full datasets using the met hod described in Appendix C. We only use these fits to extract the irreducible entropy terms Egy and Exje-
2. Then fit K, K. by linear regression in log space after shifting the data by Egg and Esje-
In Section 6 we also consider fits from smaller datasets where Egy is difficult to fit well given the limited sample size. In these cases, we instead let E be free parameter and fit all of K, Eng via non-lincar least squares optimization using scipy.
G Comment on theoretical implications
There is now a growing body of literature on the theory of loss scaling in large neural networks (see, e.g., Bahri et al. (2024); Lin et al. (2024); Sharma and Kaplan (2022); Maloney et al. (2022); Canatar et al. (2021); Dohmatob et al. (2024); Hutter (2021); Wei et al. (2022); Michaud et al. (2023); Jain et al. (2024); Bordelon et al. (2020); Atanasov et al. (2024); Nam et al. (2024): Bordclon al. (2024b); Paquette al. (2024) and references therein). For example, Lin et al. (2024) derives an expression for the loss scaling finite model size and dataset size in sketched linear model and single-pass SGD setting. Bahri et al. (2024) and Atanasov et al. (2024) considered similar problem in analogous student-teacher network setting, but in the asymptotic regimes where either the dataset size or model size was taken to infinity.
However, there is comparatively less theoretical work on understanding the effects of the data distribution the scaling laws, and on disentangling the two different types of scaling laws in Equation (1) and Equation (3). This is partially because in the asymptotic regime when N + 0O or D + oc, both forms given rise to the same scaling in the other variable and because empirically both result in "reasonable" fits to the data. Works like Lin al. (2024) derive bounds which include cross terms involving both and D, but remains unclear if these cross terms can be interpreted as those coming from the polynomial form of Equation (3).
In this work, we use the scaling law in Equation (3) since yiclds valid scaling law translations (though our results do not necessarily rule out other parametrizations). This leads ask if existing theoretical models prefer the functional form of Equation (3) versus, e.g., Equation (1). In this section, consider this
29

---

Published in Transactions on Machine Learning Research (02/2025)
question in simple linear model that has been considered in many previous works to theorize about scaling laws (Bordelon et. al., 2020; Maloney et al., 2022; Lin et al., 2024). Our goal here is not to derive novel result, but rather to show that a simplified version of the train-to-train (in-domain) loss transfer emerges in the existing theory, and that the scaling law is qualitatively described by an oquation that is roughly analogous to Equation (2). However, we defer analysis of the train-to-test and test-lo-test, setting for future work, which, to the best of our knowlodge, is not captured by any thcoretical model studied in the literature.
G.1 Generalized linear model
As in Canatar et al. (2021); Spigler et al. (2019); Cui et al. (2021); Maloney al. (2022), we consider data 2y that has M features whose covariance has spectrum that exhibits the empirically-moliiatee power-law behavior
A
1811 i: i: M.
(14)
It is straightforward to construct a features dataset that satisfies this property. For example, for any random orthogonal matrix O we can construct a datasct of dimension D-by-M by taking the covariance to be Σ = OAOT with A diag(A;), and sample D samples from E),
To avoid directly working in the large feature space, these features are projected down into a smaller set (this controls the extent which the learner can resolve the features). Mechanically, we also want to disentangle the size of the dataset D from the number of parameters of our model N. This can be achieved through linear map
M
(x)
Σ
Uni Fi,
a: a.l....,N N.
(15)
The weights are drawn from normal distribution Vai NN(0 o( The learned model is
f(x;)) - > (z)a
(16)
where are the model parameters, and for simplicity we have assumed a scalar output (i.e. a single label per sample). The labels are given by
M
y =
wp
(17)
where W/ ~ ~N(0,03)) We optimize the squared loss
[(0)= - 110=;))000;00---- v11²
(18)
Note that for simplicity we do not consider the ridge term (we will work far enough into the underparametrized regime N D, where the ridge term does not significantly contribute to the loss) and work in the limit of
zero label noise. The analytic solution for the optimal parameters 0 is straightforward to compute and given by (Maloney et al., 2022; Atanasov et al., 2024)
0 00 [
(19)
Since there exists an exact formula for the optimal parameters, this can be seen as effectively performing infinite passes on the training data.
Once we obtain set of optimal parameters we ovaluate the loss on large held-out validation set whose samples & are also drawn from (O, E):
(((0))
= B (1 r||²
2
(20)
2 Ea [+((; - sw T
=
30

---

Published in Transactions on Machine Learning Research (02/2025)
The number of features is larger than the number of parameters and dataset size M > N, D, such that the loss on the validation set decreases as the size of the train set is made larger. The expectation can be evaluated in closed form and is given by (Maloney et al., 2022)
of ( = L(N,
(
(INN/D)
A
(21)
2 1
where the quantity satisfics the trace equation
1 ur EAM + NZ)
(22)
In the eigenbasis, we can write this as
1= 1=Σ A A
(23)
Plugging in Equation (14) for our eigenvalue scaling, we therefore have
M
1
(24)
A
+N
When the spectrum is dense (M + oc) we can approximate this as?
de
1
3 2
N
1
2F
1,
(25)
A
BA
3 + 1
B+1
A
where 2F1 is the hypergoometric function. When N >> A we find that
2+1
cscl (Tx1) A (a) A 8+
(29)
I+ + 1)+
Plugging this back into our expression for the loss in Equation (21), we find that for any given eigenvalue scaling B and N < D D M,
9+1
ANN, M, D) ~ 0 N 2 N/D
8+1 CSC( 3+1
(30)
I++(
The comparison of this theoretical prediction of the loss as function of D to the numerical simulation can be in Figure 19 for different choices of the scaling exponent B. We see that the predictions get slightly worse for smaller values of 3. This is expected as the numerical simulations must be carried out with some finite but large value of M (1.2 x 10 in these plots). As B + 0, the approximation in Equation (25) requires a correspondingly larger value of M to correctly capture the tail behavior of Equation (23). We also compare the prediction Equation (30) to numerical data a function of N, for fixed values of D in Figure 20.
This result immediately implics:
Note that this approximation requires that M be much larger than other scale. In particular, when & is close to zero, the sum in Equation (24) converges very slowly, and is only approximated by the integral when M is sufficiently large. "The validity of this limit can be argued as follows: note that we break up the integral in Equation (25) into two regimes: one where the first term in the denominator dominates and one where the second term dominates. The transition point where this happens is at z = 2o where Agft 6+1 xe N, and so
de
de
dz
da A ++
(26)
A 9+1
20
Evaluating, we thus have
1 15 1+ 3 1 B N A x CN-8
N
(27)
and
(28)
where C= [(1 + 8+1
31

---

Published in Transactions on Machine Learning Research (02/2025)
N = 200
N = 300
109
10
10-2
B 0.3
8 0.4
theory
8= 0.7
empirical
8 1.0
10
loss
1000
1200
1400
1600
1800
2000
1000
1200
1400
1600
1800
2000
N 400
500
10
10-1
10-3
1000
1200
1400
1600
1800
2000
1000
1200
1400
1600
1800
2000
train dataset size D
Figure 19: Shows the validation loss plotted as function of train dataset size for different choices of the eigenvalue scaling 3. Each subplot is different choice of N, the number of model parameters. Solid line indicates numerical data while dashed line indicates theoretical prediction Equation (30). The mumerics were carried out with M 1.2 10 a, = ow and averaged over 2000 random scods.
The losses between any two distributions parametrized by eigenvalue scalings 1/ and 1/28"+1 for the same values of N, D, and M will be related to each other via L/L' = A (8) / (8") Note that this ratio is independent of D. We must therefore have that the log-losses on these two distributions will have slope when plotted against each other and intercepl log A A This somewhat. different from what we observe in the real datasets, where the slope can be data-dependent (see, e.g., the variation in across datascts in Table 9.) Nevertheless, the lincar model does show that the eigenvalue scaling constrains the behavior of the in-domain losses.
The dependence of the loss on N and D is not trivial, and does not optically resemble Equation (1) or Equation (3). However, can study it in different limits to connect it to the usual formulation of scaling laws. In particular, we can expand in the joint limit of N, D + oo with the ratio N/D << fixed. In this limit we find
L(N,M.D) R
1 N
DN + + O(N/D)) (8 + 11++T*+
8+1
cscl
8+1
(31)
2
B
We can see that the term in the parantheses includes a cross term between N and D. This cross term precisely the leading term would obtain if expanded scaling law of the form ( + B) at
32

---

Published in Transactions on Machine Learning Research (02/2025)
large D if A and = B This indicates that Equation (2) with a/B correctly describes the scaling of this model in the underparametrized regime, consistent with the result presented in Maloney et al. (2022).
Taken logether, these results suggest that this theoretical model captures some of the observed phenomena, but that some richer component of the real dataset setting still missing. In particular, we cannot establish similar result on train-to-test transfer, since the model manifestly does not capture any information out-of-distribution.
10
D=1300
D-2000
6 10
loss
4
3
theory
empirical
200
250
300
350
400
450
500
550
600
number of parameters N
Figure 20: Shows the validation loss as a function of the number of model parameters N, for fixed values of the train dataset size and / = Solid line indicates numerical data while dashed line indicates theoretical prediction Equation (30), with the same choice of hyperparameters as Figure 19.
33


Reviews:
[{"review_id": "sTa05bpQ3M", "reviewer": "Reviewer_D86T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper extends scaling law analysis by studying loss-to-loss prediction. The authors identify consistent loss-to-loss curves across a wide range of pre-training datasets and downstream tasks. This includes datasets with significantly different content (e.g., code vs. non-code). \n\n- By identifying and validating shifted power law relationships, the paper provides a model for predicting losses across distributions, which saves the computation for directly fitting the scaling law curves. \n\n### Weaknesses \n\n- It would be better to provide more details in the experimental setup, as this paper primarily focuses on empirical results. Although they may seem standard, they will help readers interpret results. For example, how are the scaling law curves and loss-to-loss curves fitted? How many models are trained? For a certain FLOP budget and model size, how are the experiments implemented? In addition, there are downstream datasets. What is the setup for the downstream evaluation or fine-tuning? \n\n- It would be better to provide more quantitative results in evaluating scaling laws and loss-to-loss curves. For example, for all the curves reported in the paper, it would be better to define an error metric between the curve and data points (how accurate are the curves)? How many points are used to fit the curves (sample complexity)? Why is the chosen loss-to-loss parameterized function better than other options (as mentioned in Section 4.1)? \n\n- It would be better to provide more analysis from the theoretical perspective. As the scaling laws are closely related to generalization theory, why is one function favored over other functions? Can the authors provide more intuition from the existing generalization theory literature?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper explores how scaling laws can be extended to account for changes in data distribution. This approach examines relationships between losses across datasets, aiming to predict one loss from another. The authors identify three main relationships: (1) Train-to-Train: Predicts training loss across two datasets paired by the same compute; (2)  Train-to-Test: Relates training loss on one dataset to test loss on another. (3) Test-to-Test: Links test losses across models trained on different datasets. These relationships are modeled using polynomial functions. \n- This paper provides extensive empirical results across diverse datasets, including those with drastically different content (e.g., code vs. non-code datasets) and eleven downstream tasks.\n- This paper provides applications for fitting such loss-to-loss curves: (1) Translating scaling laws can optimize predictions for new datasets, reducing the computations. (2) Loss-to-loss prediction could guide data mixing and designing efficient training strategies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be better to provide more abundant experimental setup details. Please see the weaknesses above. \n- For each curve, it would be better to provide how accurately the curve fits the underlying empirical data points. \n- In Section 4.2, are the models fine-tuned on the downstream datasets? \n- It would be better to provide algorithmic boxes for methods fitting each curve (in each section). It would help readers to understand the procedure of the paper. \n- In Section 5, it would be better to provide the actual computational cost of fitting each curve, for example, GPU hours and FLOPs.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kdv3NhkIAu", "reviewer": "Reviewer_4pEG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\n- The paper is well written, with clear notations, and motivations;\n- The proposed approach is well explained with thorough empirical validations;\n- The related work is comprehensive and covers the literature well;\n- The experiments are large-scale and the ablation studies are very complete;\n- I particularly appreciate the explanations of the authors regarding their methodological and implementation choices compared to the literature. In addition, they perform the additional experiments needed to justify their choices;\n- Finally, the detailed discussion on the limitations and future work helps the reader to properly assess the contributions of the paper and what it brings to the current literature. This is very much appreciated. \n\n**Weaknesses**\n\nOverall, I find the paper very good. I only have one small issue with the current structure that, in my opinion, can \"lose\" the reader a little bit as the benefits of the approach are not highlighted enough. I think that Sections 4 and 5 lack some transitions to better explain to the reader the benefits of the approach presented in Section 4. I would suggest presenting the generic loss-to-loss prediction of Eq. (9) at the beginning of Section 4 with some explanations of why it matters and then detailing the three cases before starting Section 5 with the practical use cases of the approach. I believe this would strengthen the usefulness of the approach for the reader. What do the authors think about this suggestion? I would appreciate it if the authors could clarify that with me in case I am missing something that would better explain the current organization.\n\nOverall, I think this paper has strong contributions and is very well written which greatly helps in understanding it. It tackles an important problem and tries to provide leverageable takeaways while also discussing its potential shortcomings, which is not done enough in my opinion in the ML field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "*Idea*:  The paper presents a method to predict one loss from another (train to test/ train to train/ train to test) and applies this method to predict losses across different pre-training datasets and between pre-training data and a given downstream test dataset. \n\n*Method*: The authors propose a formulation to translate scaling laws between datasets and connect it to a more generic scaling law formulation akin (but more general) to scaling laws in the literature. The authors detail the formulation for three cases: train to train losses, train to test losses and test to test losses, and show that there are simple shifted power law relationships between the different losses in all those cases. Finally, the authors discuss the implications of their approach to better understand scaling laws, transfer learning, and generalization to downstream tasks.\n\n*Experiments*: The authors perform large-scale experiments with 88 models to train in those three settings on 6 pre-training datasets and 11 downstream tasks to determine the coefficients of their scaling laws and they show that leveraging data from multiple pre-training datasets can yield better predictions about the loss when the model is trained on new datasets compared to fitting independent scaling laws.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No strong requested changes but a small suggestion presented in the weaknesses part.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Io1a6zmiB9", "reviewer": "Reviewer_xxrN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written and well motivated\n- The experimental setup is convincing (variety of datasets and downstream tasks).\n- All the hyper-parameters are listed. \n- Really appreciate the discussion section. Authors are really straightforward with the limitations of their methods. \n- The additional loss to accuracy analysis in the appendix is very valuable. \n- Good literature review. \n\nI think this is a solid paper, with evidences supporting the claims and that will be definitively be of interest to the TMLR audience.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on scaling laws. Instead of focusing on analyzing the scaling properties on a single training data distribution, the authors are analyzing what are the scaling-laws relationships between different training data distribution. They first present results on what they called loss-to-loss relations in which, the power-law relationship is analyzed between different training data distribution for a given compute budget. Then, they analyze the relationship train-to-test when the training dataset is different from the test dataset which is helpful to predict performance on downstream tasks. Lastly, they present test-to-test scaling law relationships where they compare the loss on the same test set while the model are trained on different datasets. The authors ran experiment across several NLP and code datasets. They are able to show that some scaling law properties translate from text data that does not contain code to code data which mean that there approach could generalize even in slight shift in the data distribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No changes requested on my side.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers unanimously agreed to accept this work and stated that it is novel and interesting. The authors have done a great job in experiments and writing.
Decision: Accept (poster)