A Hitchhiker's Guide to Scaling Law Estimation
Authors: Leshem Choshen
Yang Zhang
Jacob Andreas
Venue: ICML 2025 poster (2025)
Keywords: Scaling laws
recycling
efficiency
LLMs
Large Language Models
NLP
Natural Lagnuage Processing
Primary area: deep_learning->large_language_models
TL;DR: A guide to efficiently fit scaling laws, for example by using the whole loss trajectory

Abstract: Scaling laws predict the loss of a target machine learning model by extrapolating
from easier-to-train models with fewer parameters or smaller training sets. This
provides an efficient way for practitioners and researchers alike to compare pre-
training decisions involving optimizers, datasets, and model architectures. Despite
the widespread use of scaling laws to model the dynamics of language model
training, there has been little work on understanding how to best estimate and
interpret them. We collect (and release) a large-scale dataset containing losses and
downstream evaluations for 485 previously published pretrained models. We use
these to estimate more than 1000 scaling laws, then derive a set of best practices
for estimating scaling laws in new model families. We find that fitting scaling laws
to intermediate checkpoints of training runs (and not just their final losses) substan-
tially improves accuracy, and that—all else equal—estimates of performance are
generally most accurate when derived from other models of similar sizes. However,
because there is a significant degree of variability across model seeds, training
multiple small models is sometimes more useful than training a single large one.
Moreover, while different model families differ in scaling behavior, they are often
similar enough that a target model’s behavior can be predicted from a single model
with the same architecture, along with scaling parameter estimates derived from
other model families.

A Hitchhiker's Guide to Scaling Law Estimation
Leshem Choshen 123
Yang Zhang 23
Jacob Andreas
Abstract
L(t)
Scaling laws predict the loss of target language model by extrapolating from easier-to-train mod- els with fewer parameters or smaller training sets. This provides an efficient way for practi- tioners and researchers alike choose optimiz- ers, datasets, and model architectures. Despite the widespread use of scaling laws model the dynamics of language model training, there has been little work on understanding how to best estimate and interpret them. We collect (and re- lease) a large-scale dataset containing losses and downstream evaluations for 485 previously pub- lished pretrained models. We use these estimate more than 1,000 scaling laws, then derive set of best practices for estimating scaling laws in new model families. We find that fitting scaling laws to intermediate checkpoints of training runs (and not just their final losses) substantially improves accuracy, and that-all else equal estimates of performance generally most accurate when de- rived from other models of similar sizes. However, because there is significant degree of variabil- ity across model seeds, training multiple small models is sometimes more useful than training a single large one. Moreover, while different model families differ in scaling behavior, they are often similar enough that a target model's behavior be predicted from single model with the same chitecture, along with scaling parameter estimates derived from other model families.
Model Performance
ARE
L(t)
fmodel
Model #toks
Model #params
Figure 1: Illustration of scaled family, estimated scaling law, and its prediction error for target model.
models' architecture or training data. There is thus acute need for efficient decision-making aids that can evaluate new methods without full-scale training. A large body of work motivates or evaluates these changes using small models (Warstadt et al., 2023; Hu al., 2024a: Hillier et al., 2024), synthetic tasks (Akyürek et al., 2024; Wortsman et al., 2023) theory (Jelassi et al., 2024). But one of the most important tools for current practitioners is the estimation of scaling laws for LLMs (Ivgi et al., 2022; Dubey et al., 2024).
A scaling law extrapolates the performance of target model from the performance of a set of models with fewer pa- rameters smaller training sets. Typically, this requires models belong the same model family, differing only in parameter count and training size, but using the same ar- chitecture and training distribution. A high nigh-quality scaling law accurately predicts the target model's test performance (Rosenfeld et al.; Kaplan et al., 2020; Hoffmann et al., 2022). Most past work fixes model family and exhaustively trains models relate scale and performance through new scal- ing law. One question that has received comparatively little attention is how to create such laws in the first place.
1. Introduction
Substantial effort and cost are required to train even a single large language model (LLM), and even greater cost and effort are required to evaluate proposed changes to language
'MIT' MMMT-MMIBB IBM Watson Al Lab 'IBM Research, Correspon- mit.edu>.
dence Leshem Choshen
This paper offers a practical guide to when, how and which small models to use, to efficiently obtain meaningful predic- tions about large models' behavior maximizing prediction reliability while minimizing the budget for preliminary ex-
Proceedings of the ngnd International Conference on Machine I Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
See our repository for code, data, and experimental results.

---

A Hitchhiker's Guide to Scaling Law Estimation
perimentation, which necessarily involves tradeoffss between the number of preliminary models trained, the size of the largest preliminary model, and size of the dataset used to train it. To answer those questions one requires analysis across model families and scaling law procedures.
2. Defining scaling law
A scaling law estimates the loss of a costly model by training cheaper ones (see Fig. 1) typically with fewer parameters (#param)) and training tokens (#toks), A scaling law is a function that predicts target model's loss on held-out data when setting the value of one hyperparameter (Kaplan et al., 2020) or both (Rosenfeld et al.; Hoffmann et al., 2022). Comparing laws' predictions about different pretraining choices (e.g. data Ge et al., 2024) allows informed decisions about which large-scale model to train. A scaling law also enables finding the optimal choice of hyperparameters under computational constraints on pretraining (Hoffmann et al., 2022) inference (Touvron et al., 2023; Sardana al.).
We begin by collecting data to perform a large-scale meta- analysis of scaling laws ($3). Usually, scaling law research relies on single collection of closely related models, or alters only a minimal aspect of pretraining (e.g. data size; Muennighoff et al., 2024). We instead gather a large and diverse set of scaled families, to allow this and future meta- analysis of scaling laws that generalize across architectures, datasets and settings.
The rest of the paper uses this data to analyze a number of key questions around scaling law estimation:
Formally, we call a model f any single neural language model with a specific set of parameters. Different seeds, or even different checkpoints from the same training run, correspond to different models. We define a scaled model family F as set of models, with each f E F differing only in size #params(1 and number of tokens # tfoks(f). We note that change in size is usually applied in systematic manner that affects the number of attention heads, width, depth and such network characteristics that in all our data changes with it in a one-to-one mapping.
1. What reliability is achievable and expected from scal- ing laws? Variation between LLM initializations pro- duce unpredictable changes of up to 4% in loss. Most published controlled experiments on pretraining deci- sions, report changes between 4% and 50% ($4).
2. How much does the shape of scaling laws vary across model families? Different model families have scaling laws with different functional dependence model size. However, transformer LLMs are similar enough that, with a single model from target family and a scaling law from a different model family, it is some- times possible to accurately estimate target model per- formance. ($5).
Two subsets of scaled model families will be especially im- portant in our analysis. First, the maximal parameter fam- ily F F contains only models with the largest number of parameters. Formally, define m = fEF #params(]] then FMaxP = {f E F : #params(f) = m}. FmaxP will generally contain the target model(s) whose behavior we wish to predict E Frarget. Second, the q-maximal token family F#ttokig contains all models trained on at least a q-sized fraction of the training set. Formally, de- fine t = q (max)EF #toks(f)); then F#tok>4 {f e F : #toks(J) > 4). Note that this definition does not distinguish between partially trained models, and models trained to convergence on a subset of the training set. These two types of models generally differ, but as current theory does not predict in what manner and the former are cheap substitutes, we test empirically if those are similar enough to make good predictions in Section 6.
3. Must scaling laws be estimated only from fully trained models? Even though optimization procedures are typically sensitive to the full size of training run, (unprincipled) estimation of scaling laws from interme- diate training checkpoints greatly improves scaling law fit ($6). It is generally possible to estimate a model's final loss beginning roughly 1/3 of the way through train- ing.
4. How large must models be to produce reliable scaling laws? All else equal, experimenting with large models is typically more useful than with small models ($7), but may be outweighed by the benefits of reduced variance from training more, smaller models ($8).
A scaling law i.(f F) estimates the performance of a new model f given a model family F. (We will simply write i(s) when the family is clear from the context.) All experiments in this paper use the widely used functional form proposed by Hoffmann et al. (2022):
5. Taken together, cost-effective estimation of a scaling law should consider the number of models, the size of the models, and the number of training tokens for each model. We highlight those size, tokens and number of models effects in Fig. 2.
B
Our experiments also provide insight into the functional form of scaling laws themselves, suggesting that they may have fewer degrees of freedom than typically assumed ($9). We conclude with discussion of other work on scaling law estimation that may be of interest to practitioners ($10).
i()) = #
# #toks(
(I)
Here E captures the scaled family's 's general performance; A, a and B, describe the scaling effect of #params and # tooks respectively. The parameters are in an exponent to ensure positivity, i.e., more training data improves the
2

---

A Hitchhiker's Guide to Scaling Law Estimation
2016
-11
-50
1.35
mms modes poodiee
22
GPT3
0033)
24
20.0
0C1
Livee evei www.ereeeeee aodeted
5034
5:90
6011
1o00
Pythia
1-400 102 0021 022
1000
18
Pacanlage
(a) Scale up vs. Train Percentage
(b) #Models vs. Train Percentage
(c) #Models vs. Scale predicted
Figure 2: The effects of variables on scaling law accuracy. Each cell corresponds to a single scaling law estimated from a set of model checkpoints Firains with the color denoting the error when predicting the largest model. Each column shows a subset of the three axes along which these training sets differ: (1) the number of tokens used to train each LM in Ferain (expressed as a fraction the full training corpus), (2) the number of distinet models trained; and (3) the size of the largest model trained (expressed laws estimated from four models. In (c) all laws use the full corpus. Orange lines show iso-FLOP contours (sets of scaling laws whose training sets require the same computational cost to produce). * represent the most efficient ways obtain 15%, 10% and 5% ARE. One of the most immediate conclusions from these plots is that scaling law estimation is quite noisy-thh the inclusion of single badly-behaved model in the estimation procedure small model families етrоr does not reliably decrease with additional computation. However because of noise- it is often preferable to extrapolate from a large number small, partially trained models rather than a small number of large models.
scale-up factor the ratio between the target model and the largest model in Furain). In (a), all
produce large errors, and in
scaling. These parameters are estimated by first collect- ing set of training models Firaius then minimizing the reconstruction emror
3. Data for 1000+ scaling laws and more
As part of this work, we have collected and released the largest-scale public dataset describing scaling behavior across model families. This dataset aggregates informa- tion from large number of LLM training efforts that have released information about the behavior of multiple models of different sizes scales. While experiments in this paper focus scaling laws that measure loss, the dataset also includes information about model performance on down- stream evaluation benchmarks where available. We have focused on language models where the largest one is more than 3B parameters and where data was shared publicly or in private correspondence. Our repository accepts further contributions and requests for additions. In addition those, we have manually extracted some data from papers that did not release models but reported losses in figures.
Σ (L(f) Feaan
arg min B,
where L(F) denotes the empirical negative log-likelihood of some held-out data under the model f.
In this sense, scaling law is an ordinary parametric model, and we may ask many of the questions about L that we ask about LLMs themselves-wte collect? How estimate accuracy? However, to provide empirical answers to these questions, we first require data.
training data (Firain) to
We believe many of findings apply to other functional forms (Caballero et al.), and even suggest new ones in $9.
Other data sources. We want to highlight other sources for data on model results. Resources on the training check-
3

---

A Hitchhiker's Guide to Scaling Law Estimation
points and dynamics are scarce and perhaps the only other collection of such will be in the data-limited babyL.M mod- els of 2025 (Charpentier et al., 2025). There are some efforts to collect large sets of downstream evaluations for models that have been useful in other works. Those include DO which aims to collect all LLM evaluations (Habba et al., 2025), and data collected to create observational scaling laws (Maia Polo al., 2024; Ruan et al., 2024).
We additionally experimented with an L-BFGS- solver but found it to be less stable. It converged to similar results, but often did not converge, was slow and required multiple trials. Some past work has Huber loss to improve the ro- bustness of estimates; we repeat the analysis from the main paper with Huber loss in $E and find the same trends as in our main analysis. We only estimate scaling laws for families that contain at least three models.
3.1. Data sources
Evaluation To evaluate estimated scaling laws reliably, we need to account for loss fluctuations during large-scale model training. Thus, we test against a few checkpoints near the end of training: we choose as target models Fearget the 30%-maximal token family from the set Fotok>330% de- fined in the previous paragraph-thi that is, we take Frarget FP, 300% We then report the mean absolute relative error (ARE) Fe [L(f) i( Firain))/ (I.()) be- tween the empirical loss L and the loss É predicted by the scaling law.
For each model in this dataset, we collect any downstream evaluation and loss during training that was reported, as well as #toks for each, links matching checkpoints when available, links to data sources, and information about com- putational cost (in FLOPs) and number of training epochs (i.e. passes over the training set). Each model is identified by a unique name, a type (e.g. Ilama), # toks, #par ams, architecture type (e.g. encoder-decoder), and seed.
Models (families) in this dataset include Pythia (Biderman et al., 2023), OPT (Zhang al., 2022, collected thanks to Xia et al., 2023; Biderman et al., 2023), OLMO (Groeneveld et al., 2024), Amber (Liu et al., 2023). K2 (LLM360 Team, 2024), Mamba (Liu et al., 2023), RedPajamas(Together Modul mixture of experts (Shen et al., 2023), overtrained models (Gadre et al., 2024), Mamba, Llama and hybrid architecture variations from Poli et al. (2024), trans- former architectures (Alabdulmohsin et al., 2022), Bloom (Le Scao et al., 2023). T5-Pile (Sutawika et al., 2024), Pandey (2024) models, GPTT-fmill models with different data regimes (Muennighoff et al., 2024), Gopher (Hoffmann et al., 2022) and GPT-3 (Brown et al., 2020).
4. How well should scaling laws predict?
4% is the best ARE rypically obtained; ARE up to 20% can still distinguish between many modeling choices.
To establish how accurate a scaling law must be to be useful to practitioners, we first assess what changes in model ac- curacy have been considered meaningful in past work. We have surveyed experiments in the literature where an A/B test was performed, i.e., two models were trained similarly, manipulating one attribute to see how it affects scores. Em- pirically, we found no widely adopted modeling changes that were motivated with less than 4% relative difference between models. Additionally, reported variance across ran- dom restarts of the same model architecture reaches up to 3.5% (c.f.$$; Sellam et al., 2021). We take this to mean that this is approximately the minimal effect size experimenters care, as well the minimal effect that can be reliably mea- sured. Accordingly, this bounds the best goodness of fit we should expect or require of scaling laws.
The data consists of 1.9M steps of training evaluated on loss or perplexity, usually on multiple data sources belonging to 485 unique pretrained models, and more than 40 scaled families. We hope this will provide useful resource for the community and plan to extend it further as models get released and their training dynamics are shared. We see such a resource as a facilitator to more research on model development (e.g. A/B testing), scaling laws, downstream scaling laws (Gadre et al., 2024; Ruan et al., 2024; Owen, 2024; Isik et al., 2024), training dynamics (Choshen al., 2022) and more.
To offer several concrete points of comparison: Pythia 6.9B models fixed inconsistencies in their code and hence have two versions (c.f. App. B; Biderman et al., 2023) which differ in loss by 40%. They also provide a data deduplication A/B test that decreased loss by roughly 5%. Gadre et al. (2024) tested the effect of training 400M parameter models for different # toks. The smallest modification (doubling the number of training tokens) decreased loss by roughly 4%, while 30x more training tokens produced a 50% loss difference. Instead of varying the amount of data or epochs, Ge et al. (2024) found that training on a different kind of data incurred ARE of approximately 10% and different data mixes led to 6% changes or less.
3.2. Scaling law estimation
In the rest of the paper, we present findings from estimating hundreds of scaling laws follows:
Fitting For each model family F, we identify the maximal parameter family FaaxP, and estimate scaling law L us- ing the remaining models F Firain = F | F.axP. Estimation of scaling law parameters uses the curve_fit function in scikit-les (Pedregosa et al., 2011), with square loss.
4

---

A Hitchhiker's Guide to Scaling Law Estimation
1.0
Decoder
0.5
Encoder- Decoder
2
w -10
Tyypl
Jiee
-20
10
20
10
20
A
B
Scaled model family (discreet)
(a) Size parameters
(b) Token parameters
(c) Intercept
Figure 3: Parameters differ between scaled model families. Surprisingly, however, the pairs of parameters controlling the influence of model and training set size have similar ratios. The legend shows model architecture (left), scaling families (center) and per-family intercept (right).
5. When I train new model, do I even need a new scaling law?
less than 1% error estimating the accuracy of a 7B model from a collection of 1B model checkpoints. We find that predictions generalize, and constant #params scaling fac- tor is enough for most models (except the encoder-decoder TS-Pile). However, error rates are larger than in the source family, and predictions for larger models are worse (most conspicuous in OPT's error of %, 25% and 15% when extrapolating from 8.7B, 13B and 30B 175B).
Different model families exhibit different scaling behavior, but performance can sometimes be estimated using a single model in new family.
Scaling laws relate performance to scalar training parame- ters like model or dataset size. For discrete decisions (e.g. the choice of nonlinearity), it is unclear how to pool infor- mation across models that differ in these traits (see Ruan et al., 2024; Maia Polo et al., 2024, for concurrent work that performs this pooling based on downstream task behavior). Clearly, discrete choices affect loss of pretrained models with the same #pparas and # toks. But how do they affect the form of scaling laws?
5.1. Can train the target model a bit instead of many small models?
Yes, but obtaining reliable estimates in this way requires up 30% of the full training run.
The above results (last row of Fig. in App. A) also suggest the possibility of predicting losses not with just smaller mod- els, but with partially trained versions of the target model itself. When predicting inside the same #params family- that is, estimating i Fearget {(}}) the #aarams term in Equation (1) is constant, and extrapolation is only required for toks. As seen in the figures, this form of estimation is informative if permitted by computational con- straints. Beyond the immediate usefulness of this approach, it a promising avenue for future research on alternatives to scaling the number of layers.
One way to answer this question is to look at the param- eter estimates for scaling law parameters E, A, B and B differ across model families. These results are shown in Figure 3, where can be seen that there are often dramatic differences in all five parameters across families. In this sense, even the rate which additional data or parameters improve model performance depend underlying architec- tural details, suggesting that understanding the behavior of a new model family may require a new scaling law.
But another way to answer this is to ask how reliably we can predict final model accuracy when borrowing (or pooling) some parameters of scaling laws between families-ever even if these result in poor parameter estimates, they may predict large-scale model behavior within the range of meaningful differences identified in Section 4. To do so, we set the #params ams scaling parameters (A, a) to fixed values reported in past work, and estimate remaining parameters for indi- vidual model families. We take the variable values found by Muennighoff et al. (2024) (see Besiroglu et al., 2024; Porian al., 2024 for discussion of estimates from earlier work including Hoffmann et al., 2022). We find (see Fig. 6 in App. A) that in some cases only a single training run in new model family is necessary to obtain accurate scaling law predictions. In the OLMO family, for example, we obtain
5.2. Are even simpler baselines enough?
Some extrapolation is necessary: scaling laws can produce accurate estimates even when the target model vastly outper- forms any training model.
To provide another form of comparison for the predicted scaling laws, we compute two baselines. Both base- lines adopt a pessimistic evaluation assuming that the tar- get model is no better than the best model in the small model family used to estimate a scaling law. Specif- ically, the baselines are the best performance L(Ø I Firain) = min L(f) and the performance of the most-trained model, consuming the most compute for train-
5

---

A Hitchhiker's Guide to Scaling Law Estimation
30
#Irain modelss
#Train models
ATnis modeds
25
20
10 9
0.99
0.3 -Min. percentage
0.7
0.5
0.2
0.1
-in. pereanrage
-Minn percentige
(a) OPT
(b) GPT3
(c) Pythia
Figure 4: The effect of fitting on more of the training trajectory. Each cell represents the absolute relative error estimating scaling laws from a given number of models (vertical axis) trained on a subset of the final checkpoints from a training run (so scaling laws on the left are estimated using checkpoints, and on the right using only the final 10% of checkpoints). White cells failed fit. As long as the first 100% of checkpoints are discarded, final loss can often be predicted accurately.
ing, i.e. 1( Pirain) arg fe Fursi #paaram( (J) # (f) Those baselines might be the best one can pect without fitting law to scaling.
tokens and sizes depending the choice of learning rate schedule. But it is unknown whether these differences large enough to impact the estimation of scaling laws. We now test whether this finding extends to larger families-i.e whether including intermediate checkpoints from all models in a model family reduces ARE.
We find (See App. 5.2) that the best performance baseline is closer to L(Firgget), which is to be expected, as the target model performance is better than any other model in F and this is the better of the two. In both cases, even with the full F, the baselines suffer more than 15% error, mostly above 10%, almost never get below 5%, and 18% ARE on average across all scaled families we study.
Results are shown in Figure 4, which plots ARE for scaling laws estimated from data subsets of the form F for varying q. We find that including full training curves in scal- ing law estimation can predict losses well. In fact, relying merely on the end of training (left in Figure 4) produces significantly worse performance across the board. Our re- maining experiments thus fit scaling laws using all these intermediate checkpoints, and not final performance alone.
6. I have some data; what portions should I use?
Estimate scaling laws from intermediate checkpoints, not just fully trained models!
6.1. Should use all intermediate checkpoints?
Most past work on scaling behavior of language models (e.g., Gadre et al., 2024; Muennighoff et al., 2024) has trained a separate model for each value of #toks studied. This is based on the assumption that changes in the learning rate schedule, which depend on size of the full dataset that will be used for training, render losses from intermedi- checkpoints uninformative.
Almost all, but drop very early checkpoints.
In Fig. 4, plot ARE for different F F#tok> -maximal token families serving as F, i.e., when fitting only with the end of training runs. There is not a clear trend indicating whether we should use all data (as might be suggested by GPT-3 results alone) or only some of it. But is rarely the case that best estimates are obtained from the end of training alone.
However, some recent work has demonstrated the effective- ness of learning schedules that do not require prior access to the size of the training set (Hu et al., 2024b), and some work has questioned whether careful choice of the learning rate decay even necessary for reliable scaling laws (Porian al., 2024). Together, these findings motivate revisiting the assumption that only single useful datapoint may be ob- tained from each training run. In $5.1, observed the value of intermediate checkpoints when only single #params family is used to fit a scaling law. In general, there may be differences between models trained on the same number of
There is, however, distinctly uninformative phase the beginning of training, as can be seen in the loss curves (App. B) and noted in the literature (e.g., Chen et al.). We observe that this period is more likely to contain significant spikes or an increase loss (worse performance) despite additional training. We hence hypothesize this part should always be removed from data used to estimate scaling laws.
Indeed, our experiments depicted in Fig. 5 compare scaling law AREs with and without including models trained on less than 10B tokens in F. Evidently, the very beginning
6

---

A Hitchhiker's Guide to Scaling Law Estimation
of training (often not even reported in logs and graphs) is sometimes harmful the prediction. Specifically, we run the same experiments with and without ignoring the
-3
26
OPT
first 10B tokens seen. We find that for some models (e.g.,
OPT and Pythia) the ARE exceeds 15% even when using the whole data, but drops to 4-10% when ignoring those tokens. In preliminary experiments, we found that cutting fewer tokens gave noisier results, and cutting more had negligible effect.
#Train
15
-00
83
64
8.5
05
10
0.2.0.0. 0. 0.6
persentage
-35
7. How big a model should I train?
Pythia
*Trak model
25
-20
20
Larger models are better, but not necessary. Mainly, beware of specific models that might give noisy results.
10
10
In Figure 2 we compare scaling laws when controlling the amount, percentage, or size of the models (2 at a time). We find that choosing models closer in #param ram to the target model is generally effective (e.g.. Fig. 2a, 2c), but the effect is neither strong monotonic. For example, in all cases fitting on all F provides one of the lowest ARE. However, in GPT, Gopher and OPT, predicting with the smallest 4 models available is already enough to achieve less than 10% error. In Pythia, the smallest models are not predictive but the rest of the models provide similar fit. While relying on
9.1 0.0.2..3 0.4 G.5 0.8 1.0 peroentage
Peoorrage
(a) Cut 10B first tokens
(b) Fit all data
Figure 5: The effect of fitting with/without the beginning 10B tokens seen. Each cell represents the absolute relative error when estimating scaling law from a given number of models (vertical axis) trained on a given subset of check- points from the beginning of training (horizontal axis).
a larger model is beneficial, predicting many scales up (e.g., the behavior of x4 larger model in Pythia) is still reliable, especially if accounting for other factors we discuss next.
8. How many models for reliable predictions?
5 models is a safe bet, more would improve the results' robust- ness. These models can be small.
In fact, training additional, larger models before fitting a scaling law may sometimes decrease accuracy due to in- creased variance in large model performance- ample. Pythia 2.8B in Fig. 1. Unfortunately, it is difficult to identify whether seed is exceptionally high or low- performing without additional information. For example, cross-validation on F fails to detect it (see App. D).
We have seen that predicting with larger models and hence extrapolating less yields better results. However, given com- pute constraints (and additional hardware constraints like memory), practitioners may generally wish use smaller models when possible. Consider for example Fig. 2b where we compare fitting on 4 models but vary their size. We find that more models reduce ARE even without being big- ger models. As discussed in $7, adding a larger model to current scaled family serves two goals, increases the proximity to the predicted model, as well as increases the number of models seen.
for ex-
Instead, this instability can be addressed by accounting for seed variability. A wasteful way to do so would to train every model several times. better alternative is to diversify and train each model on differing hyperparameters as possible and to maximize the information gained (a common practice in efficiency coverrae scenarios. e.g., Perlitz et al., 2024). Hence, we suggest training more models of differing sizes each accounting for both size and seed changes, rather than training multiple seeds. We further discuss the effects of number of models (F $8.
We separate the contribution of size and number of models. In Fig. 2c, we predict with the largest model being held constant and add (at minimal cost) smaller models. We see again that larger models do benefit predictions. For exam- ple, the small models part (left) of the graph indicates large errors (bright). However, we also see again the unwanted effects a single model may have on the overall prediction. Consider for example the figure's 's diagonal in Pythia. Cells in a diagonal share group of models and each row adds another one to F. Evidently this specific group hurts results, even when larger models are added to F. With enough mod- els (bottom of diagonal), the negative decreases. Switching the model (next column) also removes the negative effect.
Given the choice of the largest model and the number of models, it is unclear how to optimally space the model sizes, whether linearly, log-scale, otherwise. We leave that optimization problem for future work.
7

---

A Hitchhiker's Guide to Scaling Law Estimation
Moreover, across all rows the tendency is never monotonic, implying larger models do not not ensure better predictions.
model training. Among the aspects studied are total train- ing costs including inference (Sardana et al.), effects of sophisticated data selection (Sorscher et al., 2022; Ge et al., 2024), training time (Inbar & Sernau, 2024), transfer of learned skills (Hernandez et al., 2021), behavior of models in other modalities (Mikami et al., 2022; Abnar et al.; Al- abdulmohsin et al., 2024; Hesslow et al., 2022) mixtures of experts (Ludziejewski et al.), data mixing (Ge et al.. 2024), downstream performance (Muennighoff et al., 2024), vocab- ulary size (Tao et al., 2024), and architecture comparisons (Tay et al., 2023; Poli et al., 2024) including small models (Muckatira et al., 2024) or other phenomena like finctuning (Zhang et al.) and the loss in different positions in the train- ing sequences (Xiong et al., 2024). Especially relevant to our context is Ruan et al. (2024); Maia Polo et al. (2024) that rely on multiple pretraining settings for creating scaling laws that generalize across models kinds of losses.
But in general, we see that increasing the number of mod- els tends to improve prediction. For example, in GPT3 the best predictions are with many models. Perhaps intu- itively, adding a larger model and improving both #params and number of models aspects improves quite consistently (Fig, 2b and diagonals of Fig. 2c).
9. Are all scaling law parameters crucial?
Scaling laws might have fewer degrees of freedom than de- scribed in the literature.
Assuming do not try account for aspects other than # toks and #pparams (see $10), one might wonder if some of the observed errors come from model misspecification an incorrect functional form for f., which (with a small number of exceptions including Caballero et al.) has gener- ally gone uncontested since it was first proposed (Rosenfeld et al.; Hoffmann et al., 2022). Here we specifically evaluate whether scaling laws empirically exhibit fewer degrees of freedom than has been proposed. First, we compute the principal components of the 5 learned parameters and find that 3 components explain 99.49% of the variance between the 5 parameters. Inspection reveals that two of these com- ponents tightly couple the pairs of parameters dealing with the same training parameter (#params ams and # toks). Plot- ting values of A against and of B against 3 (Fig. 3), we see clear linear relationship between these variables de- spite ther non-linear interaction in Eq. 1. There are few exceptions: the Encoder-Decoderr model TS-Pile shows a different behavior from the rest of the scaled families, and four additional scaled families show a different relationship between and B. In fact, all these families share the com- mon feature that they were trained using multiple passes over a single training set Gadre et al. (2024). The outlier point with 3= > 4 is 70m baseline of Pythia for a continual training intervention experiment (Biderman et al., 2023). Future work may consider different function forms tying some of the parameters or introducing other ones instead.
Another line of works that can be seen as scaling law discusses the relation between model width and hyperpa- rameter choices (rather than loss) (Yang et al., 2022; 2021; Blake et al., 2024; Lingle, 2024).
11. Limitations
Our primary metric, ARE, does not distinguish between over- or under-estimation of performance. When using scaling laws to choose between candidate models to train, these error estimates may be unnecessarily conservative (e.g. if both families' laws are biased in the same direction).
Another difficulty is aggregating information across model families. As most published families evaluate models of in- comparable scales, often over incomparable ranges, we were unable to produce an informative version of Figure 2 that aggregated information across all models available, and was thus able to give general recommendations about compute- optimal choice of preliminary experiments.
12. Discussion
This paper provides a first study of open questions in the estimation of scaling laws and their relation to large-scale pretraining decisions. We expect that many of these conclu- sions could be sharpened or extended with the availability of additional information about model training, and we call on other leaders of large-scale training efforts to share train- ing losses and evaluation results from multiple checkpoints during pretraining even in cases where model parameters themselves cannot be released.
Another function form change that future work should con- sider is accounting for the learning rate schedule, as our experiments assumed it was negligible, and previous works disregarded the training trajectory. A mismatch between the form and the real dependence might explain the inconsisten- cies in using the beginning of training. As noted in 1, the beginning is not fitting as well as later on.
Our findings leave open many important questions, from efficient predictions by fitting on many model families to scaling laws of the deltas between a/b test (e.g. on optimizer choice), and to other methods that efficiently compare archi-
10. Related work
This work builds on large number of recent studies re- lating scaling law estimation and decision-making about
8

---

A Hitchhiker's Guide to Scaling Law Estimation
tectures without relying on multiple models (e.g. continual learning). In addition, our results in $9 suggest other scaling law parameterizations might better fit data.
Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 2397-2430. PMLR, 23-29 Jul 2023. URL https::/proceedd/e mlr. wwwpppss//v22/bdderreeeeee
html.
Practical reccomendations
$4
Set an estimation goal and a budget.
Blake, C., Eichenberg, C., Dean, J., Balles, L.., Prince, L. Y., Deiseroth, B., Cruz-Salinas, A. F., Luschi, C., Weinbach, S., and Orr, D. u-mu p: The unit-scaled maximal update parametrization. arXiv preprint arXiv:2407.17465 2024.
$5.1
If budget allows, train the whole model for 30%.
SA
If extremely constrained, predict from one model.
$6
Use all training losses (except the beginning).
$7
Train as big possible, but limit tokens.
$7
Train more models, not just larger ones.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Impact Statement
This paper presents work whose goal is to advance LLM research. There are many potential societal consequences of our work, most of which lean towards the positive, environ- mental (efficient), open, and collaborative aspects. There- fore, it is hard to imagine specific issues with the progress allowed by this work.
J. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S.. Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020.
Caballero, E., Gupta, K., Rish, I., and Krueger, D. Bro- ken neural scaling laws. In The Eleventh International Conference on Learning Representations.
Acknowledgments
Charpentier, L.,, Choshen, L., Cotterell, R., Gul, M. O., Hu, M., Jumelet, J., Linzen, T., Liu, J., Mueller, A., Ross, C., Shah, R. S., Warstadt, A., Wilcox, E., and Williams, A. Babylm turns 3: Call for papers for the 2025 babylm workshop, 2025. URL htttps////aaattt org/ abs/ a 10645.
This work was funded by the MIT-IBM Watson AI Lab, and by a Sloan Research Fellowship to JA.
References
Abnar, S., Dehghani, M., Neyshabur, B., and Sedghi, H. Exploring the limits of large scale pre-training. In Inter- national Conference on Learning Representations.
Chen, A., Shwartz-Ziv, R., Cho, K., Leavitt, M. L., and Saphra, N. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms. In The Twelfth International Conference on Learning Represen- tations.
Akyürek, E., Wang, B., Kim, Y., and Andreas, J. In-context language learning: Arhitectures and algorithms. arXiv preprint arXiv:2401.12975 2024.
Choshen, L., Hacohen, G., Weinshall, D., and Abend, O. The grammar-learninn trajectories of neural language models. In Muresan, S., Nakov, P., and Villavicen- cio. A. (eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8281-8297, Dublin, Ire- land, May 2022. Association for Computational Lin- guistics. doi: 10. wttp:oggg////.e/oo acl- 568. https://aclanthhllee http:///aa.mmttooo...orgg///2222
Alabdulmohsin, I. M., Neyshabur, B., and Zhai, X. Revisit- ing neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300- 22312, 2022.
Alabdulmohsin, I. M., Zhai, X., Kolesnikov, A., and Beyer, L. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Process- ing Systems, 36, 2024.
URL
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark. A., Rao, A.. Zhang, A., Rodriguez, A., Gregerson, A. Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garriia-Olano D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova,
Besiroglu, T., Erdil, E.,, Barnett, M., and You, J. Chinchilla scaling: A replication attempt, 2024. URL https:// arxiv.org/abs/2444. 10102.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, I.., and Van Der Wal, O. Pythia: A suite for analyzing large language models across training and scaling. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International

---

A Hitchhiker's Guide to Scaling Law Estimation
E., Dinan, E.,, Smith, E. M., Radenovic, F., Zhang. F., Synnaeve, G., Lee, G., Anderson, G. L.., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H.. Touvron, H., Zarov, I., Ibarra. I. A., Kloumann, I, Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park. J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang. J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., AI- wala, K. V., Upasani, K., Plawiak, K., Li, K. Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., R-Raalaanaaaaan L., van der Maaten, L.,. Chen, L., Tan, L., Jenkins, L., Martin, L.,, Madaan, L., Malo, L.,, Blecher, L.., Landzaat, L.., de Oliveira, L., Muzzi, M., Pa- supuleti, M., Singh, M., Paluri, M., Kardas, M. Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P.. Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P.. He, Q., Dong, Q., Srinivasan, R., Ganapa- thy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R.. Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie. S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T.. Speckbacher, T., Mihaylov, T. Xiao, T. Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do. V. Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang, X., Gold- schlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y. Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z.. Singh, A., Grattafiori, A.. Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A. Menon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Lcon- hardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C.,, Burton, C., Mejia, C., Wang, C., Kim, C., Zhou, C., Hu, C. Chu. C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E.,, Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute, E., Dunbar,
E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Ozgenel, F,, Caggioni, F., Guzmán, F., Kanayer, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern G., Thattai, G. Herman, G., Sizov, G., Guangyi, Zhang, Lak- shminarayanan, G., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Molybog, I. Tufanov, I., Veliche, I.-E., Gat, I. Weissman, J.. Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J.. Tang. J., Chan, J., Zhen, J.. Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang. J., Cummings, J.. Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Prasad, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakho- tia, K., Huang, K., Chen, L., Garg. L., A, L.. Silva, L., Bell, L.,, Zhang, L... Guo, L.,, Yu, L., Moshkovich, L.. Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Tsim- poukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vy- atskov, M., Samvelyan, M., Clark, M., Macey, M., Wang. M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Sing- hal, N., Egebo, N., Usumier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng. N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj. P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li. R., Hogan, R., Battey, R., Wang. R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S.,, Chugh, S. Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S. Yamamoto, S., Ramaswamy, S.. Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chin- tala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Cho, S., Virk. S., Subrama- nian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang. X., Wang. X., Wu, X., Wang, X., Xia, X., Wu, X.. Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Hao, Y., Qian, Y., He, Y., Rait, Z., Vito, Z.,, Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, Z. The llama 3 herd of models, 2024. URL https: //arxiv....iv/////a org/ hb/p/407. 2407.21783.
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint
10

---

A Hitchhiker's Guide to Scaling Law Estimation
arXiv:2403.08540, 2024.
Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vas- silvitskii, S., and Koyejo, S. Scaling laws for down- stream task performance of large language models. arXiv preprint arXiv:2402.04177, 2024.
Ge, C., Ma, Z., Chen, D., Li, Y., and Ding, B. Data mix- ing made efficient: A bivariate scaling law for language model pretraining, 2024. URL https://arxiv. org/ abs/2405. 4908.
Ivgi, M., Carmon, Y., and Berant, J. Scaling laws under the microscope: Predicting transformer perfor- mance from small scale experiments. arXiv preprint arXiv:2202.06387. 2022.
Groeneveld, D., Beltagy, I,, Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu. Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V. Ravichan- der, A., Schwenk, D., Shah, S., Smith, W. Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge. J.. Lo, K., Sol- daini, L., Smith, N. A., and Hajishirzi, H. Olmo: Ac- celerating the science of language models, 2024. URL https://arxiv> herrg/bbs/444g 00838.
Jelassi, S., Brandfonbrener, D., Kakade, S. M., and Malach, E. Repeat after me: Transformers are bet- ter than state space models at copying. arXiv preprint 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint Stti:2000108836/ 2020.
Habba, E., Arviv, O., Itzhak, I., Perlitz, Y., Bandel, E., Choshen, L.,, Shmueli-Scheues. M., and Stanovsky, G. Dove: A large-scale multi diteesiinmnii predictions dataset towards meaningful Ilm evaluation, 2025. URL https://art /avv://///aiiee.eooeoooaeoe org/ aat/a/b//225.. 01622.
Le Scao, T., Fan, A., Akiki, C., Pavlick, E., Iliè, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176h-parameter open-access multilingual language model. 2023.
Lingle, L. A large-scale exploration of mu-transfer. arXiv preprint arXiv:2404.05778, 2024.
Hernandez, D., Kaplan, J., Henighan, T., and McCan- dlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
Liu, Z., Qiao, A., Neiswanger, W., Wang, H., Tan, B., Tao,
Hesslow, D., Zanichelli, N., Notin, P., Poli, I,, and Marks, D. Rita: a study on scaling up generative protein sequence models. arXiv preprint arXiv:2205.055888 2022.
T., Li, J., Wang. Y., Sun, S., Pangarkar, O., Fan, R., Gu, Y., Miller, V., Zhuang, Y., He. G., Li, H., Koto, F., Tang. L., Ranjan, N., Shen, Z., Ren, X., Iriondo, R., Mu, C., Hu, Z., Schulze, M., Nakov, P., Baldwin, T., and Xing,
Hillier, D., Guertler, L., Tan, C., Agrawal, P., Chen, R. and Cheng, B. Super tiny language models. ArXiv, abs/2405.14153 2024. URL https://api. semanticscholar orgg///rrpuSID::266222222
E. P. Llm360: Towards fully transparent open-source Ilms, 2023.
LLM360 Team. Llm360 k2-65b: Scaling up fully transpar- ent open-source 1lms. 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L.., Hendricks, L. A., Welbl, J.. Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Ludziejewski. J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygóádž, T., Sankowski, et Scaling laws for fine- grained mixture of experts. In Forty-first International Conference on Machine Learning.
Hu, M. Y., Mueller, A., Ross, C., Williams, A., Linzen, T., Zhuang, C., Cotterell, R., Choshen, L., Warstadt, A., and Wilcox, E. G. Findings of the second babylm challenge: Sample-efficient pretraining on developmentally plausi- ble corpora, arXiv preprint 2rX 2024a.
Maia Polo, F., Somerstep, S., Choshen, L., Sun, Y., and Yurochkin, M. Sloth: scaling laws for I1m skills to predict multi-benchhark performance across families. arXiv preprint arXiv:2410.1 11840, 2024.
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long. X., Zheng, Z.,, Fang, Y., Huang, Y., Zhao, W., et al. Minicpm: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint arXiv:2404.N0 06635, 2024b.
Mikami, H., Fukumizu, K., Murai, S., Suzuki, S., Kikuchi, Y., Suzuki, T., Maeda, S.-i., and Hayashi, K. A scaling law for syn2real transfer: How much is your pre-training effective? In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 477-492. Springer, 2022.
Inbar, and Sernau, L. Time matters: Scaling laws for any budget. arXiv preprint arXiv:2406. 18922, 2024.

---

A Hitchhiker's Guide to Scaling Law Estimation
Muckatira, S., Deshpande, V., Lialin, V., and Rumshisky, A. Emergent abilities in reduced-scale generative language models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguis- fics: NAACI. 2024, pp. 1242-1257, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1e..1166611222444 10.11666//1//2224fnonnn------ hiwpoo244.iinnnnn--aaaaa
Sardana, N., Portes, J., Doubov, S., and Frankle, J. Beyond chinchilla-optimal! Accounting for inference in language model scaling laws. In Forty-first International Confer- ence on Machine Learning.
Sellam, T., Yadlowsky, S., Tenney, I., Wei, J., Saphra, N., D' Amour, A., Linzen, T., Bastings, J., Ture, I. R., Eisen- stein, J., et al. The multiberts: Bert reproductions for ro- bustness analysis. In International Conference on Learn- ing Representations, 2021.
URL https: //aclanthology org/ wrgg/024, findings-naacl. 79.
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling cont-connt language models. Advances in Neural Information Processing Systems, 36, 2024.
Shen, Y., Zhang. Z.,, Cao, T., Tan, S., Chen, 7.,, and Gan, C. Moduleformer: Learning modular large language models from uncurated data. arXiv preprint Stfi:2300.046444 2023.
Owen, D. How predictable is language model benchmark performance? arXiv preprint 2rX4v2240.004757, 2024.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Mor- cos, A. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536, 2022.
Pandey, R. gzip predicts data-dependent scaling laws. arXiv preprint ar arviv:2405. 16684, 2024.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830) 2011.
Sutawika, L., Komatsuzaki, A., and Raffel, C. Pile-t5, 2024. URL https://blog. w.w.ther.aa/pileee.... ai/pile-t5/. Blog post.
Perlitz, Y., Bandel, E., Gera, A., Arviv, O., Ein-Dor, L., Shnarch, E., Slonim, N., Shmueli-Scheuer, M., and Choshen, L. Efficient benchmarking (of language mod- els). In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pp. 2519-2536, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL https: //aclanthology.o hwwp:/ooo.ooorg/////e www.///.....2..../ naa.llllcll.nn.. 139.
Tao, C.,, Liu, Q., Dou, L., Muennighoff, N., Wan, Z., Luo, P., Lin, M., and Wong, N. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407. 13623, 2024.
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., and Met- zler, D. Scaling laws vs model architectures: How does inductive bias influence scaling? In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Associa- tion for Computational Linguisties: EMNLP 2023, pp. 12342-12364, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.1133//112002 findings-emnlp.822 URL https://aclanthology org/2023. findings-emnlp. 825.
doi: 10. 10.186633w1/20244maaczl
Poli, M., Thomas, A. W., Nguyen, E., Ponnusamy, P., Deis- eroth, B., Kersting, K., Suzuki. T., Hie, B., Ermon, S., Ré, C., Zhang, C., and Massaroli, S. Mechanistic de- sign and scaling of hybrid architectures, 2024. URL https:/ https://rrxiz http://aaxvvvv.oabb.bb.ss/
org/abs/2405/ 17844.
Together. Releasing 3b and 7b redpajama-incite family of models including base, instruction-tumed chat models, May 2023. URL https: htt.://wwwttttteerrre www.:/tttotteeee.bbbllloee ai/blog/ redpajamas mod-alels-m
Porian, T,, Wortsman, M., Jitsev. J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute- optimal scaling of language models. arXiv preprint 0rXiv:2406.19144 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Rosenfeld, S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. In International Conference Learning Representations.
Ruan, Y., Maddison, J., and Hashimoto, T. Observational scaling laws and the predictability of language model performance, 2024.
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T. al. Findings of the babylm challenge:
12

---

A Hitchhiker's Guide to Scaling Law Estimation
Sample-efficient pretraining on developmentally plau- sible corpora. In Proceedings of the BabyLM Chal- lenge at the 27th Conference ral Language Learning pp. 1-34, 2023. URL https: / //aclanthology. org/ org/2023.coo wwtg/222..conl-000 www.oooabllllmlll.. 1/.
full training. Scaling is constant and follows the literature (Muennighoff et al., 2024) and the largest model stands as target model (so the bottom line in each figure represents predicting from the beginning of training). In parallel to this paper, an even more efficient work on predicting with I model was suggested, and the two should be incorporated (Maia Polo et al., 2024).
Computational Natu-
Wortsman, M., Liu, P. J., Xiao, L.. Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., et al. Small-scale proxies for large-scale trans- former training instabilities. In The Twelfth International Conference on Learning Representations, 2023.
B. Loss curves and predictions
We provide in Fig. 7 graphs of the loss during training of the target models per originating source (e.g., paper) together with the predictions by using different percentage of the training.
Xia, M., Artetxe, M., Zhou, C.,, Lin, X. V., Pasunuru, R., Chen, D., Zeitlemoyer, L., and Stoyanov, V. Training trajectories of language models across scales. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1371 1-3333, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10218653//1/2023. acl-long.700 767. URL acl-long. 767.
C. Is scaling working only upwards?
No. Small models usually show consistent and predicatable performance.
Usually, one does not use a scaling law to extrapolate to a smaller model as one can just train the small model. How- ever, under observational scaling laws, where one wants to research a phenomenon without scaling at all (Ruan et al., 2024; Maia Polo et al., 2024), or when many models were trained and one wishes to create smaller models for various reasons (Hillier et al., 2024; Warstadt al., 2023), scaling down might prove useful. Moreover, in the context of tra- ditional scaling laws this may act as a baseline. Such an experiment may shed another light on the number of models FF versus their size #parame ams. If large models are better because they are more stable or otherwise fit laws more ro- bustly, few models will be enough, if the number of models scale down difference from the prediction, it will show similar behaviour scaling up. See more in $8.
Xiong, Y., Chen, X., Ye, X., Chen, H., Lin, Z., Lian, H., Niu, J., and Ding. G. Temporal scaling law for large language models. 2024. URL sttpps:/aap1 http:::://apssssnnniicsshhee semanticscholar. org/Corpus ID: 110/0rpps11:2284499884 269449894.
Yang, G., Hu, E., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tuning large neural networks via zero-shot hyperparameter trans- fer. Advances in Neural Information Processing Systems, 044:178441199 17097, 2021.
Yang, G. Hu, E. J., Babuschkin, I. Sidor, S., Liu, X., Farhi, D., Ryder, N.. Pachocki, J., Chen, W., and Gao, J. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.00 03466, 2022.
To test this we reverse the order of models and predict with the largest models the loss on the smallest models. This means that for example in the case of models, we predict the smallest model's loss and fit the scaling law relying on the largest models. As before, we break the results by the percentage of training done and do not reverse it.
Zhang, B., Liu, Z. Cherry, C., and Firat, O. When scaling meets Ilm finetuning: The effect of data, model and fine- tuning method. In The Twelfth International Conference on Learning Representations.
As shown in Fig. 8, the number of models plays an important role in fitting well and a minimum of 30-40% of the training is necessary for good fit, more than that often improves further.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mi- haylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. Opt: Open pre-trained transformer language models, 2022. URL https::////teeeee orgg hatp://bs/22205/ 01068.
D. Can we detect bad models to fit on?
If so, not through cross validataion.
A. Scale up with 1 model
In $7, we raise the issue of instability of scaling law predic- tions, with single model vastly changing the results. We tried to see if, without knowing the ARE, we could remove
We bring errors of data from fitting from a single model on a given percentage of training to the largest model with
13

---

A Hitchhiker's Guide to Scaling Law Estimation
35
35
0.12
30
2.8
30
1.3
Train model sizze
25
Train model size
25
6.7
20
13.0
15
30,0
10
175.0
Percentage
Perventage
(a) OPT
(b) GPT2 (trained on C4)
35
1.0 0.41 0.160007
30
1.0
Tran model sizze
25
Train model size
20
20
15
15
2.8
7.0
10
10
12.0 6.9
-
5
0.1 0.2 0.3
0.6 Percentage
0.7 0.8
0.1
0.2 0.3
0.4
0.6 Perventage
0.5
0.7
0.9
1.0
OLMO
(d) Pythia deduped VO
35
0.02
16
0.07
30
0.16
size
25
Train model
1.4
1.0
1.3
15
2.8
1.4
10
6.9
2.8
-5
6.9
12.0
12.0
0
0.1 0.2 0.3 0.4 0.5 0.6 0.7
0.8 0.9
1.0
0.1 0.2 0.3
0.4 0.5
0.6 Percentage
0.7
0.8
0.9
1.0
Percentage
(e) Pythia VO
(f) Pythia deduped
33
33
0.25
30
0.41
Train model size
0.78
25
Truin modee
20
2.8
2,85
15
10
6.9
11.14
-
12.0
0.3 0.4
0.5
0.9
1.0
0.1 0.2 0.3 0.4 0.6 0.7 0.8 0.9 Percentage
Percentage
(g) Pythia
(h) TS-Pile
Figure 6: Fitting scaling laws under the assumption that all models scale similarly. Thus, a single model needed predict. The last row in each Figure represents predicting a model the beginning of its training.
14

---

A Hitchhiker's Guide to Scaling Law Estimation
44
2.5g
2.3
Volccn
Tikezeoce
(a) Overtrain
(b) Datablations
Laprrapt 2R0 14160
32
C cntrrro CIS 230
ge tha ths Shcettet
2.
29
31
3.
244
2.43
2.25
245
15335
J.31
To
(c) GPT-3
(d) Pythia
origall pip
algad papertakiin Eajocacia
Exiliig, anjocttiis C4154
24
3.9
1.07
14
1.53
gar 4 pop
19
(e) TS-Pile
(l) Training Trajectories
Figure 7: In each figure all losses from a specific source and predictions of the scaling loss with different percentage of the #toks and all models. Predictions are points where the X axis is the available data for prediction and the prediction value, lines are the actual value. One scaled family per paper was sampled as a representative.
bad models from the prediction. We hypothesized that mod- els that we can't predict would mean models that would skew our predictions when fitted upon. We performed a cross-validation on the #params families in F each time setting the models with most tokk as target ans exclusing the #params family from F. Our hypothesis was found to be incorrect. Such cases of hard-to-predict models were found to indicate that the models left in F' are bad predictors and not that the target is very dissimilar (a "bad" training). In 58% of the cases removing that model from the scaling law created the worst ARE possible the actual target, more than removing any other model.
E. Huber replication
Huber loss is sometimes used instead of ARE (Hoffmann et al., 2022). Huber loss is defined as
1 ' Lo(a)= = 8. ((a) 18)
for |a| S 5, otherwise.
2
We use ò = e10-3 3 as done in (Hoffimann et al., 2022). The overall results are similar but for completeness report them in Fig. 9.
15

---

A Hitchhiker's Guide to Scaling Law Estimation
30
25
#1 models
20
15
10
0.1
0.3
0.4
0.5
0.6
0.9
1.0
percentage
Percentage
(a) OPT
(h) Pythia deduped VO
35
33
#Tain models
#Tran models
5
20
20
5
10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Percentage
Percentage
(c) Pythia VO
(d) Pythia deduped
30
30
iTaain models
#Tmin models
15
15
5
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Percentage
Percentage
(c) Pythia
(f) TS-Pilc
Figure 8: Fitting scaling laws trying predict the smallest model, with the largest (Y-axis) models trained on a percentage of the data (X-axis).
16

---

A Hitchhiker's Guide to Scaling Law Estimation
1.0
(X500)
#Tnin models (Sode up predicted)
(X171)
(X230)
(X75)
0.8
models up
(K14)
(X134)
(X64)
(X12)
(X26)
(X9)
(X13)
(<30)
0.1 0.2 0.3
944 0.5 0.6
0.7 0.8 099 1.0
0.1 02 0.3
04
0.5 0.5
0.7
0.K 09
1.0
Percentage
Poreenlage
GPT3
(b) Pythia
#Tain models (Seak up prodicted)
"TTain models (Scale up prodetea)
1.0
(X44)
0.8
0.6
16
(X5)
0.2 0.3
0.4
0.5
0.7 Ponennaage
0.6
0.8
0.9
1.0
0.3 0.2 0.3
0.4 0.5 0.5 0.7
0.8
0.9
1.0
Percentage
GPT2
Llama
Figure 9: Scaling laws under Huber loss. The line represents most efficient setting to recieve <0.05. Comparison of several models given different amount of models and percentages of training.
17


Reviews:
[{"review_id": "2l9uGVPqFB", "reviewer": "Reviewer_Hw8m", "summary": "The paper is a meta-analysis of scaling law fitting, focusing on a popular power law relating the pretraining loss to the model size and number of training tokens. The paper offers guidelines for how accurate these scaling laws can and should be and how to best estimate them given available resources.\n\n## Update after rebuttal\n\nThe authors have addressed my primary concern, and I have raised my score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the deep read, care for details and supportive stance.\n\nThe main concern raised by the review had to do with the estimation of scaling law parameters (and why some were reported as negative).  As discussed below, this was (fortunately) a mistake in our presentation rather than the estimation procedure itself.\n\n**Estimation of scaling parameters:**\nGreat catch! Indeed, revisiting the graph, the caption does not mention that we have the positivity constraint. Following past work, we fit (and plot) log E, not E, to ensure positivity. We will clarify this in our revised version of the paper. \n\nSimilarly, we will clarify that when we fit with L-BFGS (which we found to be less stable as stated in “Fit” Section), we search the parameters using the code from Muennighoff (2023) (which is improved over Hoffman).\n\nWe also discuss other mentioned comments.\n   \n**Parametric form has limitations:** \nAs the reviewer states, “(1) is popular in the literature”; and as we state in the paper, this is our reason to build upon it for this meta-study. Based on preliminary experiments,  expect that many of our findings would be similar in other functional forms. \n\nNote that, while previous work focuses on identifying new functional forms for scaling laws, we here are analyzing how to fit these laws by evaluating generalization on held-out models. Consequently, our results report the extent to which the scaling laws generalized under different settings and allow others to know what to expect. \n\nBesiroglu (2024) notes issues with the fitting reported in the original paper, but those are related to rounding and other issues that are not relevant to our case, especially not with the repetitions across settings, that do not share the same issues.\n\n**Pretraining loss vs. downstream task:**\nYou are correct that various other kinds of quantities can be predicted with scaling lows. We believe this would be an excellent topic for a follow-up paper! \n\n**Comparisons to observational scaling law papers:**\n\nWe are happy to add more discussion of the relationship of this work to those! We value their contribution to the field (as can be seen by citing them in various contexts). We wish to emphasize that our papers have different goals: Ruan focuses on predicting scaling across datasets  and Polo proposes a new kind of scaling laws that fits across multiple models together; both given fixed data for estimating a scaling law. Here we’re focused specifically on identifying *what data to collect*---a question that could also be asked about observational laws in future work. Our approach does not provide a new scaling law, but discusses fitting scaling laws, when creating new models, efficient use of the information available etc. Possibly, information from previous models can aid there as well (but it is unclear, as the distinctions in A/B tests might be smaller than the ones between full models that they focus on).\n\nModel size:\nThank you, we will clarify based on your suggestions.\n\nTitle:\nThanks, that’s reassuring!", "claims_and_evidence": "The paper adequately supports its claim with evidence. ~However, I am concerned that a flaw in the analysis (described below under “Experimental Designs Or Analyses”) might invalidate the evidence.~", "methods_and_evaluation_criteria": "The parametric form (1) at the center of the paper has limitations. In addition to potentially being incorrectly or over-parameterized as discussed in Sec. 9 of the paper, eq. (1) is also difficult to fit - As discussed in Besiroglu et al. (2024), Hoffmann et al.’’s fit for (1) had multiple issues, leading to incorrect prediction of compute optimal scaling. The two other approaches studied in Hoffman et al. sidestepped the need to assume (1) holds and instead directly predicted the compute optimal scaling, leading to consistent results validated by subsequent work. \n\nMore broadly, pretraining loss is arguably not the most important thing to predict using scaling - predicting optimal design choices such as model size, number of experts, and dataset composition as a function of compute budget is operationally much more meaningful than predicting the resulting loss of the model. Consequently, it would have been more interesting to study how the accuracy of those predictions varies with the specifics of the training procedure. \n\nHowever, I acknowledge that the parameter form (1) is popular in the literature, and therefore I do not consider focusing on it a critical flaw in the paper.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "~I have concerns about the method used to fit the central power law in this paper. In particular, it appears to miss two crucial components used in most prior work:~\n1. ~A positivity constraint on $A,B$ and $E$, implemented in prior work by fitting the log of the scaling exponents and coefficients. Alarmingly, Figure 3 shows $E$ fitted to negative values for some model families!~\n2. ~A grid search for the initialization to L-BFGS is important for preventing convergence to suboptimal local minima. The authors make no indication of using this technique in their scaling law fit.~\n\n (Addressed by author rebuttal)", "relation_to_broader_scientific_literature": "The paper revisits the fitting of the power law (1), but unlike prior work it uses a large and diverse body of scaling experiment results, and experiments with degrees of freedom like using intermediate checkpoints from training, and transferring scaling laws between model families, which to my knowledge were not explicitly explored in prior work.", "essential_references_not_discussed": "I am missing a more detailed comparison between the dataset of experiments results compiled in this paper and the ones collected for recent meta-analyses such as Ruan et al. (2024) and Maia Polo et al. (2024). Such comparison would be valuable for future practitioners trying to decide which compilation to use in their studies.", "other_strengths_and_weaknesses": "Strength: The publicly released compilation of scaling experiments results collected as part of this paper could be useful for subsequent work. However, most of the files in the anonymous repository linked to the papers were not available, which preventing me from gauging the usability of the dataset.", "other_comments_or_suggestions": "The definition of model family in line 74 (right) is not clear. What does it mean for models to “differ only in size?” Even for standard Transformer models, models of different sizes necessarily differ by other hyperparameters such as depth, d_model, attention heads, etc. I think what you meant is that a family is a mapping from (model size, num tokens) to model checkpoints.\n\nAlso, I really liked the title of the paper.", "questions_for_authors": "~My main question is whether the fitting of the scaling law is valid, given the concerns I raised under “Experimental Designs Or Analyses” above.~", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBQAcXhuJ7", "reviewer": "Reviewer_43ps", "summary": "This paper provides a comprehensive analysis of scaling laws in large language model (LLM) training, focusing on how to estimate and interpret scaling laws effectively. The authors construct and release a large-scale dataset containing training losses and evaluations from several pre-trained models, enabling them to derive over 1000 scaling laws. The study presents best practices for cost-efficient scaling law estimation and discusses trade-offs between the number of preliminary models, their sizes, and the dataset used.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and the accurate points made are encouraging that some deep points about scaling laws indeed pass to readers. \n\nRegarding the two questions:\n1. How well do these findings generalize to architectures beyond transformers?\nOur experiments focused on two classes of Transformer models (encoder–decoder and decoder-only). Scaling laws for mixtures of experts seem to work relatively well (while not reported in the paper, we do have MoEs included in the scaling law dataset that we will release). We also tested some small Mamba models; they were noisy as they didn’t have as many checkpoints and sizes, but they generally seemed to behave in the same way (in the sense that fitting laws behaved similarly). However, those are only anecdotes and not a robust study.\n1. What are the computational savings when using this method vs. naive scaling?\nAs we provide a guide, it depends on individual time, cost, and hardware constraints on training. For example, if you just train on the beginning of your largest model, a 3x speedup is achievable but requires the same hardware as the large model (see e.g. Fig. 6). If you train on many small models then it depends on how well you need your estimation to be, look for example for the orange lines in the main figure (Fig. 2), they highlight places with approximately the same amount of compute, so you can see for example that often you save more than any amount of training on the large model (so more than 10X), usually, much more. Note that an additional benefit from running smaller models is the hardware, one can use smaller GPUs, less memory and other hardware that is easier to come by, in addition to the FLOPs saved.\n\nWe thank the reviewers again for the appreciative review and thoughtful comments. It seems like no major issues preventing the paper from being accepted. We humbly ask if they can raise their score to signify it to the AC.", "claims_and_evidence": "* The paper makes strong, well-supported claims backed by demonstrating scaling behavior across multiple model families.\n* The claim that intermediate checkpoints improve scaling law estimation is well-supported with empirical evidence.\n* The assertion that smaller models can sometimes provide better estimates than a single large model is backed by statistical analysis.", "methods_and_evaluation_criteria": "The experimental setup is well-designed with varying model sizes, training checkpoints, and dataset choices.", "theoretical_claims": "The functional form of scaling laws follows prior work (Hoffmann et al., 2022). The paper provides empirical validation for using intermediate checkpoints but lacks a formal theoretical justification. The trade-offs between training fewer large models vs. many small ones are explained well empirically, but a formal learning-theoretic analysis would strengthen the argument.", "experimental_designs_or_analyses": "The statistical robustness of their findings is high, as it aggregates data across 1000+ scaling laws. Extrapolation to unseen models is tested.\n\nInference efficiency comparisons are not provided.", "relation_to_broader_scientific_literature": "This paper builds on scaling law research from Kaplan et al. (2020) and Hoffmann et al. (2022), but extends their findings to diverse model families. Unlike prior studies that focused on fixed model families, this work demonstrates that cross-family extrapolation is feasible, opening new possibilities for scaling predictions across architectures.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above comments.", "other_comments_or_suggestions": "See below questions.", "questions_for_authors": "1. How well do these findings generalize to architectures beyond transformers?\n2. What are the computational savings when using this method vs. naive scaling?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dJpmVBj1NQ", "reviewer": "Reviewer_buHt", "summary": "The authors address challenges in scaling law estimation by compiling a dataset of training losses and evaluations from 485 pretrained language models. Through extensive empirical analyses, they establish concrete best-practice guidelines for efficiently predicting the performance of new, larger models without fully training them. Key recommendations include leveraging intermediate training checkpoints, excluding unstable early-stage training data, using multiple smaller models to reduce estimation variability, and generalizing scaling parameters across related model families. Overall, the paper serves as a practical guide for efficiently and reliably estimating scaling laws in large language model training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and feedback and for proposing that the paper should be accepted. In essence, we also agree that this paper is a meta-analysis trying to figure out what current trends tell us about the world rather than prove that this has to be the case forever. We do believe, however, that with the vast number of cases presented, most of the results will hold until a significant paradigm shift would enter the way we pretrain models, making the knowledge shared here valuable. We believe that the answer to question 2, might shed a lot of light on that issue as well.\n\nWe address the weaknesses mentioned:\n1. “Contributions are primarily empirical with limited theoretical novelty.”\nThe contributions are indeed not a mathematical theory, while guarantees would be nice, we see scientific value also in empirical knowledge.\n1. “Focuses exclusively on current families of language models, leaving general applicability uncertain.”\nWe see the replication across settings as a strength of this paper rather than a weakness. While most scaling law papers deal with a single family of models and describe a “law of nature” observed in it, we describe what is consistent across settings and is hence expected to generalize.\n1. “Methodological assumption that a single scaling law fits entire training trajectories overlooks hyperparameter influence.”\nWe kindly disagree that this is an assumption we make. Instead, we hypothesize that this is the case and bring supportive evidence that this is a useful simplification of the setting and that the benefits from the additional data outweigh the downsides, across many models trained in very different hyperparameter settings.\n\nResponses to questions asked:\n1. “Have you investigated how different learning rate schedules affect scaling-law estimation accuracy, particularly when using intermediate checkpoints? “\nTo some extent. Our results aggregate results across multiple models trained with different LR schedules; characterizing their effects in a fine-grained way would be an interesting question for followup work (and one only possible to answer using the dataset released with this paper). \n1. “How are the hyperparameters (like the learning rate) selected?”\nWe do not select any hyperparameters throughout the paper. All the models we report are hyperparameter choices from pretrained models that publicly shared their pretraining loss, like GPT 3 (we extracted) or Pythia. This is also the reason why we expect everything to generalize, as, in essence, it already generalizes across papers and models that each had their own set of choices.\n1. Can you provide additional evidence or analyses on the sensitivity of your guidelines to hyperparameter variations and random seeds?\nIf you consider the fact that every model size is initiated differently, and has the data shuffled separately, and that we have models across different papers. Then, you can see that seeds are changing a lot here and we account for them. See paragraph from line 347 on for the main place where this appears in the context of the paper.", "claims_and_evidence": "The empirical evaluation supporting the paper’s main claims is solid, demonstrating that their practical guidelines—such as using intermediate checkpoints, excluding early-stage noisy data, and training multiple smaller models—improve prediction accuracy within the studied scenarios.\n\nHowever, the paper's findings depend heavily on implicit assumptions about training dynamics, particularly that hyperparameter choices (e.g., learning rate schedules) do not significantly disrupt scaling behavior. Since the authors do not systematically examine how these hyperparameters affect their conclusions, the generality and practicality of these guidelines beyond the presented setup remain uncertain. Additionally, without deeper theoretical insights or a clear understanding of what fundamentally constitutes robust scaling behavior, the long-term applicability of these empirical guidelines may be limited.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate and well-chosen for estimating scaling laws. The authors cover nearly all major publicly available LLMs. Their evaluation criterion is sensible too.", "theoretical_claims": "The contributions are empirical.", "experimental_designs_or_analyses": "The experimental designs and analyses are generally sound, systematically varying key factors such as model size, dataset coverage, and checkpoints. However, the authors do not fully explore sensitivity to variations such as random seeds or specific hyperparameter choices, which could significantly affect results. While their recommendation to use multiple smaller models appears beneficial in their tested scenarios, further validation of robustness to these factors would strengthen confidence in the general applicability of their conclusions.", "relation_to_broader_scientific_literature": "The paper extends prior scaling law research by focusing on practical estimation strategies rather than deriving new theoretical formulations. It consolidates insights across a broad set of models, validating empirical heuristics like using intermediate checkpoints and model family generalization.", "essential_references_not_discussed": "The related work is well-grounded and sufficiently supports the paper’s contributions.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Conducts comprehensive empirical analysis addressing practical questions in scaling law estimation.\n2. Offers clear, actionable guidelines (e.g., using intermediate checkpoints, excluding noisy early training).\n3. Clearly structured and accessible, effectively using visualizations and succinct takeaways.\n\nWeaknesses:\n\n1. Contributions are primarily empirical with limited theoretical novelty.\n2. Focuses exclusively on current families of language models, leaving general applicability uncertain.\n3. Methodological assumption that a single scaling law fits entire training trajectories overlooks hyperparameter influence.", "other_comments_or_suggestions": "-", "questions_for_authors": "1. Have you investigated how different learning rate schedules affect scaling-law estimation accuracy, particularly when using intermediate checkpoints?\n\n2. How are the hyperparameters (like the learning rate) selected?\n\n3. Can you provide additional evidence or analyses on the sensitivity of your guidelines to hyperparameter variations and random seeds?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxdtjKVdxT", "reviewer": "Reviewer_CJfP", "summary": "This work investigates the challenge of fitting scaling laws in the language model domain, in particular:\n- How accurate we should expect scaling laws to be? (The authors show ~ 4% Absolute Relative Error (ARE) at best).\n- How does the shape of scaling laws vary with architecture, in order for the community to efficiently use scaling laws to propose architectures modifications? (The authors show \"similar enough to be useful and transferrable\").\n- Should we only use final checkpoints? (The authors show we should use checkpoints from ~> 30% of training data, and that only using final checkpoints can be detrimental).\n- How large should models be? (The authors show a nuanced trade-off between variance of larger models and proximity of larger models to extrapolated models of interest).\n\nThe authors do the above through a meta-analysis of existing open pretrained Language Models (LMs).\n\n### Update after rebuttal\n\nI increased my score to 4: accept. The critical issues I had with the paper (the numerical values of the coefficients) were addressed, and turned out to be only a presentation issue. I see no issues in accepting this work, as it is correct to the best of my understanding, and can provide a valuable resource for the community.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their interest and for deeming our work “extremely valuable” as well as for the sincere effort in ensuring all the details are in place. We believe the two most pressing issues found are, in fact, simpler to explain than one might expect and hope the reviewer will agree with us on this point.\n\n\nRegarding the ARE vs. residuals: the main difference is that residuals explain the ability to fit the training data (goodness of fit), and in the ARE we measure the prediction error on a test model. We will clarify this in the paper to avoid confusion with related works (such as the one mentioned).\n\n\nRegarding estimated parameter values: as noted in our response to Hw8m  noted, this was a typo: we estimate (and plot) the log of these scaling parameters (as in previous work) to improve stability and enforce positivity. .  Thus, this is a minor error in presentation not an issue with the experimentation. We will fix it in the final version of the paper.  We will also explain more about the rest of the fitting details as requested. For your interest in fitting pre-existing parameters, we do fit the token parameters, and regarding comparing only to the last model, results are very similar, just slightly more noisy when compared per cell in the tables. \n\n\nRegarding the number of checkpoints, note that the number of checkpoints used is much larger than 3: it is the number of unique pretraining *runs* that is relatively small, we discuss the contribution of single seeds and the effect or number of model sizes used in Section 7,8 . \nWe agree that the question about chinchilla is interesting, but as they did not release the necessary checkpoints to answer it, we can only assume based on the rest of the experiments that it would not change much (as they have a lot of data already). We did, however, run experiments with the data they did offer (or that we can extract as done in the mentioned paper). The results match the rest of our findings in the paper (when relevant).\nThanks for the reference to the hyperparameter search, we missed it and will discuss it in the final version.\n\n\nRegarding the questions about fitting with L-BFGS: it is not that one cannot use L-BFGS; with appropriate hyperparameter choices, it does indeed converge. But, the optimization scheme used in the paper was equally effective and required less tuning. We did have various initial experiments with L-BFGS, and they ended up with similar results. We will clarify this in the paper.\n\n\nRegarding the question about the interplay between choices for data, model size and the scaling laws. We believe that the impact here is not great, because we also fit scaling laws on the partial training runs. This means that models were often far from optimality. The only attribute they still share is that the scheduler might converge at the optimal point. Note that not all papers we used even rely on compute optimal thresholds (For example, Pythia is under-trained, the overtrain paper trains models on varying amounts of data, and OLMO models trained with the same amount of tokens regardless of the model size)", "claims_and_evidence": "### Claim 1: Scaling laws predict at best 4% ARE\n\n#### Supported (previously partially supported)\n\nEdit: authors clarified difference between residuals and ARE.\n\nSection 4 of the current paper provides a good discussion of the source for this value. This Figure 4, where the typical residual has an absolute value of 0.02. With Chinchilla losses ~2, this corresponds to an ARE of ~ 1%. This analysis is open source and reproducible. The authors should comment on the discrepancy between their result and the result in [1].\n\n### Claim 2:  Scaling laws can be strategically reused across model families\n\n#### Supported (was previously supported)\n\nEdit: authors clarified misunderstandings here, including reporting log values.\n\nIn Section 5, the authors note that $E, A, B, \\alpha, \\beta$  vary significantly over model families (Figure 3). I find the numerical values in Figure 3 concerning:\n1. $A$ and $B$ are more than an order of magnitude different to typically reported results (see e.g. [1]) for very similar setups.\n2. Negative values of $E$ do not correspond to a well-behaved asymptotic limit, as cross-entropy loss is lower bounded from zero.\nThis hints at an issue related to the optimization process for the parameters. I note that the scaling law estimation is done using `curve_fit` in `sklearn` (section 3.2), and the authors comment they could not get the L-BFGS based solver to be stable. Additionally, the authors optimize square loss, instead of Huber loss. They state that Huber loss produces similar trends as stable loss (Appendix E), yet, they also see throughout their work the effect of single poor runs dominating scaling law fitting (Section 7). I suggest the authors investigate the code provided in [1] as a way of robustly identifying scaling law coefficients, which can be sensitive to outliers and initial conditions, and may prevent robust identification. Because of potential numerical challenges, this claim, and remaining claims depending on derived scaling laws are only partially supported.\n\nBeyond the above, the authors take pre-existing $E, A, \\alpha$ values, and fit only the data term $B, \\beta$ values. In Figure 6 the corresponding error is shown.\n\nIt is unclear what fitting process is being performed here to produce the shown errors. Which cell(s) correspond to the fitting of a single model? Does it matter which single model? It would be useful for the authors to explicitly describe the fitting process for this Figure in the accompanying Appendix.\n\nWhat are the resulting $B,\\beta$ values? Can the authors also provide the Absolute Relative Error (ARE) for the model at the end of training, rather than an average over the $\\geq 30\\%$ checkpoints, as the latter value is the primary one of interest for drawing conclusions about.\n\n### Claim 3: Use $\\geq 30\\%$ checkpoints, not only final checkpoints\n\n#### Supported (was partially supported)\n\nEdit: authors clarified number of runs versus number of checkpoints understanding. Some comments about Chinchilla are still outstanding, however, the required data for that analysis is not available, and not a blocker for the main conclusions/contributions of the paper.\n\nIn Section 6.1 and Figure 4, the authors show how varying the checkpoints used for the fit changes the ARE. The behavior shown in Figure 4 does support the authors claims. \n\nThe baseline number of models being used for the fit in each plot in Figure 4 is low. Can the authors explain how it is possible to achieve the fitting of an equation with five free parameters $E, A, B, \\alpha, \\beta$ to ~3 models (Figure 4b, top left cell) such that the resulting scaling law reliably generalizes? Is this true for all choice of 3 models? What is the variance of the scaling result over resampling of the chosen 3 models? It would be useful the the authors to perform the relevant bootstrap analysis here.\n\nDoes the claim remain true when a significant number of models are included in the fit. E.g. Chinchilla uses >200 models in the fit. Is it still beneficial to include intermediate checkpoints in this scenario, or do the checkpoints now result in a biased estimate?\n\nIn addition to the above question regarding robustness of generalization, partially supported because of numerical issues discussed in Claim 2.\n\n### Claim 4: There is a trade-off between using fewer larger models or more smaller models\n\n#### Supported (was partially supported)\n\nEdit: issues above now resolved.\n\nSection 7 and Figure 2 support the authors claims. Partially supported because of numerical issues discussed in Claim 2.\n\n[1] Chinchilla Scaling: A replication attempt, https://arxiv.org/abs/2404.10102", "methods_and_evaluation_criteria": "The approach of a meta-analysis of many scaling studies is a sensible approach. The evaluation criteria is reasonable, and corresponds to questions of interest in the scaling laws community. The baseline scaling laws chosen in Section 5.2 are reasonable.", "theoretical_claims": "The paper contains no theoretical claims.", "experimental_designs_or_analyses": "I checked the experimental design in the paper. The meta-analysis experimental design is sensible. \n\nThere are a number of ways the experimental design could have been improved:\n- It would be beneficial if the authors more fully detailed their precise optimization scenario (learning rates used)\n- The data from [1] should have also been included, as it provides a large number of final checkpoints within a single family, an important use case not shared with other data sources used by the investigation.\n- The numerical methods chosen (see discussion in Claim 2 above) are potentially problematic, and yield coefficients that don't correspond to well-behaved scaling laws (Fig 3c) or coefficients consistent with prior approaches. The issues faced by the L-BFGS solver should be understood. (For example, did the authors provide the Jacobian to the solver?).\n- There is no discussion of which combinations of model size $N$ and data $D$ to pick to fit a scaling law. Some choices of $N$ and $D$ yield pathological solutions and do not allow Identifiability. One common problem is the temptation to use only Chinchilla-optimal models for fitting the scaling laws. In which case, $N = k D$ and the scaling law can be written as a one-dimensional scaling law in either $N$ or $D$, which produces redundancies around the solutions for $A,\\alpha$ and $B,\\beta$ as they cannot be disentangled. Doing a decomposition of the type described in Section 9 would reveal that 3 components in this case would explain most of the variance, as the authors find. IsoFLOP protocols, as in Chinchilla, are designed to mitigate this pathology/redundancy. Two questions are then:\n\t- If we use IsoFLOP models to fit the scaling law, are the conclusions of Section 9 still true?\n\t- Is the observation that using checkpoints beyond the final checkpoint is beneficial an artifact of the models being analyzed satisfying $N\\propto D$, and the utility from early checkpoints arising due to requiring a disentangling between $A,\\alpha$ and $B,\\beta$, which might then disappear in the case where the models being analyzed are IsoFLOP?\n- More generally, the paper and experimental design would benefit from a discussion regarding which $(N,D)$ combinations to choose subject to a compute budget, taking into account this identifiability issue.\n\n[1] Chinchilla Scaling: A replication attempt, https://arxiv.org/abs/2404.10102", "relation_to_broader_scientific_literature": "The work contains many contributions that, assuming correct, are extremely valuable for the scaling community. In particular, Claim 2 \"Scaling laws can be strategically reused across model families\", would enable more rapid investigation of scalable methods. It would be great to see a deeper dive into this particular claim.\n\nClaim 3 is consistent with findings in [2] as is discussed in the main text.\n\nClaim 1 has potential conflicts with existing literature (see above). Claim 4 - that it may be more effective to train a number of small models than a single large model - is interesting, but to be made fully usable, the authors should provide some guidance/rules-of-thumb for the reader if possible.\n\n[2] https://arxiv.org/abs/2406.19146v1 Resolving Discrepancies in Compute-Optimal Scaling of Language Models", "essential_references_not_discussed": "The re-usability of scaling laws across model families to iterate on architecture choices is well-captured by [3], which builds the scaling law into a hyperparameter search procedure, and is relevant for the ideas relating to Claim 2.\n\n[3] https://arxiv.org/abs/2302.00441 Scaling Laws for Hyperparameter Optimization", "other_strengths_and_weaknesses": "The paper is a joy to read. It is clearly written, and provides a clean set of definitions and notations that, on their own will aid in community discussion of scaling laws.\n\nThe open nature of the project further increases its utility, both in the provided code, and the assembled data from many sources.", "other_comments_or_suggestions": "Figure 6. Explain what \"Percentage\" means in caption, to help reader (prevent requirement from re-referencing main text.)", "questions_for_authors": "Edit: Majority of questions responded to. Some outstanding questions corresponding to \"model training strategies\" for compute-optimal identification of scaling laws, but not a blocker for acceptance.\n\n1. Can you explain the discrepancy between the 4% ARE discussed in Section 4, and the 1-2% ARE observed in Chinchilla [1]? [this discrepancy should be understood as it conflicts with one of the primary contributions in the scaling law field]\n2. Optimization of scaling law related [the potential optimization difficulties that may be present in the paper raise questions about the primary claims in the paper, as those claims follow from the optimization, i.e. scaling law fitting]:\n\t1. Can you explain the negative values for $E$ in Figure 3?\n\t2. Can you explain the discrepancy of > an order of magnitude between your values for $A$ and $B$ and those of prior work, e.g. [1]?\n\t3. Do you still see the effect of poor runs affecting scaling law fits (Section 7) in your Huber Loss analysis? \n\t4. Why didn't you use Huber loss throughout (which would be standard standard scaling law practice), instead of square loss? Do you have a quantification of the difference?\n\t5. How stable is your scaling law fitting procedure under i) resampling experimental data for fitting, and ii) change of initial conditions for the fit? (i.e. what are your confidence intervals under bootstrap?)\n\t6. In your failed L-BFGS solver, did you provide the Jacobian? If not, this can be done using autograd, (e.g. JAX), and can greatly help the solver.\n3. What happens to Claim 3 if you increase the number of models available (e.g. in [1], >200 models were used)? If the conclusion/guidance the same, or now in favor of using only final checkpoints? If the latter, at what point can we discard earlier checkpoints?\n4. Do the conclusions of Section 9 still hold if the scaling laws are fit to an IsoFLOP experimental protocol? (this can be tested synthetically)\n5. More broadly, the paper mostly works with models of different sizes that happen to be trained on certain amounts of data. Can the authors comment on the importance of the interaction between $N$ and $D$, and how certain combinations of $N$ and $D$ for a model may be optimal given a compute budget, and if any of their conclusions would change if a strategic choice of models with combinations of $N$ and $D$ are trained by a practitionr?\n\n[1] Chinchilla Scaling: A replication attempt, https://arxiv.org/abs/2404.10102", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work collects a dataset containing upstream losses and downstream evaluations for 485 models to facilitate research in scaling law estimation and validation. Based on these, they come up with a recommended set of best practices. The focus here is on extrapolating performance from small models to large ones. Among other findings, the authors argue that with a single model from a target family and a scaling law from a different model family, it is sometimes possible to accurately estimate target model performance reasonably well. Also, because one can reduce variance by training many models, training many small models and averaging their results can give more accurate scaling laws than including larger models. 

Overall, reviewers found the paper to be a valuable contribution to the community. However, I urge the authors to revise the paper in light of the reviewers' feedback. 

For example, while the authors claim that their results are robust to the training hyperparameters, some reviewers raised concerns about this. I think this assumption is tricky. For example, switching to a cosine learning rate schedule often results in a loss curve that violates the power law assumption (with a rapid improvement near the middle of training). There are of course other popular lr schedules that violate the power-law assumption even more, such as decaying the lr by a factor of 10 at various points during training (see for example https://arxiv.org/pdf/1912.11370). The authors should be cautious in claiming generality of findings.

Second, since one primary contribution of this work is releasing a benchmark dataset, the authors should mention similar past efforts. See for example the benchmarks released in [1] and [2].  It appears that in the current version of the paper, there is no mention of the fact that other benchmarks exist.

[1] Besiroglu, T., Erdil, E., Barnett, M., & You, J. (2024). Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102.

[2] Alabdulmohsin, I. M., Neyshabur, B., & Zhai, X. (2022). Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35, 22300-22312.
Decision: Accept (poster)