(Mis)Fitting Scaling Laws: A Survey of Scaling Law Fitting Techniques in Deep Learning
Authors: Margaret Li
Sneha Kudugunta
Luke Zettlemoyer
Venue: ICLR 2025 Poster (2025)
Keywords: survey
scaling laws
large language models
foundation models
Primary area: foundation or frontier models, including LLMs
TL;DR: We survey over 50 papers on scaling laws and discuss how systematic underreporting of details can change the conclusions of a paper

Abstract: Modern foundation models rely heavily on using scaling laws to guide crucial training decisions. Researchers often extrapolate the optimal architecture and hyper parameters settings from smaller training runs by describing the relationship between, loss, or task performance, and scale. All components of this process vary, from the specific equation being fit, to the training setup, to the optimization method. Each of these factors may affect the fitted law, and therefore, the conclusions of a given study. We discuss discrepancies in the conclusions that several prior works reach, on questions such as the optimal token to parameter ratio. We augment this discussion with our own analysis of the critical impact that changes in specific details may effect in a scaling study, and the resulting altered conclusions. Additionally, we survey over 50 papers that study scaling trends: while 45 of these papers quantify these trends using a power law, most under-report crucial details needed to reproduce their findings. To mitigate this, we we propose a checklist for authors to consider while contributing to scaling law research.

Published as a conference paper at ICLR 2025
(MII))ITTING: A SURVEY OF SCALING LAWS
Margaret Sneha Kudugunta*, Luke Zettlemoyer (margsli, snehark) (os. washington. edu
ABSTRACT
Modern foundation models rely heavily on using scaling laws to guide crucial training decisions. Researchers often extrapolate the optimal architecture and hyper parameters settings from smaller training runs by describing the relationship between, loss, or task performance, and scale. All components of this process vary, from the specific equation being fit, to the training setup, to the optimization method. Each of these factors may affect the fitted law, and therefore, the conclusions of a given study. We discuss discrepancies in the conclusions that several prior works reach, questions such the optimal token parameter ratio. We augment this discussion with our own analysis of the critical impact that changes in specific details may effect in a scaling study, and the resulting altered conclusions. Additionally, we survey over 50 papers that study scaling trends: while 45 of these papers quantify these trends using a power law, most under-report crucial details needed to reproduce their findings. To mitigate this, we we propose checklist for authors to consider while contributing to scaling law research.
INTRODUCTION
Training at the scale seen in recent large foundation models (Dubey et al. 2024 OpenAI 2023
Reid al., (2024) is an expensive and uncertain process. Given the infeasibility of hyperparameter tuning multi-billion parameter models, researchers extrapolate the optimal training setup from smaller training runs. More precisely, scaling laws (Kaplan et al. 2020) are used to study many different aspects of model scaling. Scaling laws can guide targets for increasing dataset size and model size in pursuit of desired accuracy and latency for a specific deployment scenario, study architectural improvements, determine optimal hyperparameters and assist in model debugging.
Scaling laws are often characterized as power laws between the loss and size of the model and dataset, and are seen in several variations (Sec- tion (2). These laws are found empirically by training models across a few orders of magni- tude in model size and dataset size, and fitting the loss of these models to a proposed scaling law. Each component of this process varies in the reported literature, from the specific equa-
Table 1: We provide summary of the papers veyed, highlighting the reproducibility challenges endemic scaling law papers.
# Papers...
Surveyed
51
With quantified scaling law
45
With a description of training setup
40
With definition of equation variables
36
tion being fit, to the training setup, and the op- timization method, as well as specific details for
With a description of evaluation
29
With description of curve fitting
28
selecting checkpoints, counting parameters and
With analysis code
19
the objective loss optimized during fitting.
With metric scores checkpoints provided
17
Changes this setup lead to significant changes to the results, and therefore completely different conclusion to the study. For example, Kaplan et al. (2020) studied the optimal allocation of compute budget, and found that dataset size should be scaled more slowly than model size
(D x N0.74 D is dataset size, N is model size). Later, Hoffmann al. (2022) contradicted this finding, showing that model size and dataset size should be scaled roughly equally for optimal scaling. They highlight the differences in setup which lead to them showing that large models should be
Equal contribution

---

Published as a conference paper at ICLR 2025
SCALING LAW REPRODUCILIBIIIT CHECKLIST
Scaling Law
Training Setup ($4
Data Collection ($5)
Fitting Algorithm ($6)
Hypothesis
models
Checkpoint Open Sourcing
Objective (Loss)
Form
Model size range
Algorithm
Variables (Input).
# Checkpoints per Power Law
Dataset source & size
Optimization
Parameters
Hyperparameterss
Parameter/ FLOP Count Calculation
Derivation and Motivation
Evaluation Dataset & Metrie
Optimization Initialization
Hyperparameter Choice
Data Usage Coverage
Metrie Modification
* Assumptions
Other Settings
& Code
Validation of Law(s)
Code Open Sourcing
We introduce checklist for researcher to use for scaling laws research. In Appendix B we include an expanded version of the checklist that may be used as a template.
Figure I:
trained for significantly more tokens: particularly, they point to using later checkpoints, training larger models, a different learning rate schedule and changing the number of training tokens used across runs. Multiple followup works have focused either reproducing
explaining the differences between these two papers (Besiroglu et al.) 2024, Porian et al.) 2024 Pearce & Song, (002bb). The authors find it challenging reproduce results of previous papers refer the reader to Section for a further discussion on these replication efforts.
Motivated by this, survey and architectures, and find that essential details needed to reproduce scaling law studies are often underreported. We broadly categorize these details as follows:
50 papers scaling laws across variety of modalities, tasks
What form are we fitting? Researchers may choose any number of power law forms relating any set of variables, which they fit the data extracted from training runs. Even seemingly minor differences in form, may imply critical changes in assumptions for example, about certain interactions between variables which are excluded, the definitions of these variables or error terms which are deemed significant enough to include.
Section 3
Section 4 How do we train models? In order fit scaling law, one needs to train range of models spanning orders of magnitude in parameter count and/or dataset size. Each model requires a multitude of hyperparameter and parameter choices, such the specific model/dataset sizes use, the architecture shape, batch size or learning rate schedule.
Section How do we extract data after training? Once these models are trained, downstream metrics like perplexity must be obtained from the intermediate final checkpoints. This data may be also scaled, interpolated or bootstrapped to create more datapoints to fit the power law parameters. Section 6 How are we optimizing the fit? Finally, the variable must be fit with an objective and optimization method, which may turn have their own initialization and hyperparameters
choose.
To aid scaling laws researchers in reporting details necessary to reproduce their work, we propose checklist (Figure - an expanded version may be found in Appendix ). Based on this checklist, we summarize these details for all 51 papers in tabular form in Appendix We find that important details are frequently underreported, significantly impacting reproducibility, especially in cases where there is no code only 19 of 42 papers surveyed have analysis code/code snippets available. Additionally, 23 (a little over half) of surveyed papers do not describe the optimization process, and 15 do not describe how training FLOPs or number of parameters are counted, which has been found to significantly change results (Porian al. 2024). In addition, we fit our own power laws to further demonstrate how these choices critically impact the final scaling law (Section $7p.
PAPERS ON SCALING LAWS
Researchers have proposed scaling laws to study the scaling of deep learning across multiple domains and for several tasks. Studies of the scaling properties of generalization error with training data size and model capacity predate modern deep learning. [Banko & Brill (2001) observed a power law scaling of average validation error on a confusion set disambiguation task with increasing dataset size. The authors also claimed that the model size required to fit given dataset grows log linearly. As
2

---

Published as a conference paper at ICLR 2025
for larger scale models, (Amodei al. (2016) observe a power-law WER improvement on increasing training data for a 38M parameter Deep Speech 2 model. Hestness et al. (2017) show similar power law relationships across several domains such machine translation, language modeling, image processing and speech recognition. Moreover, they find that these exponential relationships found hold across model improvements.
Kaplan et al. (2020) push the scale of these studies further, studying power laws for models up to I.5B parameters trained on 23B tokens determine the optimal allocation of fixed compute budget.
Later studies (Hottmann et al. 2022 Hu et al. 2024) revisit this and find that Kaplan et al. (2020) greatly underestimate the amount of data needed train models optimally, though major procedural differences render it challenging to attribute the source of this discrepancy. Since then, researchers have studied various aspects of scaling up language models. Wei et al. (2022) examine the emergence of abilities with scale that are not present in smaller models, while Hernandez et al. (2021) study the scaling laws for transfer between distributions in a finetuning setting. |Henighan
al. (2020)
consider possible interactions between different modalities while recently, (Aghajanyan et al. 2023
study scaling in multimodal foundation models. Tay et al. (022) show that not all architectures scale equally well. highlighting the importance of using scaling studies to guide architecture development. Poli et al. (2024) scale hybrid architectures like Mamba GGu & Dao (023), showing the efficacy of this new model family. Other researchers also formulate specific scaling laws to study other Transformer based architectures. For example, Clark et al. (2022) and Frantar et al. (2023) introduce new scaling laws to study mixture of expert models (Fedus al., 2022; Shazeer et al.) 2017) and sparse models (Zhu & Guptal (017) respectively. Researchers have also used scaling laws to study encoder-decoder models for neural machine translation (Ghorbani et al., 202T; Gordon et al. [02T and the effect of data quality and language on scaling coefficients (Bansal et al. 2022 Zhang et al. 2022). While language models form the majority of the papers surveyed, we also consider papers that study VLMs (Cherti et al. 2023, Henighan et al. )2220) vision (Alabdulmohsin et al. 2022 Zhai et al., 2022). reinforcement learning (Hilton et al. 2023 Jones) 2021; Gao et al. 2023) and
recommendation systems (Ardalami et al. 2022). We further discuss different forms of scaling laws researchers introduce for the specific research questions they wish to answer in Section 3
A majority of the surveyed papers study Transformer (Vaswani 2017) based models, but a few consider different architectures. For example, Sorscher et (2022) investigate data pruning laws in ResNets, and some smaller scale studies use MLPs or SVMs (Hashimoto) (021). This overrepresen- tation is perhaps partially a result of Transformer-basee models achieving higher scale than other architectures; ResNet101 has 44M parameters, while the largest Llama model has 405B.
Replication Efforts Besiroglu et al. (204) seek to reproduce the parameter fitting approach used by Hoffmann al. (2022). They unable to recover the scaling law from Hoffmann al. (2022),
and demonstrate that the claims of the original paper are inconsistent with descriptions of the setup. They then seek to improve the fit of the scaling law by initializing from the parameters found in Hoffmann et al. (022) and modifying parts of the power law fitting process.
Porian et al. (2024) isolate several decisions as primarily responsible for the discrepancy between the
recommendations of Kaplan et al. (2020) and Hoffmann et al.) (2022): : (1) learning rate scheduler warmup, (2) learning rate decay, (3) inclusion of certain parameters total parameter count, and (4) specific training hyperparameters. By adjusting these factors, they are able to reproduce the results of Kaplan et al. (2020) and Hoffmaan et al. (2022). ). However, they only use 16 training runs to fit their scaling laws, each designed to match one targeted setting (c.g., replicating Kaplan et al. (2020). Instead of using raw loss values, they fit to loss values found by interpolating between checkpoints. Like Pearce & Song (2024b), they apply a log transform and linear regression to fit their law.
3 WHAT form ARE WE FITTING?
A majority of papers we study fit some kind of power law (f(x) = az -k). That is, they specify an equation defining the relationship between multiple factors, such that a proportional change in one results in the proportional change of at least one other. They then optimize this power law to find some parameters. A few efforts do not seem to fit power law, but may show line of best fit,
obtained through unspecified methods (Rae et al. (002 Dettmers et al. 2022 Tay et al. 2022 Shin
et al. 2023 Schaeffer et al. 2023 Poli et al. 2024).
3

---

Published as a conference paper at ICLR 2025
The specific form may be motivated by researcher intuition, previous empirical results, prior work, code implementation. or data availability, More importantly, the form is often determined by the specific question(s) paper investigates. For example, one may attempt to predict the performance achieved by scaling up different model architectures, or the optimal ratio for model scaling vs data scaling when increasing training compute (Kaplan et al., 2020; Hoffmann al.) (022). Based on this, we loosely classify scaling laws by their form as performance prediction and ratio optimization approaches. We indicate this classification for all surveyed papers in Appendix C
3.1 RATIO OPTIMIZATION
The simplest scaling law forms usually predict the relation between two variables in optimal setting. For example, approaches and 2 from Hoffmann et al. (2022) fit to the optimal (i.e., lowest loss) D and N values for particular compute budget C. Porian al. (2024), aiming to resolve these inconsistencies, defines p² Nr
D and writes this relationship
NO No (C) D3 ;"; Pb Ca
(1)
They assume C' G,D, and thus only need to fit the first equation; the other power laws can be inferred. This simplicity is deceptive in some cases, as collecting (N (C"). C²) pairs may not be trivial. It is possible to fix C and follow binary search approach to train multitude of models, then bisect to approximate the performance-optimal N. D pair. However, this quickly grows prohibitively costly. In practice, it is common to interpolate between a of fixed results to estimate the true N (C) $5 This adds to the complexity of this approach, and introduces a hidden dependency on the performance evaluation, yet it does not actually predict the performance of the optimal points. If only the performance of the optimal-ratio model is of interest, it is possible to fit a second power law (C), (C") = a C Most papers we survey choose fit power law which directly predicts performance.
3.2 PERFORMANCE PREDICTION
Kaplan et al. 22200) proposes a power law between Loss L, number of model Parameters N, and number Dataset tokens D:
ey a
Q.D
N
D
D)
+
De
On the other hand, Approach of Hoffmann et al. (2022) proposes
A
L(N, = E
(3)
No
D
In both of the above, all variables other than L, N, and D are parameters to be found in the power law fitting process. Though these two forms quite similar, they differ in some assumptions. Kaplan et al. (2020) constructs their form on the basis of 3 expected scaling law behaviors, and Hoffmann al. (022)) explains in their Appendix D that their form based on risk decomposition.
The resulting Kaplan et al. (2020) form includes an interaction between N and D) in order to satisfy constraint requiring assymmetry introduced by one of their expected behaviors. The Hoffmann et al.) (2022) form, on the other hand, consists of 3 additive sources of error, E representing the irreducible error that would exist even with infinite data and compute budget, as well as two terms representing the error introduced by limited parameters and limited data, respectively.
Power laws for performance prediction can sometimes yield closed form solutions for optimal ratios well. However, the additional parameters and input variables, introduced by the need to incorporate the performance metric term, add random noise and dimensionality. This increases the difficulty of optimization convergence, so when prediction performance is not the aim, ratio optimization approach is frequently a better choice.
Many papers directly adopt one of these forms, but some adapt these forms study relationships with other input variables. Clark et al. (2022). for example, study routed Mixture-of ofxture-of-Expertt models, and propose scaling law that relates dense model size (effective parameters) N and number of experts E
4

---

Published as a conference paper at ICLR 2025
with biquadratic interaction (log L(N, E) - a log N + blog E + c clog N log E + d). Frantar et al. (2023) study sparsified models, and propose a scaling law with an additional parameter sparsity S. the optimal value of which increases with N (L(S, N, D) (1 55 cg) by
bp +c).
Other papers change the form to model variables in the data setup. [Aghajanyan et al. 2023 consider interference and synergy between multiple data modalities (L(N, D2) = E3 + No
D2
L(N, D;, D2) = L (NN.D.)-UNND..) Ca. +
B.., Aud + [D:I+ID, 1., ) while Goyal et al. (024). N
Fernandes et al. (2023) and Muennighoff et al. (2024) add terms to their scaling law formulatin catmuaations
which represent mixing data sources and/or repeated data, using notions such as diminishing utility,
comprehensive list of the power law forms in the surveyed papers may be found in Table 6
4 HOW DO WE train models?
In order to fit a scaling law, one needs to train a range of models across multiple orders of magnitude in model size and/or dataset size. Researchers must first decide the range and distribution of N and D values for their training runs, in order to achieve stable convergence to a solution with high confidence, while limiting the total compute budget of all experiments. Many papers did not specify the number of data points used to fit each scaling law; those that did range from 4 to several hundred, but most used fewer than 50 data points. The specific N and D values also skew the optimization process towards a certain range of N/D ratios, which may be too narrow to include the true optimum. Some approaches, such as using IsoFLOPs (Hoffmann et al. 2022). additionally dictate rules for choosing N and D values. Moreover, using minimum N or D value may result in outlier values that
may need to be dropped (Porian et al. 2024 Shin et al., 2023 Henighan et al. 2020). We investigate
this choice in Section 7..
The definition of N, D. or compute cost C' can affect the results of a scaling study. For example, if a study studies variation in tokenizers, a definition of training data size based on character count may be more appropriate than one based on token count (Tao et al., 2024). The inclusion or exclusion of embedding layer compute and parameters, may also skew the results of a study a major factor in the different in optimal ratios determined by Kaplan et al.] (0200) and [Hffmann et al. (2022) has been attributed to not factoring embedding FLOPs into the final compute cost (Pearce & Song 2024b Porian et al. (2024). Given the increase in extremely long context models (12 Reid jt al. (024)), the commonly used training FLOPSs approximation C = 6N D (see Appendix (C) may not hold for such models, given the additional cost proportional to the context length and model dimension Bi ct al. (2024) introduce new terms non-embedding FLOPs/token to account for this.
Scaling law fit depends on the performance of each individual checkpoint, which is highly dependent on factors such as training data source, architecture and hyperparameter choice. [Bansal et al. (2022) and Goyal al. (2024), for instance, discuss the effect of data quality and composition on power law exponents and constants. Repeating data has also been found to yield different scaling patterns in large language models (Muennighoff et al. 2024 Goyal et al. 2024).
Researchers have also studied the effect of architecture choice on scaling Hestness et al.) (2017) find that architectural improvements only shift the irreducible loss, while Poli al. (2024) suggest that these improvements may be more significant. The way in which a model is scaled can also affect results. Within the same architecture family, Clark al. (2022) show that increasing the number of experts in a routed language model has diminishing returns beyond a point, while Ghorbani et al. (202T) find that scaling the encoder and decoder have different effects on model performance. Scaling embedding size can also drastically change scaling trends (TTo et al. 2024).
The optimal hyperparameters to train model changes with scale. Changing batch size, for example, can change model performance McCandlish et al. (2ala) et al. (2020). Optimal learning rate is another hyperparameter shown to change with scale, though techniques such as those proposed in Tensor Programs series of papers (Yang et al. 2022) can keep this factor constant with simple changes to initialization. More specifically, changing the learning rate schedule from a cosine decay to constant learning rate with cooldown (or even changing the learning rate hyperparameters) has been found to greatly affect the results of scaling laws studies (Hu et al.) 2024 Porian et al.) 2024 Hägele al. 2024 Hoffmann al. 2022
5

---

Published as a conference paper at ICLR 2025
One common motivation for fitting scaling laws is extrapolation to higher compute budgets. However, there is no consensus on the orders of magnitude up that one can project a scaling law and still find it accurate. nor on the breadth of compute budgets that should be covered by the data. We find that the range of model size N and dataset size D greatly varies, with the maximum value of N in each paper ranging from 10M parameters to around 7B and that of 7) being as large as 400B tokens. For most papers we survey, the scales are relatively modest: 13 of 51 papers train models beyond 2B parameters; most only train models smaller than 1B parameters. It has been shown, with some controversy Schaeffer et al. (2023), that scaling to significantly larger scales can result in new abilities that did not appear smaller models (Wei et al. (022). Forecasting limits to extrapolation and the appearance of new abilities at new scales is an open question.
How DO WE collect data FROM MODEL TRAINING?
To evaluate the range of models trained to fit a scaling law, train or validation loss are most commonly
used, but some works consider other metrics, such as ELO score (Jones) 2021 Neumann & Gros
2022), reward model score (Gao et al. 2023), or downstream task metrics like accuracy or classifica-
tion error rate (Henighan et al. 2020 Zhai et al. 2022, Cherti et al., 2023 Goyal et al. 2024 Gao
ct al. 2023). J. This choice is non- coiviaivial while some papers show that there is power law relation
between the predicted loss found by using validation loss and different downstream task (Dubey
ct al. 2024), it is possible for the results of a study to change completely depending on the metric used. Schaeffer al. (2023), for example, find that using linear metrics such as Token-edit distance instead of non-linear metrics such as accuracy produces smooth, continuous predictable changes in model performance, contrary to an earlier study by Wei al. (2022). Moreover, Neumann & Gros (2022) find that they are unable to use test loss instead of Elo scores to fit a power law.
While is most straightforward to evaluate only the final checkpoint on the target metric, some studies may use the median score of the last several checkpoints of each training run Ghorbani et al. 2022]. or multiple intermediate checkpoints throughout each training run for various reasons. One common reason is that this is the only computationally feasible way to obtain a fit with sufficient confidence intervals (Besiroglu et al. 2024). For instance, the ISOFLop approach to finding the optimal D/N ratio in Hoffmann et al. (202)) requires training multiple models for each targeted FLOP budget- this would be computationally prohibitive to do without using intermediate checkpoints. Hoffmann et al.) (2022), in particular, use the last 15% checkpoints. Some papers also report bootstrapping values (vgi et al. 2022). This detail is often not specified in scaling law papers, with only 29 of 51 papers reporting this information we point the reader to Appendix Cior an overview.
A related technique is performance interpolation. Porian et al. (022) do not aim to exactly match the desired FL.OP counts when evaluating model checkpoints mid-training. They instead interpolate between multiple model checkpoints to estimate the performance of a model with the target number of FLOPs. Hoffmannn al. (2022) and Tao al. (2024) also interpolate intermediate checkpoints. Hilton et al. (2023), relatedly, smooth the learning curve before extracting metric scores.
As discussed in Section 4 training models with too little data or too few parameters can skew the results. To prevent this issue, several works report filtering out data points before fitting their power law. [Henighan et al.] (2020) drop their smallest models, while Hilton al.] (2023) and Hoffmmann] et al. (2022) exclude early checkpoints. MMuennggotff et al. (2024) remove outlier datapoints that perform badly due to excess parameters or excess epochs. Similarly, lvgi et al. (022) remove outlier solutions after bootstrapping.
How ARE WE optimizing THE FIT?
The optimization of a power law requires several design decisions, including optimizer, loss, ini- tialization values, and bootstrapping. We discuss each in this section. Over half of the papers we analyze do not provide any information about their power law fitting process, or provide limited information only and fail to detail crucial aspects. Specifically, many papers fail to describe their choice of optimizer or loss function. In Table we provide an overview of the optimization details (if specified) for each paper considered.

---

Published as a conference paper at ICLR 2025
Piper
Caree litinng Method
Loss Otysctive
Hyperpammeeess Repertea?
Inmaliiztiio
Are staling laws validaed?
Loot Squeer Regnooion
Cuoom omer torn
NIA
Rondom
Nos linear Least Squares is log log space NA
NIA
N/A
NA
NA
Mtnute X
L-BFOS
Squaees Regeosion
SIA
Grid Search, optinize
L-IFUS
Hober Loss
Ond Search
Lood Squerre Regrosion
NIA
N.S.
KS
NS
2
M5
11111
NS
RMSE
N
NS
NS
NS
NS
NS
NS
NS
Nus-lincar Least Squares
NA
Nan-augmmmted pernttte
scppy
Cusooa
Navier
Lincer Lesst Sqpanos
Variess
N/A
NFA
Pocyzomial Regiessioz (Quadratic)
N.S.
N
N.S.
Pe/re C01r 10/46
Pocytomil Least Squares
MSB on Log loss
NIA
N0A
Lincer Leaal Squaros
MSE
NIA
N/A
Woightdd Linor Regresson
torighted Sl Log loss
NIA
NIA
Squares Respossion
MSE
N.S
N.S
N.5
K.S
M04
L-RPOS
Lun--loss
optinioo all
Noue
None
NIA
NOA
NA
NA
NA
NA
KS
NS
NS
N5
NS
E1tooaatre 15
NUME
RS
NS
NS
14
NS
NS
NS
NS
NS
NS
NS
NS
NS
NS
NS
NS
Grid Search
enoo
NA
Lar--koss
optnica all
NS
NS
Truz Region Reffective algonitimm Squares
Soft-I. Loss XS
Fixed
NS
NS
NS
UMA-FS+ Liner Regression
1.2 lng loes
Fixed
Plenar si
BPGS
Habee on Log, Joss NS
N Randon Trials
NS
NS
NS
NS
Adkmn
Cheurr Lina
Lesal Squanos LugeLar ypuar
MIE
NA
NA
L-IEUS Leatt Square
Log loss
Random Thaak from Grd
LBBGGS
NS
N5
NS
NS
K8
NS
NS
NS
NS
Tallinsc Ce Zuramoyy 1035
NA
NA
NA
NA
NA
NS
NS
NS
NS
L-BPOS
Hakee on Log-low
Gnd Stanch, aptinias all
NS
Low
NS
Table 2: We provide an overview of which papers provide specific details required reproduce how they fit their scaling law equation.
Optimizer Power laws are most commonly fit with variety of algorithms designed to optimize non-linear functions. One of the most common is the BFGS algorithm, a variation L-BFGS (Liu & Nocedal 1989). Some papers (Hashimoto) |021 Covert et al. 2024) use Adam, Adagrad, or other optimizers common in machine learning, such as AdamW, RMSProp and SGD. Though effective for LLM training, these are sometimes ill-suited for the purpose of fitting a scaling law, due to various factors limiting their practicality, such as data- hungriness. Goyal al. (2024) forgo the use of an optimizer altogether due to instability of solutions (see initialization) and rely exclusively on grid search to fit their scaling law parameters.
Seyyden-Fletcher-Gieee -haanno))
Some scaling law works (Rosenfeld et al. 2019) opt to use linear method, such as linear regression, which is generally much simpler. To do this, they typically convert the hypothesized power law to a linear form by taking the log. For example, for a power law n = c x + d, use the form 3-log(y) = 7---ooooo (z: instead. For loss prediction, this results in a form similar to (LN, D)) a log + 3- -log + E. This trick is employed even when using an optimizer capable of operating on non-linear functions (Hashimoto) 2021). Though this conversion may sometimes work in practice, it is generally not advised because the log transformation also changes the distribution of errors, exaggerating the effects of errors at small values. This mismatch increases the likelihood of poor fit (Goldstein et al. 2004). We found this approach to be very common among the papers we study.
Loss
Various loss functions have been chosen for power law optimization, including variants on MAB (mean absolute error), MSE (mean squared error), and the Huber loss (Huber 1992). which is identical to MSE for errors less than some value 8 (a hyperparameter), but grows linearly, like MAE, for larger errors, effectively balancing the weighting of small errors with robustness to outliers. Of
the papers which specify their loss function, most use variant of MAE (Ghorbani al. (021). MSE 2023
(Goyal et al. 2024 Hilton et al.) 2023). Huber loss (Hoffmann et al. 2022 Aghajanyan et al.
Frantar al.
2023
Tao et al. 2024 Muennighoff et al. 2024), custom loss (Covert et al.
2024).
7

---

Published as a conference paper at ICLR 2025
Initialization Initialization can have substantial impact final optimization fit (S7). One approach is to iteratively train with different initializations, selecting the best fit at the termination of the search. This is typically a grid search over choices for each parameter (Aghajanyan et al. Muennighoff et al. 2024), or a random sample from that grid (Frantar et al.) 2023] Tao et al. 2024). Alternatively, the full grid of potential initializations can be evaluated on the loss function
2023
without training, and the most optimal & used for initialization and optimization (Caballero et al.
2022). Finally, if hypothesis exists, either from prior work or expert knowledge about the function, this hypothesis may be used instead of search, or to guide the search (Besiroglu et al. 20024)
Validating the Scaling Law A majority of the papers surveyed do not report validating the scaling law in any meaningful way. Knowing this is critical understanding whether the results of the scaling laws study are valid, given the examples given throughout the paper of scaling laws study conclusions changing depending on the process details. Porian et al. (2024) and Alabdulmohsin et al. (2022) use confidence intervals and goodness of fit measures to validate their scaling laws. Ghorbani ct al. (202T) and Bansal et al. (2022) also do this. Otherwise, a majority of the papers that we report as validating their scaling laws mainly extrapolate to models a few orders of magnitude larger and observe the adherence to the scaling law obtained.
7 OUR REPLICATIONS AND ANALYSES
Each of the choices discussed above in Sections 3 may have a crucial impact on the result, yet it remains common to critically underspecify the setup for fitting a power law. Scaling law works often fail open source their model and code, making reproduction infeasible, and likely contributing to contradictory conclusions as discussed in Section 2 Though some efforts have been made (Porian et al. 2024, [esiroglu al., 2024) to reconeile such discrepancies, there is still only sparse understanding of the impact of each of the decisions we discuss.
To investigate the significance of these scaling law optimization decisions, we vary these choices to fit our own scaling laws. We fit both the Chinchilla-scraped data from Besiroglu et al. (2024), and data from our own models.
Reconstructed Chinchilla data (Besiroglu et al., 2024) This data is extracted from a vector-based figure in the pdf of Hoffmann al. (2022), who claim that this includes all models trained for the paper. It consists of 245 datapoints, each corresponding to a final checkpoint collected at the end of model training. There a potential risk of errors in this recovery process.
Data from Porian et al. 2024) This data includes training losses for Transformer LMS ranging in size from 5 to 901 million parameters, each trained on Open WebText2 (Gao et al., 2020) and RefinedWeb data (Penedo al. 2023), across variety of data and compute budgets, from 5 million to 14 billion tokens, depending on parameter count. Each model is trained with different peak learning rate and batch size setting, found by fitting a separate set of scaling laws.
Our models We train a variety of Transformer LMs, ranging size from 12 million to billion parameters, on varied data and compute budgets and hyperparameter settings. Details about our setup, including hyperparameters, are listed Appendix A We open source all of our models, evaluation results, code, and FLOP calculator at whttts:: /gggttttttbb/ggg com/ www.:hhaaaaaahaaaaaaaaaaaaaaaa hataaahassacaaaaaee Laws
We fit a multitude of power laws and study the effects of: (1) power law form; (2) model learning rate; (3) compute budget, model size, and data budget range and coverage; (4) definition of N and C: (5) inclusion of mid- dridnirannnn checkpoints; (6) power law parameter initialization; (7) choice of loss and (8) optimizer.
We plot all results in Appendix Figure 2 In our initial analysis of (5), we find substantial differences. sometimes improvements, in stability of fit when including mid-trad -rd-rraiiing checkpoints. As result, include analyses and corresponding figures for power law fitting with only data from final checkpoints, as well with all mid-trainin checkpoints, when applicable.
Based on our observations, we also make some more concrete recommendations in Appendix DD with the caveat that following the recommendations cannot guarantee good scaling law fit.
8

---

Published as a conference paper at ICLR 2025
7.1 FORM (
We consider the (1) baseline Hofimann et al. (2022) form (Approach 3) and then (2) apply the trick employed by Muennighoff et al. (2024) setting a = 8 in the Hoffmann al. (2022) form, which assumes the optimal D/N ratio stays roughly constant L(N, D) = E + No D We also compare with (3) an ISOFlop approach (Approach of [ooffman et al. (002)). D. which we fit an optimal N for each compute budget C, (C', which can then be used to fit the predicted loss L(N, D). This approach usually necessitates the usage of mid-traine aid-training checkpoints (discussed further in $7.3) as is infeasible to train large enough number of models for each FLOP budget considered. However, we apply it here without using only final model checkpoints, and extend to mid-training checkpoints in $7.3). We adapt the implementation from Porian et al.) (2024), which contains more details about interpolation of data points and specific hyperparameters. In all data sets, (2) approaches the law reported by Hoffmann al. (2022). but (I) only does so for the data from Porian et al. (2024) (Figure 2a).
B
We experiment with the power law form reported by Kaplan et al. (2020), but this consistently yields a law which suggests that the optimal number of data points is I, even when varying many aspects of the power law fitting procedure. The difficulty of fitting to this form might be partially a result of severe under-reporting in Kaplan et al. (2020) with regard to procedural details, including hyperparameters for both model training and fitting.
7.2
TRAINING 4$4
Hu al. (2024) study the effects of several model training decisions, including batch size, learning rate schedule, and model width. Their analysis focuses on optimizing hyperparameters, not on the ways hyperparameter and architecture choices affect the reliability of scaling law fitting. Observed variations between settings suggest that suboptimal performance could skew the scaling law fit.
To substantiate this further, we simulate the effects of not sweeping the learning rate in our models. As baseline. (I) we sweep at each (N, D) pair for the optimal learning rate over a range of values, most multiple of 2 apart. Next, (2) we use a learning rate of 1e-3 for all N, the optimal for our 1 billion parameter models, and do the same for (3) 2e-3 and (4) 4e-3, which is optimal for our 12 million parameter models. Lastly, we use all models across all learning rates at the same N and D. Results vary dramatically across these settings. Somewhat surprisingly. using all learning rates results in a very similar power law to sweeping the learning rate, whereas using a fixed learning rate of le-3 or 4e-3 yields the lowest optimization loss or closest match to the Hoffmann et al. 2022 power laws, respectively (Figure 2b).
We also study the effects of limiting model scale range, data scale range (implicitly), and data-to- parameters range by filtering all datasets: we compare (I) using all N D scales, (2) only models with N up about 100 million (3) 400 million parameters, and including (4) only models with D/N < 18 or (5) D/N > 22. These ranges are designed to exclude D/N = 20, the rule of thumb based on Hoffmann et al. (2022). The minimum maximum D/N ratio tested does skew results; above 10 FLOPs, (4) and (5) fit to optimal ratios D/N < 18 and D/N > 22, respectively. Removing our largest models in (2) also creates a major shift in the predicted optimal D/N (Figure 2c).
Related to choice of N, D and C' values, we investigate different ways of counting N and C. specifically whether to include embedding parameters and FLOPs. We compare (1) our baseline, which includes embeddings in both N and C, with (2) excluding embeddings only in N, (3) excluding embeddings only in C, (4) excluding embeddings in both N and C. We also compare to (5) using the = 6NN D approximation, including embedding parameters. In all other settings, we calculate the FLOPs in manner similar to Hoffmann et al. (2022), and we open source the code for these calculations. With both datasets, the exclusion of embeddings in FLOPs has very little impact on the final fit. Similarly, using the C' = 6.N D approximation has no visible impact. For the Porian et al. (2024) models, the exclusion of embedding parameters in the calculation of N results in scaling laws which differ substantially, and with increasing divergences at large scales (Figure 2dh.

---

Published as a conference paper at ICLR 2025
7.3 DATA COLLECTION ($5)
We compare using data from (1) only the final checkpoints of each model with (2) using all available mid hid-training checkpoints. We also consider removing checkpoints collected during the (3) first 10%, (4) 20%, and (5) 50% of training. Using mid-training checkpoints sometimes results in more stable fits which are similar to the Hoffmann et al. (2022) scaling laws, but the effect is unreliable and is dependent on other decisions. Following this finding, we also run all other analyses using setting (2), and find that these scaling law fits are often more consistent when varying other aspects of the fitting procedure (Figure 2e
7.4 FITTING ($6)
We vary the initialization method for our power law fitting procedure: (1) our baseline replication
Hoffmann et al. (2022) Approach 3, which conducts the full optimization process on a grid of
6x6x5x5x5=4500 initializations (Hoffmann et al. 2022). ; (2) searching for the lowest loss initialization point (Caballero et al. (2022) and optimizing only from that initialization, (3) optimizing from the 1000 lowest loss initialiiations, (4) randomly sampling k(=100) points (Frantar et al. 2023 Tao et al.. 2024), and (5) initializing with the coefficients found in Hoffmann et al. (2022), as Besiroglu et al.
(2024) does. With the Besiroglu et al. (2024) data, (5) yields a fit nearly identical to that reported by Hoffmann al. (2022). although (1) results in the lowest fitting loss. With the Porian et al. (2024) data, all approaches except (2) yield a very similar fit, which gives a recommended D/N ratio similar to that of Hoffmann et al. 20222. However, using our data, (2) optimizing over only the most optimal initialization yields the best match to the Hoffmann et al. (2022) power laws, followed by (5) initialization from the reported Hoffmann et al. (2022) scaling law parameters. Optimizing over the full grid yields the power law that diverges most from the Hoffmann et al. (2022) law, suggesting the difficulty of optimizing over this space, and the presence of many local minima (Figure 21).
Then, we analyze the choice of loss objectives, including (1) the baseline log-Huber loss, (2) MSE, (3) MAE, and (4) the Huber loss. We find that there is somewhat less variance in the resulting power laws than we have seen in other power law fitting decisions, but the recommended optimal data budgets still span a wide range (Figure 2g).
Finally, we consider the choice of optimizer. We first fit a power law to using the original (1) L-BFGS, with settings matching those described by HHoffmaan et al. (2022). L-BFGS and BFGS implementations have an early stopping mechanism, which conditions on the stability of the solution between optimization steps. We experiment with setting this threshold for L-BFGS to a (2) higher value le-4 (stopping earlier). We find that using any higher or lower values results in the same solutions or instability for all three datasets. L-BFGS and BFGS also have the option to use the true gradient of the loss, instead of an estimate, which is the default. In this figure, we include this setting, (3) using the true gradient for L-BFGS. We compare these L-BFGS settings to (4) BFGS. We test the same tolerance value and gradient settings, and find that none of these options change the outcome of BEGS in our analysis. Finally, we compare to (5) non-linear least squares and (6) pure grid search, using a grid 5 times more dense along each axis as that which we used for initialization with other optimizers. This density is chosen to approximately match the runtime of L-BFGS. Many of these optimizer settings converge similar solutions, but this depends on the data, and the settings which diverge from this majority do not follow any immediately evident pattern (Figure 2h
CONCLUSION
We survey over 50 papers on scaling laws, and discuss differences in form, training setup, evaluation, and curve fitting, which may lead to significantly different conclusions. We also discuss significant under-reporting of details crucial to replicating the conclusion of these studies, and provide guidelines in the form of checklist aid researchers in reporting more complete details. In addition to discussing several prior replication studies in literature, we empirically demonstrate the fragility of this process, by systematically varying these choices on available checkpoints and models that we train from scratch. We choose to avoid overly prescriptive recommendations, because there is no known set of actions which can guarantee a good scaling law fit, but we make some suggestions based on patterns in our findings (Appendix (D) Despite our preliminary investigations, our understanding of which decisions may skew the results of a scaling law study is sparse, and defines the path for future work.
10

---

Published as a conference paper at ICLR 2025
This work discusses how lack of reproducibility and open-sourcing may be harmful for scaling laws research, given that the factors in a study setup that may change research conclusions vary widely.
Ethics Statement
Reproducibility Statement The model checkpoints and analysis code required to reproduce the results discussed in Section are at httttps://githbb wttp://a....ooaaaaaaaaaaaaaaaaa _aws
ACKNOWLEDGMENTS
We thank Aaron Defazio, Divyansh Pareck, Aditya Kusupati and Tim Althoff for their valuable feedback. We also acknowledge the computing resources and support from the Hyak supercomputer system at the University of Washington.
REFERENCES
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-moda language models. In International Conference on Machine Learning, pp. 263-279. PMLR, 2023.
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-22333 22312,, 2022.
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-182. PMLR, 2016
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm technical report. arXiv preprint arXiv:2305. 10403, 2023.
Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. Under- standing scaling laws for recommendation models. arXiv preprint arXiv:2208.084. 08849,88,. 2022.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy Sciences, 121(27), June 2024. ISSN 1091-6490. doi: 10. 10.1073/pnas.231 1878121. URL http: doi org/ /t.g//1..11 1073 2311878121
Michele Banko and Eric Brill. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th annual meeting of the Association for Computational Linguistics, pp-
26-33, 2001.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effeet of noise and architecture. In International Conference on Machine Learning, pp. 1466-1482. PMLR, 2022.
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102 2024.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.022 2029444 2024.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds. Jonathan Tow, Ben Wang, and Samuel Weinbach. 2pt-neox-20b: An open-source autoregressive language model, 2022. URL https: : arriv. orq/orggaa.... htt//bb//200 /2004 06745
11

---

Published as a conference paper at ICLR 2025
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. arXiv preprint (tti::2210.189911 2022.
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 2829, 2023.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference machine learning, pp. 4057-4086. PMLR, 2022.
lan Covert, Wenlong Ji, Tatsunori Hashimoto, and James Zou. Scaling laws for the value of individual data points in machine learning. arXiv preprint arXiv:2400.204456 2024.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pp- 7750-7774. PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07333, 2022.
Jasha Droppo and Oguz Elibol. Scaling laws for acoustic models. arXiv preprint arXiv:2106.09450 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprini arXiv:2407.2 1783, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232-5270, 2022.
Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation. arXiv preprint arXiv:2302.09653 2023.
Matthew J Filipovich, Alessandro Cappelli, Daniel Hesslow, and Julien Launay. Scaling laws beyond backpropagation. arXiv preprint arXiv::2200 10. 14593, 2022.
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connectee foundation models. arXiv preprint arXiv:2309.08522 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL htt.:ttss:://ett /ttp:ss//arxvvvvoo www.r/////////o..eee.oeeeeeeee/e abs/2101 00027
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Leo Gao, Tom Dupré Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https
//axxiv.org/ abs/ /200a22246..00 04093
Jonas Geiping, Micah Goldblum, Gowthami Somepalli, Ravid Shwartz-Ziv, Tom Goldstein, and Andrew Gordon Wilson. How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. arXiv preprint arXiv:2210.06441, 2022.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740. 2021.
Michel L Goldstein, Steven A Morris, and Gary G Yen. Problems with fitting the power-law distribution. The European Physical Journal B- B-Condensed Matter and Complex Systems, 41: 255-258, 2004.
12

---

Published as a conference paper at ICLR 2025
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5915-5922, 2021.
Sachin Goyal, Pratyush Maini, Zachary Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering- data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22702-22711, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752. 2023.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv preprint arXiv:2405. 18392, 2024.
Tatsunori Hashimoto. Model performance scaling with multiple data sources. International Conference on Machine Learming, pp. 4107-4116. PMLR, 2021.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Danny Hemaandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01290) 2021.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint 2tti::2225..0888 2022.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1112.00409, 2017.
Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301 MXXiv:2301.1344:: 13442, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint 22Xiv:2203. 15556, 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.066. 06395, 2024.
Peter Huber. Robust estimation of location parameter. In Breakthroughs in statistics: Methodology and distribution, pp. 492-518. Springer, 1992.
Maor Ivgi, Yair Carmon, and Jonathan Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. arXiv preprint arXiv:2202.06388, 2022.
Andy Jones. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03113, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. 2020.
T Kudo. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing, arXiv preprint arXiv:1808.062: 2018.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(D):5035228, 1989.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of latch-batch training. arXiv preprint arXiv: 162 2018.
13

---

Published as a conference paper at ICLR 2025
Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi Maeda, and Kohei Hayashi. A scaling law for syn-to-real transfer: How much is your pre-training effective?
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Oren Neumann and Claudius Gros. Scaling laws for multi-agent reinforcement learning model. arXiv preprint arXiv:2210.0084/ 2022.
OpenAL. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023.
Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. arXiv preprint arXiv:2406. 12907, 2024a.
Tim Pearce and Jinycop Song. Reconciling kaplan and chinchilla scaling laws, 2024b. URL
https: wwwp:/iwww.rrgaa//////oo//
12907
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023.
URL https: /ttpsi//aaxxvv.oo org/ abs 2ttrr///b//33060000/ 01116
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. httttss://aiiii.///a////
abs 220.22466.1.. 175577
Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Bjöm Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures. arXiv preprint arXiv:2403.17844 2024.
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-opimal scaling of language models. arXiv preprint UrXiv:2406.191460 2024.
Gabriele Prato, Simon Guiroy, Ethan Caballero, Irina Rish, and Sarath Chandar. Scaling laws for the few-shot adaptation of pre-trained image classifiers. arXiv preprint arXiv:21 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 12. 2021.
Colin Raffel Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across seales. arXiv preprint arXiv: 12673, 2019.
Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938, 2024.
Nikhil Sardana and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint 1.04442411.00448, 2023.
14

---

Published as a conference paper at ICLR 2025
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? arXiv preprint arXiv:234..15000 2023.
Noam Shazeer. Glu variants improve transformer, 2020, URL https://axxivoo//////
2002. 05202
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of -fxeeer-of-experts layer. arXiv preprint arXiv: :00i::/700.066500 2017.
Kyuyong Shin, Hanock Kwak, Su Young Kim, Max Nihlén Ramström, Jisu Jeong, Jung-Woo Ha, and Kyung-Min Kim. Scaling law for recommendation models: Towards general-purpor purposs representations. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 4596 4604, 2023.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19523-19536, 2022.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint X1v:22407.13623, 2024.
Yi Tay, Mostafa Dehghami, Samira Abnar, Hyung Won Chung. William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207. 10551, 2022.
Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint N.XY::2206.076222. 2022.
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.000 2233003666, 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22104-12113 2022.
Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan Firat. Examining scaling and transfer of language model architectures for machine translation. In International Conference on Machine Learning, pp. 26176-26192. PMLR, 2022.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 171000188,, 2017.
15

---

Published as a conference paper at ICLR 2025
A OUR MODEL TRAINING (S7)
We train a varicty of Transformer LMs, ranging in size from million to billion parameters, on Web (Penedo et al. 2024), tokenized with the GPT--Neo--22 tokenizer (Black al. 2022).
which a BPE tokenizer with vocabulary size of 50257, trained on the Pile (Gao et al. 2020)
All models were trained combination of NVIDIA GeForce RTX 2080 Ti, Quadro RTX 6000,
NVIDIA A40, and NVIDIA L40 GPUs. These transformers follow the standard architecture, and pre-layer RMSNorm (Kudo 2018), the SwiGLU activation function [Shazeer] 2020), and rotary positional embeddings (Su et al. 20244. We use batch size of 512 with sequence length of 2048. The learning rate is warmed up linearly over 50 steps the peak learning rate and then follows cosine decay to 10% of the peak. We use an Adam optimizer with 31 0.9, 32 0.95. We sweep over learning rates and data budgets each model scale. For our evaluation metric, we use perplexity on validation set of C4 (Raffel al.) 2020). See Table and details and hyperparameters, including data budget and learning rate.
and Table I0 for additional architecture
Hyperparameter
Value
Vocabulary size
50K
Batch Size
512
Sequence Length
2048
Attention Head Size
64
Learning Rate
Swept (0.1,2.33 -3
Feedforward Dimension
4 hidden dimension
LR Schedule
Cosine Decay
LR Warmup
50 steps 0.1 x Peak LR Adam (31, 32 0.9, 0.95)
End LR
Optimizer
Table 3: Hyperparameter details for models trained in 7
Model size
# layers
Hidden Dim
# Attn Heads
# Steps
12M
448
7
(100, 200, 250, 360, 500, 750, 1,000, 4,000)
17M
448
7
(100, 200, 250, 500, 750, 1,000, 1,250, 1,500)
25M
8
512
8
{250, 360, 500, 750. 1,000, 1,500, 2,000, 8, 8,000, 16,000)
35M
9
576
9
{200, 250, 360, 500, 750, 1,000, 1,250, 1,500, 4,000, 16,000)
50M
10
640
10
[250, 500, 750, 1,000, 1,250, 1 1,500.
1,800, 2,000, 2,500, 4,000, 16,000)
70M
12
704
11
(250, 500, 750, 1,000, 1,250, 1,500,
2,000, 2,500, 3,000, 5,000)
100M
14
768
12
{250, 500, 1,000. 1,500, 2,000,
2,500, 3,000, 4,000, 6,000, 12,000)
200M
18
960
15
(500, 1,500, 6,000, 9,000)
300M
19
1152
18
{4,000}
400M
20
1280
20
(3,000, 6,000)
IB
26
1792
28
{20,000)
Table 4: Architecture and Data budget details models trained in
16

---

Published as a conference paper at ICLR 2025
B FULL CHECKLIST
We define each category as follows:
Scaling Law Hypothesis: This specifies the form of the scaling law, that of the variables and parameters, and the relation between each.
Training Setup:
This specifies the exact training setup of each of the models trained to test the scaling law hypothesis.
Data Collection: Evaluating various checkpoints of our trained models to collect data points that will be used to fit scaling law the next stage.
Fitting Algorithm: Using the data points collected in the previous stage to optimize the scaling law hypothesis.
SCALING LAW REPRODUCILIBILITY CHECKLIST
Scaling Law Hypothesis ($3)
What is the form of the power law?
What are the variables related by (included in) the power law?
What are the parameters fit?
On what principles is this form derived?
Does this form make assumptions about how the variables related?
Training Setup
How many models trained?
which sizes?
On how much data each? On what data? Is any data repeated within the training for model?
How are model size, dataset size, compute hudget size counted? For example, are parameters of the model counted? Are any parameters excluded (e.g., embedding layers)?
How is compute cost calculated?
Are code/code snippets provided calculating these variables if applicable?
How are hyperparameters chosen (e.g., optimizer, learning rate schedule, batch size)? Do they change with scale?
What other settings must be decided (e.g., model width vs. depth)? Do they change with scale?
Is the training code open source?
Data Collection
Are the model checkpoints provided openly?
How many checkpoints per model are evaluated each scaling law? Which ones, if
What evaluation metric is used? On what dataset?
Are the raw evaluation metrics modified? Some examples include loss interpolation, centering around mean or scaling logarithmically.
If the above done, is code for modifying the metric provided?
Fitting Algorithm ($6)
What objective (loss) is used?
What algorithm is used fit the equation?
What hyperparameters are used for this algorithm?
How this algorithm initialized?
Are all datapoints collected used to fit the equations? For example, are outliers dropped? Are portions of the datapoints used fit different equations?
How the correctness the scaling considered? Extrapolation, Confidence Intervals,
Goodness Fit?
17

---

Published as a conference paper at ICLR 2025
C FULL SHEET
We provide an overview of all the papers surveyed Tables 6
7
8
and
Paper
Domain
Training Code?
Analysis Code?
Checkpuints?
Metric Scures?
Rnsenfeld al
2019
Vision, LM
N
N
N
Mikami et al.
Vision
Schaeller
2023
LM
N
N
N
Sariana Frankle
2023
LM
N
Sorscher
2072
Vision
N
N
Y
abutero et al
2022
LM
N
Y
LM
Y
Y
Leadinn mi.
2021
NMT
Y
Y
Y
Bansel
NMT
Hestuess
NMT. LM, Vision, Speech
N
N
Bi al. 21144
LM
21224
Vision
N
Geiping cl. 21722
Vision
LM
N
Ti cl
2124
LM
Y
Hashimoto
Küinh ot
2024
LM
Y
Attil at
2023
LM
N
2024a
LM
N
Lherit et 2125
4125
VLM
LM
Y
Atadduhnollsit
2022
LM, Vision
N
Gão | 2024
Mugimicholl et
2024
LM
[ite al
21..11111 2041
LM
Shin 24225
RecSys
Hernandez ot
2022
LM
Filipovien
LM
Neumaan Lrbs
20
RL
Y
Drogpoo
Speech
Honigha el
L.M. Vision, Video, VLM
Goval et 202-
Vision, VLM
2023
Multimodal LM
Kaplan et al 2020
LM
2012
NMT
Lag al. 2123
RL/MM
N
Frintar et
M12-
LM, Vision
N
Pralo al.
2021
Vision
20/4
LM
Y
N
Hermandez
cl
1202
LM
N
NLP
N
Tay et a.
2122
LM
N
N
LM
Jones 2021
RL.
Y
I
INN
7hai el
2023
Vision
Detmers Zeitlemoyer
2023
LM
N
Dibey 2(
2024
LM
Hollmann
2022
LM
N
Ardalant cLai 2022
RecSys
N
LM
Table 5: Details domain of experiments and availability of code by category for each paper surveyed.
18

---

Published as a conference paper at ICLR 2025
Pedeened Pockran
S Bir A X Ra) arpeaa AogAANA
Fiztemad Prokran
Mill (1-(2)
For
Grbe
100 [E)* 8J
Perlenand Focksan
Pectenanc Pocksan
Koti.
Put es
Pectonaac Pocktan
shilt sars aa a- (((evaal
Optaal Bptic
71441
I De
Me -e Ae mothor
Pertorman
150 181
1000g i +
Oul
12 ((e) ++]
BuB cat ".Lu4 old wirl al (Siggnl (M
Optinal
( (8)" (o.
Mi Iull <en 1nd (4) (H))
Bac De
(1)) Fod
Optinal Tederaato Prodecios
Peclemand
Quiau Keti. Fudociaoo
Table 6: Details on power law for each paper surveyed.
Dra
NA
sproady
21/0
28 Clac an os
3155
142
6.75
CL.P L1A- 2000 AGN
Mld
3-3
NA
1758 LUONN EMAL
01/0m 15%
364
176
1309 cale
Table 7: Details on training setup for each paper surveyed.
19

---

Published as a conference paper at ICLR 2025
End
34
96.
400 116
10
1.30
4/4
500
36
Bada
36
BLLCK
34a0la allad SI
355
(10
35
204
304
2000
55
35
155
30-56
Table 8: Details on data extraction for each paper surveyed.
Came-- Mathed
Lost Dijectise
Hyperpeenenn
Insilcaion
Am.Scolle Villdinc?
Lead Squarce Hearovnzee
Czares cIme
NGA
Sundhen
MDT
NA
NA.
KA
NA
NA
Rlleer Lows
Gld Search
Lees Squaes Regrosiin
MSLE
XIA
Gri Seatch ept.epptiiiii
Y
Rluber Lows
Y
Gl Searih
RMSE
Noe Ulvor Leas Squares
XA
Not eigeeetel peranctes
cameiti
Alaggal
Custre Los
Xaviet
Polyaoeaul Betroative (Quddaazc)
N.S.
5.5
Polonsenal Leai Squaree
MCSE Logoloo
XGA
2/A
N
Woighted Mapeusiea
NGA
26A
Leaal Scuaro Hegrovsiee
MOSE:
3.3
1.3505
Hluher
Sunth Sonth.ptnnii. di
Noet
Noce
26A
N/A
N.
NS
35
355
NS
NS
NS
35
35
NS
Cird Search
CTEE
NA
Soath optniii di
N/S
NS
X5
XS
N
Trun Rapio Reflactie alpetihhs Lend Square
Sects Low
CMA T5--1 liver Regrewion
1.7 Toos
Fland
Lagelon
Critree Towe
Y
XS
quacz
NA
Nck.
NA
NA
L-SEES, Lewe Squares
Eluter Tog loss
Rardeee Trisks frves Gil
L-3
Buter Log-on
Gnd Search optiiic al
Y
Table Details on optimization for each paper surveyed.
20

---

Published as a conference paper at ICLR 2025
D RECOMMENDATIONS
As seen in our analyses, many decisions in our checklist have a number of reasonable options, but those reasonable choices lead wide range of scaling law fits, and the observed variations do not follow any clear pattern. It is probable that variations would be even harder to predict when varying model architectures or other design decisions, removing the possibility of a universal set of best practices. However, it is certainly possible to determine that some scaling law fits are plausible or highly implausible, and to observe any recommendations can not guarantee good scaling law fir, we can make some more concrete recommendations based on these observations:
stability of the fitting procedure. With the caveat that following
Scaling Law Hypothesis
Fitting fewer scaling law parameters at time typically results in greater stability. In some cases, may be beneficial to decompose the scaling law fitting problem into two separate procedures. Examples of this approach are the IsoFL.P procedure from Hoffmann et al. (2022), as well as fitting first the relation between L and C, then finding the optimal and D for C, as seen in Porian et al. (2024).
Training Setup
The trained models should include a wide range of input variables settings. For example, when the input variables to the scaling law are L, N, D, the included models should include a wide range of N and D values for each C, or equivalently, should include wide range of D/N ratios. If the included settings do not include the true optimum, the procedure will struggle to fit to the optimum.
Sweeping for the optimal learning rate results in a less stable than fixing the learning rate. We hypothesize that this may be because the true optimal learning rate for each model and data budget size not any of the options we consider, and thus, each model varies in the difference between its true and approximate optimal learning rate. This may introduce additional noise to the data. Due to resource constraints, we are unable to fully test this hypothesis, and it may not hold at significantly larger scale, but recommend fixing the learning rate, or changing it according to, say, model size, according to a fixed formula.
Data Collection
Results across tasks or datasets should not be mixed. Neither performance predictions optimal D/N ratios are fixed across different evaluation settings for the same set of models.
Fitting Algorithm
Scaling law fitting is sensitive to initialization; most known optimization methods for scaling laws are only able, in practice, to the shift parameters near to their initialization values. Thus, dense search over initialization values is necessary. If there is a strong hypothesis guiding the choice of one specific initialization, such as a previously fit and validated scaling law, this will also limit the set of possible final scaling law parameter values.
Different losses emphasize the contribution of errors from certain datapoints. The chosen loss should be suited to the distribution of datapoints and sources of noise.
A simple grid search is unlikely to result in good scaling law Additionally, optimizers designed to fit linear relations may make assumptions about the distribution of errors and should not be used to power law.
D.I EXAMPLE CHECKLIST
We provide one possible of responses our checklist, reflective of some recommendations enumerated above, and loosely based on Hoffmann et al. (2022). These answers roughly correspond subset of the experiments in
21

---

Published as a conference paper at ICLR 2025
FITSIFITTING:
SCALING LAW REPRODUCIBILITY CHECKLIST
Scaling Law Hypothesis ($3)
What is the form of the power law? L(N, E
What are the variables related by (included in) the power law? N: the number of model parameters,
D: the number of data tokens, and L: the model's 's validation loss
What are the parameters to fit? A, B, E, o, 3
On what principles is this form derived? This is taken from Holfmann et al. 2022)) who hypothesize this form the basis of prior work in risk decomposition.
Does this form make assumptions about how the variables are related? This form inherently assumes that N and D do not have any interaction in their effect on the scaling of L. For some experiments,
the assumption 3 simplify optimization.
Training Setup ($4)
How many models are trained? Refer to Table 10
which sizes? Refer Table 10
On how much data each? On what data? Is any data repeated within the training for a model? Refer to Table 10.
How are model size, dataset size, compute budget size counted? For example, are parameters of the model counted? Are any parameters excluded (e.g., embedding layers)? How is compute cost calculated? We include the results inchuding and excluding embedding layers for both the total parameter count N and the total FLOP count C. We also include, for comparison, results using the estimate C
D.
Are code/code snippets provided for calculating these variables if applicable? https:/7gi Chub com/ hadasah/ www.sialllnaaaaeeeeee laws
How are hyperparameters chosen (e.g.. optimizer, learning rate schedule, batch size)? Do they change with scale? Most hyperparameters are chosen based on best practices in current literature: several are taken directly from the settings in MHHoffmman et al. (2022) For learning rate, conduct extensive hyperparaneter search across 2-3 orders of magnitude, multpplyng by 2-2.5, and then conduct training at 3 learning rates, including the optimum, for nearly all (N, D) configurations.
What other settings must be decided (c.g., model width depth)? Do they change with scale? Refer Table 10
Is the training code open source? Yes
Model size
# layers
Hidden Dim
# Alin Heads
# Steps
12M
5
448
7
(100, 200, 250, 360, 500, 750, 1,000, 4,000)
17M
7
448
(100, 200, 250, 500, 750, 1,000, 1,250, 1,500)
25M
8
512
8
(250, 360, 500, 750, 1,000, 1,500, 2,000, 8,000, 16,000)
35M
9
576
9
(200, 250, 360, 500, 750, 1,000, 1,250, 1,500, 4,000, 16,000)
50M
10
640
10
{250. 500. 750, 1,000, 1,250, 1,500, 1,800, 2,000, 2,500, 4,000, 16,000)
70M
12
704
(250, 500, 750, 1,000, 1,250, 1,500, 2,000, 2,500, 3,000, 5,000)
100M
14
768
12
{250, 500, 1,000, 1,500, 2,000,
2,500, 3,000, 4,000, 6,000, 12.000)
200M
18
960
15
(500, 1,500, 6,000, 9,000)
300M
19
1152
18
(4,000)
400M
20
1280
20
3.000, 6,000)
1B
26
1792
28
(20,000}
Table 10: Architecture and Data budget details for models trained in
22

---

Published as a conference paper at ICLR 2025
Data Collection (5
Are the model checkpoints provided openly? Yes,
https://qi thub. com/ www/coom/aacceeeceeoceccecccccca http://aaaaaaaaaaa Seaadaaahsscaanng.. scaiing laws
How many checkpoints per model evaluated fit each scaling law? Which ones, if so? Unless clearty denoted otherwise, one checkpoint per model evaluated: last checkpoint. By default, no mid-training checkpoints are used, i.e. from before the termination of the cosine learning rate schedules.
What evaluation metric is used? On what datasct? We use cross-entropy loss, measured held-out validation subset of the Common Crawl (Raffel et al. (020) dataset.
Are the evaluation metries modified? Some examples include loss interpolation, centering around mean scaling logarithmically. No.
If the above is done, is code for modifying the metric provided? Yes.
Fitting Algorithm ($6)
What objective (loss) used? try various loss objectives (1) log-Hubee loss, (2) MSE, (3) MAE and (4) Huber loss
What algorithm is used to fit the equation? Mainly L-BFGS, but also experiment with BFGS, non-line uin-linear least squares and grid search.
What hyperparameters are used for this algorithm? Thresholds of {le 4, 6, le 8}.
exact gradient
How is this algorithm initialized? We initialize with 4500 initializations similar to Hofimann et al. (2022)
Are datapoints collected used to fit equations? For example, are any outliers dropped? Are portions of the datapoints used to fit different equations? No outliers are dropped in general, but we do show some results on specific subsets of models. For example, we compare the result of a scaling law fit when using only models trained at peak learning rate of 1e 4 4e 4.
How is the correctness of the scaling law considered? Extrapolation, Confidence Intervals, Goodness of Fit? Currently, we do not evaluate the correctness beyond comparing results literature (Hoffmann el al. 2022 Kaplan al. 2020
23

---

Published as a conference paper at ICLR 2025
E FULL ANALYSIS PLOTS
10
NoTmam aphumbitt
Wefraan (pparssii
Capann onporte
NT:21
insprted
4650
100e
1.01
1.00
27
N: 20
10a
1290
10th
Taning compute (1LDP)
Hoflmann et al. 2022 Besirogla et
2024
rutirant Farm
Pulimaat Farm
Power law oottool para 10000
2012
OPT rest rewot))
Ca treacted)
Pontr love ostinal pura Nommees
2012
Hoftmann
Chinchi (eaattee)
Kap Japo tod)
Dr 5008
2014
son
308
D- 230
100
10y
D: 10ce
700
30a
2.19
12
10
1520
10a
10t0
10.4
10
10
1010
10
10th
120
109
1073
2014
10th
1010
Faining compute (FLOM
Vairing computs (FLON)
Porian ct al. 20244)
Porian et al. (004) (all checkpoints)
Hafframn
608
Hoffmann Ferm Hotfroen mn Marta-bee bebs loel (wekgrnst)
Io0tO0 jneigredd
sottronn Ni Net
10t
Power fam ossino prametes
2014
Chibeiil (masted) GPT uet report)
Insertiet
T08
Power faw optna prommtees
20th
Iispritel Chinchi (nsarted) G973 (ee en meoottt Usira jepaneei
By 800g
peparaci
128
10t
3:8
14
107
10c0
309
10*
107
2074
Training compute
Tririnng compute (FLOP)
Ours
Ours (all checkpoints)
(a) $3 $7.1
2024 (left) and effects of fitting to power law form used Approach of Hoffmann et al. (2022) with the variant used by
Using data from both Besiroglu
models (right), compare the
Muennighoff et al. (024)) which assumes that the exponents o, 3 are equal cquivalently, that N (C) and
DCC scale about linearly with each other. When using only the performance of final checkpoints from both
Besiroglu al. (2024) and (2022). experiments. taking this assertion results in law much closer to the our own models, we also show results when using IsoFLOP approach from Hoffmann et al, (2022). As we are using only results from final model checkpoints, the size of the data input to the IsoFLOP approach in this case is reduced.
reported by Hoffmann
24

---

Published as a conference paper at ICLR 2025
1014
Zo-3
20-3
Power low ontrrel lartnnecee
1011
Poes UB fo
ST
Power low octirel Laranecces
Foed 6e3
B 2000
Yeastes)
yeasses)
2013
bs (eaettt (msurte))
CPT2 nat reportal
Iespraec
2005
10t4
Gm nas en reporal
Pepaneal
2010
14 508
D:330
10B
10*
608
10"
10a
150 Tratring compute (FL:P))
Triining compute (TL.0P)
Ours
Ours checkpoints)
sweeping the learning rate. baseline, (1) we sweep at each (N, D) pair for the optimal learning rate over a range of values at most multiple 2 apart. Next, (2) we use learning rate of lc-3 for all N, the optimal for our billion parameter models, and do the same for all models across all N and D. Results vary dramatically across these settings. Somewhat surprisingly. using all learning rates results in very similar power sweeping the learning rate, whereas using fixed learning rate of 1e-3 or 4e-3 yields the lowest optimization loss or closest match to the Hoffmann et al. 2022) power laws, respectively.
(b) $4 7.2 With our models, we simulate effects
(3) 20-3 and (4) 4c-3, which is optimal for our 12 million parameter models. Lastly, learning rates the
25

---

Published as a conference paper at ICLR 2025
10u
qe 4EON
186N
20 12
(sparidd) inporte
1011
grii 2nd reserts
Lama rmpottd)
1000
47
1008
10a
16
10th
10ah
1017
10.7
1077
10
Taiiing compute SPLOP)
Hoffmann et al. x002 Besiroglu et al. (2024)
Almazeb
Almazes
Power 3e sptna 1000 7405
Power oe optinal 7ede's
DVN
07
70th
DW x-
roftrum Insprtedd Pazereed
Kapin Juzz
Rofmarn Iuporied
500n
1011
GPT ieppreett 0u4th
Lame 4228 Pepariet
1014
30: 10-10
20*
2007
2007
2000
50M
10a
307
2034
121>
1010
1520
10
20th
1014
10th
10at
1014
20tH
10
122
1099
2077
2014
10th
10M
Tririing (FLOP)
Training (T1.0))
Porian et al. 22044)
Porian et al. 2024) (all checkpoints)
1019
Almsser
10
Almsse
Nem 120M
1299
1014
1ii 420M
10
43gu
Power the rentinal parsinete5
108
30th
300
Hoffronn
Hoffronn
Kepkn heaeted)
Karinn yearoes!
Chexteto treseteal
Checte reearted)
ren repertal
Power epttmall
GP reat reportal
no0s
1014
100
10y
508
102
Taining compute
Tririnng compute (FLOP)
Ours
Ours (all checkpoints)
From all datasets, choose subsets with D) values which fit particular method one might have of setting up training. We fit with (1) all models, which for our dataset, ranges from 12 million to billion parameters, then with (2) only models of up to about 100 million parameters or (3) up to 400 million parameters. We also compare the effects of a higher or lower hypothesis about the optimal D N ratio, including (4) only models with D/N 18 (5) D/N 22. These ranges designed exclude D/N 20, the rule of thumb based on Hoffmann et al. (2022). The minimum or maximum D/N ratio tested does skew results; above 10a2 FLOPs, (4) and (5) lit to optimal ratios D/N < 18 and D/N >> 22, respectively. Removing our largest models in (2) also creates major shift in the predicted optimal D/N.
(c) $4 7.2
26

---

Published as a conference paper at ICLR 2025
K. A.Cejanbo arbul
N.C anbed
2005
C-Warbbd
Wearted
Power lan oothnol aUOO HONT
20:3
Putrom inperted
10
Patlnamn Inported
Kapen teaste3)
Karzn teas
1011
Chicct (resrted)
Tower faw opthmal
30
2009
papsrtss
01-47
10t4
07238
107
10*
430
1:0m
107
101
Porian et al. 2024
Porian al. 2024 (all checkpoints)
1014
Wnerbad
10t4
Power lem oothaa tormneeee
N.C.o utied
N. Ciz unbed
N: si
2014
CethD espronmatiin
10th
C-ER0 esprosmation
(easted)
Caa
00 308
Chacttla tremeredd am tos en repebb
OPtiz nsi en crnpeeaal
Power 0odmaa
Ieparesi
10th
pepaneei
90.3008
10t
238
307
508
30a
7074
107
20
15a0
10a
201
100
1071
1074
107
10
Tairing compute (FLOP)
Taarinng compute (FLOP)
Ours
Ours checkpoints)
(d) $ 47.2 With our own data and those from Porian et al. (2024), we power laws to the same sets of models, while varying the ways we count N and C. We compare (1) including embeddings, which is our baseline, with (2) excluding emheddings only in N, (3) excluding embeddings only in C, (4) excluding embeddings in both N and C. We also compare using the C 6.N D approximation. including embedding parameters. Throughout work, we calculate FLOPs in manner similar to Hoffmann al.) (2022). and we open source code for these calculations. With both datasets, the exclusion embeddings in FLOPs has very little impact on the final fit. Similarly, using the C = 6NJ D approximation has no visible impact. For the Porian et al. (2024) models, the exclusion of embedding parameters in the calculation of N results in scaling laws which differ substantially, with increasing divergences large scales.
27

---

Published as a conference paper at ICLR 2025
Real ene-ppiitt
1614
All choopeeia
30% buinig coen
mainng cone
N:
1.0
Captor oxpotes:
Oinecla ieported
1041
G712
2030
10
19a
100
Taming compute FFLD
Porian 2024) (all checkpoints)
Rral Enedgeeition
635
1071
26h minid dree
10% buinig coem Inparial
Ohiicilss ixperted
0m12 recerti
Dama 5 40bd regostad
.018
1.09
1014
1211
1220
1071
1077
3077
1.34
1.71
10"
Thming compute (71)P))
Ours (all checkpoints)
(c) $ $7.3 With models from Porian et al. 2024) and our own dataset, we compare (1) fitting power law using only the final checkpoint with (2) using all mid-training checkpoints (3) using all checkpoints, starting 10% through training. (4) the same, starting 20% through training, and (5) the same again, starting 50% through training. observe
(2)-5) consistently yields power laws more similar that reported by Hoffmann) et al. [2022), so we also repeat all analyses in Figures (a 2d Using mid-training checkpoints sometimes results in more stable fits which are similar to the Hoffmann al. (2022) scaling laws, but the effect is noisy and dependent other decisions
28

---

Published as a conference paper at ICLR 2025
Optintes fill find
1011
Search Gri.l Optiminn
Searh Optamies 2000
1013
inbsitn Chivchill
Pullmam inpprtted
Kanin pesolie))
N-1008
(reasted
Uarr
Power lsw
2014
0,388
1049
10tr
10m
100
104
1040
Hottmann et al. (022
Besiroglu et al. 2024
Opomaa Gnd
Opdintes
1344
Searh Optimize 1000
Searh 8-91 Optinies
10t
Sear drd Dutivibe
Soar ded Ortivibe 3300
mp au Bundam
Power fow aodnee LRTSINCIIS
Optrtees fandem
2201
sofhmon orpertie
2013
Hothnomn
202008
1011
Ouston impantes
Chandella Inspotuss
sdse
16:
3r20on
101h
Chextt teasrted)
Kerinn Jean ted)
not reperis
50.9008
2,30
208
10tt
425
D: 1008
30a
10*
2015
100
1001
1071 Tairinn compute (FUOP)
Taining compute (PLOP)
Porian al. (0024)
Porian 2024
(all checkpoints)
2007
10n
Optrres hul One
Seath Opdritt 1000
1023
Search Grd. Opttvibe
1515
1el skze Toen Shandall
Nardom
Power ba opiinal Nomeeeer
10
Optisec 200 Random
In from Charcealo
ripprtted
6 2008
Power lan SpSnN
Chinchlla Insprtect GP1-2 jet on masinn) Uaana resummst
1011
Chiechl Iraarted! GPT tet rewort) Lsve paparnas
40: 3108
2000
0:3
1339
20.
0:: 120
10'
1.24
2015
2074
1071
1091
1077
1074
1231
1014
2014
20tH
2019
2.27
1220
2071
2014
2079
10M
Training compute (ILO))
Tririnng compute (PLOP)
Ours
Ours (all checkpoints)
( $6. $7.4 We fit data from all 3 datasets to experiment with the initialization of parameters in the power law: We start with (1) optimizing every point in grid search of 6x6x5x5x5=4500 initializations (Hotfmann al. 2002)), (2) randomly sampling from only a single initialization in this grid, (3) searching for the lowest loss initialization point (Caballero et al. 2022). (4) randomly sampling 100 points, and (5) initializing with the coefficients found in Hoffmann et al. 2022). as Besiroglu et al. (2024) does. With the |Bsiirollu et al. (024) data, (5) yields fit nearly identical to that reported hy Hoffmann et al. (2022). although (1) results in the lowest. fitting loss. With the Porian al. (024) data, all approaches except (2) yield very similar fit, which gives recommended D/N ratio similar to that ofHoffmann et al. (2022). However, using our data, (2) optimizing over only the most optimal initialization yields the best match to the Hoffmann et al. (2022) power laws, followed by (5) initialization from the reported Hoffmann et al. 2022 scaling law parameters. Optimizing over the full grid yields the power law which diverges most from the Hoffmann al. 2022) law, suggesting the difficulty of optimizing over this space, and the presence of many local minima.
29

---

Published as a conference paper at ICLR 2025
Lgg Futer
2.01
1o Trruu Ooporaad)
Caplan Inporeed
2008
1.11
Uama 4550 rapattdd
1000
6016
1.00
1.0
1014
3011
120
1001
1077
1027
104
1071
16
Taiiigg compute (FLOPS
Hoffmann et al. 2022
Besirog lu et al. (2024
Huzer
Haber
MIE
10
Fower fan gothnol piro 1005
2012
Kavion yeasttsll
2008
1017
Kapin yea0
2018
Cainch trecctedd
Chiect (raorted)
oat repertal
Fower Powerbowe oitnol
10t1
regorta
10t
N:
1.0
207
20*
TOOM 2308
307
1000
Porian et al. 2024
Porian al. 2024) (all checkpoints)
M-300
M:
MIE
Power thm ontine par Minetto
2013
MAZ
3:
STT
Power law ozdine Darmmettrs
MAE
soffrann imporeed
Impored
Dr 2000
1011
2 rosera)
Usre 4358 hepomeci
M:238
D:: 7ce
10th
CP nol reperbl
Lama 4155 bepanesi
3008
1014
23h
2014
200
300
309
10a
2074
107
20
15a0
1020
2014
20
1677
1071
1074
107
10ah
Tairinn compute (FLOP)
Traiinng compute (FLOP)
Ours
Ours (all checkpoints)
(g) 6 $7.4 We fit power law to data from from all datasets, minimizing different objective functions: (1) the baseline log-Huber loss, (2) MSF, (3) MAF, and (4) the Huber loss. The resulting power laws are less disparate than when varying many of the other factors discussed above and generally fall near the power law parameters reported by Kaplan al. (2020) and Hoffmann et (2022), but this is still wide range of recommended optimal parameter counts for each compute budget. Overall, the loss function behavior not predictable, given the differences between loss functions when looking at the power laws resulting from these three sources of data.
30

---

Published as a conference paper at ICLR 2025
10at
10e
(gndient))
SPCS
10th
Nostmar Lend, Stuerto
Search
Caplar Importel
Cheiceta irpprted
gP122 2nd. resert)
Lama roported)
1.00
10
1.0
0007
1014
10th
1555
1011
2077
2677
104
1079
103
Taainng compute (LL0))
Hoffmann et al. 2022 ; Besirog lu et al. (2024
UPP5
UPPSS
LOnGE (grndann)
LIPeS Prigh
10th
UPCS 2rgh toi
Power bae ondnal parsmerrs
1022
G81 South
Power Now andral DaEamece
30
odd Soarch
N. 3008
rofirum Insprteed Pezeredd
Poftman Iispotiel Kapin
1011
10
5..5008
GPT mewnr
GPT3 reports
2014
Usva 4358 imponsai
Unma 4255 papanear
307
107
con
1007
301
10m
1023
2014
10.0
1000
Tairing
Porian et al. 2024
Porian al. (2024) (all checkpoints)
1007 858
10:4
UIPCS (guadiunk)
LBFGS
308
1014
UEPOS (gradiennt
LIFES 2rich
N: 27
D: 1008
Powerii spkinaa paramitee
20th
BP25
Power spénna partiettes
10th
Hoffram
Hoffmann
Kabin yearte3)
10TH
Karin year to3)
30a
Clich4 (resetee)
Chiccd (esertel)
nen reperta)
regiotal
300
500007
307
30*
200007
2307
3674
1079
30
301
3077
1074
107
10a
Taatinng compute (FLOP)
Triiinn compute (FLOP)
Ours
Ours (all checkpoints)
(h) $6 $7.4 a power law to data from all 3 datasets using various optimizers, beginning with the original (1) L-BGGS. L-BFGS and BFGS implementations have an carly stopping mechanism, which conditions on the stability of the solution between optimization steps. We set this threshold for L-BFGS to a (2) higher value le-4 (stopping earlier). We found that using any higher or lower values resulted in the same solutions or instability for all three datasets. L-BFGS and BFGS also have the option use the true gradient the loss, instead of an estimate, which is default,
figure, we include this setting, (3) using true gradient L-BFGS. We compare these L-BFGS settings to (4) BFGS. We test the same tolerance value and gradient settings, and find that none of these options change the outcome of BFGS in our analysis, and omit them from this figure for legibility. Finally, we compare to (5) non-linear least squares and (6) pure grid search, using grid 5 times more dense along each axis match the runtime L-BFGS. Many these optimizers do converge to similar solutions, but this depends on the data, and the settings which diverge from this majority do not follow any immediately evident pattern.
used initialization with other optimizers. This density is chosen approximately
31

---

Published as a conference paper at ICLR 2025
Figure 2: ($7) We study the effects of various decisions in the fitting of a power law, as outlined in our checklist (Appendix B) and detailed in $3 (6 For each set of analyses, the scaling laws found by (Kaplan et al. 2020) and (Hoffmann et al. 2022) for comparison. We also include markers indicating existing models for comparison purposes: Llama 3 405B (Dubey al. 2024), the Chinchilla model (Hoffmann et al. 2022), and an estimate of the 1.5B GPT-2 model (Radford et al. 2019), for which know details of the dataset storage size and word count, but which we estimate at 100B. We additionally annotate, at the compute budget C' for each of these reference points, the maximum and minimum predicted (i.e. extrapolated) optimal model parameter count N Nopt and data budget Dopt from the fitted power laws. We use a thicker, solid line for the method each plot which achieves the lowest optimization loss, with the exception of the plots comparing power law form, those comparing loss functions and those comparing optimizers, for which this would be nonsensical. We find overall, throughout our analyses, that all the decisions explore have impact the final fit of the power law, supporting conclusion that more thorough reporting of these decisions is critical for scaling law reproducibility.
exact count of data BPE tokens,
32


Reviews:
[{"review_id": "ThAHe6Mxoz", "reviewer": "Reviewer_PhzZ", "summary": "This paper surveyed/ more than 50 papers about the scaling law of language models. Authors discussed different aspects of scaling law including fitting forms, model training, data extraction, and fitting optimization. Based on that, authors provided a checklist, which helps to transparent settings for reproducible results in future research. Experiments also were conducted to verify their replication and analyses.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper has several strengths:\n- This paper considers a timely topic, the scaling law of language models. Understanding this topic will help to effectively train LLMs, avoiding resource overuse. \n- Authors discussed the discrepancies in experiment settings of different papers and empirically verified it. Results are aligned with previous works.\n- Authors open-sourced their code to reproduce results which benefits the community since the source code is usually absent from previous papers.", "weaknesses": "Despite these strengths, this paper has several weaknesses:\n- The scale of the model in experiments is not big enough. Authors consider only models with less than 400M params, ignoring the existence of larger models with billions of params. \n- The writing in some parts of the main paper causes confusion. E.g., Section 5 is about data extraction after training, I was confused by which kind of data could be extracted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k8zYkpPcjc", "reviewer": "Reviewer_vpGv", "summary": "The authors survey a large corpus of papers that involve scaling laws, and find that many papers underreport necessary details for reproducibility, which they demonstrate with experiments that demonstrate a large variability depending on those exact choices of details. They propose a checklist for authors to consider when publishing scaling laws.", "questions": "I have already listed direct questions in the section above, and I would be open to discuss this in the rebuttal. I hope the authors see the comments to be constructive, and can clarify or improve the distinctive value of the paper.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper gives a good overview of many papers on scaling laws, and nicely categorizes the important steps: functional form, training setup, data(points) extraction, curve fitting. The checklist provides a clear way of reproducibility and quality assessment of scaling experiments. I think the topic of scaling law studies is important and relevant, and the writing is clear.", "weaknesses": "The main concern for me is the following: what is the main goal the authors are trying to convey? To me, there are two obvious takeaways, which is 1) changes to the scaling law setup can change the results drastically, and 2) previous papers very much underreport crucial details. However, both of these things are rather clear already to the community and also illustrated by published papers: for example, point 1) is shown by Porian et al., and point 2) is a broader critique of reproducibility problems, which (unfortunately) is a generic problem. I do not see a clear and actionable interpretation beyond that. For instance, how do the different choices of fitting actually affect the scaling laws? (The assessment is mostly just “the results vary dramatically” — but how?) What should I as a researcher now do for my future scaling studies, having read your paper, beyond using the checklist? Are there clearly ‘wrong’ or ‘right’ choices? Was there a most predictive scaling law (e.g. when you leave out some experiments as a validation set)?\n\nTo be clear, I very much believe there is merit in a survey or pointing out these problems; as it stands, however, the paper is foremost “just” a survey, and I am not convinced this merits publishing at the conference.\n\nSome additional comments: \n * The paper template says ICLR 2024\n * The Figures are unfortunately of low quality (very pixelated), especially considering the fact that it’s natural to zoom in to compare the many lines and details. I suggest the authors include the pdf forms for proper rendering.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hOh13BST2C", "reviewer": "Reviewer_kn4j", "summary": "This work revisits scaling laws and factors influencing the results reported in recent papers. The paper takes a \"review approach\", outlining some of the debates in recent literature as well as common strategies. To draw conclusions and emphasize their points, the last pages are dedicated to an in-depth analysis by the authors on small to moderate size transformers.", "questions": "-", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I really like this paper and think it is of great value for the community to \"recap\" scaling law results and provide a critical discussion, complemented by experiments showing which factors matter when choosing a scaling law. It was really a pleasant read, quality of writing is good and the motivation clear.", "weaknesses": "One could maybe claim the paper is not too constructive, as it shows that choices (optimizer, fitting method, lr annealing, data) matter when fitting a scaling law: there is no correct answer. However, this conclusion demystifies the topic, which I like very much: there is no magic, just common choices and \"usual\" results. This said, there are a couple of very minor points.\n\n1) Proposing a checklist is helpful, but, as the authors themselves seem to hint, the number of factors to account for is potentially infinite. What about Adam beta2? What about weight decay? What about hybrid algorithms? What about qk norm and new tricks? The reality this paper points out is that, indeed, such choices matter, and I do not think any checklist can be conclusive.\n\n2) section 7.1: why did you decide to set alpha=beta?\n\n3) The paper lacks a bit of conclusions: what should researchers do? should we trust scaling laws? what are the things that hold true despite changing the setting? Is there some practical rule for scaling that holds approximately in your experiments? (would have been interesting alpha and beta)\n\ntypo spot: \"was was\" in the abstract, repetition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "60QgKKfgdJ", "reviewer": "Reviewer_pBpK", "summary": "The paper presents a survey on the fitting of scaling laws, and argue that current practices are lacking in scientific rigor. Apart from an extensive survey, the authors presents a reproducibility checklist, and compare 51 papers to this checklist. They generally find that important details are underreported, e.g. the method to calculate model parameters might not be given. They also provide a replication study of Hoffman, using data extracted from the paper PDF and data they’ve collected themselves. Here they find that subtle choices in the curve fitting can result in significantly different conclusions.", "questions": "- could you provide explicit recommendation regarding how to perform the curve fitting? I think this is different from a checklist which allows reproduction. \n- Could you expand section 7?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Scaling laws is an important topic, and scientific rigor here can benefit the research community.\n- The authors illustrate how subtle choices in the curve fitting can cause significant results\n- Section 7 is great.", "weaknesses": "- Significant parts of the papers are dedicated to a survey. I’m not sure survey papers are the right fit for ICLR main track.\n- There are not so many empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)