Reconciling Kaplan and Chinchilla Scaling Laws
Authors: Tim Pearce
Jinyeop Song
Venue: Accepted by TMLR ()

Abstract: Kaplan and Chinchilla studied the scaling behavior of transformers trained on next-token language prediction. These studies produced different estimates for how the number of parameters ($N$) and training tokens ($D$) should be set to achieve the lowest possible loss for a given compute budget ($C$). Kaplan: $N_\text{optimal} \propto C^{0.73}$, Chinchilla: $N_\text{optimal} \propto C^{0.50}$. This paper finds that much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale. Simulating the Chinchilla study under these conditions produces biased scaling coefficients close to Kaplan's. Hence, this paper reaffirms Chinchilla's scaling coefficients, by explaining the primary cause of Kaplan's original overestimation. As a second contribution, the paper explains differences in the reported relationships between loss and compute. These findings lead us to recommend that future scaling studies use total parameters and compute.

Published in Transactions on Machine Learning Research (11/2024)
Reconciling Kaplan and Chinchilla Scaling Laws
Tim Pearce Microsoft Reseorch
Jinyeop Song MIT
Reviewed on OpenReview: https: // openreveew.eee.... formm?? id= NLOoLyuUUF
Abstract
Kaplan et al. 2020) ('Kaplan") and Hofimann al. (2022 ('Chinchilla') studied the scaling behavior of transformers trained next-token language prediction. These studies pro- duced different estimates for how the number of parameters (N) and training tokens (D) should be set to achieve the lowest possible loss for a given compute budget (C). Kaplan: Noptimal x CO.73, Chinchilla: Noptimal xx CO.50 This paper finds that much of this discrep- ancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale. Simulating the Chinchilla study under these conditions produces biased scaling coefficients close to Kaplan's. Hence, this paper reaffirms Chinchilla's scaling coefficients, by explaining the primary cause of Ka- plan's ss original overestimation. As second contribution, the paper explains differences in the reported relationships between loss and compute. These findings lead us to recommend that future scaling studies use total parameters and compute.
Find compute optimal Fruntieria in terms of total parameters Ny and compete Chinchilla
Find power law staling osefficiento 0.51, close Chinchilla's 0.50
Start from fitted mode Chinchilla's 's training curves
182
2085
Loss(Nr, D)
1.82
N0.35
37
Generate tuining for model used in Kaplan's is suudy (Ik 1.58 params))
Nr C
Find compute optimal Frontier in non-embedding paramelurs Nis compute Kaplan
Find local power low scaling coefficient.o 0.78, Kaplas's
NRp 2 C1
Figure 1: Overview of the approach used to reconcile the two studies.
1 Introduction
('Chinchilla') provided two influential studies measuring the impact of scale in large language models (LLMs). Both informed large-scale efforts on how to trade off model parameters (N) and training tokens (D) for given compute budget (C), but with conflicting advice. Kaplan's finding that Noptimal x 0.73 Doptimal o 0.22 led to the conclusion that "big models may more important than big data ", and LLMs trained in the ensuing years committed more resources to
Kaplan al. (2020) ('Kaplan)) and Hoffmann et al. (2022)
Code analysis:
Chinchilla Seaa___sa________ee Laws

---

Published in Transactions on Machine Learning Research (11/2024)
model size and less to data size. The subsequent Chinchilla study found Noptimal & CO.50 Doptimal CO.50 leading to their main thesis "for many current LLMs, smaller models should have been trinnd on more tokens to achieve the most performant model", sparking a trend towards LLMs of more modest model sizes being trained on more data.
What. was the cause of the difference in these scaling coefficient estimates that led to vast amounts of compute (plus emissions and finances) being used inefficiently? There have been suggestions that differences could be explained by different optimization schemes (Hoffmann et al., 2022) datasets (Bi et al. 2024). This note finds these suggestions incomplete, and offers simple alternative explanation; much of the discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale.
We find that this methodological difference also plays a part in differences in the reported relationship between compute and loss. Kaplan: Ioptimal x C0.067, Chinchilla: Loptimal E CO.178
Concretely, this paper offers the following contributions.
We develop an analytical approach comparing the scaling relationships reported in Kaplan and Chinchilla (Section 3). This approach finds that Kaplan's reported relationship is locally consistent with Chinchillass, s, if non-embedding parameters are used, and al smaller scale.
Secondly, we study the reported relationships between compute and loss (Section 5 embedding parameters and smaller scale models are the cause of Kaplan's biased estimate, combined with the absence of an offset term in their compule-loss form.
Again non-
We recommend that going forward the scaling community measures total parameters and total compute, and uses an offset in the compute-loss relationship.
2
Preliminaries
This section introduces some background knowledge and definitions (Section 2.1 h outlines the analysis
approach for our main result (Section 2.2 and documents assumptions (Section 2.3
2.1
Set Up
Neural scaling laws. Kaplan et al. (2020) & Hoffmann et al. (202)) investigated empirically modeling the relationships between the number of parameters (N), training tokens (D)) training compute (C) and loss (L) in transformers for language modeling. The main functional form of relationship considered was power law, y ax², which is widely used throughout the sciences to describe relationship between two quantities (x & y) that holds over many orders of magnitude, e.g. (Kello et al. 2010).
Definition of N & C. One difference between the two studies the definition of N & C. Kaplan studied relationships in terms of non-embedding parameters (Nig) and non-embeddding compute (Cie), excluding the contributions of the embedding layers for the vocabulary and position indices (Ne) By contrast, Chinchilla studied total parameters (Nr) and total compute (Cr). We define,
Ny = N2 + N, NE = (h + v)d,
(1)
(2)
where d is the dimension of the transformer residual stream, is vocab size, h is context length (only included when positional embeddings are learned). Using the common approximation for training compute FLOPs C = GND (a factor of covers a forward and backward pass), we define total and non-embedding compute,
CT 6N, 6(NE N,E)
(3)
C GNV
(4)
2

---

Published in Transactions on Machine Learning Research (11/2024)
Compute optimality. This definition of compute C 6ND suggests direct trade off between parameters and training tokens for given compute budget. The two studies focus on 'compute optimal' configurations of these quantities. That is, for a given compute budget, the parameters and training tokens that lead to the lowest possible loss. For total paramcters this is written (using to signify 'optimal'),
argmin L(NT, Cr). Ny s.t. G,
(5)
Given this notation, the estimated scaling coefficients can be more precisely written,
Kaplan: Ni AE
(6)
Chinchilla: Ni oc CO.50
(7)
(Note that whilst this work focuses on the scaling coefficient for parameters, by subscribing to 6ND the data coefficient is implied; N C = C/D Ca = D x C²-a.)
Functional form. An important functional form relating Nr, D, L used in Chinchilla is given by,
No No D3
De
L(NT,)) D)
+
E,
(8)
where Ne: De,a, B 0 are empirically fitted constants and E the 'irreducible' loss inherent in language. This form conveniently gives rise power law relationships; N₃ x Of with a a+ Dj x C3 with
o3 0433 and Lip E CT with 7 o-B
There are two possible specifications from the constants in Eq. those originally reported in Chinchilla, and those reported in a re-analysis conducted by Besiroglu et al. (0024) claiming to correct minor errors in the fitting procedure. Our work reports results with both specifications.
Chinchilla specification: No 406.4, De 410.7, 0.3392, 3 0.2849, E 1.693 Nj CO.46
(9) Epoch AT specification: N₂ = 482.0, De 2085.43. a 0.3478, 3 0.3658, E 1.817 = N X CT
(10)
2.2 Analysis Overview
Our analysis uses information and data from the Chinchilla and Kaplan studies to estimate the scaling laws that would emerge the Chinchilla relationship had been expressed in terms of N E & C,., and this had been done over the smaller model sizes used in Kaplan, as sumunarized in Figure [
We will see that for large Nr, NE becomes negligible portion of the model's parameters and compute cost. Hence in the large parameter regime the two coefficients directly conflict with each other. At smaller values of NT, Nø is not negligible (this is the regime considered in Kaplan's 's study 768 to 1.5B parameters). We find that at the smaller end of this range, the relationship between N & C is not in fact a power law. However, fitting a "local" power law at this small scale, produces a coefficient that is close to Kaplan's, and hence roughly reconciles these two results.
Our approach Section 3 is broken down as follows.
Step 1. Fit suitable function predicting Mi from Nr.
Step 2. Incorporate this function into a model predicting loss in terms of Ny & Cr.
Step 3. Analytically derive the relationship between NE & CVE-
Simulate synthetic data from the Chinchilla loss model over the model sizes used Kaplan. Fit local power law for No in terms of CAB-
Step 4.
3

---

Published in Transactions on Machine Learning Research (11/2024)
Section experimenttllyyyyy verifies analysis by training set of language models at tiny scale and conducting scaling law analyses under various settings. Simply changing the basis Nr to N, produces coefficients inline with Chinchilla and Kaplan respectively, while multiple token budgets and decay schedules does not.
Section 3 presents a second, related contribution. We reconcile differences the relationship between loss and compute proposed by the two studies. We leverage similar analysis as above, but Step 3 & 4 are now performed with respect the relationship between the optimal loss L and compute CBB- So again, we begin from Chinchilla data, and correct for the exclusion of embedding parameters and compute, combined with the smaller model sizes used in Kaplan's study, and additionally differing choice of fitting function. Through these corrections, we are able to roughly recover Kaplan's compute-loss coefficient, and hence reconcile the two studies.
2.3 Assumptions
For transparency, we list the assumptions and approximations made in our analysis.
We assume CEE = 6.N and Cr D.
We assume a fixed functional form between total and non-embedding parameters in Eq. and fit w empirically using Chinchilla model configurations.
We assumee a fixed functional form between loss, total parameters and training data given by Eq. We report results using both the Chinchilla (Eq. 9) and Epoch AI (Eq. [0] fitted constants.
We approximate Kaplan's models with 20 logarithmically spaced model sizes from 0.79k 1.58B non-embedding parameters.
3 Analysis: Compute-Parameteo Scaling Coefficient
This Section presents our main analysis. We show that a local scaling coefficient of 0.74 to 0.78 (close Kaplan's 0.73) can arise when computed in terms of non-embedding parameters in the amall-parameter regime, whilst being consistent with Chinchilla's coefficient.
Step 1. Fit suitable function predicting NiE from Ny.
We require a suitable function relating non-embedding and total parameters. We propose to use the form
Ny = Ni + WN E
(11)
for some constant w > 0. Aside from having several nice properties (strictly increasing and lini, Ny = N E can be motivated from both the Kaplan and Chinchilla study.
Kaplan perspective. Consider Kaplan's method for parameter counting.
NT 12 + Ne:
(12)
where | is number of layers. Whilst Kaplan do not list their model configurations, they do study varying aspect ratio A d/l for a fixed size model. They find that models of a given size perform similarly over a range of aspect ratios, and this not affected by model scale (their Figure 5). Hence, we could assume a sizing scheme with fixed aspect ratio (A 40 appears sensible from their plots). Assuming this sizing allows state (with ( d/A in Eq. 12
12 - d + N
NT A
(13)
"Proof. Nx = Ni. + wA/
= = Examining the r.h.s., lim + WN- =1, hence we
(8
conclude the l.h.s 1o NiE/Nr = No Ny.
4

---

Published in Transactions on Machine Learning Research (11/2024)
Observing that N. (12/A)d² d= (N)n(A/12)) and combining with NE (v h)d,
1/3
= + (U+
A
N N
(14)
12
EE
11 with = (u+ h) (4) 1/3
This is the same form as Eq.
10
Chichilla modei contixs
Nic
Nr 47491M!'
10"
Nus. www.mbliiiiii prrmnctes
10
10
16
10
10
10
10
10
panmetiors
Figure 2: Total parameter count vs. non-embedding parameter count for the suite of models sizes used the Chinchilla study, along with fitted approximation. Note the curvature model sizes below around 200M parameters.
Chinchilla perspective. We empirically fit function N₂ Ni + wN (note the learnable exponent) the Chinchilla model configurations listed in Table A9 Hoffmann et al. (2022) for range of Nr (44M to 16B). We compute Ng from Eq. [ using the reported vocab size of 32,000, but ignore the context length 2,048 since Chinchilla used non-learnable position embeddings (though their inclusion effects coefficients only slightly).
Figure shows the configurations and relationship from model fitted with numpy's polyfit, which produces coefficients, w = 47491 & = 0.34. The exponent has come out close to 1/3, and an implied aspect ratio A 39.2 (inferred from w). Hence, this further supports the form in Eq. I1
Step 2. Incorporate this function into a model predicting loss in terms of Nv 6 Cr.
Recall that whilst interested in the dependence of N Cr, this arises only via their mutual relationship with loss
N7 argmin L(Nr, Cr). Ny s.t. Cr=
(15)
In order analytically study their scaling relationship, we need an analytical form of loss, for which we use the functional form in the Chinchilla study. Taking Eq. & and using Cr. 6,,
N No
Dr + E. (Cr/6NT)
L(Nr, Cr) =
+
(16)
By differentiating Eq. 16 wrt Ny and setting to zero, then rearranging in terms of Ny we find
( No
Cr
N
simply N Cafa.
(17)
B De

---

Published in Transactions on Machine Learning Research (11/2024)
We now modify Eq. 16 to be in terms of non-embedding parameters and compute. Note whilst Ny requires Eq. 11 from step 1, the second term avoids this as D = CT /6NT Cig/NN
No
Do
LINE CGB)= (Ne + WN (C +E
(18)
Step 3. Analytically derive the relationship between NE 8 C.E.
To find the relationship between NE & CiE we take the derivative of Eq- rearrange,
18
wrt NiE set to and
6 (Ni + ME //5) 4 (Nie (N tNNe 1/3)
4a
3 De
1
(19)
a N
This shows that in general the relationship between NE & C&E is not a power law. However, we can consider a "local" power law approximation. That is, for some particular value Nig; there some constant g giving a first order approximation (denoted by 2) MI x C.E. where g is defined
1 dloo((CiE) =
1 N 2/3
N 2/3
2
(20)
g (No
B No
2/3
+
B
No
2/3
+ w
Working is given in Appendix A.1 Figure
3 plots Eq.
19 &
20
using coefficients for a, 3, Ne, Do from the
Epoch AI specification, and w = 47491. There are three phascs.
At small scale, limn; 1 +0 a/3+8
NE X C
B
At large scale, limn; 0o 9
1 at2 ME X Cars
in the Ny case in Eq. 17
There is also transition phase, where g briefly increases. This happens in between the two limits, when 2/2/3 is of the same order as w. Indeed exactly the point N2/3 w, we have Nr EE
E
N + wN 1/3 = Nr = 2N or 50/50 split between embedding and non-embedding parameters.
In Figure 3 we see this transition region occurs around this point; N = 3/2 47491 3/2 = 10
Step Simulate synthetic data from the Chinchilla loss model over the model sizes used Kaplan, Fil a local power law for NC in terms of C.F.
By reading g off Figure 3. could estimate local power law and hence scaling coefficient for given value of NE However, it's not clear what Ni point value is representative of the Kaplan study. We opt for more faithful estimation procedure, generating synthetic training curves from Eq. 18 across the range of model sizes used in Kaplan, and fit coefficients using models falling on the compute efficient frontier. This will also verify our analytic expression for N F & CAE in Eq. 19
Figure 4 shows the synthetic training curves generated. We simulated 20 models with N ranging from 790 parameters to 1.58B (Kaplan reports using model sizes "ranging in size from 768 to 1.5 billion non- embedding parameters"), "). For other constants in Eq. 18 we adopt the Epoch AI specification (Eq. 10 and = 47491, though we report final results for the Chinchilla specification (Eq. g also.
Main result. Figure 5 shows the estimated scaling coefficient when fitting a power law to the compute optimal frontier (Chinchilla's Method 1) produced by these synthetic training curves. This marks our main
Proof.. As No + O. we can ignore No terms and 1/g = (1/9) (a 1)/38 a/38. Prroff. As No - on, we ignore terms and 1/g (1/9) + (a + 1//88=111///

---

Published in Transactions on Machine Learning Research (11/2024)
MIE
BMB
10
91
10
10"
10
10
16"
16a
16
10"
Cis
Figure 3: Visualization of Eq.
19
20
using the Epoch Al specification.
result beginning from a model taken from Chinchilla's study, and modifying two aspects to align with Kaplan's study (Nr Mi small model sizes 0.79k 1.58B parameters), find local scaling coefficients,
Epoch AI specification: NE 2 QVE Chinchilla specification: Me X CEE
(21)
(22)
which are close to the Kaplan cocfficient of 0.73. Hence, this shows that the Chinchilla coefficient is roughly consistent with Kaplan's coefficient, given these two adjustments. This constitutes the paper's main result, reconciling these two apparently conflicting results.
4 Experiments: Compute-Paramette Parameter Scaling Coefficient
We provide brief experiments verifying that our claims hold for models trained small scale (millions of parameters).
Experiment 1. Firstly, we verify whether scaling coefficients come out close to Chinchilla's and Kaplan's when using N7 and NE respectively.
We trained five models of sizes, Nr E [0.8M, 1,,,, 2.1M, 3.3M, 4.6M] on the BookCorpus dataset. We used the GPT-2 tokenizer with vocab size of 50,257, and a context length of 16 (whilst much smaller than typical, our experiments suggest scaling coefficients are not affected by context length). Chinchilla's 's Method 1 was G.
used to fit. scaling coefficients, with the approximation
Models were trained for updates E [4000, 4000, 4000, 8000, 8000], batchsize was 65,536 tokens per update, for total training tokens D E [262M, 262M,, 262M, 524M, 524M). The best learning rate for each model size was chosen E [0.001, 0.005, 0.01, 0.05) and no annealing was applied.
Result 1. Table shows that when coefficients are fitted to Nr, we find Ny x C and for Ny we find NiE x CO.74 These match closcly with the Chinchilla and Kaplan coefficients.
Experiment 2. We provide ablation of optimization schemes demonstrating that using multiple training budgets per model affects coefficients only marginally (counter to Chinchilla's explanation).
Scheme 1. A single learning rate of 0.001 set for all models. A single model trained per size, and no annealing applied.
Scheme The best learning rate is chosen per model. single model trained per size, and no annealing applied. (As in Ny Nog comparison.)

---

Published in Transactions on Machine Learning Research (11/2024)
lig 0.79K
Compute efficient frontier
Mig 1.7k
Mip
Mr 7.85K
lig 84k
Mk 36
Mi 17.53K
10'
10
lig m 0.360
lee 0.7754
1.6M1
Loos
3M533
Lows
7.577
My MMb.2.
11 74.76M
Byy m 344.27M
lbe 1584.99M
10'
10'
10"
10
10
10
10"
10
CiE
Figure 4: Synthetic training curves from Eq. 18 fitted to the Chinchilla data using the Epoch Al specification. models matching Kaplan's size range. Left in terms of
Curves are generated for 20 logarithmically-spaeee training tokens, right. in terms of non-embedding compute, as used in the Kaplan study. Hence the right plot can be viewed as Chinchilla's 's loss curves, adjusted to match Kaplan's 's model sizes and non-embedding measurements.
10"
Compute offeieni frentier
Analyccal Ni: vs. Cit
(ogth( (
Large Nig rogime Gr log(N",) = 0.51lopiC,.) - 3.16
10
10a
10
16"
Ci
Figure 5: Using the synthetic training curves generated in Figure 4 we empirically fit the frontier of compute efficient models using Chinchilla's Method 1. This gives our main result; synthetic training curves generated from Chinchilla's study, adjusted for model sizes and non-embedding compute, produce a local scaling coefficient N x CO.78 close to Kaplan's reported coefficient of 0.73. The analytical function from Eq. CE is also verified.
Scheme 3. The best learning rate is chosen per model. A single model trained per size, and cosine annealing applied at the update budget. (Kaplan study used this.)

---

Published in Transactions on Machine Learning Research (11/2024)
Scheme 4. The best learning rate is chosen per model. Six models trained per size at different budgets E [0.25 0.5D, 0.75/, 1.00, 1.5D, 2.0D]. and cosine annealing applied. (Chinchilla study used this.)
Result 2. Table shows that optimization scheme has a smaller impact on scaling coefficients than switching from Nr to N.E. Using single set of models with no anncaling (schome 2) produces the same coefficients as using the more computationalyy expensive scheme 4. Counter to Chinchilla's comment that moving from Kaplan's scheme 3 scheme would reduce the scaling coefficient, our experiment suggests the opposite is the case, increasing from 0.46 to 0.49. This might explain our slight overestimation of the scaling coefficients in Eq. 21 & 22
Table 1: Comparison of different scaling coefficients from our experiments. Note that the change moving from Nr to Ni has a much larger effect than moving between optimization schemes.
Experiment
where Noptimal C
where Doptimal & C*
Chinchilla, N/ Kaplan, N
0.50
0.50
0.73
0.27
Ablating Ny vs NiF
Ours, NT & Cr Ours, ME & CVE
0.49
0.51
0.74
0.26
Ablating optimization scheme
Ours, Nr, scheme 1, single Irate, no anneal
0.58
0.42
Ours, Nr, scheme 2, best Irate, anneal Ours, Nr, scheme 3, best Irate, single-cosiue anneal
0.49
0.51
0.46
0.54
Ours, Nr, scheme 4, best Irate, multi-cosine anneal
0.49
0.51
5 Analysis: Compute-Loss Scaling Coefficient
As well as analyzing the compute-optinal parameter scaling, Kaplan and Chinchilla also described the scaling relationship between compute and loss, assuming paramcters were scaled optimally. This optimal loss was again given in terms of non-embedding compute by Kaplan, and lotal compute by Chinchilla,
LE = min s.t. CLEENNED ED (MME,,EE)
(23)
I = min L(Nr,C,, Cr). s.t. Cr=
(24)
Concretely, the two studies reported the following forms and coefficients linking optimal loss and compule.
Kaplan compute-loss form: Li C
(25)
Co
Kaplan compute-loss fit: Li x C. 0.057
(26)
Chinchilla compute-loss form: Li =
+ E
(27)
Co
Chinchilla compute-loss fit, Epoch AI spec: L E CT
0.178
(28)
Chinchilla spec: L E CT 0.115
(29)
(See Section A.3 for Chinchilla's compute coefficient.) Similar to the compute-paramellee scaling coefficient, on the surface Kaplan's coefficient of 0.057 appears quite far from Chinchilla's of 0.155 However, we will again show that by beginning from the Chinchilla study, and adjusting for Kaplan's non-embedding compute, smaller scale, and additionally their computo-loss form, these two coefficients can be roughly reconciled.

---

Published in Transactions on Machine Learning Research (11/2024)
-0.01
Amaittical K
Kaplan repoctod depe. y 0.057
-0.02
alope. y
0.03
0.04
-0.05
00.0
00.07
+1.08
10'
10
10"
10
10
16
16
Giz
Figure 6: Visualization of Eq.
30
using the Epoch Al specification.
Our analysis follows the same four-step approach as in Section 3 We can directly reuse Steps & 2, while Steps & are now modified study the relationship between optimal loss and compule, rather than optimal parameters and compute as previously.
Step 3. Analytically derive the relationship between LE & CA..
We find that the relationship between Me & CVE is not power law (derived in Section A.2
Nio f/w (NiE) 1/3
d dlog(Li (T dlog(G, (B.) Li
g
CiE
1
-aN
1/3) arT + BD.
(30)
6N
NNE w (Nip)
d'og((iie) dlog(((s))
However, again we can consider local first-order approximation, De X CK where k visualize this in Figure 6
We
Step Simulate synthetic data from the Chinchilla loss model over the model sizes used Kaplan, Fit a local power law, for E terms of Ci using Kaplan's compute-loss form.
As in Section 3 could use Eq. 30 to compute point estimate for k in the relationship Lig X C However, again we opt for the more faithful procedure of simulating data from the loss curves. We then fit Kaplan's compute-loss form LE
C.E For the two specifications, these give the following models,
Co
Epoch Al specification: X C, -0.000
(31)
Chinchilla spocification: X C- 0.006
(32)
which are roughly inline with Kaplan's reported coefficient of LAF X CT D.005
0.057
Figure visualizes the fit of the Chinchilla compute-loss form vs. the Kaplan compute-loss form, when one counts total compute vs. non-embelding compute. We see that Kaplan's form provides a good fit of the data in the non-embedding compute plot at small scale, over the range of model sizes they considered. We speculate that this might be the motivation for Kaplan's selection of this simpler compute-loss form.
10

---

Published in Transactions on Machine Learning Research (11/2024)
10
Loss
Lows
Compute efticient froatier
Compure effisient frontisr
Kaplan compule- lorm:
Kaplan compule-loss form:
log(L;)) 0.097log(C+) + 5.32
log(L 0.069/og(Cii) +
4.11
10"
Chinchilla compute-loss form:
Chinchilla compute loss form:
log(L,") 1.82
(.178log(C;)+ 7.90
log (g) 1.82 0.133/og(C.e)
5.91
10
10'
10
t
10ah 10a
10
10'
10
10
10"
Cr
CIE
Figure 7: Synthetic data with the Epoch Al specification, as in Figure Here visualize the quality of the fits using both the Kaplan and Chinchilla compute-loss forms. Notice that Chinchilla's form provides a better fit under total compute, while Kaplan's form is a better match under non-embedding compute.
6
Related work
Following early works formalizing how language models improve with parameters, data, and training compute
[Rosenfeld et al. 2019 Kaplan et al. 2020 Hoffmann al. 2022 there has been investigation into whether
these scaling laws arise in other domains (Hnnihaa et al. 2020), and to explain their existence from
theoretical standpoint (Hutter 2021
Maloney et al. 2022 Bahri et al. 2024
Closer in spirit to our paper are several concurrent works that have investigated the influence various design decisions on scaling law analyses. (Su et al. (2024) revisít. the methodology used to find scaling coefficients. [Hägele al. (2024) found that multiple independent cosine schedules could be reproduced more efficiently through constant learning rate with multiple short decays, stochastic weight averaging. Our finding is subtly different; simple fixed learning rate will recover very similar compute-parameee scaling coefficients as multiple cosine schedules. Bi et al. (2024) study the effect of various hyperparameters on scaling laws. They observe that different text datasets produce slightly different optimal coefficients, with 'cleaner' data leading to more parameter-hungr scaling behavior, which they speculate could partially explain the difference between Kaplan and Chinchilla coefficients.
Porian al. (2024) provide concurrent work with the same objective as our paper explaining the differ- ences between the Kaplan and Chinchilla coefficients. Through set of large-scale experiments reproducing Kaplan's study, they determine that responsibility for the discrepancy can be attributed, in decreasing order of significance, to; 1) Kaplan counting non-embedding rather than total compute. 2) Kaplan using fixed- length warmup period that was too long for smaller models, making them appear less efficient. 3) Kaplan not fully tuning optimization hyperparameters. We see these findings as complimentary to our own. We have been able to identify the primary 'first-order' reason using only information that was publicly available the two papers, with fully analytical approach. (Tiny-scale experiments were run host-hoc as verification.) This iltustrates the promise of applying mathematical approaches to the empirical science of scaling.
11

---

Published in Transactions on Machine Learning Research (11/2024)
7 Discussion
This paper aimed to explain the difference between the Kaplan and Chinchilla scaling coefficients. We found two issues in Kaplan's study that combined to bias their estimated scaling coefficients; their choice to count only non-embedding parameters, and studying smaller sized model sizes. This means there is curvature in the true relationship between Ne & Nr (Figure 5 At larger values of Nr, the embedding parameter counts become negligible, Nr N. and differences would not arisc. Alternatively, had Kaplan studied relationships directly in terms of Nr, this issue would also not arise, even at this smaller scale (confirmed by our Experiment finding Nr C9.49 even for Nr < 5.M). The form Kaplan used to predict loss from compute further contributed to differences in the reported compute-loss scaling coefficients.
Inconsistency across scaling studies. Existing literature on scaling is not consistent in its use of non- embedding vs. total compute. Some studics (Henighan et al. 2020 Gordon et al., 2021 Ghorbani et al.,
2021 Fernandes et al. 2023 Hu et al.) 2024; Bi et al. 2024 Su et al.) 2024h follow Kaplan's 's approach, using non-embedding parameters or compute, while others (Clark et al.) 2022 Que et al., 2024 Wang et al. 2023) adhere the Chinchilla approach, using total parameters. Our work indicates that this choice can substantially alter scaling exponents, complicating cross-study comparisons. Similarly, the choice of compute-loss equal ion varies through the literature. Studies such as (Clark et al.
2022 Brown el al. 2020: Smith & Doe 2024) opt for the Kaplan compute-loss form without offsets. In contrast, (Henighan et al, 2020 Gordon ct al. 2021 Fernandes et al. 2023 Hu et al. 2024 Wang et al.) 2023) employ the Chinchilla compule-loss 1orm with non-zero ollsets. Again, our work suggests that these methodological differences can lead to significant variations in scaling predictions and interpretations.
The lack of a standardized approach in scaling studies risks making comparisons mislcading and insights less clear. We see our work as helping to understand certain decisions made in previous studies that should be standardized. Concretely, we advise future studies to report total, rather than non-embedding, parameters, and to include an offset in the compute-loss fitting models. We discuss motivation for these choices below. Furthermore, our initial evidence does not support using multiple cosine decays per model size - we find a single fixed learning rate per model size is sufficient for measuring compute-optimal parameter coefficients.
Why should embedding parameters contribute to scaling behavior? Several works evidence that embedding parameters capture meaningful language properties. Word embeddings can be factorized into semantically interpretable factors (even the shallow Word2vec) (Mikolov et al. 2013a b) Arora et al. 2018). LLMs learn linear embeddings of space and time across scales (Gurnee & Tegmark 2024). Developing such meaningful embedding structures allows LLMs to perform high-level language operations, such as arithmetic (McLeish et al.) 2024). Therefore, if one believes that the embedding layer does more than just 'translate' tokens to a vector of the correct dimension, we see no reason to exclude them in the parameter count.
Why should non-zero offset be used in loss-compute predictions? The Chinchilla compute-loss form with non-zero offset (Eq. 27 is more appropriate form from the perspective of statistical learning. This approach accounts for the concept of irreducible risk (Hernandez et al. 2023). which posits lower bound on achievable loss regardless of model or dataset size. This may arise from various factors: inherent biases or limitations in the learning algorithm, or noise in the original task. As a concrete example in language modeling, the best a model can do for the prediction of the first token in a sequence, is to estimate the marginal distribution of all tokens, which leads to a non-zero loss.
Limitations. We acknowledge several limitations of our analysis. We have aimed to capture the primary 'first order' reason for the difference between the Kaplan and Chinchilla scaling coefficients. But there are multiple other differrences between the two sludies that likely also affect scaling coefficients (Section 6 datasets (Kaplan used SpenWebTText2. Chinchilla used MassiveText)) transformer details (Kaplan used Iearnable position embeddings while Chinchilla's 's were fixed, also differing tokenizers, vocabularly sizes), optimization scheme (Kaplan used scheme 3, Chinchilla scheme 4, also differing warmup schedules), differ- ences computation counting (Kaplan used C = 6ND, Chinchilla's Method 1 & 2 used a full calculation). However, our work suggested these factors impact coefficients in a more minor way.
12

---

Published in Transactions on Machine Learning Research (11/2024)
References
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Compulational Linguistics, htggQQ8-10344
6:483-495, 2018. doi: 10.1162/tan a 000044. URL lntpp::
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, (2)00021231187,1,,, 2024.
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint 2t44 10102, 2024.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek lm:: Scaling open-source language models with longtermism. arXiv preprint ar Xrx:222401.00 02954, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Noc- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are lew-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoe, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference machine learning, pp. 4057-4086. PMLR, 2022.
Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation. In International Conference on Machine Learning, pp. 10053- 10071. PMLR, 2023.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. ar preprint or arXi::2109007744 2021.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.
2021.
Wes Gurnee and Max Tegmark. Language models represent space and time, 2024.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scal- ing laws and computc-optimal training beyond fixed training durations. arXiv preprint or orxv::2405.. 18392, 2024.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom Brown, Prafulla Dhariwal, Scott Gray, al. Scaling laws for autoregressive generative modeling. arXiv preprint Xrx:::2000.1 14701, 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, Sam McCandlish, Tom B. Brown, Andrew Carr, Dicterich Lawson, Jacob Steinhardt, Chris Olah, and Dario Amodei. Scaling laws for neural language models. doi: 10.1073 2tt33/paa.23000 231188211 URL
Proceedings of the National Academy of Sciences, 120(26), 2023.
https://www. wwwp:://wwwpppa...../ www.ppaw...pp.pppppppppp /tt/d/..111111// 1073/ pnas. 2311878121
Jordan Hoffmann, Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. ar Xie preprint arXim:2203.1555. 2022.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Miniepm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXic:2404. 06395, 2024.
Marcus Hutter. Learning curve theory. arXiv preprint or Xix: 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXie preprint arXiv:2001. 008311 2020.
Christopher T. Kello, Gordon D.A. Brown, Ramon Ferrer i Cancho, John G. Holden, Klaus Linkenkaer- Hansen, Theo Rhodes, and Guy C. Van Orden. Scaling laws in cognitive sciences. Trends in Cognitive Sciences, 14(3):223 232, 2010. ISSN 1364-6613. doi:
tis.0000... URL https://www.sciencees htt.:///.....................aaa
Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprinl arXiv:2210. 16859, 2022.
Scan McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein. Transformers can do arithmetic with the right embeddings, 2024.
Tomas Mikolov, Iya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahra- mani, and K.Q. Weinberger (eds.), Advances in Neural Information Processóng Systems, volume 26. Cur- ran Associates, Inc., 2013a. URL https://pprceedddgg/ neurips...... www.pperrccpppprrp.ppp http//pprrf.leeepppp///////////1 Ba220318222039963333333eeb--b022
pdf
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word repre- sentations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 746-751, Atlanta, Georgia, June 2013b. Association for Computational Linguistics. URL httpps::/aclatthll..oo httph/oooooooooo/11131/111 org/N13-1090
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. arXiv preprint or arXiv:2406. 19146, 2024.
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, et al. D-ept law: Domain-specifie continual pre-training scaling law for large language models. arXiv preprint arXiv:2406.cm 01377, 2024.
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. constructive prediction of the generalization error across scales. Xrxiv preprint arXiv: 1909. 12673, 2019.
J. Smith and A. Doe. The role of xyz in abc. bioRziv, 2024. doi: petts://wwiir wiw.o///ii/w...ooi......o biorxiv. www.:/oooxwv.ooooooooooooo
URL
597. ful1.pp
Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. Unraveling the mystery of scaling laws: Part i. arXiv preprint XrX::22403.066 065663, 2024.
Peihao Wang, Rameswar Panda, and Zhangyang Wang. Data efficient neural scaling law via model reusing.
In International Conference on Machine Learning, pp. 36193 36204. PMLR, 2023.
14

---

Published in Transactions on Machine Learning Research (11/2024)
A
Appendix
A.1 Derivation of 20
20
This scction derives Eq 20
1 dllo(CriE)
1 NE
2/3
w.
a +
N 2/3
=1
(33)
9 dloog NiEE
B N 2/3
+ 3
B
No
2/3
+ w
First note that
dlog(C((e) dlog (Nx).
dlog(Cie)
dNTe
(34)
dNiiF dlogg(NiE))
dlog(CiB)
ME
(35)
dN
Recall the definition of CLE from Eq. 19
= CG=GNN (NEE + w
1/3) - (NiE + +NNE 1/5)
x
Dr
3
(36)
E
N
log(O(e) log(
log
NCE + ME 1/3)
+
1
log (Nie + WNie V/)
+const.
(37)
B
term
where const. does not depend on AE We now can take the derivative of each term. Derivative of term 1.
d log N (E)
(38)
d.V
N
Derivative of term 2.
1 d dllog NNiE + 3 N 1/3
w
dlog NE + w N.
3
1/3)
dN= LE
+ +HH NN 1/3
1 + JNCE -2/3
(39)
3
dN E
dN + NN 1/3
dN
N + NN 1/3
Derivative of term 3.
a + 1 dlog (Nie + WN 1/3)
a a d log
NiE + WNEE 1/3
dN +NN 1/3
a + 1+ 3 N -2/3
3
dN
B
dN + w.N. 1/3
d.o
3 N + wN (E
(40)
Then assemble all terms and multiply by No per Eq. 35
15

---

Published in Transactions on Machine Learning Research (11/2024)
A.2 Derivation of compute-loss analytical form in Eq. 30
This section derives k, defined as,
dlog(LiE)) k: = dllog( ((E)
(41)
Expanding with the chain rule we find,
k dlog(LLE)) dLAE dNC d (log(~N())
(42)
dLiB dNi (N dlog((C) (E))
NE dLie
TE dN 9,
(43)
delgg(N(e) in Eq. 20
where we previously derived g Rl
dii This leaves us with aN to find. First note that Li is given by Eq. used, ie., Ni t ME
18 when the optimal model size is
No
De + E.
LF (NC + WN 1/31a
+ (C (Cie/GNi
(44)
g )
Before taking this derivative, we recall that C is actually a function of Nr (via Eq. the derivative in two parts. We find the first term derivative is equal to,
19
Hence, we tackle
(1+ j/w (NiE)
2/3
d
No (N 17350
-a.N
(45)
d.V. E
1/3) a+1
Ni NVEt (Niz)
The derivative of the second term, via the product rule, and spotting that
dG dNi i
Cse aN j
equals,
De
-
(CAB/6N
1
CiE
CiE 6.
BD
3D
(46)
dN* E
Ni
6N
g.
-B
1
CiE
(1
1
DD No
(47)
6.
g
Hence, combining these two terms we find,
(1+ fu (Nit))
-2/3
dLig -NNo
1
C5E
-9
(1-1)
(1
3DD
(48)
dN
NNe + (NiE)
1/3) a+1
N
6N
g
Combining this result into Eq.
1 we get,
Ni dlig
(49)
L dN
NE + fo (NiE) 1/3
-9
g
C
(1-1)
1
-aN.-
1/3) a+I + BDD
(50)
Di
NiB + w How(Nig, (Nip)
6.
g
16

---

Published in Transactions on Machine Learning Research (11/2024)
A.3 Compute-loss coefficient derivation
We know from Eq. 17 Ni x C and similarly Dj x Ca Substituting these into the loss form of Eq. and for some new constants No, De we find
N
De
L1
+
E,
(51)
N
D1 3
N
De
+
E;
(52)
od Carp
C C
-88
L1 E C
(53)
17


Reviews:
[{"review_id": "IahReYixrT", "reviewer": "Reviewer_GNQy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- The paper attempts to reconcile two fundamental scaling laws and highlights the reasons for the discrepancy between the obtained coefficients.\n- The paper reanalyzes the data based on these assumptions, highlighting that Chinchilla would also obtain similar coefficients when focusing only on non-embedding parameters, and considering smaller model sizes.\n- The paper presents toy experiments and highlight the conclusions hold in practice.\n\n### Weaknesses\n- While the paper is technically solid and interesting, the reviewer felt that the paper missed important background that is important to understand the paper. Therefore, the reviewer would suggest including a proper background section that describes all the relevant details from Kaplan and Chinchilla papers in order to make the paper more understandable and standalone such as describing the L(N, D) or L(N, C) parameterizations directly within the background.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Kaplan and Chinchilla presented different coefficients for how the number of parameters (N) should be scaled to achieve the lowest possible loss given a fixed compute budget (C). Chinchilla suggested scaling the amount of data and number of parameters in proportion, while Kaplan suggested to focus more on increasing the number of parameters in contrast to data.\n\nThis paper attempts to understand this discrepancy between the two scaling laws. In particular, the paper highlights that two fundamental reasons for this difference: (i) Kaplan focused on non-embedding parameters, while Chinchilla focused on total parameters, and (ii) Kaplan focused on smaller models (1k to 1.5B parameters) while Chinchilla considered larger models (44M to 16B).\n\nWhile other differences exist such as differences in embedding types and dataset used for training, the paper shows that they only marginally impact the conclusions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The reviewer would suggest a proper background section that sets the stage for the whole paper, without requiring the reader to be very familiar with the exact parameterizations of scaling laws. This can significantly improve the accessibility of the paper for a wider audience.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "STrC6gEOe9", "reviewer": "Reviewer_nDnt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In short, I could not find any notable weaknesses in this submission.\nThe author's hypothesis is that most of the discrepancy can be explained by accounting for embedding parameters in Kaplan's model. Given the widespread use of scaling laws, it is an important study, as the two discussed papers end up with different scaling law exponents.\n\nAll steps and derivations are well motivated and explained, and the main result looks very convincing.\n\nThe only question I have for the authors is whether they are planning to release the code for the paper (which I would highly encourage).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main goal of the paper is to explain the discrepancy between Kaplan and Chinchilla scaling laws via a theoretical model. The main result shows that sampling from the Chinchilla scaling law, and computing only non-embedding parameters, it is possible to match closely the scaling law from Kaplan et al.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Minor comments:\n* missing space in equation 5 and 6\n* page 4, above (12): \"from\" should be \"form\"\n* it might be helpful for the reader to include a few extra steps for the calculations on page 5 (possibly in the appendix)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VJfAOIVQEw", "reviewer": "Reviewer_Cahx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written. The analysis is written in clear steps, making it easy to follow.\n- The paper provides evidence for the correctness of the Chinchilla’s scaling law by showing how the two seemingly different scaling laws are related using analysis.\n- The paper provides very good theoretical analysis showing the connection between the two scaling laws.\n- The paper also provides experimental evidence showing the correctness of Chinchilla’s scaling law, the connection between the two scaling laws, and how optimization hyperparameters do not affect the scaling laws much.\n- The paper addresses a very important topic and will help designing large scale models in the future.\n\nWeaknesses:\n- This analysis in the paper is independent of the nature of the architecture used, as long as the parameters are given. Is there any insight on whether architecture has any role to play in the scaling laws and whether any further theoretical/experimental analysis can help understand whether the scaling laws apply to architectures beyond the transformers.\n- Fig 5 is a bit difficult to understand: I suggested improving the coloring scheme (the similar colors used for the lines are difficult to distinguish), and making the caption and the “Main result” in Section 3 more detailed since it is the core message of the paper and it would help the readability of the paper.\n- Since this is an analytical study of scaling laws, it would be better to write down the summary of approximations and assumptions made in the analysis. This would help readers understand how to modify the analysis in the future if the assumptions change, hence, it would avoid the mistake made in Kaplan’s scaling law (their assumption of ignoring the embedding parameters and working with small scale). This would also help because the assumptions and approximations made during the analysis steps can sometimes be hard to follow.\n- Minor: Page 4 in step 2, “we need an analytical from of loss” -> “we need an analytical form of loss”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides analytical and experimental evidence to reaffirm the Chinchilla’s scaling law. To do so, it shows that the other popular scaling law (Kaplan’s scaling law), which is in contradiction to Chincilla’s scaling law, can indeed be obtained from Chinchilla's scaling law by accounting for the number of embedding parameters. The paper starts with Chinchilla’s scaling law, and using analysis it shows that by simply removing the number of embedding parameters, one obtains Kaplan’s law for small-sized models, which leads to the issue with Kaplan’s scaling law. The paper also provides experimental evidence to back their claims showing i) the two different scaling laws for small-sized models when considering the embedding parameters vs. not considering the embedding parameters, (ii) it also shows that different optimization hyperparameters have little influence on the scaling laws.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please see the weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the discrepancy between previous studies on scaling "laws" by Kaplan and the Chinchilla paper, who reach different conclusions w.r.t. the optimal parameter count per training set size. The paper demonstrates convincingly that the discrepancy can be explained by the lack of counting embedding parameters in Kaplan, and the experiments mainly done on smaller architectures by Kaplan, which introduces a bias away from Chinchilla's results. This clears up a long-standing discrepancy in a clear and comprehensive fashion. While this contribution is worthy of a publication in itself, the execution of the paper (quality of the writing and experimental evaluation) is very good. In addition to meticulously performing Chinchilla-style analysis under Kaplan's settings (with small models and on BookCorpus), the paper provides a comprehensive theoretical analysis and has some important insights and best-practices for (future) scaling analysis for the field (e.g., focus on total parameters and total compute vs. compute-loss analysis). 

Overall, I think the paper is a clear accept. I also recommend the paper to receive a Reproducibility Certification - even though, it does not precisely repeat a previously reported study, it analyzes and clearly explains how previously reported results from two studies arise, and how a reproduction of one of the previous studies (Chinchilla on Kaplan's setting) resolves the discrepancy.
Decision: Accept (poster)