Algorithmic progress in language models
Authors: Anson Ho
Tamay Besiroglu
Ege Erdil
Zifan Carl Guo
David Owen
Robi Rahman
David Atkinson
Neil Thompson
Jaime Sevilla
Venue: NeurIPS 2024 poster (2024)
Keywords: Natural Language Processing
Primary area: evaluation
TL;DR: Progress in language model performance surpasses what we'd expect from merely increasing computing resources, occurring at a pace equivalent to doubling computational power every 2 to 22 months.

Abstract: We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 90\% confidence interval of around 2 to 22 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.

Algorithmic progress in language models
Anson Ho!*
Tamay Besiroglu .-
Ege Erdil
David Owen'
Robi Rahman'
Zifan Carl Guo
David Atkinson
Neil Thompson
Jaime Sevilla
Abstract
We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach set performance threshold has halved approximately every 8 months, with 90% confidence interval of around to 22 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase compute made even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.
1 Introduction
The field of language modeling has seen rapid advances, with recent large language models (LLMs) demonstrating strong performance in domains such as programming [Li et al., 2022, Leblond et al., 2023], mathematics [Cobbe al., 2021, Trinh al., 2024], and standardized tests [OpenAI, 2023]. But how has this progress been possible?
One salient factor is the scaling of training compute based neural scaling laws [Sevilla et al., 2022, Hoffmann et al., 2022, Kaplan et al., 2020], with state-of-the-art systems being trained for months on tens of thousands of GPUs. But this is only part of the story: another key factor has been algorithmic improvements, which result in more efficient use resources such as compute and training data. These include changes model architectures, optimization algorithms, and software frameworks.
This picture raises some important questions: How much of recent progress language models has come from algorithmic improvements during pre-training, and how much has been from scaling up models and datasets? The answers to these questions have crucial implications for the future of AI progress, and have important role in informing Al policy.
* Joint authors. Epoch. 2MT FutureTech, CSAIL,, Northeaster University. Email correspondence to tamaydepochai. org. You can find our code and data here: http://ggttuuu.ooopppeeeeeee wwwm//pocc-reeeeeeeeeeeeeee researchh 1m- wwwerrtttiii--ppeeeee.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we aim to answer these questions by following the approach first presented by Erdil and Besiroglu [2022] in computer vision. 2 To this end, we produce dataset of over 200 language models evaluated on popular language modeling datasets, and use this to fit a statistical model that helps estimate the rate of algorithmic progress. Using our model, we further quantify the relative importance of algorithms and scaling training compute, shedding light on one of the most important drivers of Al progress.
1.1 Previous work
Thus far, there have been few works investigating algorithmic progress in machine learning. Notably, Hemaadde and Brown [2020] re-implement popular open-source ImageNet models and find 44x decrease in the compute required to achieve the same performance as AlexNet. Karpathy [2022] reproduced the convolutional neural network of LeCun et al. [1989] using modern algorithmic innovations, achieving 60% reduction in error rate. Dorner [2021] measures progress in the sample efficiency of deep reinforcement learning algorithms over time with doubling times ranging from 5 to 18 months. More recently, Erdil and Besiroglu [2022] estimate algorithmic progress based on fitting a statistical model inspired by neural scaling laws. They find that algorithms and hardware contribute roughly equally to performance, and the training compute needed to reach a certain level of performance halves every months.
2 Methodology
2.1 Model definitions
We want to estimate the rate at which newer language models are able to achieve certain level of performance more efficiently than older models. We do this by fitting a model that meets two key desiderata: (1) the model must be broadly consistent with previous work neural scaling laws (e.g.
Hoffmann et al. [2022]). and (2) the model should allow for a decomposition of the main contributors to increased performance, such as improvements in how efficiently data free parameters in the model are used. In this sense, our core approach is similar to that in Erdil and Besiroglu [2022].
The starting point is the scaling law from Hoffmann et al. [2022], which relates the training loss I. of a dense transformer to number of parameters N and the training dataset size D):
A B
=
(1)
Na
DE
where L is per-token cross entropy loss on the dataset, and E, A, B, a and B are constants. E represents the 'irredueible loss' of the dataset, while the second and third terms, N/a and Dr capture the errors that are due to the finiteness of the model or dataset, respectively.
Following Erdil and Besiroglu [2022] and Hernandez and Brown [2020], we quantify algorithmic progress in terms of reductions of the resources (N and D) required to achieve the same level of performance over time. To measure this, we introduce the concepts of "effective data" Deff and "effective model size" Netr into the model:3
Nat = N exp(a' Yo)), and Der = Dexp(B -Y)),
where Y is the current year, Yo is some reference year', and a' and 3r characterize the rate of algorithmic progress for model size and dataset size, respectively. In other words, we assume that continued algorithmic progress results in exponential increase in Der and Ner over some time interval Y -o, even with fixed D and N. Plugging these into the original scaling law gives:
A Aanm
B
A
B
I=
F.
Ararr (Y-Yo
ByeelY Yo)
S elf
Nagam
D
eff
NNoe that we focus pre-training algorithmic progress, which is distinct from algorithmic progress in general. In particular, do not consider "post-training enhancements" such chain-of-though prompting [Davidson et al., 2023].
'This is not an original idea for example, Hernandez and Brown [2020] and Erdil and Besiroglu [2022] use the concept of "effeettvve compute" to calculate doubling times for compute efficiency in computer vision, and Davidson [2023] incorporates similar idea into an integrated economic model.
"Note that the "years" in our model do not need to be integers, i.e. "fractions of year" are allowed and are determined based on the specific publication date of model.

---

where A, B, Oparaam Otyear. Puata and Byear are constants. In relation to equation 2, we have that a' = Ayear/ ppram and 3 B aatn- Algorithmic progress is thus captured as a constant exponential trend that multiplies with each of the two terms. in the scaling law. In doing so, we are able to estimate the rate at which fewer 'resources' (N and D) are required to achieve the same level of performance over time. Furthermore, given that that the physical compute is approximately given by C' z 6ND [Hoffmann et al., 2022, Kaplan et al., 2020], we can similarly define an "effective compute" which is determined from the effective parameters and effective data.
2.2
Estimation approach
2.2.1 Model selection
We estimate variants of the augmented scaling law presented in equation (3) on our dataset of language model evaluations. We perform extensive vrass-valiattion exercises identify the variant of the model that fits the data best. The goal of this exercise is to consider different models that capture different effects (e.g. different scaling behavior across different model architectures, different forms of algorithmic progress, etc.).
Concretely, we consider dataset-specific coefficients (A, B), rates of algorithmic progress (e.g. ayears Syaar), different scaling coefficients for different architectures, regularization (Aparam ddaa)), and more. The model variants we consider generally do not contain an irreducible loss term (i.e. E' 0) since this is poorly estimated on our data, and because it does not change our estimated doubling times in practice we check the robustness of this change in appendix H. In total, we evaluate around 90 different model specifications through cave--one-uutcross validation and pick the models that perform best on relevant out-of- -f--ppll metrics, see Appendix for more details. In the end, the model we select is model 7, where the coefficients A and B are benchmark specific, but estimates of algorithmic progress and scaling exponents (e.g. a and (9) are not. This model achieves an I?2 of around 0.91 between predictions and held-out test data.
further important consideration the possibility of alternative forms of algorithmic progress. In particular, in section 2.1 we model algorithmic progress as causing exponential increases in an "effective" budget, e.g. of parameters. But one could also observe progress through changes in scaling exponents (i.e. Parram and (dma). There are a priori reasons to suspect that this might be the case-for for instance, one notable innovation is due a change in scaling laws such as those introduced in Kaplan et al. [2020] and Hoffmann et al. [2022]. Different model architectures, such as recurrent neural networks and transformers, are also known to have different scaling behaviors (see for instance Tay et al. [2022] and Droppo and Elibol [2021]).
We attempt to account for this possibility in the cross validation analysis. In particular, introduce three models (models 13 to 15) which account for different kinds of scaling exponents, including the possibility of changing exponents over time. Our chosen main model (model 7) outperforms these models in cross validation, but these alternatives also perform similarly well, typically with an li2 of between 0.88 and 0.91. This analysis is described in more detail in appendix J.
We also consider other factors that could potentially impact measured perplexity, and thereby mea- sured rates of algorithmic progress. For example, different tokenization schemes during preprocessing have been found to improve WT perplexity in some instances [Radford et al., 2019], and training models for multiple epochs has been common way of improving performance [Muennighoff et al., 2023). We find that our core results are broadly the same while varying these degrees of freedom-wer provide more details on these experiments in the appendices."
2.2.2 Data
Our dataset contains over 400 language models evaluated on WikiText-103 (WT103), WikiText-2 (WT2), and Penn Treebank (PTB), about 60% of which we are able to use in our analysis. In particular, relevant information was retrieved from around 200 different papers, as well as evaluations of 25 models that we performed ourselves using the framework from Gao et al. [2021]. We then consider the subset of the data that contains the information necessary to fit our proposed model structure in equation 3: leven-level test perplexity (which determines the cross-entropy loss), publication date,
In particular, consider tokenization in appendix E.2.2, epochs in appendix F, and context length in E.2.
3

---

number of model parameters, and training dataset size. This leaves us with around 231 models for analysis.
In some instances, multiple models are retrieved from the same paper, even if they constitute similar algorithmic innovations. This could pose problems around autocorrelation, which could result in underestimating the uncertainty in our individual parameter estimates. In the following main analysis, we therefore only include up to three models per paper, which results in approximately 50 more models being excluded. To verify the robustness of this approach, we also consider an alternative technique that directly accounts for autocorrelation in the analysis, which yields doubling time and confidence interval estimates that are consistent with our main results (see Appendix I).
3 Empirical results
3.1 Models require 2x less compute roughly every eight months
How quickly are the algorithms underpinning language models improving? Our core approach is to back out doubling times based on fitting the augmented scaling law introduced in equation (8), and using the definitions of effective data, effective parameters, and effective compute we introduced in
section 2.1. Here the effective data is given by Detr = D exp
Bywr (Y - Y.) so the doubling time Plxa
for Dea is determined by the time Y Yo where Dan 2D.
Thus we have:
Bdata 2.
To = Yo =
(4)
Byear
The doubling times for parameter efficiency can be determined similarly, giving
Oiparam
TN =
In 2,
(5)
O'year
which we use to work out the doubling times for effective compute. In particular, since the total compute in FLOP, C, required during training is approximately 6ND, the growth rates are related via Sc = gN + gn. Here gC is the growth rate in effective compute, gv is the growth rate in effective parameters, and gn is the growth rate in effective data. Since doubling times are inversely related to growth rates, we therefore have that
- --=(+++1))' TD
Tc = TN
where Tc, TN, and TD are the doubling times (due only to algorithmic progress in pre-training) for effective compute, effective parameters, and effective data respectively. Based on this approach, using our preferred model, we find that the median doubling time for effective compute is 6.1 months, with a 90% confidence interval of 3.3 to 11.3 months.
We further check the robustness of this result by looking at the predictions from different models. In particular, because we perform model selection using leave-one-out cross-validation, we can compare the predictions of our preferred model with the predictions from other models we considered. Concatenating the doubling time estimates from all the models in Figure 1b, we find a median doubling time of 7.5 months [95% CI: 1.7 to 22.5 months], which is consistent with our preferred model.? 7 This estimate falls within the range of confidence intervals of the estimated rates of algorithmic progress in computer vision [Erdil and Besiroglu, 2022], and sample efficiency improvements in reinforcement learning [Domer 2021].
While the structure of our model is not amenable to analyzing fine-grained speedups or slowdowns in the rate of algorithmic improvements, we can nevertheless test the possibility of one-time increase or
'Not that our preferred model model 7, whereas the model that performs best in cross validation model 10. We opt for model 7 given that it performs essentially as well in cross validation (MSE test loss of ~0.048 for model 7 compared to ~0.042 for model 10) but uses two fewer parameters. In addition, model 7 can be used to back out single rate of algorithmic progress, rather than dataset-specific rates, which makes the results easier to interpret. More details about the models and their performance can be found in appendix J.
"In both the preferred model and aggregated estimates, the lower range of the confidence interval implies extremely fast doubling times. While we believe there is reason to be sceptical of these estimates, we provide the estimates here for transparency. We elaborate this concern in Section ??.

---

Density
Doubll ng time (ycars)
3.5
0.20
Apgregated
Modian:
10
Model7
0.15
Pledians 7.5
20
0.5
15
Doubling time (months)
cross-vlidation per
cross-validation performance
(a) Core estimates. Doubling times(b) Robustness across model specification. Swarm plots showing model estimates of the from our preferre model, and aggre-rate of algorithmic progress across distinct model structures. For each model, we choose the regularization strength that performs best leave-ome-out cross validation.
gale across model specifications.
Degree of Freedom
2
5
7
9
10
11
12
13
14
15
Progress Efficiency Along
X
V
VT
V
Progress in Efficiency Along
V
V
X
V
/T
V
Dataset Specific Exponents
Dataset Specific Constants
(c) Summary of model structures and the degrees of freedom included. Efficiency guins are captured by exponential decrease relevant error terms, except models indicated by ", which have time-varying exponents. For full specificution, Tables 11 and
Figure I: Estimates of algorithmic progress of models selected by cross validation. Figure 3a shows aggregated estimates over doubling times, and Figure 3b illustrates via swarm plots sorted from left to right in order of decrcasing validation performance (increasing MSE test loss). Note that model 14 is omitted from Figure 3b we elaborate reasoning appendix J.2.
decrease in growth rates over the full time period. To this end, we consider a variant of our preferred model (model 7) where dummy variable is introduced-this is equal to O for any model that is published before the start of a certain year, and otherwise. This allows us to consider doubling times before and after certain year cutoff (e.g. 2017), and we perform this analysis for several such cutoffs.
The result is shown in Figure 2. Here we see that the difference in estimated doubling time before and after the start 2017 is very pronounced, however this is not the case for other choices of the cutoff year. In each year the median doubling time is faster after the start of the cutoff year, but usually only marginally so. Overall, this does not provide strong evidence of a drastic speedup in algorithmii progress. This does not rule out the possibility of weaker effect sizes, since our approach statistically under-powen under-powered.
304
303
200
Figure 2: Left: Comparison of estimated doubling times for effective compute from algorithmic progress, before and after set cutoff years from 2016-2020. Shorter doubling times in the "post" period relative "pre" indicate acceleration in the of algorithmic progress after that cutoff year. Longer doubling times indicate deceleration. Right: stylized illustration the relative contribution of compute scaling and algorithmic progress to effective compute. The physical compute contribution is estimated from the doubling times in Sevilla et al. [2022], and the algorithmic progress contribution is hased on the aggregated doubling time estimate across model specifications (see section 3). We further plot the physical training compute values for several notable models (c.g. GPT-2) in their publication years.
5

---

3.2 Most recent performance gains in next-foken prediction have been from compute-scaling
Naively extrapolating our estimated doubling times suggests that, between 2014 and 2023, pre- training algorithmic progress has enabled performance to improve as much as it would have with around 22,000x more compute." R At the same time, Sevilla et al. [2022] find that physical compute budgets have doubled roughly every 6 months since the start of deep learning, including in language models. This suggests that physical compute has instead grown by a factor of around mme-million-fold This paints a stylized picture where "effective compute" expanded by about 22-billion-fol since 2014, with slightly under two-thirds of the scaling being due to increased use of actual, physical computing resources.
There are reasons to be cautious about this naive extrapolation. For one, we do not directly observe gains of 22, 000x (or even 10, 000x) anywhere our dataset. However, given that it is unlikely that early researchers trained language models on very large quantities of compute, it is therefore improbable that we observe such large declines over the analyzed time period. Nevertheless, the lack of such observations still raises questions about the reliability of extrapolating these trends between long multi-year periods.
One specific reason for caution is that the extrapolation neglects the scale-dependence of algorithmic innovations. It is likely that some algorithmic innovations will become obsolete over time as models are trained at larger scales of compute-e.g. the effectiveness of specific tokenizers or hyperparameter settings may diminish, making them less useful for future, larger models. Conversely, recent innovations might fail to produce large or any benefits when implemented at much smaller scales than models today. For example. the gains from scaling laws are related the scale of compute used (see Appendix B), and older architectures, such as the LSTM and convolutional network, can exhibit higher efficiency at small scales relative to the transformer [Droppo and Elibol, 2021, Karpathy, 2022].
While naive extrapolation of doubling times predicts substantial reductions in compute requirements, our work does not provide compelling evidence that we can currently or in the future train extremely small models to achieve the performance of much larger ones by applying the full suite of modern innovations. The scale-dependence of algorithmic improvements and the lack of direct observations of such large efficiency gains our dataset suggest that further research and more comprehensive data are needed to validate these extrapolations.
Besides doubling times, we can also decompose the relative contributions from algorithms and compute scaling by evaluating our estimated models directly. We approach this using a Shapley value analysis, and the results weakly support the stylized picture above that compute scaling has been more important for explaining performance improvements than algorithmic progress since 2014.
The findings indicate that the relative contribution of algorithmic progress to performance improve- ments has diminished over time, at least within the dataset of models that have historically been close to the state-of-the-art. This observation aligns with the stylized representation in Figure 2 and the findings of Erdil and Besiroglu [2022] for computer vision, where compute scaling has shown increasing importance over time.
One explanation for the diminishing relative contribution of algorithmic progress is that investments in expanding physical compute have increased substantially, outpacing the rate of algorithmic improvements. This framing aligns with the increased emphasis on scaling large language models over the last few years, particularly since the introduction of GPT-2 in 2019 [Radford et al., 2019], relative to fundamental algorithmic or architectural changes."Figure 2 illustrates stylized version of this perspective, depicting sharp increase in physical compute scaling around 201 2-19-2019, followed by a return to previous compute scaling growth rates.
There are other potential explanations - for example, it is possible that the transformer architecture was a pivotal innovation (see section 3.3), and subsequent algorithmic advances have been less
We consider 2014 since this is publication year of the earliest model in our dataset for which the training compute is known.
"We can provide further support for this interpretation by considering the average growth in compute between pairs of systems in Table 1. This turns out to be higher for later pairs of systems that we consider: e.g. between the Transformer and GPT-3 there was an average annual growth rate of 9%, compared to an average growth rate of 2% between the 2012 RNN and GPT-2.

---

Parameter scaling
Data
Data
scaling
efficiency
RNN (2012) LSTM (2016)
16.9%
40.4%
43.8%
RNN (2012) + Transformer (2018)
48.1% %
20.6%
32.1%
RNN (2012) GPT-2 (2019)
47.7%
20.1%
32.9%
RNN (2012) 7 GPT-3 (2021)
50.2%
26.0%
24.4%
RNN (2012) - Gopher (2021)
54.8%
24.0%
21.7%
LSTM (2016) + Transformer (2018)
82.6%
0.0%
17.9%
LSTM (2016) GPT-2 (2019)
72.1%
16.1%
12.1%
LSTM (2016) + GPT-3 (2021)
69.9%
20.0%
10.3%
LSTM (2016) + Gopher (2021)
68.8%
17.6%
13.9%
Transformer (2018) + GPT-2 (2019)
56.8%
38.2%
5.0%
Transformer (2018) GPT-3 (2021)
63.1%
29.7%
7.4%
Transformer (2018) + Gopher (2021)
61.9%
25,4%
12.9%
Table 1: Attribution of progress pre-training algorithmic progress and compute scaling between model pairs
based on Shapley decomposition in linear space. Numhers may not add up to 100% due rounding. These Shapley values are based on point estimates from our preferred model and as such are meant for illustrative purposes only. We omit parameter efficiency improvements from the table since these are almost always 0% and very informative. The Transformer here is by Baevski and Auli [2018] (the earliest decoder-only transformer have in our datasct), who modify the original transformer architccture by Vaswani [2017]
be decoder-only.
significant comparison. Alternatively, this observation could also be explained by a secular decline in the rate of algorithmic innovation. However, find these two explanations less compelling than the results of Figure 2, where the rate of algorithmic progress does not clearly decrease after the release of the transformer (e.g. with 2018 cutoff). If anything, the rate increases slightly, contrary to what both of these explanations predict.
3.3 The significance of the transformer architecture
Since its introduction in 2017 [Vaswani et al., 2017], the transformer architecture has become the dominant algorithmic architecture in language modeling, forming the base of multiple notable systems. We attempt to quantify the its contribution in terms of the "compute-equee equuivlllequvvlenttt gain" over other architectures our dataset (LSTMs, RNNs, state space models. among others). This is akin to the approach outlined in Davidson et al. [2023]-in this context, the compute-equivalent gain is the amount by which training compute must be scaled to improve benchmark performance as the same amount as the introduction of the transformer. For example, Hernandez and Brown [2020] find that a transformer (2017) achieves the same performance as Seq2Seq (2014) model on the WMT- 14-EN-FR benchmark, with 61 x less compute.
To capture the improvement represented by the transformer, we modify our core model follows:
(
A
if transformer,
L
N
N
D
otherwise.
D
where R - (0, is the sigmoid function, given by (xx) = 1/(11 77 is constant and all other terms have the same meaning as in the original model. 10 The key intuition is that the transformer could enable to compute (or perhaps data) more efficiently than the architectures that precede it.
After preprocessing, our dataset contains 103 transformer models, and 127 onnttransformer models, largely consisting of recurrent networks such as the LSTM. Fitting the model on this data reveals that the transformer architecture typically lowers reducible loss proportionally by 5.4% [90% CI: 3.8%, 6.9%].
We can calculate its contribution in terms of "compute-equivalen gains" numerically: we first calculate the predicted loss for transformer with some N and D, and the predicted loss for
The sigmoid is introduced to make easier fit the model by improving optimizer stability.
7

---

non-trans non-transformer with the same imputs. We then determine reduction in N and D to match this difference loss. Compute is then approximated usual, as C' = 6N D. In short, if an innovation halves the compute needed to achieve specific loss, then that innovation has compute-equivalent gain of
Based on 100 bootstraps, we obtain median estimate of 9.6x [90% CI: 4.3x, 34.5x] ] for the transformer's compute-equivalent gain." 11 This substantial gain indicates that the efficiency offered by the transformer architecture is equívalent to around log(9. (9.6)/ log(2e4) x 23% of the total gains from algorithms in the past nine years, or nearly two years of algorithmic progress in the field." Moreover, this could understate the gains if the transformer architecture also provides a convenient vehicle through which productively channel compute, thereby facilitating some of the gains through the scaling of compute that have likely dominated the overall gains we have seen recently.
One caveat here is that the measured significance of the transformer may depend how it is evaluated. For example, transformers may be better adapted to long contexts than recurrent networks, and evaluations using longer contexts (e.g. 1000 tokens) may suggest larger improvement from transformers than evaluations using shorter contexts [Kaplan et al., 2020]. We have not explicitly controlled for context length here, and discuss the potential impact of this assumption more detail in appendix E.
Discussion and conclusion
4.1 Summary of our findings
This paper presents a comprehensive empirical analysis of algorithmic progress in language model pre-training from 2012 to 2023. By curating dataset of over 200 language model evaluations on WikiText and Penn Treebank benchmarks, we quantify the relative contributions compute scaling and algorithmic efficiency improvements to the overall performance gains. Our key findings are as follows:
First, we estimate that the compute required to reach set language modeling performance level has halved every 7-8 months on average since 2012. This supports the common intuition that language modeling is an unusually aapidl--avvanciigg field of computer science.
Parameters
Parameters
10th
2014
1031
2014
2016
1025
2016
10k
2015
1034
2018
2020
2072
1033
2020
2024
2026
101
2022
20174
10a
1021
10aa
30'
10c
10a
1046
10
1012
10P
1016
10
2012
1033
10
10a
Training Dataset Size (Tokens)
Training Dataset Size (Tokens)
Predicted requirements for GPT-2 performance
Predicted requirements for Chinchilla performance
Figure Pareto frontiers for GPT-2 [Radford et al., 2019] and Chinchilla [Hoffmann et al., 2022] level performance on WT103. We truncate the fronticrs to a factor of 1e3 greater or smaller than the existing training dataset size and parameter size of the actual model since extrapolating further out would not be reliable.
Second, our work reveals that the majority of recent advancements in language modeling stem more from scaling models and datasets than from pre-training algorithmic innovations. A Shapley
1Thhs assumes compute budgets of frontier models today, at 10a6 FLOP. At lower compute budgets, such as 20 FLOP, the gain is still substantial 8.4x 190% CI: 4.2x, 19.5x].
contribution, we also attempted to check the rate of algorithmic progress while subsetting our data to non-transformers only. However, this roughly halves the data available for fitting, and our resulting estimates are unfortunately extremely noisy: the estimated doubling times have 90% confidence interval of 11.7 8.0 months.
1'Given the magnitude

---

value-basee based analysis suggests that 60-95% of the performance gains stem from compute scaling, while algorithms contribute only 5-40%.
Third, the introduction of the transformer architecture in 2017 was a major algorithmic advance, representing between 3x and 46x in compute-equivalent gain, which accounts for more than 10% of the algorithmic innovation in pre-trained language models in the past decade. This highlights the significance of the transformer as a key architectural breakthrough in the field.
4.2 Limitations
While our analysis is advance in quantifying algorithmic progress, several limitations reduce the precision of and temper our confidence in our estimates:
Lack of estimates of gains from specific innovations. Our model is specified to quantify algorithmic progress over relatively large time periods (e.g. over several years). However, it is unable to give reliable tne-grained information, such as progress over shorter time scales, or the significance of specific innovations. Experimental work is better suited to estimating efficiency gains for specific algorithmic innovations.
Limited availability of quality data, resulting in noisy or unrealistic estimates of progress. The approach we use in our analysis relies heavily on having many data samples across many years. This proved be very challenging for a number of reasons e.g. models are not always evaluated on the same benchmark, data is relatively sparse prior to 2017, and papers may not report relevant information such as parameter counts. Among other reasons this can result in our estimates being very noisy, yielding wide confidence intervals over doubling times. In addition, algorithmic improvements and scaling have historically been introduced concurrently, and this correlation between the two in our dataset can make it hard to disentangle their relative contributions to overall effective compute growth.
Inconsistencies in model training and evaluations. Inconsistencies in evaluations are well-known. While we have excluded non-standard evaluations from our dataset, our dataset spans models with different tokenization schemes, text preprocessing. stride lengths, and other details. This introduces noise and potential bias in our estimates of algorithmic progress, as researchers might have adopted more favorable evaluation schemes over time. However, our estimated reductions in perplexity from algorithmic improvements are large; likely larger than can be accounted for by changes in evaluation procedures. We expand on these points in Appendix E.2.3.
Inability to distinguish between data quality and efficiency in data use. Reduction in data requirements could be due to both improved data quality and improved algorithms, but our model is not equipped to distinguish between these effects. Understanding the relative contributions of each could be subject of future research.
Reliance on the Chinchilla scaling law. The scaling law from which our model is derived applies to dense transformers following a GPT-3 architecture [Hoffmann et al., 2022, Rae et al., 2021]. However, we use this scaling law to model algorithmic improvements in different transformer architectures, recurrent neural networks, etc. Future algorithms might also follow different scaling laws (e.g. GPT-4 is rumored to be mixture of experts). However, we believe it is likely that our core results should still hold: for one, neural scaling is not phenomenon restricted to transformers (e.g. known to happen in RNNs as well, see Kaplan et al. [2020]). We find that a wide range of statistical model structures provide consistent estimates, and that alternative methods of estimating pre-training algorithmic progress also give similar results (see e.g. appendix A), so it is probable that our core results are robust to the use of the scaling law from Hoffmman et al. [2022].
Limited insight about future progress. While the results from this paper could be used to inform one about future progress in language modeling, our paper focuses on historical improvements. Future rates of progress could be slower (e.g. if one thinks that historical progress consisted of picking "low hanging-fruit",, but they could potentially also be faster (e.g. due to increased research interest and investment). Expectations about future progress need to account for factors such these, which we do not discuss in depth for the most part.

---

4.3 Conclusion
Using dataset of over 200 language model evaluations spanning 2012-2023 evaluated on Wikitext and Penn Treebank, we find that the compute required to reach fixed performance threshold has halved approximately every 8 months. This much faster than the rate associated with Moore's law and many other domains of computing. While algorithmic innovations have occurred rapidly, compute scaling has expanded by over a million-foll fold in this same period, exceeding the gains from algorithms and constituting the predominant source of performance improvements in recent years.
Overall, our work provides a quantitative estimate of the rapid pace of progress in language modeling. It also reveals the dominant role of scale rather than algorithms for recent gains. Future work could benefit from extending this analysis additional, specific benchmarks and more closely examining the impact of data quality improvements and the gains from additional specific innovations. Despite its limitations, this research demonstrates the valuable insights that can be gained from detailed statistical analysis of extensive datasets of machine learning results. By identifying the main drivers of performance improvements, this work lays the groundwork for further exploration and understanding of these trends in the field.
Acknowledgments
We thank Tom Davidson, Pablo Villalobos, Josh You, Lukas Finnveden, Eli Lifland, nostalgebraist, David Schneider-Joseph. Danny Hermandez, Alyssa Vance, Yafah Edelman, Ben Edelman, Matthew Barnett, Ben Cottier, Keith Wynroe, Markus Anderljung, Carl Shulman, Marius Hobbhahn and Nikola Jurković for their feedback. We thank Eduardo Roldán and Robert Sandler for helping design and implement graphs.
References
Uri Alon, Frank Xu, Junxian He, Sudipta Sengupta, Dan Roth, and Graham Neubig. Neuro-symbolic language modeling with automaton-augna augmented retrieval. In International Conference on Machine Learning, pages 468 468-485. PMLR, 2022.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. 2018. doi: 1048550/ARXIV.100... 10853. URL htttp:://artie org/abs/ hts//bb//8009 10853.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020. doi: 10.48550/ ARXIV.2005. 14165. URL https: //axxiv. org/ abs aabs/2055 14165.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. 2021. doi: 10.48550/ARXIV221100 14168.
wwtpsiiiw.rg//b////// /2100 14168.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher R'e. Flashattention: Fast and memory-efficient exact attention with io-awareness. ArXiv, abs/2205.14135, 2022. URL 2491111871.
https://api. semanticscholar hat.aarr.rrrgroorrpu...00011111
Tom Davidson. speeds. -est----cmpptt----rr--------e
What a compute-centric framework says about AI takeoff https: //wp: http://wwww.immmmffffffooooe
centric-eteeeeeee framework coopputeeeemtri---faaooooooo
says about- ai aituut-ai-takoo......oo 2023.
10

---

wtt.:////2/////////nnnannfeaaee/ Tom Davidson, Sean-Stanislas Denain, Pablo Villalobos, and Guillem Bas. Ai capabilitiess can be significantly improved without expensive retraining. arXiv preprint arXiv:2312.07413, 2023.
Harm de Vries. the long (context) run, 2023. URL httt:///wwwwwaammdrreess.ooom context-length/.
Florian E Dorner. Measuring progress in deep reinforcement learning sample efficiency. arXiv preprint arXiv:2 2021.
Jasha Droppo and Oguz H. Elibol. Scaling laws for acoustic models. In Interspeech, 2021. URL https://api. semanticscholar org/ wttg/oopppsIDD:254685551.
Ege Erdil and Tamay Besiroglu. Algorithmic progress computer vision. 2022. doi: 10.48550/ ARXIV.2212.05155.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept, 2021.
Google Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of to- kens of context. 2024. URL sssp com com/deepmind-medi a/gmmini///////////ee/emm Siciimi_v_____ppp___ .df.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. ArXiv, abs/2312.000552, 2023. URL httpss///ap..emmmnicccccoooo. wwrg/CorpusID:.. (GorpusID:2655517777 265551773.
semanticscholast
Danny Hernandez and Tom Brown. Measuring the algorithmic efficiency of neural networks. arXiv preprint arXiv:2005.04300. 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15555 2022.
Hugging Face. wwt.fformrrs/ppppexexxtttt 2023. [Online; accessed 14-Nov-2023).
Perplexity of fixed-length models. https://huggingfaces co/cc///
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08.0 100601, 2020.
Andrej Karpathy. Deep Neural Nets: 33 years ago and 33 years from now. http://karpathy: 2022. [Online; accessed 21-July-2022).
Rémi Leblond, Felix Gimeno, Florent Altché, Alaa Saade, Anton Ruddock, Corentin Tallec, George Powell, Jean-Bastien Grill, Maciej Mikula, Matthias Lochbrunner, Michael Math- ieu, Paul Caron, et al. Alphacode technical report. Technical report, Google Deep- Mind, 2023, URL https::/sstrage..... sowtoorrgeggoglleeaeeeoooooooeee googleapis.....oo.oo..o........m com wwwccomm/////eeeee homm/deepmidd--een aadeppminddeediia//llpeeeee mediaa/AlphaCod2)) Bechaaoode_TechhBeee
pdf.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied handwritten zip code recognition. Neural computation, (()):541-551, 1989.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-levee code generation with alphacode. arXiv preprint arXiv:2203. 07814, 2022.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and lan- guage with ringattention. ArXiv, abs/2402.08268, 2024. https: //api. iechanta Org/CorpusID:225 267637097.

---

Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus. Noua- mane Tazi, Sampo Pyysalo. Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. ArXiv, abs/2305.16264 2023. utt.ttps:://// //api semanticscholes org/CorpusID: 258888192.
OpenAl. GPT-4 Technical Report. 2023, URL https://cdn. openai careaiicoomppaaa www.papers/ http://pppgpptae-p.
gpt - pdf.
Dylan Patel and Gerald Wong. Gpt-4 architecture, infrastructure, training dataset, costs, vision, moe, 2023. URL attps://ww..smmtttt
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican. Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang. Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Ban Bean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. 2021. doi:
10:4850/ARXI.21122..11 1446. URL https: //arxi//xxiiv wwwaaiii.g///aaeeeeeee org/ hrrg/abs/200 2212. 11446.
Jaime Sevilla, Lennart Heim, Anson Ho, Marius Hobbhahn, Tamay Besiroglu, and Pablo Villalobos. Estimating training compute of deep learning models. Technical report, 2022.
Sho Takase, Jun Suzuki, and Masaaki Nagata. Direct output connection for a high-rank language model. ArXiv, abs/1 10144 2018. URL https://api semanticscholar Orgg/CorpuIDD 52138320.
Yi Tay, Mostafa Dehghami, Samira Abnar, Hyung Won Chung. William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Metzler. Scaling laws model architectures: How does inductive bias influence scaling? ArXiv, abs/2207.10551, 2022. URI. httpst httpstss: //api.
semanticscholar Dtrg/CorpusID:... 250920512.
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, (7777995)4476-822 2024.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. ArXiv, abs/1706.03762, 2017.
Chenguang Wang, Mu Li, and Alexander Smola. Language models with transformers. arXiv preprint arXiv: Nee:::9440000000 2019.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinay Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang. Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. ArXiv, abs/2309.16039, 2023. URL https: //ppi...... www.nticscholll...... wwwpcchhllaaroooooooooooseeeeeee org//ooppuIDD:22134422
12

---

A Observing improvements in the data
Besides the statistical model that we presented in section 2.1, we can also attempt to obtain doubling time estimates more directly. For example, we can look at LLMs that achieve close Hegatron-LM' or GPT--''s s level of performance over time, and see how much less compute is used. Doing so reveals that we need between 5-fold and 100-fold less compute per year in 2023 achieve the same performance achieved is between 2019 and 2023, amounting to halving time of between 11 and 17 months. This within the 90% confidence interval of our aggregate doubling time estimates, from 1.7 to 22.5 months.
Relative training compute 300%
Relative training compute
300%
Negatron UN
Infinty former
Comjresstre Pranifomerr
SM Becaving Fast
ALBI
We gies Trai Transtecmee Tonsfomi GFM
ininky farmer
30%
tranaiGheet,
10%
10%
521 sonneciion
3%
Doubling Time: months
SAUt Large
Doubling Time: 16.6 months
1%
2020
2021
2022
2022 Publication Date
2023
Publication Date
(a) Relative training compute needed for
(b) Relative training compute needed for ALiBi level performance
Megatron-LMN level performance
Figure Relative compute (relative to baseline model) used to train models achieve same evaluated perplexity as Megatron-LN, LM and ALiBi respectively. Doubling times of effective compute are 11.7 and 16.6 months using least squares regression for Megatron-LNN (cross-ent croos-yettropy range 2.87-3.06) and ALiBi (cross-entropy range 1.18-1.34) respectively. Circles proportional to the compute used during training.
This approach provides some insight but it has its issues, which is why we do not rely on it. For instance, it depends on the particular choice of reference performance. Moreover, this approach requires identifying models with similar or better performance at later dates, where the training compute is known. However, the data for the latter relatively limited. We thus opt for using an arguably more principled approach with our core model presented in section 2.1 for our main results.
B The gains from better scaling laws
We estimated the compute savings afforded by the Chinchilla scaling law, as proposed by Hoffmann al. [2022], in contrast to the previously dominant understanding based on the work of Kaplan et al. [2020]. First, we defined loss functions L(N, D) for both the Kaplan and Chinchilla scaling laws. Following this, minimized these loss functions across variables D and N, considering different levels of compute budget. For each specified budget, we then calculated the amount of compute required under the Chinchilla scaling law achieve loss equivalent the minimum loss obtained under the Kaplan scaling law. The Compute--qqivalens Gain (CEG) was subsequently determined as the ratio of the original compute budget to the compute required by the Chinchilla scaling match the Kaplan loss.
We find that the compute equivalent multiplier from the Chinchilla scaling laws for dense autoregres- sive transformer models is between f.25-fold (for GPT-2 scale models) and 1.6-fold (for PaLM-scale models Chowdhery al. [2023]). 13
We use PaLM as reference rather than larger more recent models such as GPT-4 because it was unlikely that GPT-4 would have been trained without an improvement in our understanding of scaling laws, whereas PaLM was likely trained prior to the development of updated scaling laws.
13

---

Compute muitiple equivalent
PaLM (5408)
GPT-3
scale
scale
1.0x
10
10ah 10a
10ah 10a4 104h
Compute
Figure 5: Compute equivalent multiplier from optimal scaling from switching from Kaplan al. [2020] Chinchilla (Hoffmann et al. [2022]) scaling laws as function of training compute for dense autoregressive transformer models. Note that GPT-3 and PaLM (540B) use around 1.7 and 1.44 tokens/parameter respectively, close to what the Kaplan scaling laws recommend, suggesting that Kaplan-scalin was close to what was practiced the time.
C Core model parameter estimates
The core model that we use was chosen based on leave-one-out cross validation, and defined similarly to equation but with a few modifications. The most important change is that A and B are estimated separately for each benchmark, whereas all other parameters are benchmark-agnostic. In minimum No and Do values in
order help with model fitting, we normalize N and D to dataset, and reparameterize and as exponentials. In full, our model is
L exple (xplo'mma Cyyaar Yo) Opaarm log( /No)] + [xxp[[( [Soms - (Y - -data (og(D/ Do)]
To estimate these in benchmark-specife fashion, we introduce dummy variables zwrz and Trrn for WT2 and PTB respectively. We then complete the model definition as follows:
d'onstt = Occonst + Orosst.TT--PTT + Occonst, WT2--W12,
B
Pennst + PTO + Deome,WT2ZWTZ.
Our parameter estimates are summarized in Table
One observation about the parameter estimates in Table 2 is that the confidence intervals for Olyear and Byear are statistically significant at the 5% significance level, while O'param and Baata are. As mentioned in section 3.2. the result is that the model fails to obtain statistically significant estimates of effective parameter and effective data doubling times. However, when we use these estimates to determine effective compute doubling times, we obtain statistically significant estimates. The reason for this illustrated in Figure 6-the estimates for Otyear and Byear are clearly negatively correlated. In particular, when ('year is positive, Byear is negative and vice versa, such that the overall estimated effective compute doubling time is always positive.
C.1 Comparing estimates to earlier work
Given that our core model is similar to previously proposed language model scaling laws, we can compare estimates
how well they correspond prior work. In particular, the estimates for O'puram and Baas in Table 2 suggest that cross entropy loss scales roughly 1/20 where C' is training compute. In comparison, Kaplan et al. [2020] find a scaling exponent of around -0.048, and Hoffmann al. [2022] estimate values of around -0.3. Given that our model is constructed based on the scaling law in Hoffmann et al. [2022], we might a priori have expected our estimated to match those more closely-so what explains the difference?
One way to understand this discrepancy to consider the scaling laws on the same plot, shown in Figure 6. Here observe that the scaling laws strongly diverge for compute values below around
14

---

Estimate
90% CI
Gcenst
(0.232)
0.903
0.647, 1.300
0.000
-0.264, 0.206
(0.139)
OcconsLW12
(0.114)
0.000
-0.213, 0.164
-0.011
Qyear
(0.021)
-0.032, 0.017
0.083
Opaam
(0.017)
0.058, 0.102
Poonst
0.791
0.197, 1.057
(0.240)
BeonsLPTB
0.190
-0.010, 0.432
(0.201)
Ben
0.163
-0.001, 0.425
(0.162)
Byear
0.038
0.017, 0.082
(0.022)
0.030
Baata
0.020, 0.050
(0.010)
Table 2: Parameter estimates from the model described in equation 8, rounded to decimal places. We report 90% confidence intervals all the parameter estimates by bootstrapping 100 iterations.
Cross entropy loss
Chinchilla
Kaplan
-
Durs
Ayer
3.66 3 FLOP
1.1 1022
FLOP
0.100
94.9%
5.1%
0.075
0.025
0.000
0.025
10
10
2
-0.1000..05-00050- 0.05 0.000 0.025 0.050 0.075
FLOP
Figure 6: (a) (Left) Comparison of scaling law predictions from our preferred model and previous work, specifically Kaplan et al. [2020] and Hoffmann et al. [2022]. The grey lines represent scaling laws based on bootstraps of our proposed model. The two vertical dotted lines indicated the 10th to 90th percentile range of training compute values in dataset. (b) (Right) Estimated values of Cyeaa and 8year from 1000 bootstraps. 94.9% the bootstrapped estimates the right of the line Cvyear Byear
18 to 1019 FLOP (and the same true for values greater than 10 FLOP). However, between these two regimes the scaling laws appear much more similar in slope.
This observation suggests the possibility that the discrepancy in estimated scaling exponents is due to the range of fitted data. Indeed, around 80% of our models with known training compute estimates lic between ~ 4 x 10 FLOP and 1022 FLOP. This suggests that large fraction of our data lies within the regime where it is hard for our model to distinguish between the exponents from Hoffmann et al. [2022] and Kaplan et al. [2020].
Another possible explanation for this discrepancy that we considered is that it is due to the omission of an irreducible loss term our core model, resulting in an omitted variable bias. However we do not much weight this explanation for our fits our robustness check using models with an irreducible loss term (see section H), we obtain very similar scaling exponents to those obtained in core model.
15

---

D Significance of the transformer architecture
Similarly to the doubling times for effective compute, we consider the predicted Compute-Equivall Gains by applying the same modification in Equation 7 to different model specifications. Most models yield estimates within similar ballpark as our core model (model 7 with 5 = 0.0025), with some models yielding relatively noisy estimates. That said, three models predict notably larger efficiency contribution from the transformer, and this suggests that there is plausibly fairly large degree of cross-model uncertainty present. These results are shown in Figure 7.
Estimate
90% CI
Cunss
(0.442)
0.358
0.000, 1.257
Occonst,PTT PTB
(0.148)
0.030
-0.234, 0.151
Occonst
0.000
(0.095)
-0.184, 0.133
Otyaar
(0.031)
0..48
-0.077, 0.018
param
(0.032)
0,117
0.052, 0.159
Aconst
1.184
0.207, 1.330
(0.381)
BeonsLITB
0.108
-0.000, 0.299
(0.161)
0.101
Beonst,
-0.000, 0.481
(0.142)
Byear
0.052
0.017, 0.085
(0.021)
Baata
0.018
0.012, 0.052
(0.014)
2.910
7
(0.209)
2.604, 3.238
Table Parameter estimates the model described equation 7, reported to decimal places. We report 90% confidence intervals for all the parameter estimates by bootstrapping 100 iterations.
Compute Equivalent Gain (CEG)
102
9,00
101
(0.005) (0.001; 10.001) (0.02)
(0.02)
(0.02)
(0.001)
(0.005)
(0.005)
(0.0025)
(0.001)
(0.000)
Model (delta)
Better cross-validation performance
Worse cross-validation performance
Figure 7: Contribution of the transformer in terms of Compute Equivalent Gain, by introducing the structure equation 7 different model specifications.
16

---

E
Dataset
E.1
Performance measure and dataset
A key component of measuring progress in machine learning algorithms is the presence of a per- formance metric benchmark. Since our focus language modeling, token-level perplexity is natural choice for this metric, for which we choose three benchmarks: WikiText- 103, WikiText-2, and Penn Treebank. Note that WT2 and WT103 are both constructed from articles on Wikipedia. The two benchmarks share the same validation and testing set, while WikiText- 103 has much larger vocabulary and training set. In total, our dataset constructed from 226 papers, from which we collect around 410 models that have reported token-level perplexity. Of these models, 370 contain sufficient information to be considered for analysis (perplexity, parameter size, publication date, and size of dataset).
parplexity
LEL
BCAN ML drippe)
UIT Veltrnd LETM
Training FLOP JuS
2NK-owlger rebe-dyraricer
LSTM
3.2
Large regataized
Translamer Emtasssing)
LLMMA-233
(1628)
2.0
(P72218420)
Turing NLO
Chrenlile
LLaM6-658
0.
2014
2010
2010
2020
2022
2024
Publication date
Figure 8: Log of perplexity models used in our work, of over 231 language models analyzed in our work spanning over 8 orders of magnitude of compute, with each shape representing model. The size of the shape is proportional to the compute used during training. Comparable perplexity evaluations are curated from the existing literature and from
evaluations.
E.2 Perplexity
A standard metric for LLM performance is the measured test perplexity on standard datasets. For a language model, this is defined in as the exponential the cross-entropy between the model predictions and the test set, i.e. Perplexity ch.
We choose this metric for two primary reasons. First, this commonly reported measure of performance, which allows to gather large dataset for our analysis. Second, the simple relation between perplexity and cross entropy allows us to easily relate our model to neural scaling laws.
a paper reports the perplexity of just one model, we collect that singular data point. However, when paper presents multiple models, only those meeting any of the following criteria are included in our dataset:
1. The model is trained or evaluated on a different benchmark dataset. A model trained on two different datasets can be used as reference understand how perplexity metrics reported on different benchmarks relate to one another. This can be helpful e.g. for data imputation
2. The model is constructed with a drastically different parameter size, as such data inform the impact of scaling
3. The model has significant difference in the algorithms used than other models in the paper
For papers presenting many (10 more) models, we exclude some from our dataset to prevent possible bias from over-representing results from few studies. We prioritize models with the lowest perplexity in their category, often highlighted in bold within tables. We also exclude minor algorithm alterations and ablations that do not impact the parameter count. Appendix we take an alternative approach by including all models from each paper and explicitly modelling the autocorrelation
17

---

structure from results from the same paper. In doing so, we find results highly similar to those we present in the paper.
E.2.1 Context length
Another consideration when analyzing algorithmic innovation in language models pertains to the context length. For one, measured perplexity on benchmarks can depend on the context length [Kaplan et al., 2020, Xiong et al.. 2023]. Different systems may have been trained or evaluated using different context lengths, and this might make model perplexity scores less directly comparable.
One way to try and quantify the magnitude of this effect is to look at studies that compare the change in perplexity given change in context length. For example, based on the scaling relations relating loss and context length from Xiong et al. [2023] and Kaplan et al. [2020], mack-of-the-envel calculations suggest that loss reductions each year due to increasing context length could be 10-60% as large as the loss reductions from algorithmic progress.
In particular, Xiong et al. [2023] finds relation between context length and validation loss for different versions of the LLaMa 2 language model. We can estimate ballpark values for context length over time based data from de Vries [2023], and use this to roughly estimate how expanding context lengths has decreased loss over time (e.g. a decrease of a few percent per year). We can then compare the magnitude of this effect the contribution of algorithmic progress decreases in loss, and we typically arrive at values between 10-60% of the overall algorithmic progress contribution. I4 We perform a similar analysis with the scaling relation described in Kaplan al. [2020], with similar results. If this rough calculation is correct, it suggests that increasing context length may have been a fairly important dimension along which algorithms have been improving.
On top of measured reductions to averaged perplexity per token, one might also consider the increasing ability of language models to perform long-contex tasks to be largely downstream of algorithmic progress in itself. Being able to handle long contexts is a key motivator for several recent algorithmic innovations [Liu et al., 2024, Gu and Dao, 2023, Gemini Team, 2024], and this has likely grown very substantially since the introduction of FlashAttention [Dao et al., 2022, de Vries, 2023]. We consider this to be important avenue for further investigation.
E.2.2 Tokenization
One way to get a sense of the impact of tokenization on measured doubling times to introduce a fixed effect that depends on the benchmark vocabulary size into our preferred model. In particular, we introduce an irreducible loss term to Equation 8, of the form y log( vocabulary size).
As with the rest of our analysis, we perform bootstraps to obtain a distribution over estimated doubling times, and further fold this model into our cross validation analysis. particular, this model predicts a median effective compute doubling time of 5.5 months, with 90% confidence interval of 2.6 to 11.8 months. In cross validation, this model performs essentially as well as our preferred model, with a MSE loss of 0.046-0.045 0.048 in both cases (depending results are very much in line with the results from our main model, lending some weight to the view that differences in tokenization schemes used in practice do not substantially change our core results.
the regularization strength). Both of these
One possible reason for this that in practice, the tokenization schemes used for evaluating language models on the considered benchmarks (i.c. WT103, WT2 and PTB) are typically highly similar, and so including a contribution from vocabulary size has only a limited effect within each dataset. For example, typical tokenizers for PTB, WT2 and WT have vocabularies of roughly 10k, 22k, and 268k tokens respectively. We illustrate this in Figure 9, where for each benchmark, the majority of vocabularies fall into the same histogram bin.
E.2.3 Inconsistencies in perplexity evaluations
Inconsistencies in benchmark evaluations a well-known issue in the machine learning community. These issues can introduce noise (if inconsistencies are 'random'), or bias if they systematically change over time. We curated data so that models were evaluated in roughly comparable ways, but often the precise details of evaluations were lacking, so that we could not verify the precise
"TThs of course depends on variables like how quickly context lengths have expanded over time- details of this calculation can be found in this spreadsheet.
18

---

Number of models
70
WT103
50
40
10
Vocabulary
Figure 9: Histogram showing the most common vocabulary sizes models in our dataset, separated by
benchmark.
evaluation procedure. However, there are many other subtleties with evaluation setups that may also cause perplexity results differ, such pretraining data, test-time adaptation [Takase al., 2018], tokenization schemes, strides, and text preprocessing.
Variation source
Significance
Training
Early models in our dataset often exclusively trained the benchmark training set before evaluation, whereas later models are generally pretrained on larger pretraining dataset. Since the majority of our dataset involves the latter type of model, we expect this effect to be minor. The direction of this effect is somewhat ambiguous: the training set data is likely 'closer' to the test distribution relative training the relevant training distribution
other internet text-corpuses, could yield lower performance at fixed budgets.
A further subtlcty is that large text corpuses often, but not always, contain Wikipedia data, Pretraining otherwise, which could be attributed algorithmic progress. A small number of existing results illustrate this effect. For example, 1.3B GPT-3 reimplemen- tation, including Wikipodia and other high-quality data in fixed-size pretraining reduced Wiki perplexity from 8.27 to 5.59 [Gao al., 2020, Table 3). In our dataset, this effect is likely show up small number of models receiving tooteethan-otherwiese perplexities in WikiText, around the time of the transition to large-scale pretraining This is true of GPT-2, for example.
such distributions could yield larger gains than
before inclusion Wikipedia data became common.
Word vs sub-word tok- enization
Changing to sub-word vocabulary can reduce perplexity substantially in an otherwise unchanged architecture, for example by ~30% in pre-GPT-2 LSTM [Wang et al., 2019]. our dataset this likely show up as one-time change, potentially exaggerating the algorithmic improvement in language models around GPT onwards.
Preprocessing and run- time adaptation
Different preprocessing of data affect results significantly. For example, inverting word-level tokenization artifacts in WikiText-103 improved perplexity by ~100 [Radford al., 2019]. Runtime adaptation sometimes has similarly large effects [Alon et al., 2022]. We expect that these sources of variation mostly act as noise in our dataset, although if these improved over time, they might inflate estimates of algorithmic efficiency.
Stride length
The move larger models led evaluations using larger stride for lower com- putational cost. This can increase perplexity, but only on the order of ~00% at realistic settings [Hugging Face, 2023]. We believe this should act as another source of variation, but without strong influence
overall findings.
Table 4: Sources of variation and their significance in language modelling evaluations.
Overall, we do not see these issues substantially undermining our results. Our mainline estimates imply that 1 year of pre- pre-training algorithmic progress amounts to reduction of perplexity of around
19

---

Estimate
90% CI
Acenst
(0.060)
1.126
1.008, 1.217
CT
(0.058)
0.172
0.070, 0.219
CooostLW12
(0.061)
0.086
-0.003, 0.187
Qyear
(0.008)
0.009
-0.009, 0.016
0.052
0.040, 0.067
param
(0.008)
Beonst
0.349
0.221, 0.595
(0.125)
POonsLPTB
0.002
-0.001, 0.120
(0.086)
Beonst
0.171
-0.000, 0.412
(0.131)
Byear
0.043
0.020, 0.091
(0.022)
Bdata
0.037
0.015, 0.060
(0.017)
Table 5: Parameter estimates for our main model, fitted to dataset of models trained on the training set of WikiText PTB, but not the corresponding test sets. We report 90% confidence intervals for all of the parameter estimates by hootstrapping 100 iterations.
10%, which is much larger reduction than seems plausible explain by changes in the average year-to-year evaluation procedures.
Given that perplexity on these datasets has persisted standard measure of language modeling performance in the literature, we expect that differences in perplexity will broadly reflect genuine underlying differences in model capabilities.
We further consider one particular source of variation from Table in more detail, namely the training set. To do this, went through all the models in our dataset and classified them into one of three categories: (I) not trained on either the training or test sets of WikiText, (2) trained on the training set but not the test set of Wiki'Text, or (3) trained both the training and test sets of WikiText." 15 We then repeated this procedure for PTB.
Based on this classification, around 2% of models fall into the first category, around 85% of models are in the second category, and the remaining models are the third category. To test whether our results are robust to this change in categories, we fit our model to models only in the second category. This yields effective compute doubling times of 5.7 months in the median case, with 90% confidence interval of 11.7, 12.6] months, consistent with our core results. The parameter estimates and associated uncertainties are listed in Table 5.
E.3 Dataset Size & Epochs
In reporting training dataset size, we generally record the size of pre-training datasets as well fine- tuning datasets if the model in question has been fine-tuned. The number of epochs is usually sourced from the papers which describe the language model in question, but if not provided, we estimate it via
num jum_epochs- context pearext__ennghh__c 1ength_tokens tokens batch size training_steps
pretrain_tokes tokens
We adjust our epoch and dataset size calculations accordingly for models like GPT-Neo and Pythia, which report the effective number of tokens seen in training.
E.4 Parameter Size
We use the reported parameter size value if stated the paper. We impute the parameter size from the previous models for papers that do not specify parameter size but follow known, otate-of-the-art
Note that the test sets for WT103 and WT2 are the same, so we group them together here to consider "WikiText".
20

---

models. Otherwise, we rely on other papers referring to the model's parameter size or manually compute parameters for certain RNN and LSTM models based on provided details about the model architecture (e.g. the number and size of hidden layers).
E.5 Inclusion and exclusion criteria
We exclude several models in the dataset from the analysis based whether they meet any of the following primary criteria: (1) use of a retrieval mechanism, (2) use of model compression or pruning, (3) use of neural architecture search, (4) use of teacher-learner or knowledge distillation mechanisms, (5) use of cache models. These models are excluded because we expect these models to exhibit significantly different scaling behaviors from other models we analyze. In particular, they can substantially change the ratio of parameters and data that would "optimally" minimize loss given some compute budget.
E.6 Dataset review
In a previous version of this paper, our core results had a narrower confidence interval as well as slightly longer median estimates." One reason for this change is that reviewed the data second time and fixed any identified errors while this changed our results, was well within the uncertainty ranges we had previously identified in our original paper.
That said, we further decided to modify our model aggregation approach to better represent cross- model uncertainty. In particular, rather than aggregating over the top 10 model-delta pairs in cross validation, we instead aggregate across different model numbers (with the regularization strength that yields the best cross validation performance).
F Quantifying training data D
One important degree of freedom in the modeling process is how to define the "training data". In particular, there we consider three possible definitions:
1. Training dataset size: The number of tokens in the dataset on which the language model was trained. This definition ignores the possibility of improvements in performance from training for multiple epochs, and is the approach taken in Erdil and Besiroglu [2022].
2. Tokens seen: The total number of tokens seen during the course of training language model-this is the definition adopted in Hoffmann et al. [2022]. This is equivalent to the training dataset size if the model is trained for one epoch on the training set, which is fairly common practice but not totally ubiquitous. models are being pushed towards training with multiple epochs on the same data e.g. GPT-4 was reportedly trained for epochs on text and 4 epochs on code [Patel and Wong. 2023]. In cases where the tokens seen is not directly reported in paper, estimate it via tokens seen R num. epochs x training dataset size.
fact, it is possible that more recent
3. Tokens seen with diminishing returns: One problem with the previous approach that seeing data repeatedly may yield diminishing returns. Muennighoff et al. [2023] find that the benefits in loss reduction drop significantly when training on more than epochs.
In order to test the robustness of our most important result (compute doubling times) to this degree of freedom, repeat our doubling times analysis in section 3.1 but account for the number of training epochs. We then replace D either with tokens seen (epochs times training dataset size) or tokens seen with diminishing returns. For each case we consider two possibilities-eitlee datapoints for which the epoch number is unknown (this drops around 100 datapoints), or we impute the epoch number as 1. We report our core parameter estimates in Tables 7 and 8, and our doubling time estimates in Table 6.
we drop
From the results in Table 6, we see that there is substantial overlap between model estimates, but the estimates are highly noisy. The most similar are the original model based on dataset size, and model
"In particular, our core model had a median doubling time estimate of 8.4 months [4.5, 14.3], and our aggregate model had median of 7.8 months [1.5, 17.6). In contrast, our core model now has a central estimate of 6.1 months [2.4, 14.8] and our aggregate model has median estimate of 7.5 months [0.9, 24.6].
21

---

Definition
Ceff doubling times (months)
Dataset size (section 3.1)
[2.6, 6.1, 14.8]
Tokens seen
[0.9, 3.0, 22.0]
Tokens seen impute
[-7.3, 1,7, 20.2]
Tokens seen w. dim. returns
[0.9, 3,4, 10.9]
Tokens seen dim. returns impute
[2.4, 5.8, 16.4]
Table 6: Estimated effective compute doubling times using core model (Equation 8), using three different
definitions of "training data". Numbers in the square brackets correspond to the [2.5th, 50th, and 97.5th percentile] after bootstrapping 100 times.
with tokens seen with diminishing returns and imputed epochs, where the latter places slightly more weight on longer doubling times.
In the case of tokens seen with imputed epochs, the model predicts no statistically significant improvement in compute efficiency. On the other hand do not observe this without imputation, which potentially suggests that our imputation strategy (i.e. assuming epoch of training when epoch counts are unknown) is suboptimal under this definition of training data.
That said, there are strong reasons to be sceptical of using "tokens seen" as definition given that diminishing returns additional training epochs are indeed observed in practice. Indeed, some papers report training for between 100- 1000 epochs, which would result in fairly large variations in dataset size estimates if diminishing returns not considered. We ultimately opt quantify training data in terms of dataset size for simplicity, although a definition based on diminishing returns is also feasible.
Estimate
90% CI
Estimate
90% CI
Occonst
(0.416)
0.215
0.088, 1.342
Occonst
(0.391)
0.914
0.000, 1.175
-0.000
-0.000
Olconst,PPB
(0.210)
-0.471, 0.205
Olgonst,
(0.188)
-0.437, 0.133
0.014
0.047
Occonst, WT2
(0.183)
-0.043, 0.462
Olconst, WT2
(0.166)
-0.076, 0.359
0.090
0.020
O'year
(0.040)
-0.101, 0.011
O'year
(0.035)
-0.101, 0.011
O'puram
(0.036)
0.168
0.077, 0.193
O'param
(0.035)
0.090
0.070, 0.172
Beonst
1,299
0.001, 1.396
Beonst
0.779
0.463, 1.316
(0.406)
(0.285)
Peonst.IB
0.152
0.001, 0.732
Beeast.PTB
0.233
0.000, 0.467
(0.210)
(0.176)
Beo
0.116
-0.468, 0.200
Beoest,
0.122
-0.189, 0.313
(0.250)
(0.194)
Byear
0.070
0.020, 0.113
Byear
0.071
0.028, 0.114
(0.030)
(0.026)
Baata
0.018
0.012, 0.055
Baata
0.026
0.009, 0.047
(0.014)
(0.012)
Table 7: D tokens seen, without imputation of Table 8: D tokens with diminishing returns and epochs.
without imputation of epochs.
Parameter estimates from the model deseribed in equation 8, 3 decimal places. We report 90% confidence intervals of parameter estimates by bootstrapping 100 iterations.
G Doubling times via optimal scaling
In the main paper we calculated doubling times for effective compute based on closed form solution for the doubling times, given by Equation 6. However, this equation was derived simply by considering changes Der. and it is not clear a priori whether or not the effective compute optimally allocated between Narr and Demr to minimize the cross entropy loss. In addition, calculating compute efficiency doubling times less straightforward for models which also include changing scale exponents Oparam and Basta (i.c. models 14 and 15 in cross validation analysis, see J). We
22

---

thus supplement our previous calculation with an alternative approach, which instead enforces the condition of compute-optimal scaling.
We approach this in two stages:
1. First, we calculate the reduction in cross entropy loss AL given doubling in compute budgets and under compute-optimaliy. Let the initial compute budget be C. We solve an optimization problem
L1 = min L(N, D), (N,D)
subject to the constraint = 6N described in Hoffmann al. [2022], which is solved with values Ni and DJ. We then perform the same optimization problem but with budget constraint of 2C = 6.N D, yielding a corresponding cross entropy loss of L2. The reduction in cross entropy loss is given by AL = L2 L1 0.
2. We then estimate the years of algorithmic progress that would be needed to achieve this same reduction AL. particular, the optimization problem is
min f()),
(10)
SER+
where we have
f = eexpp Olyaar (Y + Yo) - Qpaaram log log(N/NNo]
+ exp[Fconst Byear (Y + 5 Yo)= Bama log( DD/D)) L2 (11)
Here can be interpreted as doubling time for effective compute due to algorithmic progress in pre-training. Connst Ocoust + Coonss.TT=-FTT + W1o0st,122WW1 and P'onst Beonst + 2eo + PeomstWT22W12. where 2PTB and 2wT2 are dummy variables for PTB and WT2 respectively.
We apply this approach over 100 bootstraps of our dataset, yielding a median doubling time of 6.1 months, and a 90% confidence interval of 3.3 to 1 11.3 months. This is very similar to the doubling times estimated using the closed-form approach discussed in section 3.1, which also has a median of 6.1 months [3.3, 11 11.3) for model 7.
H Irreducible loss
Our main model for estimating doubling times does not estimate the irreducible loss. This is in part due empirical difficulties encountered in estimating plausible values for this term, and in part because the inclusion of this term does not have a bearing on our estimates of the rate of algorithmic improvements. Since the latter is the focus of our paper, we decided move forward with the outlined model without irreducible loss estimates. However, we caution against overinterpretation of these results- for instance, our parameter estimates are not reliable enough to strongly inform how to scale models compute-optimall
(although they can be somewhat illustrative).
The focus of this section is to justify the robustness of core doubling time results to this omitted variable in our model. In particular, we fit model which incorporates estimates of the irreducible loss and show that the doubling times remain in line with our previous findings. The model consider is
i = 7 /'+exp(a -lyar (Y -Y)) -Y)) log N texpo))eeexxxp...e
-Byar Byear (Y
-o) log D/Do),
(12)
where > = 7 TPTEEPTB + 2WT22w72- and the rest of the model is defined the same way as in Section 3.1. When we fit this model, we obtain compute efficiency doubling time of 5.4 months, with 90% confidence interval of 2.4 to 10.8 months, consistent with the estimates from our primary model. Our core parameter estimates are shown in Table 9.
I Autocorrelation
Our dataset was constructed by searching for papers with models that reported perplexity data on WT103, PTB, or WT2. In our initial data collection, sometimes also included multiple models
23

---

Estimate
90% CI
0.000
0.007,
(0.045)
7PB
(0.178)
0.214
1.000,0.460
-0.111
7WT2
(0.147)
-0.409, 0.001
Occont
(0.246)
0.833
0.627, 1.154
Oco
(0.097)
0.001
-0.116, 0.134
Cconst.W22
(0.103)
0.102
-0.118, 0.252
0.005
Cryaar
(0.022)
00.044, 0.013
0.085
Paraa
(0.019)
0.057, 0.099
Beonst
0.882
0.438, 1.091
(0.227)
Beunsi.TB
0.076
-0.025, 0.324
(0.161)
0.113
Beonst, WT2
-0.000, 0.476
(0.170)
Byear
0.044
0.026, 0.079
(0.018)
Bdata
0.028
0.019, 0.050
(0.009)
Table 9; Parameter estimates from the model described in equation 12, reported to 3 decimal places. We report 90% confidence intervals for all of the parameter estimates by bootstrapping 100 iterations.
originating from the same paper. In some extreme instances, more than ten models from single paper were included; for example, we incorporated 14 models from the paper "OPT: Open Pre-trained Transformer Language Models." " This poses concerns of autocorrelation, which might for instance result in us underestimating the uncertainty in our individual parameter estimates.
In the main body of the paper we approached this issue by retaining only three models per paper in our analysis, which resulted the exclusion of approximately 35 models. Here we consider an alternative approach for addressing autocorrelation, where we explicitly quantify the correlations between models from the same paper. We then use this information establish a multivariate normal likelihood function, which we maximize obtain parameter estimates.
First, let us define the residuals squared error, i.e. where the loss is given by E(c"'), c]. We want to modify our approach so that we take into account the correlations between different datapoints. The approach we take is to take an approach similar to generalized least squares and multiplicative attention- rather than consider just e" c, we consider e" Pe, where is correlation matrix.
X. The original loss function that we are using is the mean
We do this using a maximum-likelihon approach, where e placed in multivariate normal distribution, given by
exp (-her2-1e)
f(:;0) (2 det(E)
(13)
where Σ is a covariance matrix for the data. Our goal is to choose the appropriate parameters 0 such that the resulting = x maximizes this distribution. This includes the original parameters in the model from equation 3 as well as the correlation between models from the same paper. We define the loss function the negative of the logarithm of this distribution (dropping constants which do not matter for the resulting minimization problem):
1
1
((0)
2 log det Σ -x-'e
(14)
In order apply this to our data we need specify the structure of the covariance matrix 2n (and thus H-1- o p-1, We can accordingly write the loss function as
1(0) log det log o² + 202n
TTp' p-1c.
(15)
2

---

We are assuming that the models from different papers are uncorrelated, and that different models from the same paper have correlation coefficient of p. If we order the models such that all the models from the same paper form contiguous range of indices, then the correlation matrix looks block diagonal, where each block has Is on the diagonal and p for the off-diagonal terms. The matrix elements that are not in blocks are all zero. For example, one example correlation matrix is:
P
0
0
0
P
1
0
0
0
1
P
P
(16)
0
P
0
P
0
0
P
P
1
0
P
P
P
As we can see, we need to determine both the inverse and the determinant of the correlation matrix P in order to calculate the negative log-likelihood. While this can be done using standard libraries, the matrices that we are considering here are quite sparse, and thus it is more efficient to simplify our calculations here. We detail the calculations in sections I.2 and I.3.
L1 Results
To determine confidence intervals, we bootstrap this model based on clustering over 100 iterations, in similar fashion to the main results. This yields median doubling time of 4.5 months, with a 90% confidence interval of 1.5 to 9.9 months, which is consistent with our core estimates. In practice is very strong.
typically on the order of 0.45, which suggests that the degree of autocorrelation We report our parameter estimates in Table 10.
Estimate
90% CI
Crconst
(0.753)
0.355
0.802, 1.513
OrconstPTB
(0.929)
0.058
-1 -1.009, 0.267
Orconst,WT2
(0.203)
0.125
-0.133, 0.582
0..522
Oryaa
(0.068)
-0.163, 0.028
O param
(0.051)
0.120
0.068, 0.214
Beonst
1.189
-0.707, 1.471
(0.868)
BConSLPTB
0.111
-0.109, 1.623
(1.242)
Beonst WT2
0.091
2.397, 0.909
(1.247)
Byear
0.060
0.016, 0.087
(0.029)
8dsta
0.020
0.011, 0.034
(0.041)
Table 10: Parameter estimates from the model described in equation 8, but estimated using the clustering approach described in equation 15. Estimates are rounded to 3 decimal places. We report 90% confidence intervals for all of the parameter estimates by bootstrapping 100 iterations.
1.2 Determinant
order to obtain the overall determinant for P we first work this out for a single block. In particular, the determinant of each (n 1) (n 1) block B,, is given
det Bn41 (1 p10 np).
(17)
25

---

We prove this by considering the associated matrix directly:
p
p
p
Ba+1
(18)
P
P
0
1
Since det Ba+1 is unchanged under the elementary row operation of adding multiple of one row to another, write
P
p
P
p
p²
det Bx+1
det (n+1)x(x+1)
(19)
0
0²
1
1
1²
p2
p2
We can thus simplify the determinant
1
p
0
P
1+p
P
P
det Bu+1 (1 -p) det
:
(20)
nxn
P
P
1
P
P P
1+p. p
To evaluate the determinant this matrix we follow a similar procedure, where we eliminate most of the elements in the first column. We then repeat this process as the resulting matrix becomes smaller and smaller, until we reach a trivial case. The kth step in this iterative procedure has the following structure (where k = 0, ......1.. 1,n):
1+(++1)p
1+hp
T+Ap
T+Mp
k+hp
1+(++1)p
I+Ep
1+kp
Itkp
I+kp
det Bn+1 (1-p) (1 kk)) det (m-k)
0 144p
1+(K+1)p
T4hp
1+kp
T+
P 1+hp
1+(k++1pp
1+kp
1+kp
1+kp
(21)
We now substract 1=(K+1)p times the first row from all other rows. In the first column, all except the first row thus becomes zeros, and there are two main cases we need to consider for the other elements. In the first case, for diagonal elements, we have we have
(k 1)p P 1+ kp
P kp + (k + 1)p
[1 + (k + 1)p)² p²
(22)
[1 + kp + (I + 1)
+ (k: + 2)p
(23)
1 (k + 1)p
In the second case, for off-diagonal elements, we instead have
P
+ (k + 1)p - p
(24)
1+kp 1- kp 1 + (k+ 1)p 1 kp
1+ (k: + 1)p
(25)
1 (k: 1)p
Thus we have that
1+(++1)g
k+
k4
14(K42)p TF(FFT)
IF 1)p
det But1 = (1 - (1 kp) de ()x(a
1+0k++25p TIKKT
Tit Tpp
1+(k-2)p
(26)
26

---

1+(k+2)e
I+(k+1)p
1+16+1)p
1+1++1/0
I+(++150
1+(R+2)
1+(k+1)p
1+(k+1)p
1+(K+1)p
I+(K+1)p
det -1-=)) (0-(++1)x((-(x+t))
:
P
1+(+22)p
1+(k+1)p
1+(++1)p
1+(++1)p
[+
P
P
1++2-2
1+(k+1}p
1+(k+1)p
[+(++13p
[+ 1)p
(27)
If repeat this argument times (going from an (n 1) (n 1) matrix the trivial case of x n matrix, then we conclude that
det Bu+1 = (1-pP (1 np),
(28)
as desired. Now work the overall determinant of P, we simply have multiply the determinants of the individual blocks, and we are done.
1.3
Inverse
We now want determine the third term in 100), i.e. ef One way reduce the compu- tational cost of this calculation is to take advantage of P being block diagonal, where the inverse p-1 can be determined by inverting the individual blocks. That for blocks B3 the inverse of the correlation matrix is
B
0
0
0
0
B
1
0
0
p-1
(29)
0
000
0
B-1
0
n-1
0
B
To invert single block B, first observe that each block B is such that the elements are
Bi - { if
(30)
otherwise.
Now let D be the diagonal matrix with diagonal entries Du = 1 - p. Then we have that B D + puv", where is vector of ones. We can now calculate the inverse B-1 by application the Sherman-Morrison Morrison formula. In particular we have
po" D-1
B = (D + pvv -
D-L
(31)
1 pu D
Since the inverse of D is just Djj1 I-p assuming that the correlation p * 1. To simplify notation, We then have
we define c + T-p
i=
Bij
1-p o1-
(32)
i
p
and calculating the associated quadratic form
202m T p follows trivially.
27

---

Cross validation for model choice
We determined our primary model for estimating doubling times using formal model selection procedures. In particular, we considered range of 15 candidate model numbers and a range of possible regularization strengths 8 {0, 0.001, 0.0025, 0.005, 0.01, We consider each "model" as a pair, consisting of particular model number and a regularization strength 8. We then perform leave-one-out cross validation on all of these models. Here we show the results of this analysis.
Our candidate models are defined by varying three degrees of freedom:
1. Which parameters to make benchmark-specific (e.g. having three separate parameters for the exponent on training dataset size, one for each of WT103, PTB and WT2).
2. Whether to explicitly model algorithmic progress in parameters N, data D, both.
In general we do not include the irreducible loss in the model (i.e. we usually set E = 0, in the equation 3 described in more detail in section H. In Tables 11 and 12 we list the definitions of all models that we looped through in leave-one-our cross validation. For this, we first define some common terms for simplification:
O const
Olconst
+ PTB-5T.PTB:UPTB + OcconstWT22200
year
Olyear
+ Otyear PTB
0 param
at param
a param, PpT + O WT2
3
const Beunst
Pconst,Pnn/prs
Reoms, WT244/22
8
Byear + Syear +
yyaar WT2
3latn Busta + Pdu +
3dma WT22
aparam = O'paramNN (1 IT) + O'param, TT
Piam = Ba TT) BAMA.TTT
apuram Opparam + Crrate log Y
B1
Baata + Bruse log
Here ZPTB, ZwT2, and are dummy variables for PTB, WT2, and the transformer respectively. Y is the year, and Crtte and Brate are constants that determine how quickly scaling exponents (O'param and (dma) change over time.
Models Il all constructed using similar of rules:
Olcoast and Sconst determine the coefficients of the parameter and data reducible loss terms respectively
O'year and Byear capture algorithmic progress in parameters and data
O'puram and Bans determine the scaling behavior with respect to N and D respectively
All of these parameters may be set as senchmark-speciffe (e.g. by writing d'ona in lieu of Ocons).
The remaining models are defined using different set of rules:
Model 12 is defined similar fashion but 'Hicks-neutral' fashion, such that the same degree of efficiency gain is seen across both parameters and data.
Model 13 models different scaling exponents Opparam and Basta for transformer vs non- transformer models
Model 14 captures only algorithmic progress via changes in the scaling exponents aparam and Btu
Model 15 captures algorithmic progress via Olyear and Byears as well changes in the scaling exponents
28

---

Table 11: Model specifications for leave-one-out cross validation (Models 11).
No.
Model specification
1
Algorithmic progress in both N and D
L exp Occonst Ovyaar (Y Yo) Oparam (log N - log No)] expl Byear -Yo)
Baata (log D log Dn)
2
No algorithmic progress in parameters
I, = exp Oconst pprrmmm (oommm((og (log N log No)] + exp[Bconst Syear (Y -Yo) - -Baata (log D
log Do)l
3
No algorithmic progress in data
L exp Oconst Ayear (Y- Yo) Opparam (log N -log No)l + exp Jccontt Idata (log D
log Do)l
4
Benchmark-specinee in Oyear
L == exp Constt 0 yyear (Y Yo) Oparam (log log No)] + explBeemss Byear (Y -Yo)
(log D - log Do)]
5
Benchmark-specific in Byear
L exp Ccoontt Otyaar (Y Yo) Oparam (log -log No)] explBconss Byyar -Y-
Bant (log D log Do)]
manchmark-speciffee
C'year and Byear
I = exp Cononst year (Y Yo) Oparam (log N - loo No)] + +exp Byear (Y (Y-Yo)
fanta (log D - log Do)]
7
Benchmark-specine in Oleonst and Boonst
= exp 0 Okyaar (Y Yo) Oparam (log N - log +exp Byear (Y Yoo
Bann Blog D log Do)]
8
Occona and Bconst, with no algorithmic progress in L = exp Cconst Oparam (log N log No)] + exp[32 (3onst 3year (YearrY Yo) Betaaa (log D
Benchmark-specifee
log Do)l
9
Benchmark-specific in Occonst and Aoonst with no algorithmic progress in D)
L exp expla'cost Ayear Yo) Opparam (log N log No)l + (emm Pdata (log D log Do)l
10
Benchmark-speciff in Olconst; Otyear Aconst: 3year ==exp exp const a Avear Yo) Opaaram (log log No)] + explsconss Byyar (Y -Yo)
(log D - log Do)]
11
Benchmark- specific O'oonst, Oyear, Oparam Boonst Byear and 3data L exp const a'ear (Y Yo) aparam (log log No)] expl -year -Y-)o Blam (log D log Dn)]
Models 16 and 17 do capture algorithmic progress, and are used baseline for goodness-of-fi comparisons. Model 16 is modeled off equation 34, and model 17 modifies this to include transformer-speciff scaling exponents. This described in more detail in Appendix J.2
Model 18 is a model that only considers the total compute, via the approximation C = 6NN. This discussed in more detail in Appendix J.I.
Model 19 includes a term that accounts for different vocabulary sizes. We discuss this in Appendix E.2.2.
Model 20 the same as model 7, but rather than using purely the training dataset size, defines "training data" in way that accounts for the number of epochs of training. Where the number of training epochs is unknown, we impute an epoch count of This model presented in more detail in Appendix F.
29

---

Table 12: Model specifications for leave-one-out cross validation (Models 12 20).
No.
Model specification
12
"Hicks-nuttrll model
L (exple cong + Oparam (log N - log No)] + expl@ + Santa (log D - log Dn)l)
expl -Okeer Yo]]
13
Transformer vs non-transformer scaling
I, = exp (0 O'yarr (Y Yo) param (log N - log No)] + +expl Byyar (Y -Yo)
Pian (log D log Do)]
14
Progress only via changing scaling exponents
I. = exp (ornst O'param (log N log No)] + exp[2tomst
Blana (log D logg Do)l
15
Changing ayears Syear, Ciparam, and Sdata
L = exp (oost Otyear (Y - Yo) Oparam (log N - log +exp Byear (Y Yo)
8 (log D log Do)]
16
Benchmark specific Hoffmann al. [2022] scaling law (for comparison only)
L = cxp const Opparam (log N log No)] + expl (log D - log Do)l
17
Transformer-specine scaling law (for comparison only)
L = exp (o - Pprram (log N log No)] + exp[S - Pasa D - loo Do)l
18
Compute-only model (for comparison only) = exp(p (xppaacomu Ayear( (Y Yo) 4coompute (log(6NN)) log(6NNoDo))
19
Vocabulary fixed-effects (for comparison only)
L y log( /ocab) expla
Olyaa (Y- Yo) aparam (log N log No)] expl
Byear (Y Yu) - Bama (log D - log Do)l
20
Same as model 7 but with imputed epochs (for comparison only)
== exp d'oostt Ovear (Y - Yo) Oparam (log N - loog No)] + eexp Byyar (Y -Yo)
Baau (log D log Do)]
We chose to include models 16 to 20 in the cross validation analysis to help the robustness checks performed in different appendices.
J.1 Compute-only model
Given that our core focus is on estimating doubling times in effective compute, one natural parame- terization is to directly consider total training compute C, in particular,
== + explo - Cyear (Y Yo) Oremmpute (log C - log Co)]-
(33)
Here 7/ is defined similarly the "primed" constants above, i.c. 7 y/= = + "PTBTTTT + 7wT2"wr22 and we have the approximate relation that C = 6ND [Hoffmann et al., 2022]. However, this model has tendency to yield implausible results, in particular with a very small scale exponent Ocompute and very short effective compute doubling time (on the order of 1-5 months).
We omit this model because believe there are two reasons to expect this model be strongly misspecified. The first reason is that the model scaling does not accurately reflect complementarities between scaling parameter and data. For illustration, compare the following two equations:
L-
(34)
No
D
A
L= E
E
(35)
C
(6ND)
where the irreducible loss, and A, B, or, 3 are constants. equation 34, reductions the loss are bottlenecked by lack of sufficient data D. This bottleneck does not exist in the case of 35, where scaling N arbitrarily would bring you the irreducible loss F.. This model thus unrealistically suggests equivalent performance between 1-parameter model trained on 10 tokens, and a 10"
30

---

parameter model trained on 10 tokens. Of course, such extreme choices of parameters N and dataset size D are implausible in practice, but nevertheless we do see clear variation in how these values are chosen in practice. For instance, Figure 10 shows plot log (N/D) D) for models across different years, spanning over five orders of magnitude over this time period. This is in spite of the estimate by Hoffmann et al. [2022] that N/D should be ((1) for compute-optimal training.
Tokens parameter D/N
Chinchilia
101
GPT-3
10"
10
10-3
2012
2014
2018
2018
2020
2022
2024
Publication date
Figure 10: Ratio parameters N to dataset size D for models in our dataset. We emphasize the ratios corresponding to GPT-3 [Brown et al., 2020] and Chinchilla models [Hoffmann et al., 2022]. which are of historical importance in determining how to choose this ratio when scaling language models.
second way in which equation 33 is misspecified due to its unrealistic scaling behavior. To see why this is the case, compare the difference between the reducible loss terms in equations 34 and 35, by writing them as a single fraction with the same denominator (ND)), For illustration purposes, also assume that conclusion. we then factor out the coefficient A, we then have that
and A B, which simplify argument without changing the core
1 Lu x Na
No D
(36)
D
(ND)a
6
Le (6ND)a (ND)
(37)
where Ly is the loss predicted from the scaling law in Hoffmann et al. [2022], and Le is the loss from only considering compute. Here we observe that in the case of Chinchilla-scaling (equation 36), as N D is increased the value of the numerator increases, whereas the opposite is true in the compute-only case (equation 37). As a result, fits of the compute-only model tend to have very small values of a, since the difference in scaling behaviour between the two expressions tends be more pronounced for larger values of a. Indeed, estimates using the compute-only model yield very small values of a, at 0.004 with 95% CI of [0.002, 0.012]. We illustrate the difference between the scaling behavior of equations 36 and 37 in Figure 11.
We nevertheless test the compute-only model in cross validation and find that it performs very poorly on oft-of-sample prediction, far worse than any of the other models that we consider. The results of this exercise are elaborated upon in section J.3.
J.2 Algorithmic progress through changes in the scale exponents Oparam and 3
As mentioned in Section 2.1, the final model that we use for our core results does not explicitly account for efficiency improvements through changes in scaling exponents (i.e. Oparam and Bass).
The primary reason for our decision is that while this form of algorithmic progress is theoretically plausible, our model fits to the available data suggest that this effect has not been very large. For example, if we consider the estimates from model 15, which includes both the form of algorithmic progress described in Section 2.1 and improvements through changes in scaling exponents, we find that the overall contribution to algorithmic progress is dominated by the former.
31

---

Chinchilla
Compute-anly
02
-0.01
00.05
0r-03
0.0
10
10"
10"
10"
10 10
7017
10
10"
10
10"
10
10
10
107 107
104
107
30"
10
10
Parameters
Figure 11: Comparing the scaling behavior of equations 36 and 37, for a = 0.01. 0.05 and 0.3. For illustration. we choose N = D, and normalize both expressions equal when the number of parameters N = 10 The difference between the scaling laws is smallest when a is set smaller value-this is illustrated by the double-headed in cach plot, showing the largest gap between the curves.
Another piece of supporting evidence that in cross validation, model 14 appears perform roughly as well as models without any algorithmic progress at all (models 16 and 17). Furthermore, the parameters which determine the rate of changing scale exponents (O and (rae)) are generally very small (around 0.001 to 0.01). As such, the model appears to simply be approximating the equivalent models without algorithmic progress. This suggests that this form of algorithmic progress is negligible, and is also why we have excluded model 14 from Figure
J.3 Performance metries
this section we list the resulting goodness-of fit metrics from our cross validation analysis. this case report the average out-of-sample MSE loss.
Model/s-value 0
0.001
0.0025
0.005
0.01
0.02
0.04793
0,04778
0.04762
0.04762
0,04788
0.04997
0.05112
0.05111
0.05049
0.05293
0.05745
0.05793
3
0.04727
0.04732
0.04758
0.04797
0.048
0.04826
0.04716
0.04665
0.04618
0.04656
0.04696
0.04693
5
0.04457
0.04458
0.04502
0.04545
0.04606
0.04709
0.04539
0.04561
0.04529
0.04519
0.04669
0.04756
0.04795
0.04809
0.04836
0.04676
0.0468
0.04696
8 0.04671
0.04771
0.04666
0.04689
0.04745
0.0501
0.04622
0.04705
0.04732
0.0461
0.04686
0.04694
10
0.04405
0.04285
0.04217
0.04642
0.04758
0.04804
0.14378
0.04562
0.04402
0.04504
0.04518
0.04613
12
0.04845
0.04669
0.04731
0.04615
0.04656
0.04567
0.04381
0.04515
0.04525
0.04454
0.04468
0.04477
14
0.05144
0.06391
0.0637
0.06298
0.06361
0.06424
0.04751
0.04884
0.04843
0.04611
0.04612
0.0476
16
0.06448
0.0645
0.06401
0.0629
0.0632
0.06307
0.05668
0.05846
0.05879
0.05821
0.05836
0.05842
18
0.05542
0.05541
0.05543
0.05545
0.05551
0.05568
0.04643
0.04731
0.04632
0.0467
0.04669
0.04713
20
0.04564
0.04659
0.04929
0.04751
0.04765
0.04827
Table 13: Average mean squared error test loss of all model-ò combinations from cross validation. 8-values here are the regularization term in the L1 regularization set-up.
32

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1 (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providded a proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
the checklist one of the factors in their evaluation.
reported because it would be too computationally
"[No] " or "[NA] is not grounds for rejection. While the questions are phrased eeknowledge that the true answer is often more nuanced, so please just use your best, judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We describe the core approach (estimating augmented scaling laws), and the core things we investigate (the rate of algorithmic progress, the relative contributions of scaling models algorithmic progress) in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This is described under the "Liimitations" subsection of the discussion, where mention problems like the lack of quality data and the reliance the Chinchilla scaling law. We have attempted to be as transparent as possible about the model assumptions and the challenges in extrapolating our results to future AI progress.
Guidelines:
33

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texs system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The only theoretical result in this paper is in appendix on autocorrelation;
everything else is empirical. We provide the full derivation of our result in that section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We explain the model specification in section 2.1, as well as full details in the appendices (e.g. the full model equations are listed in appendix J). The data and optimization strategy used are all described section 2.2. The key results can all be tested by fitting the same model specification on the same data, the code and dataset for which provided. Guidelines:
34

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
All data and code is provided together with this paper submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
soteeeuubbsssiioooooocoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //ni cttppssscc/ppbbicguuuuc.o
/CodeSubmisst ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
35

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The methodology (including datasets, statistical model, and optimization algorithms) are all described in section 2.1. We provide more detail about these in the appendices, e.g. we go into depth about our dataset in appendix E.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide bootstrapped confidence intervals throughout the paper illustrate the uncertainty in our results. We also provide information about model uncertainty, by trying different model variants and showing how predictions vary across these variants.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula. call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma еттоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: Our empirical results are based on statistical models that do not involve running machine learning experiments. These empirical results are obtained by fitting dataset in Google Colab notebooks.
36

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http:///eeep....ppp////////////e htlpppiic//tissauieeeeeeeeeeeeee
Answer: [Yes]
Justification: Our empirical results do not involve human subjects or participants. Our dataset consists of data obtained from cublicly-accessibbe machine learning papers, such as on benchmark performance and model size, and authors are referenced in the database.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the importance of our research for understanding AI trends and for informing policy the introduction. We further emphasize potential downsides if our results are misinterpreted (e.g. extrapolated too far into the future) in the limitations section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
37

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The empirical results in our paper are from fitting a simple statistical model on publicly available benchmark results, publication dates, and information about dataset size and parameter counts. As such, our work does not pose serious risk of misuse.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: We do not use assets in the typical machine learning sense, with e.g. datasets for training neural networks or relevant code for training. Perhaps the most relevant asset we use is to collect certain results (like model size and benchmark performance) from public papers, and we link all these papers in our dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwwdee.coo..... cot.ccco//accooooce comtaatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pen e-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 'ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not create assets for machine learning training or inference, as this is not the focus of paper. The most relevant "assets" that provide the database (described detail in this paper) and IPython notebooks pertaining to each section this paper.
Guidelines:
38

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not perform crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do perform research with crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
39


Reviews:
[{"review_id": "6RWPPvqMd4", "reviewer": "Reviewer_gaW3", "summary": "This paper breaks down the driving force behind language models into two factors: scaling up the amount of compute used to train the models and algorithmic innovations. A statistical model, similar to the scaling law, is built and fitted to analyze the contributions of these two factors. The paper claims that models with the same performance level require half the compute every eight months, reflecting algorithmic improvement. The authors also find that the majority of scaling progress is due to the scaling of physical compute resources rather than algorithmic progress.", "questions": "None", "limitations": "I do not see any potential negative social impact of this work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The paper presents a very interesting and innovative approach to quantifying the algorithmic progress of language models. By covering over 200 language models from the past decade, the empirical foundation for the conclusions drawn is solid. Additionally, the author has performed extensive robustness checks to ensure the validity of their core results.", "weaknesses": "While the paper makes several assumptions to quantify algorithmic progress (including the extra coefficients in the augmented scaling law), these assumptions and the many degrees of freedom undermine the robustness of the proposed model and the conclusions drawn. I have doubts about whether the statistical model built can inform future progress in language modeling.\n\nMinor issue: The caption of Figure 1 seems to be incorrect. It should address Figures 1a and 1b instead of 3a and 3b.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R286GHYJPc", "reviewer": "Reviewer_Va8z", "summary": "This paper presents an analysis of the relative contribution of algorithmic progress to overall LM performance gains over the window of 2012-2023. The authors evaluate a large number of potential equation variants for modeling algorithmic progress using leave-one-out CV. By making use of defined notions of effective data/parameters/computer, the authors estimate that 2x less compute is needed to acheive the same performance roughly every 8 months. The authors find that roughly 2/3 of the scaling progress is due to the scaling of physical compute resources, with the remainder being attributed to algorithmic progress. The singular contribution of the Transformer architectures is individually analyzed. Thorough analysis of the techniques applied and their limitations are presented.", "questions": "None.", "limitations": "The limitations are thoroughly discussed in the paper. No relevant missing potential negative social impacts.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is clearly written and well-presented. The depth and quality of the anlaysis are exemplary. The methodology of analysis is not highly original, but its application to algorithmic progress is a novel and useful contribution to the community. The limitations of this kind of analysis are well-discussed, which is a useful contribution in its own right.", "weaknesses": "Nits:\nFigure 2: label the y axes, consider making especially the right plot a bit larger/more readable, the text is very small\nFig 1: fix the spacing between the (a) and (b) captions, they are nearly overlapping", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X4JSaY6unG", "reviewer": "Reviewer_osWG", "summary": "This paper investigates the rate of algorithmic progress on the task of language modeling, using a dataset of over 200 LM evaluations on WikiText and Penn Treebank between 2012-2023. The authors fit an augmented scaling law to the data and show that the models are requiring 2x less compute roughly every eight months -- a rate which supersedes Moore's law. Further, the authors find that more recent performance gains have been primarily due to compute scaling and that the contribution of the transformer architecture is roughly equivalent to two years of algorithmic progress in the field.", "questions": "1. Can you give examples of the ways in which the core results would change with different scaling law formulations are used?", "limitations": "The limitations were adequately addressed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper presents an interesting take on quantifying the progress on the task of language modeling using an analysis of data collected from over 200 model evaluations in the past 10-11 years.\n2. The authors clearly specific the assumptions made in conducting the analysis, which makes the approach quite readable.", "weaknesses": "1. There a number of assumptions made to elicit a quantification of algorithmic progress, and the nature of the task itself necessitates those assumptions be made. However, this also means that the resulting analysis is brittle and as such the technical contributions of the work don't rise to the level of a solid contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback on our paper. \n\nWe agree that we make several assumptions in our work quantifying algorithmic progress, and that this introduces uncertainty into our conclusions. However, we do not believe that these assumptions undermine the core results of our paper. We have performed extensive robustness checks for this purpose, e.g. we consider different ways of managing autocorrelation in Appendix I, different ways of estimating doubling times in Appendix G, etc. In each case, our robustness checks provide grounding for our core empirical results. If you do not believe that these robustness checks address your concern, could you please specify why not, and what assumptions you believe need to be addressed? \n\nTo address your question about scaling laws in particular, our work considers a range of different model specifications, as outlined in Tables 10 and 11. These models are varied across different degrees of freedom, such as algorithmic progress impacting scaling exponents, progress that is specific to each benchmark, etc. We illustrate the variation across these degrees of freedom in figure 1b.\n\nFurthermore, in appendix C.1, we compare our model parameter estimates to existing work on scaling laws (e.g. Hoffmann et al 2022 and Kaplan et al 2020). We find that our estimated scaling exponents are consistent with theirs within the time range over which the majority of our data lies. In appendix H, we analyze the impact of including an irreducible loss term in the scaling law formulation, and compare this with our core model. Again, our findings are consistent with our core results. Overall we believe that having consistent results across all these changes provides strong evidence in favor of our core results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tKqD2Yganu", "reviewer": "Reviewer_CCHC", "summary": "The authors of this paper examine the performance improvements of language models over the past decade, and investigate how much of it can be attributed to algorithmic improvements of language models.", "questions": "I would be grateful if the authors could expand on the points I raised above, regarding the insight for future directions.", "limitations": "The authors have adequately addressed the limitations of their work. Regarding negative societal impact, I do not foresee any arising from this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I note here that, given that this paper proposes a method to evaluate the historical progress of language models, I believe that the primary goal for the paper would be to provide interesting insights for the field and outline current open questions. As such, I will structure the rest of my review with that in mind.\n\n- The topic of language models is more central than ever to the broad NeurIPS community, and I think that analysis on the historical progress of the field is of great interest.\n\n- The authors obtain valuable insight on the cause of language model improvements over the years. I find their conclusion that data/compute scaling has been a major driving force for improvement over the past  few years interesting, if somewhat expected.\n\n- The result on the importance of the transformer is also interesting, and provides a nice retrospective justification of the broad adoption of this architecture.\n\n- The analysis of data from past works is also sound.", "weaknesses": "- The most crucial weakness of this paper is that, despite the analysis of past trends, it does not provide clear insights or suggestions on where the field should direct its efforts, moving on. The authors acknowledge this limitation, but it seems to me that such discussion Is crucial, in order for this paper to be of interest to the community.\n\n- It is also not clear to me how the insights provided by the paper could extrapolate in the future. While there has been a lot of effort and improvements gained in performance from data scaling, this is not something that can reliably keep on - at some point, sources of data and compute limitations catch up. As such, it is not clear to me how the insight that the effective compute for language models doubles every set period of time can be useful for many years down the line.\n\n- On more minor note, I think some points can be improved for clarity of presentation:\n\n  - The captions of Figures 1a and 1b are joined together - some spacing between them is required.\n\n  -  It would be better if the authors clarified the statistical models used in the main paper, rather than in the appendix.\n\n\nOverall, my key concern for the paper is that it does not provide enough insight for future directions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback – we address your questions and concerns below. \n\nWe fully agree with you that there is substantial uncertainty about how our results could extrapolate into the future. Indeed, we have mentioned this point in the limitations section, and pointed out that our core focus has always been to estimate historical rather than future rates of progress.\n\nThat said, although future extrapolations are not within the primary scope of our paper, our results still represent a substantial step up compared to the prior status quo. For one, it provides a rate of progress that can be extrapolated in the first place, and it paves a clear path forward for future work. E.g. We agree with your point that compute limitations might be relevant for understanding future algorithmic progress, and this provides a clear next step of trying to quantify the significance and timing of this bottleneck. \n\nIn fact, we feel that this strongly relates to your point about providing insight for future directions. Our work is most strongly directed towards people interested in understanding trends and drivers of progress in ML. Thus while it is of general interest for ML practitioners, by far the most important future directions pertain to future research on ML trends, such as in understanding future compute/data bottlenecks. We also mention that further research could extend the analysis to specific benchmarks or different domains, or consider the impact of individual algorithmic innovations by performing ML experiments. As we alluded to in the related work section, there has been relatively little work studying these important high-level questions about progress in ML and we believe our work points strongly towards additional work in this area. \n\nAs such, we completely agree with you that the primary goal of the paper should be to provide interesting insights and outline future directions. We believe that our work achieves these two criteria most directly for people interested in studying ML trends/progress, as per the previous discussion.\n\nAs a final point, thanks for the suggestions regarding improving the paper’s presentation – we will incorporate these changes when we next update the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors of this paper try to quantify the rate of progress in deep learning over the course of the last decade and disentangle progress from scaling the amount of compute and progress due to algorithmic advances. The research question the authors are trying to tackle is an interesting one and the paper does an adequate job shedding some light on it. There are quite a few assumptions in the work and the estimates might be a bit off, but overall the reviewers found the paper worthy of acceptance and the AC concurs.
Decision: Accept (poster)