Title: Algorithmic progress in language models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Natural Language Processing
Abstract: We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 90\% confidence interval of around 2 to 22 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.

Algorithmic progress in language models

Abstract

We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 90% confidence interval of around 2 to 22 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.

1 Introduction

The field of language modeling has seen rapid advances, with recent large language models (LLMs) demonstrating strong performance in domains such as programming [Li et al., 2022, Leblond et al., 2023], mathematics [Cobbe et al., 2021, Trinh et al., 2024], and standardized tests [OpenAI, 2023]. But how has this progress been possible?

One salient factor is the scaling of training compute based on neural scaling laws [Sevilla et al., 2022, Hoffmann et al., 2022, Kaplan et al., 2020], with state-of-the-art systems being trained for months on tens of thousands of GPUs. But this is only part of the story: another key factor has been algorithmic improvements, which result in more efficient use of resources such as compute and training data. These include changes in model architectures, optimization algorithms, and software frameworks.

This picture raises some important questions: How much of recent progress in language models has come from algorithmic improvements during pre-training, and how much has been from scaling up models and datasets? The answers to these questions have crucial implications for the future of AI progress, and have an important role in informing AI policy.

In this paper, we aim to answer these questions by following the approach first presented by Erdil and Besiroglu [2022] in computer vision. 2 To this end, we produce a dataset of over 200 language models evaluated on popular language modeling datasets, and use this to fit a statistical model that helps estimate the rate of algorithmic progress. Using our model, we further quantify the relative importance of algorithms and scaling training compute, shedding light on one of the most important drivers of AI progress.

1.1 Previous work

Thus far, there have been few works investigating algorithmic progress in machine learning. Notably, Hernandez and Brown [2020] re-implement popular open-source ImageNet models and find a 44 × decrease in the compute required to achieve the same performance as AlexNet. Karpathy [2022] reproduced the convolutional neural network of LeCun et al. [1989] using modern algorithmic innovations, achieving a 60% reduction in error rate. Dorner [2021] measures progress in the sample efficiency of deep reinforcement learning algorithms over time with doubling times ranging from 5 to 18 months. More recently, Erdil and Besiroglu [2022] estimate algorithmic progress based on fitting a statistical model inspired by neural scaling laws. They find that algorithms and hardware contribute roughly equally to performance, and the training compute needed to reach a certain level of performance halves every 9 months.

2 Methodology

2.1 Model definitions

We want to estimate the rate at which newer language models are able to achieve a certain level of performance more efficiently than older models. We do this by fitting a model that meets two key desiderata: (1) the model must be broadly consistent with previous work on neural scaling laws (e.g. Hoffmann et al. [2022]), and (2) the model should allow for a decomposition of the main contributors to increased performance, such as improvements in how efficiently data or free parameters in the model are used. In this sense, our core approach is similar to that in Erdil and Besiroglu [2022].

The starting point is the scaling law from Hoffmann et al. [2022], which relates the training loss L of a dense transformer to its number of parameters N and the training dataset size D :

where L is per-token cross entropy loss on the dataset, and E , A , B , α and β are constants. E represents the 'irreducible loss' of the dataset, while the second and third terms, A N α and B D β , capture the errors that are due to the finiteness of the model or dataset, respectively.

Following Erdil and Besiroglu [2022] and Hernandez and Brown [2020], we quantify algorithmic progress in terms of reductions of the resources ( N and D ) required to achieve the same level of performance over time. To measure this, we introduce the concepts of 'effective data" D eff and 'effective model size" N eff into the model: 3

where Y is the current year, Y 0 is some reference year 4 , and α ′ and β ′ characterize the rate of algorithmic progress for model size and dataset size, respectively. In other words, we assume that continued algorithmic progress results in an exponential increase in D eff and N eff over some time interval Y -Y 0 , even with fixed D and N . Plugging these into the original scaling law gives:

where A , B , α param, α year , β data and β year are constants. In relation to equation 2, we have that α ′ = α year /α param and β ′ = β year /β data. Algorithmic progress is thus captured as a constant exponential trend that multiplies with each of the two terms in the scaling law. In doing so, we are able to estimate the rate at which fewer 'resources' ( N and D ) are required to achieve the same level of performance over time. Furthermore, given that that the physical compute is approximately given by C ≈ 6 ND [Hoffmann et al., 2022, Kaplan et al., 2020], we can similarly define an 'effective compute" which is determined from the effective parameters and effective data.

2.2 Estimation approach

2.2.1 Model selection

Weestimate variants of the augmented scaling law presented in equation (3) on our dataset of language model evaluations. We perform extensive cross-validation exercises to identify the variant of the model that fits the data best. The goal of this exercise is to consider different models that capture different effects (e.g. different scaling behavior across different model architectures, different forms of algorithmic progress, etc.).

Concretely, we consider dataset-specific coefficients ( A,B ), rates of algorithmic progress (e.g. α year , β year), different scaling coefficients for different architectures, regularization ( α param , β data), and more. The model variants we consider generally do not contain an irreducible loss term (i.e. E = 0 ) since this is poorly estimated on our data, and because it does not change our estimated doubling times in practice-we check the robustness of this change in appendix H. In total, we evaluate around 90 different model specifications through leave-one-out-cross validation and pick the models that perform best on relevant out-of-sample metrics, see Appendix J for more details. In the end, the model we select is model 7, where the coefficients A and B are benchmark specific, but estimates of algorithmic progress and scaling exponents (e.g. α and β ) are not. This model achieves an R 2 of around 0.91 between predictions and held-out test data.

A further important consideration is the possibility of alternative forms of algorithmic progress. In particular, in section 2.1 we model algorithmic progress as causing exponential increases in an 'effective" budget, e.g. of parameters. But one could also observe progress through changes in scaling exponents (i.e. α param and β data). There are a priori reasons to suspect that this might be the case-for instance, one notable innovation is due to a change in scaling laws such as those introduced in Kaplan et al. [2020] and Hoffmann et al. [2022]. Different model architectures, such as recurrent neural networks and transformers, are also known to have different scaling behaviors (see for instance Tay et al. [2022] and Droppo and Elibol [2021]).

We attempt to account for this possibility in the cross validation analysis. In particular, we introduce three models (models 13 to 15) which account for different kinds of scaling exponents, including the possibility of changing exponents over time. Our chosen main model (model 7) outperforms these models in cross validation, but these alternatives also perform similarly well, typically with an R 2 of between 0.88 and 0.91. This analysis is described in more detail in appendix J.

We also consider other factors that could potentially impact measured perplexity, and thereby measured rates of algorithmic progress. For example, different tokenization schemes during preprocessing have been found to improve WT103 perplexity in some instances [Radford et al., 2019], and training models for multiple epochs has been a common way of improving performance [Muennighoff et al., 2023]. We find that our core results are broadly the same while varying these degrees of freedom-we provide more details on these experiments in the appendices. 5

2.2.2 Data

Our dataset contains over 400 language models evaluated on WikiText-103 (WT103), WikiText-2 (WT2), and Penn Treebank (PTB), about 60% of which we are able to use in our analysis. In particular, relevant information was retrieved from around 200 different papers, as well as evaluations of 25 models that we performed ourselves using the framework from Gao et al. [2021]. We then consider the subset of the data that contains the information necessary to fit our proposed model structure in equation 3: token-level test perplexity (which determines the cross-entropy loss), publication date,

number of model parameters, and training dataset size. This leaves us with around 231 models for analysis.

In some instances, multiple models are retrieved from the same paper, even if they constitute similar algorithmic innovations. This could pose problems around autocorrelation, which could result in underestimating the uncertainty in our individual parameter estimates. In the following main analysis, we therefore only include up to three models per paper, which results in approximately 50 more models being excluded. To verify the robustness of this approach, we also consider an alternative technique that directly accounts for autocorrelation in the analysis, which yields doubling time and confidence interval estimates that are consistent with our main results (see Appendix I).

3 Empirical results

3.1 Models require 2 × less compute roughly every eight months

How quickly are the algorithms underpinning language models improving? Our core approach is to back out doubling times based on fitting the augmented scaling law introduced in equation (8), and using the definitions of effective data, effective parameters, and effective compute we introduced in section 2.1. Here the effective data is given by D eff = D exp [ β year β data ( Y -Y 0 ) ] , so the doubling time for D eff is determined by the time Y -Y 0 where D eff = 2 D . Thus we have:

The doubling times for parameter efficiency can be determined similarly, giving

which we can use to work out the doubling times for effective compute. In particular, since the total compute in FLOP, C , required during training is approximately 6 ND , the growth rates are related via g C = g N + g D . Here g C is the growth rate in effective compute, g N is the growth rate in effective parameters, and g D is the growth rate in effective data. Since doubling times are inversely related to growth rates, we therefore have that

where T C , T N , and T D are the doubling times (due only to algorithmic progress in pre-training) for effective compute, effective parameters, and effective data respectively. Based on this approach, using our preferred model, we find that the median doubling time for effective compute is 6.1 months, with a 90% confidence interval of 3.3 to 11.3 months.

We further check the robustness of this result by looking at the predictions from different models. In particular, because we perform model selection using leave-one-out cross-validation, we can compare the predictions of our preferred model with the predictions from other models we considered. 6 Concatenating the doubling time estimates from all the models in Figure 1b, we find a median doubling time of 7.5 months [95% CI: 1.7 to 22.5 months], which is consistent with our preferred model. 7 This estimate falls within the range of confidence intervals of the estimated rates of algorithmic progress in computer vision [Erdil and Besiroglu, 2022], and sample efficiency improvements in reinforcement learning [Dorner, 2021].

While the structure of our model is not amenable to analyzing fine-grained speedups or slowdowns in the rate of algorithmic improvements, we can nevertheless test the possibility of a one-time increase or

0

5

10

15

20

25

Doubling time (months)

0.00

0.05

0.10

0.15

0.20

Density

Aggregated

Median: 6.1

Model 7

Median: 7.5

10

11

13

15

7

9

12

5

8

6

4

3

1

Statistical model structure

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Doubling time (years)

Better cross-validation performance

Worse cross-validation performance

(c) Summary of all model structures and the degrees of freedom included. Efficiency gains are captured by exponential decrease in the relevant error terms, except models indicated by T , which have time-varying exponents. For a full specification, see Tables 11 and 12.

decrease in growth rates over the full time period. To this end, we consider a variant of our preferred model (model 7) where a dummy variable is introduced-this is equal to 0 for any model that is published before the start of a certain year, and 1 otherwise. This allows us to consider doubling times before and after a certain year cutoff (e.g. 2017), and we perform this analysis for several such cutoffs.

The result is shown in Figure 2. Here we see that the difference in estimated doubling time before and after the start of 2017 is very pronounced, however this is not the case for other choices of the cutoff year. In each year the median doubling time is faster after the start of the cutoff year, but usually only marginally so. Overall, this does not provide strong evidence of a drastic speedup in algorithmic progress. This does not rule out the possibility of weaker effect sizes, since our approach is statistically under-powered.

2016

2017

2018

2019

2020

Year

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Doubling Time

Pre

Post

2014

2016

2018

2020

2022

10 0

10 2

10 4

10 6

10 8

10 10

10 12

Effective compute (Relative to 2014)

Algorithmic

progress

Physical

compute scaling

LSTM

GPT-2

OPT-175B

2.2 × 10 4

1.7 × 10 7

Year

3.2 Most recent performance gains in next-token prediction have been from compute-scaling

Naively extrapolating our estimated doubling times suggests that, between 2014 and 2023, pretraining algorithmic progress has enabled performance to improve as much as it would have with around 22,000 × more compute. 8 At the same time, Sevilla et al. [2022] find that physical compute budgets have doubled roughly every 6 months since the start of deep learning, including in language models. This suggests that physical compute has instead grown by a factor of around one-million-fold. This paints a stylized picture where 'effective compute" expanded by about 22-billion-fold since 2014, with slightly under two-thirds of the scaling being due to increased use of actual, physical computing resources.

There are reasons to be cautious about this naive extrapolation. For one, we do not directly observe gains of 22 , 000 × (or even 10 , 000 × ) anywhere in our dataset. However, given that it is unlikely that early researchers trained language models on very large quantities of compute, it is therefore improbable that we observe such large declines over the analyzed time period. Nevertheless, the lack of such observations still raises questions about the reliability of extrapolating these trends between long multi-year periods.

One specific reason for caution is that the extrapolation neglects the scale-dependence of algorithmic innovations. It is likely that some algorithmic innovations will become obsolete over time as models are trained at larger scales of compute-e.g. the effectiveness of specific tokenizers or hyperparameter settings may diminish, making them less useful for future, larger models. Conversely, recent innovations might fail to produce large or any benefits when implemented at much smaller scales than models today. For example, the gains from scaling laws are related to the scale of compute used (see Appendix B), and older architectures, such as the LSTM and convolutional network, can exhibit higher efficiency at small scales relative to the transformer [Droppo and Elibol, 2021, Karpathy, 2022].

While a naive extrapolation of doubling times predicts substantial reductions in compute requirements, our work does not provide compelling evidence that we can currently or in the future train extremely small models to achieve the performance of much larger ones by applying the full suite of modern innovations. The scale-dependence of algorithmic improvements and the lack of direct observations of such large efficiency gains in our dataset suggest that further research and more comprehensive data are needed to validate these extrapolations.

Besides doubling times, we can also decompose the relative contributions from algorithms and compute scaling by evaluating our estimated models directly. We approach this using a Shapley value analysis, and the results weakly support the stylized picture above that compute scaling has been more important for explaining performance improvements than algorithmic progress since 2014.

The findings indicate that the relative contribution of algorithmic progress to performance improvements has diminished over time, at least within the dataset of models that have historically been close to the state-of-the-art. This observation aligns with the stylized representation in Figure 2 and the findings of Erdil and Besiroglu [2022] for computer vision, where compute scaling has shown increasing importance over time.

One explanation for the diminishing relative contribution of algorithmic progress is that investments in expanding physical compute have increased substantially, outpacing the rate of algorithmic improvements. This framing aligns with the increased emphasis on scaling large language models over the last few years, particularly since the introduction of GPT-2 in 2019 [Radford et al., 2019], relative to fundamental algorithmic or architectural changes. 9 Figure 2 illustrates a stylized version of this perspective, depicting a sharp increase in physical compute scaling around 2018-2019, followed by a return to previous compute scaling growth rates.

There are other potential explanations - for example, it is possible that the transformer architecture was a pivotal innovation (see section 3.3), and subsequent algorithmic advances have been less

significant in comparison. Alternatively, this observation could also be explained by a secular decline in the rate of algorithmic innovation. However, we find these two explanations less compelling than the results of Figure 2, where the rate of algorithmic progress does not clearly decrease after the release of the transformer (e.g. with a 2018 cutoff). If anything, the rate increases slightly, contrary to what both of these explanations predict.

3.3 The significance of the transformer architecture

Since its introduction in 2017 [Vaswani et al., 2017], the transformer architecture has become the dominant algorithmic architecture in language modeling, forming the base of multiple notable systems. We attempt to quantify the its contribution in terms of the 'compute-equivalent gain" over other architectures in our dataset (LSTMs, RNNs, state space models, among others). This is akin to the approach outlined in Davidson et al. [2023]-in this context, the compute-equivalent gain is the amount by which training compute must be scaled to improve benchmark performance as the same amount as the introduction of the transformer. For example, Hernandez and Brown [2020] find that a transformer (2017) achieves the same performance as a Seq2Seq (2014) model on the WMT-14-EN-FR benchmark, with 61 × less compute.

To capture the improvement represented by the transformer, we modify our core model as follows:

where σ : R → (0 , 1) is the sigmoid function, given by σ ( x ) = 1 / (1 + e -x ) . γ T is a constant and all other terms have the same meaning as in the original model. 10 The key intuition is that the transformer could enable us to use compute (or perhaps data) more efficiently than the architectures that precede it.

After preprocessing, our dataset contains 103 transformer models, and 127 non-transformer models, largely consisting of recurrent networks such as the LSTM. Fitting the model on this data reveals that the transformer architecture typically lowers reducible loss proportionally by 5.4% [90% CI: 3.8%, 6.9%].

We can calculate its contribution in terms of 'compute-equivalent gains" numerically: we first calculate the predicted loss for a transformer with some N and D , and the predicted loss for a

non-transformer with the same inputs. We then determine reduction in N and D to match this difference in loss. Compute is then approximated as usual, as C ≈ 6 ND . In short, if an innovation halves the compute needed to achieve a specific loss, then that innovation has a compute-equivalent gain of 2.

Based on 100 bootstraps, we obtain a median estimate of 9.6 × [90% CI: 4.3 × , 34.5 × ] for the transformer's compute-equivalent gain. 11 This substantial gain indicates that the efficiency offered by the transformer architecture is equivalent to around log(9 . 6) / log(2e4) ≈ 23% of the total gains from algorithms in the past nine years, or nearly two years of algorithmic progress in the field. 12 Moreover, this could understate the gains if the transformer architecture also provides a convenient vehicle through which to productively channel compute, thereby facilitating some of the gains through the scaling of compute that have likely dominated the overall gains we have seen recently.

One caveat here is that the measured significance of the transformer may depend on how it is evaluated. For example, transformers may be better adapted to long contexts than recurrent networks, and evaluations using longer contexts (e.g. > 1000 tokens) may suggest a larger improvement from transformers than evaluations using shorter contexts [Kaplan et al., 2020]. We have not explicitly controlled for context length here, and we discuss the potential impact of this assumption in more detail in appendix E.2.1.

4 Discussion and conclusion

4.1 Summary of our findings

This paper presents a comprehensive empirical analysis of algorithmic progress in language model pre-training from 2012 to 2023. By curating a dataset of over 200 language model evaluations on WikiText and Penn Treebank benchmarks, we quantify the relative contributions of compute scaling and algorithmic efficiency improvements to the overall performance gains. Our key findings are as follows:

First, we estimate that the compute required to reach a set language modeling performance level has halved every 7-8 months on average since 2012. This supports the common intuition that language modeling is an unusually rapidly-advancing field of computer science.

10 7

10 8

10 9

10 10

10 11

10 12

Training Dataset Size (Tokens)

10 8

10 9

10 10

10 11

Parameters

10 9

10 10

10 11

10 12

10 13

10 14

10 15

Training Dataset Size (Tokens)

10 10

10 11

10 12

10 13

10 14

10 15

10 16

10 17

Parameters

Predicted requirements for Chinchilla performance

Figure 3: Pareto frontiers for GPT-2 [Radford et al., 2019] and Chinchilla [Hoffmann et al., 2022] level performance on WT103. We truncate the frontiers to a factor of 1e3 greater or smaller than the existing training dataset size and parameter size of the actual model since extrapolating further out would not be reliable.

Second, our work reveals that the majority of recent advancements in language modeling stem more from scaling models and datasets than from pre-training algorithmic innovations. A Shapley

value-based analysis suggests that 60-95% of the performance gains stem from compute scaling, while algorithms contribute only 5-40%.

Third, the introduction of the transformer architecture in 2017 was a major algorithmic advance, representing between 3x and 46x in compute-equivalent gain, which accounts for more than 10% of the algorithmic innovation in pre-trained language models in the past decade. This highlights the significance of the transformer as a key architectural breakthrough in the field.

4.2 Limitations

While our analysis is an advance in quantifying algorithmic progress, several limitations reduce the precision of and temper our confidence in our estimates:

Lack of estimates of gains from specific innovations . Our model is specified to quantify algorithmic progress over relatively large time periods (e.g. over several years). However, it is unable to give reliable fine-grained information, such as progress over shorter time scales, or the significance of specific innovations. Experimental work is better suited to estimating efficiency gains for specific algorithmic innovations.

Limited availability of quality data, resulting in noisy or unrealistic estimates of progress . The approach we use in our analysis relies heavily on having many data samples across many years. This proved to be very challenging for a number of reasons-e.g. models are not always evaluated on the same benchmark, data is relatively sparse prior to 2017, and papers may not report relevant information such as parameter counts. Among other reasons this can result in our estimates being very noisy, yielding wide confidence intervals over doubling times. In addition, algorithmic improvements and scaling have historically been introduced concurrently, and this correlation between the two in our dataset can make it hard to disentangle their relative contributions to overall effective compute growth.

Inconsistencies in model training and evaluations . Inconsistencies in evaluations are well-known. While we have excluded non-standard evaluations from our dataset, our dataset spans models with different tokenization schemes, text preprocessing, stride lengths, and other details. This introduces noise and potential bias in our estimates of algorithmic progress, as researchers might have adopted more favorable evaluation schemes over time. However, our estimated reductions in perplexity from algorithmic improvements are large; likely larger than can be accounted for by changes in evaluation procedures. We expand on these points in Appendix E.2.3.

Inability to distinguish between data quality and efficiency in data use . Reduction in data requirements could be due to both improved data quality and improved algorithms, but our model is not equipped to distinguish between these effects. Understanding the relative contributions of each could be a subject of future research.

Reliance on the Chinchilla scaling law . The scaling law from which our model is derived applies to dense transformers following a GPT-3 architecture [Hoffmann et al., 2022, Rae et al., 2021]. However, we use this scaling law to model algorithmic improvements in different transformer architectures, recurrent neural networks, etc. Future algorithms might also follow different scaling laws (e.g. GPT-4 is rumored to be a mixture of experts). However, we believe it is likely that our core results should still hold: for one, neural scaling is not a phenomenon restricted to transformers (e.g. it is known to happen in RNNs as well, see Kaplan et al. [2020]). We find that a wide range of statistical model structures provide consistent estimates, and that alternative methods of estimating pre-training algorithmic progress also give similar results (see e.g. appendix A), so it is probable that our core results are robust to the use of the scaling law from Hoffmann et al. [2022].

Limited insight about future progress . While the results from this paper could be used to inform one about future progress in language modeling, our paper focuses on historical improvements. Future rates of progress could be slower (e.g. if one thinks that historical progress consisted of picking 'low hanging-fruit"), but they could potentially also be faster (e.g. due to increased research interest and investment). Expectations about future progress need to account for factors such as these, which we do not discuss in depth for the most part.

4.3 Conclusion

Using a dataset of over 200 language model evaluations spanning 2012-2023 evaluated on Wikitext and Penn Treebank, we find that the compute required to reach a fixed performance threshold has halved approximately every 8 months. This is much faster than the rate associated with Moore's law and many other domains of computing. While algorithmic innovations have occurred rapidly, compute scaling has expanded by over a million-fold in this same period, exceeding the gains from algorithms and constituting the predominant source of performance improvements in recent years.

Overall, our work provides a quantitative estimate of the rapid pace of progress in language modeling. It also reveals the dominant role of scale rather than algorithms for recent gains. Future work could benefit from extending this analysis to additional, specific benchmarks and more closely examining the impact of data quality improvements and the gains from additional specific innovations. Despite its limitations, this research demonstrates the valuable insights that can be gained from a detailed statistical analysis of extensive datasets of machine learning results. By identifying the main drivers of performance improvements, this work lays the groundwork for further exploration and understanding of these trends in the field.

Acknowledgments

We thank Tom Davidson, Pablo Villalobos, Josh You, Lukas Finnveden, Eli Lifland, nostalgebraist, David Schneider-Joseph, Danny Hernandez, Alyssa Vance, Yafah Edelman, Ben Edelman, Matthew Barnett, Ben Cottier, Keith Wynroe, Markus Anderljung, Carl Shulman, Marius Hobbhahn and Nikola Jurkovi´ c for their feedback. We thank Eduardo Roldán and Robert Sandler for helping design and implement graphs.

References

Uri Alon, Frank Xu, Junxian He, Sudipta Sengupta, Dan Roth, and Graham Neubig. Neuro-symbolic language modeling with automaton-augmented retrieval. In International Conference on Machine Learning , pages 468-485. PMLR, 2022.

Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. 2018. doi: 10.48550/ARXIV.1809.10853. URL https://arxiv.org/abs/1809.10853 .

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020. doi: 10.48550/ ARXIV.2005.14165. URL https://arxiv.org/abs/2005.14165 .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1-113, 2023.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. 2021. doi: 10.48550/ARXIV.2110. 14168. URL https://arxiv.org/abs/2110.14168 .

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R'e. Flashattention: Fast and memory-efficient exact attention with io-awareness. ArXiv , abs/2205.14135, 2022. URL https://api.semanticscholar.org/CorpusID:249151871 .

Tom Davidson. What a compute-centric framework says about AI takeoff speeds. https://www.alignmentforum.org/posts/Gc9FGtdXhK9sCSEYu/ what-a-compute-centric-framework-says-about-ai-takeoff , 2023.

Tom Davidson, Jean-Stanislas Denain, Pablo Villalobos, and Guillem Bas. Ai capabilities can be significantly improved without expensive retraining. arXiv preprint arXiv:2312.07413 , 2023.

Harm de Vries. In the long (context) run, 2023. URL https://www.harmdevries.com/post/ context-length/ .

Florian E Dorner. Measuring progress in deep reinforcement learning sample efficiency. arXiv preprint arXiv:2102.04881 , 2021.

Jasha Droppo and Oguz H. Elibol. Scaling laws for acoustic models. In Interspeech , 2021. URL https://api.semanticscholar.org/CorpusID:235458551 .

Ege Erdil and Tamay Besiroglu. Algorithmic progress in computer vision. 2022. doi: 10.48550/ ARXIV.2212.05153. URL https://arxiv.org/abs/2212.05153 .

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept , 2021.

Google Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. 2024. URL https://storage.googleapis.com/deepmind-media/gemini/ gemini_v1_5_report.pdf .

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. ArXiv , abs/2312.00752, 2023. URL https://api.semanticscholar.org/CorpusID:265551773 .

Danny Hernandez and Tom Brown. Measuring the algorithmic efficiency of neural networks. arXiv preprint arXiv:2005.04305 , 2020.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Hugging Face. Perplexity of fixed-length models. https://huggingface.co/docs/ transformers/perplexity , 2023. [Online; accessed 14-Nov-2023].

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Andrej Karpathy. Deep Neural Nets: 33 years ago and 33 years from now. http://karpathy. github.io/2022/03/14/lecun1989/ , 2022. [Online; accessed 21-July-2022].

Rémi Leblond, Felix Gimeno, Florent Altché, Alaa Saade, Anton Ruddock, Corentin Tallec, George Powell, Jean-Bastien Grill, Maciej Mikuła, Matthias Lochbrunner, Michael Mathieu, Paul Caron, et al. Alphacode 2 technical report. Technical report, Google DeepMind, 2023. URL https://storage.googleapis.com/deepmind-media/AlphaCode2/ AlphaCode2_Tech_Report.pdf .

Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541-551, 1989.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814 , 2022.

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. ArXiv , abs/2402.08268, 2024. URL https://api.semanticscholar. org/CorpusID:267637090 .

Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. ArXiv , abs/2305.16264, 2023. URL https://api.semanticscholar.org/CorpusID: 258888192 .

OpenAI. GPT-4 Technical Report. 2023. URL https://cdn.openai.com/papers/gpt-4.pdf .

Dylan Patel and Gerald Wong. Gpt-4 architecture, infrastructure, training dataset, costs, vision, moe, 2023. URL https://www.semianalysis.com/p/gpt-4-architecture-infrastructure .

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. 2021. doi: 10.48550/ARXIV.2112.11446. URL https://arxiv.org/abs/2112.11446 .

Jaime Sevilla, Lennart Heim, Anson Ho, Marius Hobbhahn, Tamay Besiroglu, and Pablo Villalobos. Estimating training compute of deep learning models. Technical report, 2022.

Sho Takase, Jun Suzuki, and Masaaki Nagata. Direct output connection for a high-rank language model. ArXiv , abs/1808.10143, 2018. URL https://api.semanticscholar.org/CorpusID: 52138320 .

Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? ArXiv , abs/2207.10551, 2022. URL https://api. semanticscholar.org/CorpusID:250920512 .

Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature , 625(7995):476-482, 2024.

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv , abs/1706.03762, 2017.

Chenguang Wang, Mu Li, and Alexander J Smola. Language models with transformers. arXiv preprint arXiv:1904.09408 , 2019.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas O˘ guz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. ArXiv , abs/2309.16039, 2023. URL https://api.semanticscholar.org/CorpusID:263134982 .

A Observing improvements in the data

Besides the statistical model that we presented in section 2.1, we can also attempt to obtain doubling time estimates more directly. For example, we can look at LLMs that achieve close to Megatron-LM's or GPT-2's level of performance over time, and see how much less compute is used. Doing so reveals that we need between 5-fold and 100-fold less compute per year in 2023 to achieve the same performance achieved is between 2019 and 2023, amounting to a halving time of between 11 and 17 months. This is within the 90% confidence interval of our aggregate doubling time estimates, from 1.7 to 22.5 months.

2020

2021

2022

Publication Date

300%

100%

30%

10%

3%

1%

Relative training compute

Megatron-LM

Compressive Transformer

ALiBi

Infinity former

Transformer + RD

B2T connection

SRU++ Large

Doubling Time:

11.7 months

(a) Relative training compute

needed for

Megatron-LM level performance

2022

2023

Publication Date

300%

100%

30%

10%

3%

1%

Relative training compute

ALiBi

Infinity former

NMST+GPT-2

Decaying Fast

Weights Transformer

Transformer

+ GFM

B2T connection

Doubling Time:

16.6 months

(b) Relative training compute needed for ALiBi level performance

This approach provides some insight but it has its issues, which is why we do not rely on it. For instance, it depends on the particular choice of reference performance. Moreover, this approach requires identifying models with similar or better performance at later dates, where the training compute is known. However, the data for the latter is relatively limited. We thus opt for using an arguably more principled approach with our core model presented in section 2.1 for our main results.

B The gains from better scaling laws

We estimated the compute savings afforded by the Chinchilla scaling law, as proposed by Hoffmann et al. [2022], in contrast to the previously dominant understanding based on the work of Kaplan et al. [2020]. First, we defined loss functions L ( N,D ) for both the Kaplan and Chinchilla scaling laws. Following this, we minimized these loss functions across variables D and N , considering different levels of compute budget. For each specified budget, we then calculated the amount of compute required under the Chinchilla scaling law to achieve a loss equivalent to the minimum loss obtained under the Kaplan scaling law. The Compute-Equivalent Gain (CEG) was subsequently determined as the ratio of the original compute budget to the compute required by the Chinchilla scaling to match the Kaplan loss.

We find that the compute equivalent multiplier from the Chinchilla scaling laws for dense autoregressive transformer models is between 1.25-fold (for GPT-2 scale models) and 1.6-fold (for PaLM-scale models Chowdhery et al. [2023]). 13

Compute multiple equivalent

10 19

10 20

10 21

10 22

10 23

10 24

10 25

Compute

1.0x

1.2x

1.4x

1.6x

1.8x

2.0x

GPT-2

scale

GPT-3

scale

PaLM

(540B)

scale

C Core model parameter estimates

The core model that we use was chosen based on leave-one-out cross validation, and is defined similarly to equation 3 but with a few modifications. The most important change is that A and B are estimated separately for each benchmark, whereas all other parameters are benchmark-agnostic. In order to help with model fitting, we normalize N and D to some minimum N 0 and D 0 values in our dataset, and reparameterize A and B as exponentials. In full, our model is

To estimate these in benchmark-specific fashion, we introduce dummy variables x WT2 and x PTB for WT2 and PTB respectively. We then complete the model definition as follows:

Our parameter estimates are summarized in Table 2.

One observation about the parameter estimates in Table 2 is that the confidence intervals for α year and β year are not statistically significant at the 5% significance level, while α param and β data are. As mentioned in section 3.2, the result is that the model fails to obtain statistically significant estimates of effective parameter and effective data doubling times. However, when we use these estimates to determine effective compute doubling times, we obtain statistically significant estimates. The reason for this is illustrated in Figure 6-the estimates for α year and β year are clearly negatively correlated. In particular, when α year is positive, β year is negative and vice versa, such that the overall estimated effective compute doubling time is always positive.

C.1 Comparing our estimates to earlier work

Given that our core model is similar to previously proposed language model scaling laws, we can compare our estimates to see how well they correspond to prior work. In particular, the estimates for α param and β data in Table 2 suggest that cross entropy loss scales roughly as C -1 / 20 , where C is training compute. In comparison, Kaplan et al. [2020] find a scaling exponent of around -0.048, and Hoffmann et al. [2022] estimate values of around -0.3. Given that our model is constructed based on the scaling law in Hoffmann et al. [2022], we might a priori have expected our estimated to match those more closely-so what explains the difference?

One way to understand this discrepancy is to consider the scaling laws on the same plot, shown in Figure 6. Here we observe that the scaling laws strongly diverge for compute values below around

10 14

10 16

10 18

10 20

10 22

FLOP

2

3

4

5

6

Cross entropy loss

3.6 × 10 17

FLOP

1.1 × 10 22

FLOP

Ours

0.100

0.075

0.050

0.025 0.000

0.025

0.050

0.075

year

0.050

0.025

0.000

0.025

0.050

0.075

0.100

year

94.9%

5.1%

10 18 to 10 19 FLOP (and the same is true for values greater than 10 22 FLOP). However, between these two regimes the scaling laws appear much more similar in slope.

This observation suggests the possibility that the discrepancy in estimated scaling exponents is due to the range of fitted data. Indeed, around 80% of our models with known training compute estimates lie between ∼ 4 × 10 17 FLOP and 10 22 FLOP. This suggests that a large fraction of our data lies within the regime where it is hard for our model to distinguish between the exponents from Hoffmann et al. [2022] and Kaplan et al. [2020].

Another possible explanation for this discrepancy that we considered is that it is due to the omission of an irreducible loss term in our core model, resulting in an omitted variable bias. However we do not put much weight on this explanation for our fits-in our robustness check using models with an irreducible loss term (see section H), we obtain very similar scaling exponents to those obtained in our core model.

D Significance of the transformer architecture

Similarly to the doubling times for effective compute, we consider the predicted Compute-Equivalent Gains by applying the same modification in Equation 7 to different model specifications. Most models yield estimates within a similar ballpark as our core model (model 7 with δ = 0 . 0025 ), with some models yielding relatively noisy estimates. That said, three models predict a notably larger efficiency contribution from the transformer, and this suggests that there is plausibly a fairly large degree of cross-model uncertainty present. These results are shown in Figure 7.

Compute Equivalent Gain (CEG)

10 (0.005)

11 (0.001)

15 (0.001)

7 (0.02)

9 (0.02)

12 (0.02)

5 (0.001)

8 (0.005)

6 (0.005)

4 (0.0025)

3 (0.001)

1 (0.001)

2 (0.001)

Model (delta)

10 0

10 1

10 2

10 3

10 4

Better cross-validation performance

Worse cross-validation performance

E Dataset

E.1 Performance measure and dataset

A key component of measuring progress in machine learning algorithms is the presence of a performance metric or benchmark. Since our focus is language modeling, token-level perplexity is a natural choice for this metric, for which we choose three benchmarks: WikiText-103, WikiText-2, and Penn Treebank. Note that WT2 and WT103 are both constructed from articles on Wikipedia. The two benchmarks share the same validation and testing set, while WikiText-103 has a much larger vocabulary and training set. In total, our dataset is constructed from 226 papers, from which we collect around 410 models that have reported token-level perplexity. Of these models, 370 contain sufficient information to be considered for analysis (perplexity, parameter size, publication date, and size of dataset).

E.2 Perplexity

A standard metric for LLM performance is the measured test perplexity on standard datasets. For a language model, this is defined in as the exponential of the cross-entropy L between the model predictions and the test set, i.e. Perplexity = e L .

We choose this metric for two primary reasons. First, this is a commonly reported measure of performance, which allows us to gather a large dataset for our analysis. Second, the simple relation between perplexity and cross entropy L allows us to easily relate our model to neural scaling laws.

If a paper reports the perplexity of just one model, we collect that singular data point. However, when a paper presents multiple models, only those meeting any of the following criteria are included in our dataset:

The model is trained or evaluated on a different benchmark dataset. A model trained on two different datasets can be used as a reference to understand how perplexity metrics reported on different benchmarks relate to one another. This can be helpful e.g. for data imputation

The model is constructed with a drastically different parameter size, as such data inform the impact of scaling

The model has a significant difference in the algorithms used than other models in the paper

For papers presenting many (10 or more) models, we exclude some from our dataset to prevent possible bias from over-representing results from a few studies. We prioritize models with the lowest perplexity in their category, often highlighted in bold within tables. We also exclude minor algorithm alterations and ablations that do not impact the parameter count. In Appendix I, we take an alternative approach by including all models from each paper and explicitly modelling the autocorrelation

structure from results from the same paper. In doing so, we find results highly similar to those we present in the paper.

E.2.1 Context length

Another consideration when analyzing algorithmic innovation in language models pertains to the context length. For one, measured perplexity on benchmarks can depend on the context length [Kaplan et al., 2020, Xiong et al., 2023]. Different systems may have been trained or evaluated using different context lengths, and this might make model perplexity scores less directly comparable.

One way to try and quantify the magnitude of this effect is to look at studies that compare the change in perplexity given a change in context length. For example, based on the scaling relations relating loss and context length from Xiong et al. [2023] and Kaplan et al. [2020], back-of-the-envelope calculations suggest that loss reductions each year due to increasing context length could be 10-60% as large as the loss reductions from algorithmic progress.

In particular, Xiong et al. [2023] finds a relation between context length c and validation loss for different versions of the LLaMa 2 language model. We can estimate ballpark values for context length over time based on data from de Vries [2023], and use this to roughly estimate how expanding context lengths has decreased loss over time (e.g. a decrease of a few percent per year). We can then compare the magnitude of this effect to the contribution of algorithmic progress to decreases in loss, and we typically arrive at values between 10-60% of the overall algorithmic progress contribution. 14 We perform a similar analysis with the scaling relation described in Kaplan et al. [2020], with similar results. If this rough calculation is correct, it suggests that increasing context length may have been a fairly important dimension along which algorithms have been improving.

Ontop of measured reductions to averaged perplexity per token, one might also consider the increasing ability of language models to perform long-context tasks to be largely downstream of algorithmic progress in itself. Being able to handle long contexts is a key motivator for several recent algorithmic innovations [Liu et al., 2024, Gu and Dao, 2023, Gemini Team, 2024], and this has likely grown very substantially since the introduction of FlashAttention [Dao et al., 2022, de Vries, 2023]. We consider this to be an important avenue for further investigation.

E.2.2 Tokenization

One way to get a sense of the impact of tokenization on measured doubling times is to introduce a fixed effect that depends on the benchmark vocabulary size into our preferred model. In particular, we introduce an irreducible loss term to Equation 8, of the form γ log( vocabulary size ) .

As with the rest of our analysis, we perform bootstraps to obtain a distribution over estimated doubling times, and further fold this model into our cross validation analysis. In particular, this model predicts a median effective compute doubling time of 5.5 months, with a 90% confidence interval of 2.6 to 11.8 months. In cross validation, this model performs essentially as well as our preferred model, with a MSE loss of 0.046-0.048 in both cases (depending on the regularization strength). Both of these results are very much in line with the results from our main model, lending some weight to the view that differences in tokenization schemes used in practice do not substantially change our core results.

One possible reason for this is that in practice, the tokenization schemes used for evaluating language models on the considered benchmarks (i.e. WT103, WT2 and PTB) are typically highly similar, and so including a contribution from vocabulary size has only a limited effect within each dataset. For example, typical tokenizers for PTB, WT2 and WT103 have vocabularies of roughly 10k, 22k, and 268k tokens respectively. We illustrate this in Figure 9, where for each benchmark, the majority of vocabularies fall into the same histogram bin.

E.2.3 Inconsistencies in perplexity evaluations

Inconsistencies in benchmark evaluations is a well-known issue in the machine learning community. These issues can introduce noise (if inconsistencies are 'random'), or bias if they systematically change over time. We curated data so that models were evaluated in roughly comparable ways, but often the precise details of evaluations were lacking, so that we could not verify the precise

Number of models

1e4

3e4

1e5

3e5

1e6

0

10

20

30

40

50

60

70

PTB

1e4

3e4

1e5

3e5

1e6

Vocabulary

WT2

1e4

3e4

1e5

3e5

1e6

WT103

evaluation procedure. However, there are many other subtleties with evaluation setups that may also cause perplexity results to differ, such as pretraining data, test-time adaptation [Takase et al., 2018], tokenization schemes, strides, and text preprocessing.

Overall, we do not see these issues substantially undermining our results. Our mainline estimates imply that 1 year of pre-training algorithmic progress amounts to a reduction of perplexity of around

10%, which is a much larger reduction than seems plausible to explain by changes in the average year-to-year evaluation procedures.

Given that test perplexity on these datasets has persisted as a standard measure of language modeling performance in the literature, we expect that differences in perplexity will broadly reflect genuine underlying differences in model capabilities.

We further consider one particular source of variation from Table 4 in more detail, namely the training set. To do this, we went through all the models in our dataset and classified them into one of three categories: (1) not trained on either the training or test sets of WikiText, (2) trained on the training set but not the test set of WikiText, or (3) trained on both the training and test sets of WikiText. 15 We then repeated this procedure for PTB.

Based on this classification, around 2% of models fall into the first category, around 85% of models are in the second category, and the remaining models are in the third category. To test whether our results are robust to this change in categories, we fit our model to models only in the second category. This yields effective compute doubling times of 5.7 months in the median case, with a 90% confidence interval of [1.7, 12.6] months, consistent with our core results. The parameter estimates and associated uncertainties are listed in Table 5.

E.3 Dataset Size & Epochs

In reporting training dataset size, we generally record the size of pre-training datasets as well finetuning datasets if the model in question has been fine-tuned. The number of epochs is usually sourced from the papers which describe the language model in question, but if not provided, we estimate it via

We adjust our epoch and dataset size calculations accordingly for models like GPT-Neo and Pythia, which report the effective number of tokens seen in training.

E.4 Parameter Size

We use the reported parameter size value if it is stated in the paper. We impute the parameter size from the previous models for papers that do not specify parameter size but follow known, state-of-the-art

models. Otherwise, we rely on other papers referring to the model's parameter size or manually compute parameters for certain RNN and LSTM models based on provided details about the model architecture (e.g. the number and size of hidden layers).

E.5 Inclusion and exclusion criteria

We exclude several models in the dataset from the analysis based on whether they meet any of the following primary criteria: (1) use of a retrieval mechanism, (2) use of model compression or pruning, (3) use of neural architecture search, (4) use of teacher-learner or knowledge distillation mechanisms, (5) use of cache models. These models are excluded because we expect these models to exhibit significantly different scaling behaviors from other models we analyze. In particular, they can substantially change the ratio of parameters and data that would 'optimally" minimize loss given some compute budget.

E.6 Dataset review

In a previous version of this paper, our core results had a narrower confidence interval as well as slightly longer median estimates. 16 One reason for this change is that reviewed the data a second time and fixed any identified errors - while this changed our results, it was well within the uncertainty ranges we had previously identified in our original paper.

That said, we further decided to modify our model aggregation approach to better represent crossmodel uncertainty. In particular, rather than aggregating over the top 10 model-delta pairs in cross validation, we instead aggregate across different model numbers (with the regularization strength δ that yields the best cross validation performance).

F Quantifying training data D

One important degree of freedom in the modeling process is how to define the 'training data". In particular, there we consider three possible definitions:

Training dataset size: The number of tokens in the dataset on which the language model was trained. This definition ignores the possibility of improvements in performance from training for multiple epochs, and is the approach taken in Erdil and Besiroglu [2022].

Tokens seen: The total number of tokens seen during the course of training a language model-this is the definition adopted in Hoffmann et al. [2022]. This is equivalent to the training dataset size if the model is trained for one epoch on the training set, which is fairly common practice but not totally ubiquitous. In fact, it is possible that more recent models are being pushed towards training with multiple epochs on the same data-e.g. GPT-4 was reportedly trained for 2 epochs on text and 4 epochs on code [Patel and Wong, 2023]. In cases where the tokens seen is not directly reported in paper, we estimate it via tokens seen ≈ num. epochs × training dataset size.

Tokens seen with diminishing returns: One problem with the previous approach is that seeing data repeatedly may yield diminishing returns. Muennighoff et al. [2023] find that the benefits in loss reduction drop significantly when training on more than 4 epochs.

In order to test the robustness of our most important result (compute doubling times) to this degree of freedom, we repeat our doubling times analysis as in section 3.1 but account for the number of training epochs. We then replace D either with tokens seen (epochs times training dataset size) or tokens seen with diminishing returns. For each case we consider two possibilities-either we drop datapoints for which the epoch number is unknown (this drops around 100 datapoints), or we impute the epoch number as 1. We report our core parameter estimates in Tables 7 and 8, and our doubling time estimates in Table 6.

From the results in Table 6, we see that there is substantial overlap between model estimates, but the estimates are highly noisy. The most similar are the original model based on dataset size, and model

with tokens seen with diminishing returns and imputed epochs, where the latter places slightly more weight on longer doubling times.

In the case of tokens seen with imputed epochs, the model predicts no statistically significant improvement in compute efficiency. On the other hand we do not observe this without imputation, which potentially suggests that our imputation strategy (i.e. assuming 1 epoch of training when epoch counts are unknown) is suboptimal under this definition of training data.

That said, there are strong reasons to be sceptical of using 'tokens seen" as a definition given that diminishing returns to additional training epochs are indeed observed in practice. Indeed, some papers report training for between 100-1000 epochs, which would result in fairly large variations in dataset size estimates if diminishing returns are not considered. We ultimately opt to quantify training data in terms of dataset size for simplicity, although a definition based on diminishing returns is also feasible.

Parameter estimates from the model described in equation 8, to 3 decimal places. We report 90% confidence intervals for all of the parameter estimates by bootstrapping 100 iterations.

G Doubling times via optimal scaling

In the main paper we calculated doubling times for effective compute based on a closed form solution for the doubling times, given by Equation 6. However, this equation was derived simply by considering changes in D eff, and it is not clear a priori whether or not the effective compute is optimally allocated between N eff and D eff to minimize the cross entropy loss. In addition, calculating compute efficiency doubling times is less straightforward for models which also include changing scale exponents α param and β data (i.e. models 14 and 15 in our cross validation analysis, see J). We

thus supplement our previous calculation with an alternative approach, which instead enforces the condition of compute-optimal scaling.

We approach this in two stages:

First, we calculate the reduction in cross entropy loss ∆ L given a doubling in compute budgets and under compute-optimality. Let the initial compute budget be C . We solve an optimization problem

subject to the constraint C = 6 ND described in Hoffmann et al. [2022], which is solved with values N ∗ 1 and D ∗ 1 . We then perform the same optimization problem but with a budget constraint of 2 C = 6 ND , yielding a corresponding cross entropy loss of L 2 . The reduction in cross entropy loss is given by ∆ L = L 2 -L 1 ≤ 0 .

We then estimate the years of algorithmic progress that would be needed to achieve this same reduction ∆ L . In particular, the optimization problem is

where we have

. Here δ can be interpreted as a doubling time for effective compute due to algorithmic progress in pre-training. α ′ const = α const + α const,PTB x PTB + α const,WT2 x WT2 and β ′ const = β const + β const,PTB x PTB + β const,WT2 x WT2, where x PTB and x WT2 are dummy variables for PTB and WT2 respectively.

We apply this approach over 100 bootstraps of our dataset, yielding a median doubling time of 6.1 months, and a 90% confidence interval of 3.3 to 11.3 months. This is very similar to the doubling times estimated using the closed-form approach discussed in section 3.1, which also has a median of 6.1 months [3.3, 11.3] for model 7.

H Irreducible loss

Our main model for estimating doubling times does not estimate the irreducible loss. This is in part due to empirical difficulties encountered in estimating plausible values for this term, and in part because the inclusion of this term does not have a bearing on our estimates of the rate of algorithmic improvements. Since the latter is the focus of our paper, we decided to move forward with the outlined model without irreducible loss estimates. However, we caution against overinterpretation of these results-for instance, our parameter estimates are not reliable enough to strongly inform how to scale models compute-optimally (although they can be somewhat illustrative).

The focus of this section is to justify the robustness of our core doubling time results to this omitted variable in our model. In particular, we fit a model which incorporates estimates of the irreducible loss and show that the doubling times remain in line with our previous findings. The model we consider is

where γ ′ = γ + γ PTB x PTB + γ WT 2 x WT 2 , and the rest of the model is defined in the same way as in Section 3.1. When we fit this model, we obtain a compute efficiency doubling time of 5.4 months, with a 90% confidence interval of 2.4 to 10.8 months, consistent with the estimates from our primary model. Our core parameter estimates are shown in Table 9.

I Autocorrelation

Our dataset was constructed by searching for papers with models that reported perplexity data on WT103, PTB, or WT2. In our initial data collection, we sometimes also included multiple models

originating from the same paper. In some extreme instances, more than ten models from a single paper were included; for example, we incorporated 14 models from the paper "OPT: Open Pre-trained Transformer Language Models." This poses concerns of autocorrelation, which might for instance result in us underestimating the uncertainty in our individual parameter estimates.

In the main body of the paper we approached this issue by retaining only three models per paper in our analysis, which resulted in the exclusion of approximately 35 models. Here we consider an alternative approach for addressing autocorrelation, where we explicitly quantify the correlations between models from the same paper. We then use this information to establish a multivariate normal likelihood function, which we maximize to obtain parameter estimates.

First, let us define the residuals as ϵ = x -ˆ x . The original loss function that we are using is the mean squared error, i.e. where the loss is given by E [ ϵ T ϵ ] . We want to modify our approach so that we take into account the correlations between different datapoints. The approach we take is to take an approach similar to generalized least squares and multiplicative attention-rather than consider just ϵ T ϵ , we consider ϵ T Pϵ , where P is a correlation matrix.

We do this using a maximum-likelihood approach, where ϵ T Σ ϵ placed in a multivariate normal distribution, given by

where Σ is a covariance matrix for the data. Our goal is to choose the appropriate parameters θ such that the resulting ϵ = x -¯ x maximizes this distribution. This includes the original parameters in the model from equation 3 as well as the correlation ρ between models from the same paper. We define the loss function as the negative of the logarithm of this distribution (dropping constants which do not matter for the resulting minimization problem):

In order to apply this to our data we need to specify the structure of the covariance matrix Σ = σ 2 n P (and thus Σ -1 = σ -2 n P -1 . We can accordingly write the loss function as

We are assuming that the models from different papers are uncorrelated, and that different models from the same paper have a correlation coefficient of ρ . If we order the models such that all the models from the same paper form a contiguous range of indices, then the correlation matrix looks block diagonal, where each block has 1s on the diagonal and ρ for the off-diagonal terms. The matrix elements that are not in blocks are all zero. For example, one example correlation matrix is:

As we can see, we need to determine both the inverse and the determinant of the correlation matrix P in order to calculate the negative log-likelihood. While this can be done using standard libraries, the matrices that we are considering here are quite sparse, and thus it is more efficient to simplify our calculations here. We detail the calculations in sections I.2 and I.3.

I.1 Results

To determine confidence intervals, we bootstrap this model based on clustering over 100 iterations, in similar fashion to the main results. This yields a median doubling time of 4.5 months, with a 90% confidence interval of 1.5 to 9.9 months, which is consistent with our core estimates. In practice ρ is typically on the order of 0.45, which suggests that the degree of autocorrelation is not very strong. We report our parameter estimates in Table 10.

I.2 Determinant

In order to obtain the overall determinant for P we first work this out for a single block. In particular, the determinant of each ( n +1) × ( n +1) block B n is given by

We prove this by considering the associated matrix directly:

Since det B n +1 is unchanged under the elementary row operation of adding a multiple of one row to another, we write

We can thus simplify the determinant as

To evaluate the determinant of this matrix we follow a similar procedure, where we eliminate most of the elements in the first column. We then repeat this process as the resulting matrix becomes smaller and smaller, until we reach a trivial case. The k th step in this iterative procedure has the following structure (where k = 0 , 1 , . . . , n -1 , n ):

We now substract ρ 1+( k +1) ρ times the first row from all other rows. In the first column, all except the first row thus becomes zeros, and there are two main cases we need to consider for the other elements. In the first case, for diagonal elements, we have we have

In the second case, for off-diagonal elements, we instead have

Thus we have that

If we repeat this argument n times (going from an ( n +1) × ( n +1) matrix to the trivial case of a n × n matrix, then we conclude that

as desired. Now to work out the overall determinant of P , we simply have to multiply the determinants of the individual blocks, and we are done.

I.3 Inverse

We now want to determine the third term in l ( θ ) , i.e. 1 2 σ 2 n ϵ T P -1 ϵ . One way to reduce the computational cost of this calculation is to take advantage of P being block diagonal, where the inverse P -1 can be determined by inverting the individual blocks. That is, for blocks B i the inverse of the correlation matrix is

To invert a single block B , first observe that each block B is such that the elements are

Now let D be the diagonal matrix with diagonal entries D ii = 1 -ρ . Then we have that B = D + ρvv T , where v is a vector of ones. We can now calculate the inverse B -1 by application of the Sherman-Morrison formula. In particular we have

̸

Since the inverse of D is just D -1 ii = 1 1 -ρ , assuming that the correlation ρ = 1 . To simplify notation, we define c = 1 ρ + n 1 -ρ . We then have

̸

and calculating the associated quadratic form 1 2 σ 2 n ϵ T P -1 ϵ follows trivially.

J Cross validation for model choice

We determined our primary model for estimating doubling times using formal model selection procedures. In particular, we considered a range of 15 candidate model numbers and a range of possible regularization strengths δ ∈ { 0 , 0 . 001 , 0 . 0025 , 0 . 005 , 0 . 01 , 0 . 02 } . We consider each 'model" as a pair , consisting of a particular model number and a regularization strength δ . We then perform leave-one-out cross validation on all of these models. Here we show the results of this analysis.

Our candidate models are defined by varying three degrees of freedom:

Which parameters to make benchmark-specific (e.g. having three separate parameters for the exponent on training dataset size, one for each of WT103, PTB and WT2).

Whether to explicitly model algorithmic progress in parameters N , data D

, or both.

In general we do not include the irreducible loss in the model (i.e. we usually set E = 0 , in the equation 3)-this is described in more detail in section H. In Tables 11 and 12 we list the definitions of all models that we looped through in leave-one-out cross validation. For this, we first define some common terms for simplification:

Here x PTB, x WT2, and x T are dummy variables for PTB, WT2, and the transformer respectively. Y is the year, and α rate and β rate are constants that determine how quickly scaling exponents ( α param and β data) change over time.

Models 1 to 11 are all constructed using a similar set of rules:

α const and β const determine the coefficients of the parameter and data reducible loss terms respectively

α year and β year capture algorithmic progress in parameters and data

α param and β data determine the scaling behavior with respect to N and D respectively

All of these parameters may be set as benchmark-specific (e.g. by writing α ′ const in lieu of α const ).

The remaining models are defined using a different set of rules:

Model 12 is defined in similar fashion but in 'Hicks-neutral' fashion, such that the same degree of efficiency gain is seen across both parameters and data.

Model 13 models different scaling exponents α param and β data for transformer vs nontransformer models

Model 14 captures only algorithmic progress via changes in the scaling exponents α † param and β † data

Model 15 captures algorithmic progress via α year and β year, as well as changes in the scaling exponents

No. Model specification

1 Algorithmic progress in both N and D L = exp[ α const -α year ( Y -Y 0 ) -α param (log N -log N 0 )] +exp[ β const -β year ( Y -Y 0 ) -β data (log D -log D 0 )]

2 No algorithmic progress in parameters N L = exp[ α const -α param (log N -log N 0 )] + exp[ β const -β year ( Y -Y 0 ) -β data (log D -log D 0 )]

3 No algorithmic progress in data D L = exp[ α const -α year ( Y -Y 0 ) -α param (log N -log N 0 )] + exp[ β const -β data (log D -log D 0 )]

4 Benchmark-specific in α year L = exp[ α const -α ′ year ( Y -Y 0 ) -α param (log N -log N 0 )] +exp[ β const -β year ( Y -Y 0 ) -β data (log D -log D 0 )]

5 Benchmark-specific in β year L = exp[ α const -α year ( Y -Y 0 ) -α param (log N -log N 0 )] +exp[ β const -β ′ year ( Y -Y 0 ) -β data (log D -log D 0 )]

6 Benchmark-specific in α year and β year L = exp[ α const -α ′ year ( Y -Y 0 ) -α param (log N -log N 0 )] +exp[ β const -β ′ year ( Y -Y 0 ) -β data (log D -log D 0 )]

7 Benchmark-specific in α const and β const L = exp[ α ′ const -α year ( Y -Y 0 ) -α param (log N -log N 0 )] +exp[ β ′ const -β year ( Y -Y 0 ) -β data (log D -log D 0 )]

8 Benchmark-specific in α const and β const, with no algorithmic progress in N L = exp[ α ′ const -α param (log N -log N 0 )] + exp[ β ′ const -β year ( Y -Y 0 ) -β data (log D -log D 0 )]

9 Benchmark-specific in α const and β const, with no algorithmic progress in D L = exp[ α ′ const -α year ( Y -Y 0 ) -α param (log N -log N 0 )] + exp[ β ′ const -β data (log D -log D 0 )]

10 Benchmark-specific in α const , α year , β const , β year L = exp[ α ′ const -α ′ year ( Y -Y 0 ) -α param (log N -log N 0 )] +exp[ β ′ const -β ′ year ( Y -Y 0 ) -β data (log D -log D 0 )]

11 Benchmark-specific in α const , α year , α param , β const , β year and β data L = exp[ α ′ const -α ′ year ( Y -Y 0 ) -α ′ param (log N -log N 0 )] +exp[ β ′ const -β ′ year ( Y -Y 0 ) -β ′ data (log D -log D 0 )]

Models 16 and 17 do not capture algorithmic progress, and are used as a baseline for goodness-of-fit comparisons. Model 16 is modeled off equation 34, and model 17 modifies this to include transformer-specific scaling exponents. This is described in more detail in Appendix J.2

Model 18 is a model that only considers the total compute, via the approximation C = 6 ND . This is discussed in more detail in Appendix J.1.

Model 19 includes a term that accounts for different vocabulary sizes. We discuss this in Appendix E.2.2.

Model 20 is the same as model 7, but rather than using purely the training dataset size, it defines 'training data" in a way that accounts for the number of epochs of training. Where the number of training epochs is unknown, we impute an epoch count of 1. This model is presented in more detail in Appendix F.

No. Model specification

12 'Hicks-neutral' model L = (exp[ α ′ const + α param (log N -log N 0 )] + exp[ β ′ const + β data (log D -log D 0 )]) · exp[ -α year ( Y -Y 0 )]

13 Transformer vs non-transformer scaling L = exp[ α ′ const -α year ( Y -Y 0 ) -α ∗ param (log N -log N 0 )] +exp[ β ′ const -β year ( Y -Y 0 ) -β ∗ data (log D -log D 0 )]

14 Progress only via changing scaling exponents L = exp[ α ′ const -α † param (log N -log N 0 )] + exp[ β ′ const -β † data (log D -log D 0 )]

15 Changing α year , β year , α param, and β data L = exp[ α ′ const -α year ( Y -Y 0 ) -α † param (log N -log N 0 )] +exp[ β ′ const -β year ( Y -Y 0 ) -β † data (log D -log D 0 )]

16 Benchmark-specific Hoffmann et al. [2022] scaling law (for comparison only) L = exp[ α ′ const -α param (log N -log N 0 )] + exp[ β ′ const -β data (log D -log D 0 )]

17 Transformer-specific scaling law (for comparison only) L = exp[ α ′ const -α ∗ param (log N -log N 0 )] + exp[ β ′ const -β ∗ data (log D -log D 0 )]

18 Compute-only model (for comparison only) L = exp[ α ′ const -α year ( Y -Y 0 ) -α compute (log(6 ND ) -log(6 N 0 D 0 )]

19 Vocabulary fixed-effects (for comparison only) L = γ log( vocab ) +exp[ α ′ const -α year ( Y -Y 0 ) -α param (log N -log N 0 )] + exp[ β ′ const -β year ( Y -Y 0 ) -β data (log D -log D 0 )]

20 Same as model 7 but with imputed epochs (for comparison only) L = exp[ α ′ const -α year ( Y -Y 0 ) -α param (log N -log N 0 )] +exp[ β ′ const -β year ( Y -Y 0 ) -β data (log D -log D 0 )]

We chose to include models 16 to 20 in the cross validation analysis to help the robustness checks performed in different appendices.

J.1 Compute-only model

Given that our core focus is on estimating doubling times in effective compute, one natural parameterization is to directly consider total training compute C , in particular,

Here γ ′ is defined similarly to the 'primed" constants above, i.e. γ ′ = γ + γ PTB x PTB + γ WT2 x WT2, and we have the approximate relation that C ≈ 6 ND [Hoffmann et al., 2022]. However, this model has a tendency to yield implausible results, in particular with a very small scale exponent α compute and a very short effective compute doubling time (on the order of 1-5 months).

We omit this model because we believe there are two reasons to expect this model to be strongly misspecified. The first reason is that the model scaling does not accurately reflect complementarities between scaling parameter and data. For illustration, compare the following two equations:

where E is the irreducible loss, and A,B,α,β are constants. In equation 34, reductions to the loss are bottlenecked by a lack of sufficient data D . This bottleneck does not exist in the case of 35, where scaling N arbitrarily would bring you to the irreducible loss E . This model thus unrealistically suggests equivalent performance between a 1-parameter model trained on 10 24 tokens, and a 10 12

parameter model trained on 10 12 tokens. Of course, such extreme choices of parameters N and dataset size D are implausible in practice, but nevertheless we do see clear variation in how these values are chosen in practice. For instance, Figure 10 shows a plot log 10 ( N/D ) for models across different years, spanning over five orders of magnitude over this time period. This is in spite of the estimate by Hoffmann et al. [2022] that N/D should be O (1) for compute-optimal training.

2012

2014

2016

2018

2020

2022

2024

Publication date

10

3

10

2

10

1

10 0

10 1

10 2

Tokens per parameter D/N

GPT-3

A second way in which equation 33 is misspecified is due to its unrealistic scaling behavior. To see why this is the case, we compare the difference between the reducible loss terms in equations 34 and 35, by writing them as a single fraction with the same denominator ( ND ) α . For illustration purposes, we also assume that α = β and A = B , which simplify our argument without changing the core conclusion. If we then factor out the coefficient A , we then have that

where L H is the loss predicted from the scaling law in Hoffmann et al. [2022], and L C is the loss from only considering compute. Here we observe that in the case of Chinchilla-scaling (equation 36), as N or D is increased the value of the numerator increases , whereas the opposite is true in the compute-only case (equation 37). As a result, fits of the compute-only model tend to have very small values of α , since the difference in scaling behaviour between the two expressions tends to be more pronounced for larger values of α . Indeed, estimates using the compute-only model yield very small values of α , at 0.004 with a 95% CI of [0.002, 0.012]. We illustrate the difference between the scaling behavior of equations 36 and 37 in Figure 11.

We nevertheless test the compute-only model in cross validation and find that it performs very poorly on out-of-sample prediction, far worse than any of the other models that we consider. The results of this exercise are elaborated upon in section J.3.

J.2 Algorithmic progress through changes in the scale exponents α param and β data

As mentioned in Section 2.1, the final model that we use for our core results does not explicitly account for efficiency improvements through changes in scaling exponents (i.e. α param and β data ). The primary reason for our decision is that while this form of algorithmic progress is theoretically plausible, our model fits to the available data suggest that this effect has not been very large. For example, if we consider the estimates from model 15, which includes both the form of algorithmic progress described in Section 2.1 and improvements through changes in scaling exponents, we find that the overall contribution to algorithmic progress is dominated by the former.

10 6

10 7

10 8

10 9

10 10

10 11

10 12

0.0

0.2

0.4

0.6

0.8

1.0

Normalized value

=0.01

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Parameters

=0.05

10 6

10 7

10 8

10 9

10 10

10 11

10 12

=0.3

Compute-only

Another piece of supporting evidence is that in cross validation, model 14 appears to perform roughly as well as models without any algorithmic progress at all (models 16 and 17). Furthermore, the parameters which determine the rate of changing scale exponents ( α rate and β rate) are generally very small (around 0.001 to 0.01). As such, the model appears to simply be approximating the equivalent models without algorithmic progress. This suggests that this form of algorithmic progress is negligible, and is also why we have excluded model 14 from Figure 1.

J.3 Performance metrics

In this section we list the resulting goodness-of-fit metrics from our cross validation analysis. In this case we report the average out-of-sample MSE loss.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We describe the core approach (estimating augmented scaling laws), and the core things we investigate (the rate of algorithmic progress, the relative contributions of scaling models vs algorithmic progress) in the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This is described under the 'Limitations" subsection of the discussion, where we mention problems like the lack of quality data and the reliance on the Chinchilla scaling law. We have attempted to be as transparent as possible about the model assumptions and the challenges in extrapolating our results to future AI progress.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The only theoretical result in this paper is in appendix I on autocorrelation; everything else is empirical. We provide the full derivation of our result in that section.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We explain the model specification in section 2.1, as well as full details in the appendices (e.g. the full model equations are listed in appendix J). The data and optimization strategy used are all described in section 2.2. The key results can all be tested by fitting the same model specification on the same data, the code and dataset for which are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All data and code is provided together with this paper submission.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The methodology (including datasets, statistical model, and optimization algorithms) are all described in section 2.1. We provide more detail about these in the appendices, e.g. we go into depth about our dataset in appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide bootstrapped confidence intervals throughout the paper to illustrate the uncertainty in our results. We also provide information about model uncertainty, by trying different model variants and showing how predictions vary across these variants.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our empirical results are based on statistical models that do not involve running machine learning experiments. These empirical results are obtained by fitting a dataset in Google Colab notebooks.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our empirical results do not involve human subjects or participants. Our dataset consists of data obtained from publicly-accessible machine learning papers, such as on benchmark performance and model size, and authors are referenced in the database.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the importance of our research for understanding AI trends and for informing policy in the introduction. We further emphasize potential downsides if our results are misinterpreted (e.g. extrapolated too far into the future) in the limitations section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The empirical results in our paper are from fitting a simple statistical model on publicly available benchmark results, publication dates, and information about dataset size and parameter counts. As such, our work does not pose serious risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use assets in the typical machine learning sense, with e.g. datasets for training neural networks or relevant code for training. Perhaps the most relevant asset we use is to collect certain results (like model size and benchmark performance) from public papers, and we link all these papers in our dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not create assets for machine learning training or inference, as this is not the focus of our paper. The most relevant 'assets" that we provide are the database (described in detail in this paper) and IPython notebooks pertaining to each section of this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not perform crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not perform research with crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper examine the performance improvements of language models over the past decade, and investigate how much of it can be attributed to algorithmic improvements of language models.\n\nSTRENGTHS:\nI note here that, given that this paper proposes a method to evaluate the historical progress of language models, I believe that the primary goal for the paper would be to provide interesting insights for the field and outline current open questions. As such, I will structure the rest of my review with that in mind.\n\n- The topic of language models is more central than ever to the broad NeurIPS community, and I think that analysis on the historical progress of the field is of great interest.\n\n- The authors obtain valuable insight on the cause of language model improvements over the years. I find their conclusion that data/compute scaling has been a major driving force for improvement over the past  few years interesting, if somewhat expected.\n\n- The result on the importance of the transformer is also interesting, and provides a nice retrospective justification of the broad adoption of this architecture.\n\n- The analysis of data from past works is also sound.\n\nWEAKNESSES:\n- The most crucial weakness of this paper is that, despite the analysis of past trends, it does not provide clear insights or suggestions on where the field should direct its efforts, moving on. The authors acknowledge this limitation, but it seems to me that such discussion Is crucial, in order for this paper to be of interest to the community.\n\n- It is also not clear to me how the insights provided by the paper could extrapolate in the future. While there has been a lot of effort and improvements gained in performance from data scaling, this is not something that can reliably keep on - at some point, sources of data and compute limitations catch up. As such, it is not clear to me how the insight that the effective compute for language models doubles every set period of time can be useful for many years down the line.\n\n- On more minor note, I think some points can be improved for clarity of presentation:\n\n  - The captions of Figures 1a and 1b are joined together - some spacing between them is required.\n\n  -  It would be better if the authors clarified the statistical models used in the main paper, rather than in the appendix.\n\n\nOverall, my key concern for the paper is that it does not provide enough insight for future directions.\n\nQUESTIONS:\nI would be grateful if the authors could expand on the points I raised above, regarding the insight for future directions.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work. Regarding negative societal impact, I do not foresee any arising from this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper investigates the rate of algorithmic progress on the task of language modeling, using a dataset of over 200 LM evaluations on WikiText and Penn Treebank between 2012-2023. The authors fit an augmented scaling law to the data and show that the models are requiring 2x less compute roughly every eight months -- a rate which supersedes Moore's law. Further, the authors find that more recent performance gains have been primarily due to compute scaling and that the contribution of the transformer architecture is roughly equivalent to two years of algorithmic progress in the field.

STRENGTHS:
1. This paper presents an interesting take on quantifying the progress on the task of language modeling using an analysis of data collected from over 200 model evaluations in the past 10-11 years.
2. The authors clearly specific the assumptions made in conducting the analysis, which makes the approach quite readable.

WEAKNESSES:
1. There a number of assumptions made to elicit a quantification of algorithmic progress, and the nature of the task itself necessitates those assumptions be made. However, this also means that the resulting analysis is brittle and as such the technical contributions of the work don't rise to the level of a solid contribution.

QUESTIONS:
1. Can you give examples of the ways in which the core results would change with different scaling law formulations are used?

LIMITATIONS:
The limitations were adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

comments: SUMMARY:
This paper presents an analysis of the relative contribution of algorithmic progress to overall LM performance gains over the window of 2012-2023. The authors evaluate a large number of potential equation variants for modeling algorithmic progress using leave-one-out CV. By making use of defined notions of effective data/parameters/computer, the authors estimate that 2x less compute is needed to acheive the same performance roughly every 8 months. The authors find that roughly 2/3 of the scaling progress is due to the scaling of physical compute resources, with the remainder being attributed to algorithmic progress. The singular contribution of the Transformer architectures is individually analyzed. Thorough analysis of the techniques applied and their limitations are presented.

STRENGTHS:
The paper is clearly written and well-presented. The depth and quality of the anlaysis are exemplary. The methodology of analysis is not highly original, but its application to algorithmic progress is a novel and useful contribution to the community. The limitations of this kind of analysis are well-discussed, which is a useful contribution in its own right.

WEAKNESSES:
Nits:
Figure 2: label the y axes, consider making especially the right plot a bit larger/more readable, the text is very small
Fig 1: fix the spacing between the (a) and (b) captions, they are nearly overlapping

QUESTIONS:
None.

LIMITATIONS:
The limitations are thoroughly discussed in the paper. No relevant missing potential negative social impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15

comments: SUMMARY:
This paper breaks down the driving force behind language models into two factors: scaling up the amount of compute used to train the models and algorithmic innovations. A statistical model, similar to the scaling law, is built and fitted to analyze the contributions of these two factors. The paper claims that models with the same performance level require half the compute every eight months, reflecting algorithmic improvement. The authors also find that the majority of scaling progress is due to the scaling of physical compute resources rather than algorithmic progress.

STRENGTHS:
The paper presents a very interesting and innovative approach to quantifying the algorithmic progress of language models. By covering over 200 language models from the past decade, the empirical foundation for the conclusions drawn is solid. Additionally, the author has performed extensive robustness checks to ensure the validity of their core results.

WEAKNESSES:
While the paper makes several assumptions to quantify algorithmic progress (including the extra coefficients in the augmented scaling law), these assumptions and the many degrees of freedom undermine the robustness of the proposed model and the conclusions drawn. I have doubts about whether the statistical model built can inform future progress in language modeling.

Minor issue: The caption of Figure 1 seems to be incorrect. It should address Figures 1a and 1b instead of 3a and 3b.

QUESTIONS:
None

LIMITATIONS:
I do not see any potential negative social impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-19