Evaluating the Robustness of Chinchilla Compute-Optimal Scaling

## Abstract

Abstract Hoffmann et al. ( 2022 ) ’s Chinchilla paper introduced the principle of compute-optimal scaling, laying a foundation for future scaling of language models.
In the years since, however, valid concerns about Chinchilla have been raised: wide confidence intervals, discrepancies between its three approaches, and incongruities with other scaling laws.
This raises a critical question for the field: Can practitioners still rely on Chinchilla’s prescriptions?
Our work demonstrates the answer is yes.
We begin by uncovering that the model parameters central to Chinchilla’s analyses were ambiguous: three interpretations are possible, with relative differences between different interpretations of model parameters as high as 15.2%.
We find that, perhaps surprisingly, which model parameters are used for the analyses do not meaningfully affect key results: the scaling law estimates and the compute-optimal tokens-to-parameter ratio.
Indeed, under one interpretation, the tokens-to-parameter ratio becomes more constant with the target compute budget.
We then ask how distorted the Chinchilla model parameters could have been without meaningfully affecting the key results.
By deliberately perturbing model parameters in four structured ways, we find that key Chinchilla results are most sensitive to additive or systematic errors, which can alter the otherwise flat
trend of the optimal tokens-to-parameter ratio, but overall, Chinchilla’s key results withstand sizable perturbations.
Altogether, our findings offer the field renewed confidence in Chinchilla as a durable guide for scaling language models.

## 1 Introduction

The study of neural scaling laws, which predictably map training resources to model performance, is a cornerstone of modern language modeling research and engineering.
Hestness et al. (2017) and soon after Kaplan et al. (2020) laid the foundation by demonstrating that pretraining losses scale as power laws with the number of data points, model parameters and pretraining compute. This was followed by significant additional work in the ensuing years (Sec. [4](https://arxiv.org/html/2509.23963v1#S4) Related Work).
Such discoveries led to the modern paradigm of training extremely large language models (OpenAI et al., 2024; DeepMind et al., 2025; DeepSeek-AI et al., 2025; Qwen et al., 2025; Kimi et al., 2025; Zhipu-AI et al., 2025).

The field’s understanding of scaling models was later altered by the seminal work of Hoffmann et al. (2022), which introduced the concept of compute-optimal scaling.
By training over 400 models ranging from 44M to 16B parameters on 5B to 500B tokens, Hoffmann et al. (2022) discovered that producing the best performing model with respect to a fixed pretraining compute budget (“compute-optimal") was achieved by linearly scaling model parameters and pretraining data together.
Chinchilla established the influential “20-to-1” heuristic: that the compute-optimal amount of training data is approximately 20 tokens per model parameter (Appendix [C](https://arxiv.org/html/2509.23963v1#A3)).
Their 70B Chinchilla outperformed larger models (Rae et al., 2022), cementing the methodology as a guiding principle for the field.

In the years since, several contributions have closely scrutinized Chinchilla, raising a number of concerns:
Zhang (2023) called attention to Chinchilla’s wide confidence intervals and questioned whether such uncertain estimates can provide practical guidance.
Besiroglu et al. (2024) investigated why some of Chinchilla’s approaches yielded inconsistent results.
Lastly, Porian et al. (2024) and Pearce & Song (2024) examined why Chinchilla makes different predictions than Kaplan et al. (2020)’s earlier scaling work.
While these works are clear contributions to the science of scaling, the field has been left uncertain: can practitioners still confidently rely on Chinchilla’s prescriptions?

In this work, we aim to answer this question.
As a warm up, we uncover that the model parameters central to the Chinchilla analyses were ambiguous, with three different possible interpretations as to which model parameters were used:
(1) the model parameters reported in Hoffmann et al. (2022)’s Table A9, (2) the model parameters calculated from the reported model architectural hyperparameters (layers, dimensions, number of heads, etc.) using a “standard” formula, and (3) the model parameters calculated from a “best-fit” formula.
Although the relative error among these three sets of model parameters rises as high as 15.2%, we show that key Chinchilla results – the estimated scaling law parameters and the compute-optimal tokens-per-parameter ratio – do not meaningfully change.
In fact, the only potential consequence is that the compute-optimal tokens-per-parameter ratio becomes more constant with respect to the target compute budget, strengthening Chinchilla’s finding.

To more generally assess the robustness of compute-optimal scaling, we then study how distorted the model parameters could have been without changing Chinchilla’s key results.
We perform a sensitivity analysis by perturbing model parameters in four structured ways.
Our analyses reveals that the robustness depends on the nature of the perturbations: while multiplicative perturbations and random noise have limited effects, additive constants or systematic biases can qualitatively change the compute-optimal scaling strategy by altering the trend of the optimal tokens-to-parameter ratio.
However, overall, all four sensitivity analyses demonstrate that Chinchilla’s key results withstand sizable perturbations.
Our results reveal a clear picture: Chinchilla’s compute-optimal prescription remains robust, further justifying its widespread use as a practical scaling blueprint for practitioners.

## 2 Key Chinchilla Results Are Robust to Three Interpretations of Chinchilla Model Parameters

One of the fundamental inputs to the Chinchilla analyses are the number of parameters per model.
However, an ambiguity exists as to which exact model parameters were used, with three different possible interpretations differing by as much as 15.2%.
We uncovered this by closely examining Chinchilla’s Table A9, which reports the number of model parameters for each model alongside key architectural hyperparameters, e.g., d_model, ffw_size, kv_size, n_heads and n_layers.
We call the model parameters reported in Chinchilla’s Table A9 the reported model parameters.
We include a brief snippet in our main text (Table [1](https://arxiv.org/html/2509.23963v1#S2.T1)) and the full table in our Appendix [B](https://arxiv.org/html/2509.23963v1#A2).

However, a second interpretation of the model parameters arises from the provided architecture hyperparameters; assuming the embedding and unembedding weights are tied (Press & Wolf, 2017) and no gating is present, a standard formula for the number of model parameters is:

$$ $\displaystyle\begin{aligned} \text{Standard Formula Model Params}&\approx\text{Embedding Params}+\text{Attn Params}+\text{FFN Params}\\ \text{Embedding Params}&=\text{Vocab Size}\cdot\text{d_model}\\ \text{Attn Params}&=\text{n_layers}\cdot(4\cdot\text{d_model}\cdot\text{kv_size}\cdot\text{n_heads})\\ \text{FFN Params}&=\text{n_layers}\cdot(2\cdot\text{d_model}\cdot\text{ffw_size})\end{aligned}$ (1) $$

Comparing the standard formula model parameters with the reported model parameters reveals a mismatch for every model, with an average relative error of 7.4% but reaching as high as 15.2% and no less than 3.6% (Fig. [1](https://arxiv.org/html/2509.23963v1#S2.F1), left).
We calculate relative error as:

$$ $\text{Relative Error (\%)}=100\cdot\frac{\text{Reported Model Params}-\text{Standard Formula Model Params}}{\text{Reported Model Params}}.$ (2) $$

In an attempt to reconcile the two interpretations of model parameters, we determined a third interpretation based on a “best fit” formula that nearly matches the reported model parameters:

$$ $\displaystyle\begin{aligned} \text{Best Fit Formula Model Params}&\approx\text{Embedding Params}+\text{Attn Params}+\text{FFN Params}\\ \text{Embedding Params}&=\text{Vocab Size}\cdot\text{d_model}\\ \text{Attn Params}&=\text{n_layers}\cdot({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\textbf{5}}\cdot\text{d_model}\cdot\text{kv_size}\cdot\text{n_heads})\\ \text{FFN Params}&=\text{n_layers}\cdot(2\cdot\text{d_model}\cdot\text{ffw_size})\end{aligned}$ (3) $$

Switching from the standard formula model parameters to the best fit model parameters reduced the number of discrepancies with the reported model parameters from $50/50$ models to $6/50$ models, and reduced the largest relative error from 15.2% to 8.7% (Fig. [1](https://arxiv.org/html/2509.23963v1#S2.F1), right).

**Table 1: Three Interpretations of Chinchilla’s Model Parameters. Hoffmann et al. (2022)’s Table A9 provides the architectural hyperparameters of all models used in the Chinchilla analyses, along with the reported model parameters (specified in millions). However, two alternative interpretations of model parameters are possible: model parameters calculated from architectural hyperparameters using a “standard” formula (Eqn. [1](https://arxiv.org/html/2509.23963v1#S2.E1)) and model parameters calculated from architectural hyperparameters using a “best fit” formula (Eqn. [3](https://arxiv.org/html/2509.23963v1#S2.E3)). For the complete table, see Appendix [B](https://arxiv.org/html/2509.23963v1#A2).**
| Table A9 from Hoffmann et al. (2022) | Our Contribution |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| d_model | ffw_size | kv_size | n_heads | n_layers | n_vocab | Chinchilla’s<br>Reported<br>Model<br>Parameters (M) | Best Fit<br>Formula’s<br>Model<br>Parameters (M) | Standard<br>Formula’s<br>Model<br>Parameters (M) |
| 512 | 2048 | 64 | 8 | 8 | 32168 | 44 | 44 | 42 |
| 576 | 2304 | 64 | 9 | 9 | 32168 | 57 | 57 | 54 |
| 640 | 2560 | 64 | 10 | 10 | 32168 | 74 | 74 | 70 |
| ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ |
| 4864 | 19456 | 128 | 36 | 47 | 32168 | 13775 | 14319 | 13266 |
| 4992 | 19968 | 128 | 32 | 49 | 32168 | 14940 | 14939 | 13937 |
| 5120 | 20480 | 128 | 40 | 47 | 32168 | 16183 | 16182 | 14950 |

Figure: Figure 1: Disagreement Between Three Interpretations of Chinchilla’s Model Parameters. Each point is one of the 50 models in Hoffmann et al. (2022)’s Table A9. Left: Calculating model parameters from the provided architectural hyperparameters using a standard formula (Eqn. [1](https://arxiv.org/html/2509.23963v1#S2.E1); attention parameters = $\text{n_layers}\cdot 4\cdot\text{d_model}\cdot\text{kv_size}\cdot\text{n_heads}$) disagrees with the reported model parameters for $50/50$ models, with relative errors averaging $7.388\%$ and rising as high as 15.2%. Right: Calculating model parameters using a best fit formula (Eqn. [3](https://arxiv.org/html/2509.23963v1#S2.E3); replace $4$ with $\mathbf{5}$) matches $44/50$ of the reported model parameters, and reduces the largest relative error to 8.7%.
Refer to caption: x1.png

Figure: Figure 2: Key Chinchilla Results Are Robust to All Three Interpretations of Model Parameters. Hoffmann et al. (2022) fit a neural scaling law $L(N,D)=E+A\cdot N^{-\alpha}+B\cdot D^{-\beta}$, where $N$ is the number of model parameters and $D$ is the number of data (Eqn. [4](https://arxiv.org/html/2509.23963v1#S2.E4)). Top: The fit parameters $(\hat{E},\hat{A},\hat{\alpha},\hat{B},\hat{\beta})$ do not meaningfully change, regardless of which model parameters are used for fitting. Error bars are standard errors from 4000 bootstrapped samples. Bottom: The compute-optimal tokens-per-parameter ratio remains constant at $\approx 20$, regardless of which notion of model parameters are used in the fitting process. The slope is flattest with the standard formula model parameters ($-0.572$ per decade; best fit: $-1.049$; reported: $-1.248$). Error bars are 80% confidence intervals. Fitting and visualization were conducted using Besiroglu et al. (2024)’s [code](https://github.com/epoch-research/analyzing-chinchilla/).
Refer to caption: x2.png

We next tested how Chinchilla’s results change depending on which of these three notions of model parameters are used for fitting.
We focus on two key results in particular:
First, Chinchilla fit a neural scaling law to the pretraining loss $L$ as a function of model parameters $N$ and pretraining data $D$:

$$ $L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}},$ (4) $$

where $E$ is the irreducible error, $A$ is the parameter prefactor, $\alpha$ is the parameter exponent, $B$ is the data prefactor and $\beta$ is the data exponent. Second, Chinchilla derived from the estimated scaling law parameters a “20-to-1” heuristic for the compute-optimal ratio of tokens-per-parameters:

$$ $\text{Compute-Optimal Tokens-per-Parameter Ratio}\approx 20.$ (5) $$

Using Besiroglu et al. (2024)’s [Chinchilla fitting code](https://github.com/epoch-research/analyzing-chinchilla/), we tested how these two Chinchilla headline results change depending on which of the three interpretations is used in the fits: (1) Reported Model Parameters, (2) Standard Formula Model Parameters, or (3) Best Fit Formula Model Parameters.

Perhaps surprisingly, we found that none of the five fit parameters $\hat{E},\hat{A},\hat{\alpha},\hat{B},\hat{\beta}$ differed significantly depending on which of our three notions of model parameters were used in fitting (Fig. [2](https://arxiv.org/html/2509.23963v1#S2.F2), top).
We similarly found the compute-optimal tokens-per-parameter ratio remains constant around $20$ tokens per parameter (Fig. [2](https://arxiv.org/html/2509.23963v1#S2.F2), bottom).
Arguably, the standard formula model parameters yield a flatter trend with increasing training compute: the slope for the standard formula model parameters is $-0.572$ for each 10x increase in compute, which decreased to $-1.049$ for the best fit formula model parameters and decreased further to $-1.248$ for the reported model parameters.
However, uncertainty makes drawing strong conclusions difficult.
These results demonstrate that key Chinchilla results are robust to whichever of our three notions of model parameters is used in the fitting process.

Figure: Figure 3: Evaluating the Robustness of Chinchilla via Four Model Parameter Perturbations. We study how robust key Chinchilla results are to structured perturbations of models’ parameters. Top Left: In the first perturbation, motivated by Sec. [2](https://arxiv.org/html/2509.23963v1#S2), we perturb model parameters with a multiplicative constant $c_{m}$. Top Right: In the second perturbation, we perturb model parameters with an additive constant $c_{a}$, that could perhaps arise due to embedding parameters being included/excluded. Bottom Left: In the third perturbation, we perturb model parameters with a systematic bias: either smaller models’ parameters are larger and larger models’ parameters are smaller, or smaller models’ parameters are smaller and larger models’ parameters are larger; the systematic bias is controlled by slope $s$. Bottom Right: In the fourth perturbation, we assume the relationship of the loss with model parameters is perhaps noisy, e.g., (Frankle & Carbin, 2019), with noise strength parameterized by $\sigma$.
Refer to caption: x4.png

## 3 Robustness of Chinchilla Headline Results Depends on Type of Perturbation to Model Parameters

Given that the key Chinchilla results did not meaningfully change even when model parameters differed by as much as 15.2%, we next asked:

> How distorted could the model parameters have been without meaningfully affecting Chinchilla’s headline results?

To answer this question, we intentionally perturbed the standard formula model parameters in four structured ways: multiplicative constant, additive constant, systematic bias and log normal noise. We then reran the fitting processes using the perturbed model parameters to see what effect each type of perturbation has on the estimated scaling law parameters and compute-optimal tokens-per-parameter.
We offer visual intuition for each of the four types of perturbations (Fig. [3](https://arxiv.org/html/2509.23963v1#S2.F3)).

### 3.1 Multiplicative Constant Perturbation Increases A ^ \hat{A} Exponentially

Motivated by Sec. [2](https://arxiv.org/html/2509.23963v1#S2), for our first perturbation, we assume model parameters are systematically under/overestimated by approximately the same percentage.
To model this, we multiplied all true model parameters $\{N_{i}\}_{i}$ by constant multiplier $c_{m}$ to produce perturbed model parameters $\{\tilde{N}_{i}\}_{i}$:

$$ $\tilde{N}_{i}\operatorname{\stackrel{{\scriptstyle\text{def}}}{{\;=\;}}}c_{m}\cdot N_{i}.$ (6) $$

We swept $c_{m}$ in logspace(-3, 3, num=11). For visual intuition, see Fig. [3](https://arxiv.org/html/2509.23963v1#S2.F3), top left.

Figure: Figure 4: Robustness of Fit Neural Scaling Law Parameters Under Four Types of Model Parameter Perturbations. Each row visualizes the effect of a different perturbation on the five fit parameters of the Chinchilla scaling law ($L(N,D)=E+A\cdot N^{-\alpha}+B\cdot D^{-\beta}$). Row 1: A multiplicative constant perturbation ($c$) increases the model parameter prefactor ($\hat{A}$) exponentially, while other fit parameters remain stable. Row 2: An additive constant perturbation ($c$) linearly increases the model parameter exponent ($\hat{\alpha}$) and exponentially increases its prefactor ($\hat{A}$), with only a gentle rise in the irreducible loss ($\hat{E}$). Row 3: A systematic bias perturbation ($s$) causes the model parameter exponent ($\hat{\alpha}$) to decay as a power law ($s^{-1}$) and the prefactor ($\hat{A}$) to decline sub-polynomially. Row 4: Adding log-normal noise ($\sigma$) primarily increases the uncertainty of all fit parameters, while weakly decreasing the model parameter exponent ($\hat{\alpha}$) logarithmically and the prefactor ($\hat{A}$) polynomially. Error bars are standard errors obtained by $4000$ bootstrapped samples.
Refer to caption: x8.png

As we derive in Appendix [C.2.1](https://arxiv.org/html/2509.23963v1#A3.SS2.SSS1) and show empirically in Fig. [4](https://arxiv.org/html/2509.23963v1#S3.F4)’s first row, the fitting process compensates for this multiplicative error primarily by adjusting the model size prefactor, $\tilde{\hat{A}}$, to approximately $\hat{A}c_{m}^{\alpha}$, while the scaling exponent, $\hat{\alpha}$, remains largely unchanged, i.e., $\tilde{\hat{\alpha}}\approx\hat{\alpha}$.
This makes sense for moderate $c_{m}$: if $\hat{A}$ is the best fit for the true model parameters, then replacing the true parameters with the perturbed parameters $N_{i}\rightarrow\tilde{N}_{i}$ and rescaling $\hat{A}\rightarrow\tilde{\hat{A}}=(c_{m})^{\alpha}\hat{A}$ produces approximately the same fit.
As a consequence, Fig. [5](https://arxiv.org/html/2509.23963v1#S3.F5) Top Left shows the compute-optimal tokens per parameter remains constant with pretraining compute, but the precise constant grows as a power of $c_{m}$ if true model parameters are underestimated $(c_{m}<1)$ and shrinks if true model parameters are overestimated $(c_{m}>1)$.
The only exceptions are for the two smallest multiplicative constants ($0.001$ and $0.004$), where uncertainty in $\hat{A}$ and $\hat{\alpha}$ produced NaNs.

### 3.2 Additive Constant Perturbation Increases α ^ \hat{\alpha} Linearly and A ^ \hat{A} Exponentially

In our second perturbation, we assume model parameters have an additive term. For example, embedding parameters may be included or excluded, a key detail in previous scaling law studies (Kaplan et al., 2020; Hoffmann et al., 2022) that is partially responsible for discrepancies between estimated scaling laws’ parameters (Pearce & Song, 2024; Porian et al., 2024)). To model this, we added constant $c_{a}$ to all model parameters:

$$ $\tilde{N}_{i}\operatorname{\stackrel{{\scriptstyle\text{def}}}{{\;=\;}}}c_{a}+N_{i}.$ (7) $$

We swept $c_{a}$ in -logspace(6.6, 7.6, num=5) $\cup\,\{0\}\,\cup$ logspace(6.6, 7.6, num=5). For visual intuition, see Fig. [3](https://arxiv.org/html/2509.23963v1#S2.F3) Top Right. For additional context, the smallest Chinchilla model has $42\times 10^{6}$ parameters.

Fig. [4](https://arxiv.org/html/2509.23963v1#S3.F4)’s second row shows the effects:
*(i)* the irreducible loss $\hat{E}$ rises only gently from $1.565$ to $1.897$ ($\approx 21\%$).
*(ii)* the model parameter prefactor $\hat{A}$ grows exponentially in $c_{a}$, increasing by $\sim\!2.5x$ from the most negative to the most positive constant
*(iii)* the model parameter exponent $\hat{\alpha}$ increases linearly with $c_{a}$ from $0.199$ to $0.481$
and *(iv)* both the data prefactor $\hat{B}$ and data exponent $\hat{\beta}$ fluctuate only within their bootstrap error bars and show no systematic trend.
As a consequence, Fig. [5](https://arxiv.org/html/2509.23963v1#S3.F5) Top Right shows the compute-optimal tokens per parameter becomes less constant with the training compute: a larger positive $c_{a}$ means larger target training horizons require more tokens per parameter, whereas a larger negative $c_{a}$ means larger target training horizons require fewer tokens per parameter.

In Appendix [C.2.2](https://arxiv.org/html/2509.23963v1#A3.SS2.SSS2), we analytically explain these trends: the most critical parameter in a power law is its exponent, which corresponds to its slope in log-log space.
However, for the perturbed function, the slope is now no longer constant and depends on $N$ as $N/(N+c_{a})$.
Thus, the fitting procedure must select a single exponent that best represents the varying slope over the range of data.
When $c_{a}>0$, the factor $N/(N+c_{a})<1$, and the fitting process must select an exponent $\tilde{\hat{\alpha}}>\hat{\alpha}$; and when $c_{a}<0$, the factor $N/(N+c_{a})>1$, and to compensate, the fitting process must select an exponent $\tilde{\hat{\alpha}}<\hat{\alpha}$.

For comparison, Porian et al. (2024) found that including the model’s head parameters increased the fit model parameter scaling exponent $\hat{\alpha}$ by $0.080\,(0.072\rightarrow 0.152)$, and Pearce & Song (2024) found that including embedding parameters increased $\hat{\alpha}$ by $0.231\,(0.135\rightarrow 0.366)$.
Although assuming an additive constant is a simplification of both analyses, all three results are quantitatively similar.

### 3.3 Systematic Bias Perturbation Decreases α ^ \hat{\alpha} Polynomially and A ^ \hat{A} Sub-Polynomially

In our third perturbation, we assume the presence of a systematic bias in reported models’ parameters: either the smaller models’ parameters are truly larger and the larger models’ parameters truly smaller, or vice versa.
To model this, we define the perturbed parameters as

$$ $\tilde{N}_{i}\operatorname{\stackrel{{\scriptstyle\text{def}}}{{\;=\;}}}\mu_{\mathrm{geo}}\cdot\left(N_{i}\,/\,\mu_{\mathrm{geo}}\right)^{s},$ (8) $$

where $\mu_{\mathrm{geo}}\operatorname{\stackrel{{\scriptstyle\text{def}}}{{\;=\;}}}(\prod_{i}N_{i})^{1/\text{Num. Models}}$ is the geometric mean of the model parameters and $s$ is the systematic bias parameter: $s<1$ shrinks large models and inflates small ones, whereas $s>1$ does the reverse.
We swept $s$ in logspace(-0.5, 0.5, 11). For visual intuition, see Fig. [3](https://arxiv.org/html/2509.23963v1#S2.F3) Bottom Left.

The third row of Fig. [4](https://arxiv.org/html/2509.23963v1#S3.F4) illustrates three main effects:
*(i)* The model parameter exponent $\hat{\alpha}$ decays according to the power-law relationship $\hat{\alpha}=10^{-0.46}\cdot s^{-1}$, which is a nearly perfect fit to the data ($R^{2}>0.999$, $p\approx$5.9\text{\times}{10}^{-90}$$).
*(ii)* The model parameter prefactor $\hat{A}$ declines sub-polynomially.
*(iii)* The irreducible loss, $\hat{E}$, and the data parameters, $\hat{B}$ and $\hat{\beta}$, show no systematic trend, with fluctuations remaining within their bootstrap error bars.
Similar to the Additive Constant perturbation, Fig. [5](https://arxiv.org/html/2509.23963v1#S3.F5) Bottom Left shows that the two trends of $\hat{A}$ and $\hat{\alpha}$ together make the compute-optimal tokens per parameter ratio less constant with the target training horizon: a larger systematic bias $s$ means larger target training horizons require fewer tokens per parameter, whereas a smaller systematic bias $s$ means larger target training horizons require more tokens per parameter.

Figure: Figure 5: Robustness of Compute-Optimal Tokens-Per-Parameter Under Four Types of Model Parameter Perturbations. Shaded regions represent 80% confidence intervals. Top Left: A multiplicative constant perturbation by $c$ shifts the compute-optimal ratio by $c^{\alpha}$ but keeps the trend flat with respect to training compute. Top Right: An additive constant perturbation by $c$ makes the compute-optimal ratio less constant across the target training compute horizon. A positive slope means more tokens are needed per parameter for larger compute budgets, while a negative slope means fewer are required. Bottom Left: A systematic bias also makes the ratio less constant. A larger bias $(s>1)$ leads to fewer optimal tokens per parameter for larger models, whereas a smaller bias $(s<1)$ requires more. Bottom Right: Adding log-normal noise to model parameters increases the uncertainty and the overall magnitude of the compute-optimal tokens per parameter ratio.
Refer to caption: x12.png

In Appendix [C.2.3](https://arxiv.org/html/2509.23963v1#A3.SS2.SSS3), we mathematically derive that under the systematic bias, the model parameter exponent is multiplied by $s^{-1}$ and the model prefactor is multiplied by $\mu_{\mathrm{geo}}^{\alpha(1-s)/s}$, making the exponent in the compute-optimal ratio $(\alpha/s-\beta)/(\alpha/s+\beta)$. Thus, if $s<1$, then the exponent on $C$ becomes positive and compute-optimal ratio of tokens-per-parameter increases with compute, whereas if $s>1$, then the exponent on C becomes negative and the compute-optimal ratio of tokens-per-parameter decreases with compute.

### 3.4 Log Normal Noise Perturbation Increases Uncertainty and Decreases α ^ \hat{\alpha} Logarithmically and A ^ \hat{A} Polynomially

In our fourth perturbation, we assume the “value” of model parameters is noisily measured, perhaps due to model initializations. To model this, we added log-normal noise to the number of parameters. Specifically, for each model’s parameter count $N_{i}$, we sampled a new parameter count as:

$$ $\tilde{N}_{i}\operatorname{\stackrel{{\scriptstyle\text{def}}}{{\;=\;}}}\exp(\delta_{i})\cdot N_{i},\quad\quad\delta_{i}\sim\mathcal{N}(0,\sigma^{2}).$ (9) $$

We swept $\sigma$ from $1\text{\times}{10}^{-2}$ to $1\text{\times}{10}^{2}$. For visual intuition, see Fig. [3](https://arxiv.org/html/2509.23963v1#S2.F3) Bottom Right.

The fourth row of Fig. [4](https://arxiv.org/html/2509.23963v1#S3.F4) illustrates three main effects:
*(i)* Nearly all fit parameters have significantly larger confidence intervals, especially as the noise standard deviation $\sigma$ increases; for the highest value of $3.162$, $\hat{A}$ and $\hat{\alpha}$ are nearly unidentifiable.
*(ii)* To the extent that trends can be identified, the irreducible error $\hat{E}$ trends down weakly, while the model parameters prefactor $\hat{A}$ falls roughly polynomially and the model parameters exponent falls roughly logarithmically with the noise standard deviation $\sigma$.
Fig. [5](https://arxiv.org/html/2509.23963v1#S3.F5) Bottom Right demonstrates the consequences of the noise: fits with too high of noise create NaNs, while noise drives up the compute-optimal tokens per parameter and also increases the width of the 80% confidence intervals by $\sim 1$ order of magnitude, although the inferred values are roughly constant with target training compute.

## 4 Related Work

Due to space constraints, we defer most Related Work to Appendix [D](https://arxiv.org/html/2509.23963v1#A4) and focus here on prior research most relevant to our contribution.
The precise details of Hoffmann et al. (2022) have recently come under scrutiny, leading to a number of important replication and re-evaluation studies.
For instance, Chinchilla used three different approaches, two of which agreed with each other, but the third did not; Besiroglu et al. (2024) conducted a detailed investigation of this third analysis and found that it could be made consistent with the first two analyses by fixing optimizer issues and not rounding reported fit parameters.
In a similar vein, Porian et al. (2024) and Pearce & Song (2024) sought to resolve a discrepancy between Hoffmann et al. (2022) and Kaplan et al. (2020) on how to scale data and parameters to produce the best performing model; Porian et al. (2024) found that the discrepancy could be resolved by three differences (last layer computational cost, warmup duration, and scale-dependent optimizer tuning) while Pearce & Song (2024) found that much of the discrepancy could be attributed to Kaplan et al. (2020) counting only non-embedding parameters.

Like Besiroglu et al. (2024) and Porian et al. (2024), our work scrutinizes the seminal work of Chinchilla.
However, our analyses focuses specifically on how robust the original Chinchilla methodology and results are to different perturbations.
Our contribution concludes with a direct confirmation of the original findings, providing evidence that Chinchilla’s compute-optimal guidance is robust.

## 5 Discussion

This work began with a perhaps surprising result: three different interpretations of Chinchilla’s model parameters are possible, with discrepancies as high as 15.2%, but all three support (or strengthen) key Chinchilla results.
Neither the estimated scaling law parameters nor the widely adopted “20-to-1” compute-optimal tokens-to-parameter ratio changed meaningfully.
Indeed, our refitting using the standard formula model parameters suggests an even more stable relationship, with the token-to-parameter ratio varying even less across different pretraining compute budgets.

To understand this robustness more deeply, we systematically investigated how various hypothetical perturbations would affect key Chinchilla results.
We perturbed the parameter counts in four structured ways and re-ran the fitting analysis for each. This stress test revealed the specific ways in which different types of errors impact the scaling law parameters. A simple multiplicative error, for example, exponentially shifts the constant in the optimal tokens-per-parameter ratio, while an additive error or a systematic bias can more dramatically alter its trend with respect to the target training compute budget.

Ultimately, our findings serve as both a critical re-examination and a powerful confirmation of the original Chinchilla results.
Our subsequent analyses should give practitioners even greater confidence in Chinchilla’s compute-optimal prescription.
Its guidance withstands not only the specific interpretation used, but also a range of other potential perturbations, reinforcing its value as a durable and practical blueprint for the field.

##### Future Directions

One obvious next step is to evaluate the robustness of more recent scaling results with additional considerations such as inference constraints (Sardana et al., 2024), data constraints (Muennighoff et al., 2023) and overtraining (Gadre et al., 2024).

## References

- Abnar et al. (2025)
Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin Mohamed Elnouby Ali, Josh Susskind, and Vimal Thilak.
Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models.
*arXiv preprint arXiv:2501.12370*, 2025.
- Agarwal et al. (2024)
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie C.Y. Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle.
Many-shot in-context learning.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=AB6XpMzvqH](https://openreview.net/forum?id=AB6XpMzvqH).
- Aghajanyan et al. (2023)
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer.
Scaling laws for generative mixed-modal language models.
In *International Conference on Machine Learning*, pp. 265–279. PMLR, 2023.
- Alabdulmohsin et al. (2022)
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai.
Revisiting neural scaling laws in language and vision.
*Advances in Neural Information Processing Systems*, 35:22300–22312, 2022.
- Anil et al. (2024)
Cem Anil, Esin DURMUS, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, James Sully, Alex Tamkin, Tamera Lanham, Karina Nguyen, Tomasz Korbak, Jared Kaplan, Deep Ganguli, Samuel R. Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud.
Many-shot jailbreaking.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=cw5mgd71jW](https://openreview.net/forum?id=cw5mgd71jW).
- Arora et al. (2024)
Aryaman Arora, Dan Jurafsky, Christopher Potts, and Noah D. Goodman.
Bayesian scaling laws for in-context learning, 2024.
URL [https://arxiv.org/abs/2410.16531](https://arxiv.org/abs/2410.16531).
- Atanasov et al. (2024)
Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan.
Scaling and renormalization in high-dimensional regression.
*arXiv preprint arXiv:2405.00592*, 2024.
- Bachmann et al. (2023)
Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann.
Scaling mlps: A tale of inductive bias, 2023.
URL [https://arxiv.org/abs/2306.13575](https://arxiv.org/abs/2306.13575).
- Bahri et al. (2024)
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma.
Explaining neural scaling laws.
*Proceedings of the National Academy of Sciences*, 121(27):e2311878121, 2024.
- Barkai et al. (1993)
N Barkai, Hyunjune Sebastian Seung, and Haim Sompolinsky.
Scaling laws in learning of classification tasks.
*Physical review letters*, 70(20):3167, 1993.
- Barnett (2024)
Matthew Barnett.
An empirical study of scaling laws for transfer, 2024.
URL [https://arxiv.org/abs/2408.16947](https://arxiv.org/abs/2408.16947).
- Besiroglu et al. (2024)
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You.
Chinchilla scaling: A replication attempt, 2024.
URL [https://arxiv.org/abs/2404.10102](https://arxiv.org/abs/2404.10102).
- Bian et al. (2025)
Song Bian, Minghao Yan, and Shivaram Venkataraman.
Scaling inference-efficient language models.
In *International Conference on Machine Learning*, 2025.
- Bordelon et al. (2024a)
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan.
A dynamical model of neural scaling laws.
*arXiv preprint arXiv:2402.01092*, 2024a.
- Bordelon et al. (2024b)
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan.
How feature learning can improve neural scaling laws.
*arXiv preprint arXiv:2409.17858*, 2024b.
- Bousquet et al. (2020)
Olivier Bousquet, Steve Hanneke, Shay Moran, Ramon van Handel, and Amir Yehudayoff.
A theory of universal learning, 2020.
URL [https://arxiv.org/abs/2011.04483](https://arxiv.org/abs/2011.04483).
- Brill (2024)
Ari Brill.
Neural scaling laws rooted in the data distribution.
*arXiv preprint arXiv:2412.07942*, 2024.
- Brown et al. (2024)
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini.
Large language monkeys: Scaling inference compute with repeated sampling, 2024.
URL [https://arxiv.org/abs/2407.21787](https://arxiv.org/abs/2407.21787).
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901, 2020.
- Busbridge et al. (2025)
Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russell Webb.
Distillation scaling laws.
In *Forty-second International Conference on Machine Learning*, 2025.
- Chen et al. (2024)
Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou.
A simple and provable scaling law for the test-time compute of large language models, 2024.
URL [https://arxiv.org/abs/2411.19477](https://arxiv.org/abs/2411.19477).
- Chen et al. (2025)
Zhengyu Chen, Siqi Wang, Teng Xiao, Yudong Wang, Shiqi Chen, Xunliang Cai, Junxian He, and Jingang Wang.
Revisiting scaling laws for language models: The role of data quality and training strategies.
In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 23881–23899, 2025.
- Cherti et al. (2023)
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.
Reproducible scaling laws for contrastive language-image learning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 2818–2829, 2023.
- Clark et al. (2022)
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al.
Unified scaling laws for routed language models.
In *International conference on machine learning*, pp. 4057–4086. PMLR, 2022.
- DeepMind et al. (2025)
Google DeepMind, Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kawintiranon, Tania Bedrax-Weiss, Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovic, Ilaï Deutel, Nam Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Waghmare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj, Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, Bo Wang, Mantas Pajarskas, Joe Heyward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel,
Mingyao Yang, Jiho Park, Koji Kojima, Parsa Mahmoudieh, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov, Sudeshna Roy, Dimitris Paparas, Hadar Shemtov, Bo-Juen Chen, Robin Strudel, David Reitter, Aurko Roy, Andrey Vlasov, Changwan Ryu, Chas Leichner, Haichuan Yang, Zelda Mariet, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang, Minmin Chen, Praynaa Rawlani, Christy Koh, JD Co-Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, Mu Cai, Mohammed Badawi, Corey Fry, Ale Hartman, Daniel Zheng, Eric Jia, James Keeling, Annie Louis, Ying Chen, Efren Robles, Wei-Chih Hung, Howard Zhou, Nikita Saxena, Sonam Goenka, Olivia Ma, Zach Fisher, Mor Hazan Taege, Emily Graves, David Steiner, Yujia Li, Sarah Nguyen, Rahul Sukthankar, Joe Stanton, Ali Eslami, Gloria Shen, Berkin Akin, Alexey Guseynov, Yiqian Zhou, Jean-Baptiste Alayrac, Armand Joulin, Efrat Farkash, Ashish Thapliyal, Stephen Roller, Noam Shazeer, Todor Davchev, Terry Koo, Hannah Forbes-Pollard, Kartik Audhkhasi, Greg Farquhar,
Adi Mayrav Gilady, Maggie Song, John Aslanides, Piermaria Mendolicchio, Alicia Parrish, John Blitzer, Pramod Gupta, Xiaoen Ju, Xiaochen Yang, Puranjay Datta, Andrea Tacchetti, Sanket Vaibhav Mehta, Gregory Dibb, Shubham Gupta, Federico Piccinini, Raia Hadsell, Sujee Rajayogam, Jiepu Jiang, Patrick Griffin, Patrik Sundberg, Jamie Hayes, Alexey Frolov, Tian Xie, Adam Zhang, Kingshuk Dasgupta, Uday Kalra, Lior Shani, Klaus Macherey, Tzu-Kuo Huang, Liam MacDermed, Karthik Duddu, Paulo Zacchello, Zi Yang, Jessica Lo, Kai Hui, Matej Kastelic, Derek Gasaway, Qijun Tan, Summer Yue, Pablo Barrio, John Wieting, Weel Yang, Andrew Nystrom, Solomon Demmessie, Anselm Levskaya, Fabio Viola, Chetan Tekur, Greg Billock, George Necula, Mandar Joshi, Rylan Schaeffer, Swachhand Lokhande, Christina Sorokin, Pradeep Shenoy, Mia Chen, Mark Collier, Hongji Li, Taylor Bos, Nevan Wichers, Sun Jae Lee, Angéline Pouget, Santhosh Thangaraj, Kyriakos Axiotis, Phil Crone, Rachel Sterneck, Nikolai Chinaev, Victoria Krakovna, Oleksandr
Ferludin, Ian Gemp, Stephanie Winkler, Dan Goldberg, Ivan Korotkov, Kefan Xiao, Malika Mehrotra, Sandeep Mariserla, Vihari Piratla, Terry Thurk, Khiem Pham, Hongxu Ma, Alexandre Senges, Ravi Kumar, Clemens Meyer, Ellie Talius, Nuo Wang Pierse, Ballie Sandhu, Horia Toma, Kuo Lin, Swaroop Nath, Tom Stone, Dorsa Sadigh, Nikita Gupta, Arthur Guez, Avi Singh, Matt Thomas, Tom Duerig, Yuan Gong, Richard Tanburn, Lydia Lihui Zhang, Phuong Dao, Mohamed Hammad, Sirui Xie, Shruti Rijhwani, Ben Murdoch, Duhyeon Kim, Will Thompson, Heng-Tze Cheng, Daniel Sohn, Pablo Sprechmann, Qiantong Xu, Srinivas Tadepalli, Peter Young, Ye Zhang, Hansa Srinivasan, Miranda Aperghis, Aditya Ayyar, Hen Fitoussi, Ryan Burnell, David Madras, Mike Dusenberry, Xi Xiong, Tayo Oguntebi, Ben Albrecht, Jörg Bornschein, Jovana Mitrović, Mason Dimarco, Bhargav Kanagal Shamanna, Premal Shah, Eren Sezener, Shyam Upadhyay, Dave Lacey, Craig Schiff, Sebastien Baur, Sanjay Ganapathy, Eva Schnider, Mateo Wirth, Connor Schenck, Andrey Simanovsky,
Yi-Xuan Tan, Philipp Fränken, Dennis Duan, Bharath Mankalale, Nikhil Dhawan, Kevin Sequeira, Zichuan Wei, Shivanker Goel, Caglar Unlu, Yukun Zhu, Haitian Sun, Ananth Balashankar, Kurt Shuster, Megh Umekar, Mahmoud Alnahlawi, Aäron van den Oord, Kelly Chen, Yuexiang Zhai, Zihang Dai, Kuang-Huei Lee, Eric Doi, Lukas Zilka, Rohith Vallu, Disha Shrivastava, Jason Lee, Hisham Husain, Honglei Zhuang, Vincent Cohen-Addad, Jarred Barber, James Atwood, Adam Sadovsky, Quentin Wellens, Steven Hand, Arunkumar Rajendran, Aybuke Turker, CJ Carey, Yuanzhong Xu, Hagen Soltau, Zefei Li, Xinying Song, Conglong Li, Iurii Kemaev, Sasha Brown, Andrea Burns, Viorica Patraucean, Piotr Stanczyk, Renga Aravamudhan, Mathieu Blondel, Hila Noga, Lorenzo Blanco, Will Song, Michael Isard, Mandar Sharma, Reid Hayes, Dalia El Badawy, Avery Lamp, Itay Laish, Olga Kozlova, Kelvin Chan, Sahil Singla, Srinivas Sunkara, Mayank Upadhyay, Chang Liu, Aijun Bai, Jarek Wilkiewicz, Martin Zlocha, Jeremiah Liu, Zhuowan Li, Haiguang Li, Omer Barak,
Ganna Raboshchuk, Jiho Choi, Fangyu Liu, Erik Jue, Mohit Sharma, Andreea Marzoca, Robert Busa-Fekete, Anna Korsun, Andre Elisseeff, Zhe Shen, Sara Mc Carthy, Kay Lamerigts, Anahita Hosseini, Hanzhao Lin, Charlie Chen, Fan Yang, Kushal Chauhan, Mark Omernick, Dawei Jia, Karina Zainullina, Demis Hassabis, Danny Vainstein, Ehsan Amid, Xiang Zhou, Ronny Votel, Eszter Vértes, Xinjian Li, Zongwei Zhou, Angeliki Lazaridou, Brendan McMahan, Arjun Narayanan, Hubert Soyer, Sujoy Basu, Kayi Lee, Bryan Perozzi, Qin Cao, Leonard Berrada, Rahul Arya, Ke Chen, Katrina, Xu, Matthias Lochbrunner, Alex Hofer, Sahand Sharifzadeh, Renjie Wu, Sally Goldman, Pranjal Awasthi, Xuezhi Wang, Yan Wu, Claire Sha, Biao Zhang, Maciej Mikuła, Filippo Graziano, Siobhan Mcloughlin, Irene Giannoumis, Youhei Namiki, Chase Malik, Carey Radebaugh, Jamie Hall, Ramiro Leal-Cavazos, Jianmin Chen, Vikas Sindhwani, David Kao, David Greene, Jordan Griffith, Chris Welty, Ceslee Montgomery, Toshihiro Yoshino, Liangzhe Yuan, Noah Goodman,
Assaf Hurwitz Michaely, Kevin Lee, KP Sawhney, Wei Chen, Zheng Zheng, Megan Shum, Nikolay Savinov, Etienne Pot, Alex Pak, Morteza Zadimoghaddam, Sijal Bhatnagar, Yoad Lewenberg, Blair Kutzman, Ji Liu, Lesley Katzen, Jeremy Selier, Josip Djolonga, Dmitry Lepikhin, Kelvin Xu, Jacky Liang, Jiewen Tan, Benoit Schillings, Muge Ersoy, Pete Blois, Bernd Bandemer, Abhimanyu Singh, Sergei Lebedev, Pankaj Joshi, Adam R. Brown, Evan Palmer, Shreya Pathak, Komal Jalan, Fedir Zubach, Shuba Lall, Randall Parker, Alok Gunjan, Sergey Rogulenko, Sumit Sanghai, Zhaoqi Leng, Zoltan Egyed, Shixin Li, Maria Ivanova, Kostas Andriopoulos, Jin Xie, Elan Rosenfeld, Auriel Wright, Ankur Sharma, Xinyang Geng, Yicheng Wang, Sam Kwei, Renke Pan, Yujing Zhang, Gabby Wang, Xi Liu, Chak Yeung, Elizabeth Cole, Aviv Rosenberg, Zhen Yang, Phil Chen, George Polovets, Pranav Nair, Rohun Saxena, Josh Smith, Shuo yiin Chang, Aroma Mahendru, Svetlana Grant, Anand Iyer, Irene Cai, Jed McGiffin, Jiaming Shen, Alanna Walton, Antonious Girgis, Oliver
Woodman, Rosemary Ke, Mike Kwong, Louis Rouillard, Jinmeng Rao, Zhihao Li, Yuntao Xu, Flavien Prost, Chi Zou, Ziwei Ji, Alberto Magni, Tyler Liechty, Dan A. Calian, Deepak Ramachandran, Igor Krivokon, Hui Huang, Terry Chen, Anja Hauth, Anastasija Ilić, Weijuan Xi, Hyeontaek Lim, Vlad-Doru Ion, Pooya Moradi, Metin Toksoz-Exley, Kalesha Bullard, Miltos Allamanis, Xiaomeng Yang, Sophie Wang, Zhi Hong, Anita Gergely, Cheng Li, Bhavishya Mittal, Vitaly Kovalev, Victor Ungureanu, Jane Labanowski, Jan Wassenberg, Nicolas Lacasse, Geoffrey Cideron, Petar Dević, Annie Marsden, Lynn Nguyen, Michael Fink, Yin Zhong, Tatsuya Kiyono, Desi Ivanov, Sally Ma, Max Bain, Kiran Yalasangi, Jennifer She, Anastasia Petrushkina, Mayank Lunayach, Carla Bromberg, Sarah Hodkinson, Vilobh Meshram, Daniel Vlasic, Austin Kyker, Steve Xu, Jeff Stanway, Zuguang Yang, Kai Zhao, Matthew Tung, Seth Odoom, Yasuhisa Fujii, Justin Gilmer, Eunyoung Kim, Felix Halim, Quoc Le, Bernd Bohnet, Seliem El-Sayed, Behnam Neyshabur, Malcolm Reynolds,
Dean Reich, Yang Xu, Erica Moreira, Anuj Sharma, Zeyu Liu, Mohammad Javad Hosseini, Naina Raisinghani, Yi Su, Ni Lao, Daniel Formoso, Marco Gelmi, Almog Gueta, Tapomay Dey, Elena Gribovskaya, Domagoj Ćevid, Sidharth Mudgal, Garrett Bingham, Jianling Wang, Anurag Kumar, Alex Cullum, Feng Han, Konstantinos Bousmalis, Diego Cedillo, Grace Chu, Vladimir Magay, Paul Michel, Ester Hlavnova, Daniele Calandriello, Setareh Ariafar, Kaisheng Yao, Vikash Sehwag, Arpi Vezer, Agustin Dal Lago, Zhenkai Zhu, Paul Kishan Rubenstein, Allen Porter, Anirudh Baddepudi, Oriana Riva, Mihai Dorin Istin, Chih-Kuan Yeh, Zhi Li, Andrew Howard, Nilpa Jha, Jeremy Chen, Raoul de Liedekerke, Zafarali Ahmed, Mikel Rodriguez, Tanuj Bhatia, Bangju Wang, Ali Elqursh, David Klinghoffer, Peter Chen, Pushmeet Kohli, Te I, Weiyang Zhang, Zack Nado, Jilin Chen, Maxwell Chen, George Zhang, Aayush Singh, Adam Hillier, Federico Lebron, Yiqing Tao, Ting Liu, Gabriel Dulac-Arnold, Jingwei Zhang, Shashi Narayan, Buhuang Liu, Orhan Firat, Abhishek
Bhowmick, Bingyuan Liu, Hao Zhang, Zizhao Zhang, Georges Rotival, Nathan Howard, Anu Sinha, Alexander Grushetsky, Benjamin Beyret, Keerthana Gopalakrishnan, James Zhao, Kyle He, Szabolcs Payrits, Zaid Nabulsi, Zhaoyi Zhang, Weijie Chen, Edward Lee, Nova Fallen, Sreenivas Gollapudi, Aurick Zhou, Filip Pavetić, Thomas Köppe, Shiyu Huang, Rama Pasumarthi, Nick Fernando, Felix Fischer, Daria Ćurko, Yang Gao, James Svensson, Austin Stone, Haroon Qureshi, Abhishek Sinha, Apoorv Kulshreshtha, Martin Matysiak, Jieming Mao, Carl Saroufim, Aleksandra Faust, Qingnan Duan, Gil Fidel, Kaan Katircioglu, Raphaël Lopez Kaufman, Dhruv Shah, Weize Kong, Abhishek Bapna, Gellért Weisz, Emma Dunleavy, Praneet Dutta, Tianqi Liu, Rahma Chaabouni, Carolina Parada, Marcus Wu, Alexandra Belias, Alessandro Bissacco, Stanislav Fort, Li Xiao, Fantine Huot, Chris Knutsen, Yochai Blau, Gang Li, Jennifer Prendki, Juliette Love, Yinlam Chow, Pichi Charoenpanit, Hidetoshi Shimokawa, Vincent Coriou, Karol Gregor, Tomas Izo, Arjun Akula,
Mario Pinto, Chris Hahn, Dominik Paulus, Jiaxian Guo, Neha Sharma, Cho-Jui Hsieh, Adaeze Chukwuka, Kazuma Hashimoto, Nathalie Rauschmayr, Ling Wu, Christof Angermueller, Yulong Wang, Sebastian Gerlach, Michael Pliskin, Daniil Mirylenka, Min Ma, Lexi Baugher, Bryan Gale, Shaan Bijwadia, Nemanja Rakićević, David Wood, Jane Park, Chung-Ching Chang, Babi Seal, Chris Tar, Kacper Krasowiak, Yiwen Song, Georgi Stephanov, Gary Wang, Marcello Maggioni, Stein Xudong Lin, Felix Wu, Shachi Paul, Zixuan Jiang, Shubham Agrawal, Bilal Piot, Alex Feng, Cheolmin Kim, Tulsee Doshi, Jonathan Lai, Chuqiao, Xu, Sharad Vikram, Ciprian Chelba, Sebastian Krause, Vincent Zhuang, Jack Rae, Timo Denk, Adrian Collister, Lotte Weerts, Xianghong Luo, Yifeng Lu, Håvard Garnes, Nitish Gupta, Terry Spitz, Avinatan Hassidim, Lihao Liang, Izhak Shafran, Peter Humphreys, Kenny Vassigh, Phil Wallis, Virat Shejwalkar, Nicolas Perez-Nieves, Rachel Hornung, Melissa Tan, Beka Westberg, Andy Ly, Richard Zhang, Brian Farris, Jongbin Park, Alec
Kosik, Zeynep Cankara, Andrii Maksai, Yunhan Xu, Albin Cassirer, Sergi Caelles, Abbas Abdolmaleki, Mencher Chiang, Alex Fabrikant, Shravya Shetty, Luheng He, Mai Giménez, Hadi Hashemi, Sheena Panthaplackel, Yana Kulizhskaya, Salil Deshmukh, Daniele Pighin, Robin Alazard, Disha Jindal, Seb Noury, Pradeep Kumar S, Siyang Qin, Xerxes Dotiwalla, Stephen Spencer, Mohammad Babaeizadeh, Blake JianHang Chen, Vaibhav Mehta, Jennie Lees, Andrew Leach, Penporn Koanantakool, Ilia Akolzin, Ramona Comanescu, Junwhan Ahn, Alexey Svyatkovskiy, Basil Mustafa, David D’Ambrosio, Shiva Mohan Reddy Garlapati, Pascal Lamblin, Alekh Agarwal, Shuang Song, Pier Giuseppe Sessa, Pauline Coquinot, John Maggs, Hussain Masoom, Divya Pitta, Yaqing Wang, Patrick Morris-Suzuki, Billy Porter, Johnson Jia, Jeffrey Dudek, Raghavender R, Cosmin Paduraru, Alan Ansell, Tolga Bolukbasi, Tony Lu, Ramya Ganeshan, Zi Wang, Henry Griffiths, Rodrigo Benenson, Yifan He, James Swirhun, George Papamakarios, Aditya Chawla, Kuntal Sengupta, Yan Wang,
Vedrana Milutinovic, Igor Mordatch, Zhipeng Jia, Jamie Smith, Will Ng, Shitij Nigam, Matt Young, Eugen Vušak, Blake Hechtman, Sheela Goenka, Avital Zipori, Kareem Ayoub, Ashok Popat, Trilok Acharya, Luo Yu, Dawn Bloxwich, Hugo Song, Paul Roit, Haiqiong Li, Aviel Boag, Nigamaa Nayakanti, Bilva Chandra, Tianli Ding, Aahil Mehta, Cath Hope, Jiageng Zhang, Idan Heimlich Shtacher, Kartikeya Badola, Ryo Nakashima, Andrei Sozanschi, Iulia Comşa, Ante Žužul, Emily Caveness, Julian Odell, Matthew Watson, Dario de Cesare, Phillip Lippe, Derek Lockhart, Siddharth Verma, Huizhong Chen, Sean Sun, Lin Zhuo, Aditya Shah, Prakhar Gupta, Alex Muzio, Ning Niu, Amir Zait, Abhinav Singh, Meenu Gaba, Fan Ye, Prajit Ramachandran, Mohammad Saleh, Raluca Ada Popa, Ayush Dubey, Frederick Liu, Sara Javanmardi, Mark Epstein, Ross Hemsley, Richard Green, Nishant Ranka, Eden Cohen, Chuyuan Kelly Fu, Sanjay Ghemawat, Jed Borovik, James Martens, Anthony Chen, Pranav Shyam, André Susano Pinto, Ming-Hsuan Yang, Alexandru Ţifrea,
David Du, Boqing Gong, Ayushi Agarwal, Seungyeon Kim, Christian Frank, Saloni Shah, Xiaodan Song, Zhiwei Deng, Ales Mikhalap, Kleopatra Chatziprimou, Timothy Chung, Toni Creswell, Susan Zhang, Yennie Jun, Carl Lebsack, Will Truong, Slavica Andačić, Itay Yona, Marco Fornoni, Rong Rong, Serge Toropov, Afzal Shama Soudagar, Andrew Audibert, Salah Zaiem, Zaheer Abbas, Andrei Rusu, Sahitya Potluri, Shitao Weng, Anastasios Kementsietsidis, Anton Tsitsulin, Daiyi Peng, Natalie Ha, Sanil Jain, Tejasi Latkar, Simeon Ivanov, Cory McLean, Anirudh GP, Rajesh Venkataraman, Canoee Liu, Dilip Krishnan, Joel D’sa, Roey Yogev, Paul Collins, Benjamin Lee, Lewis Ho, Carl Doersch, Gal Yona, Shawn Gao, Felipe Tiengo Ferreira, Adnan Ozturel, Hannah Muckenhirn, Ce Zheng, Gargi Balasubramaniam, Mudit Bansal, George van den Driessche, Sivan Eiger, Salem Haykal, Vedant Misra, Abhimanyu Goyal, Danilo Martins, Gary Leung, Jonas Valfridsson, Four Flynn, Will Bishop, Chenxi Pang, Yoni Halpern, Honglin Yu, Lawrence Moore, Yuvein, Zhu,
Sridhar Thiagarajan, Yoel Drori, Zhisheng Xiao, Lucio Dery, Rolf Jagerman, Jing Lu, Eric Ge, Vaibhav Aggarwal, Arjun Khare, Vinh Tran, Oded Elyada, Ferran Alet, James Rubin, Ian Chou, David Tian, Libin Bai, Lawrence Chan, Lukasz Lew, Karolis Misiunas, Taylan Bilal, Aniket Ray, Sindhu Raghuram, Alex Castro-Ros, Viral Carpenter, CJ Zheng, Michael Kilgore, Josef Broder, Emily Xue, Praveen Kallakuri, Dheeru Dua, Nancy Yuen, Steve Chien, John Schultz, Saurabh Agrawal, Reut Tsarfaty, Jingcao Hu, Ajay Kannan, Dror Marcus, Nisarg Kothari, Baochen Sun, Ben Horn, Matko Bošnjak, Ferjad Naeem, Dean Hirsch, Lewis Chiang, Boya Fang, Jie Han, Qifei Wang, Ben Hora, Antoine He, Mario Lučić, Beer Changpinyo, Anshuman Tripathi, John Youssef, Chester Kwak, Philippe Schlattner, Cat Graves, Rémi Leblond, Wenjun Zeng, Anders Andreassen, Gabriel Rasskin, Yue Song, Eddie Cao, Junhyuk Oh, Matt Hoffman, Wojtek Skut, Yichi Zhang, Jon Stritar, Xingyu Cai, Saarthak Khanna, Kathie Wang, Shriya Sharma, Christian Reisswig, Younghoon
Jun, Aman Prasad, Tatiana Sholokhova, Preeti Singh, Adi Gerzi Rosenthal, Anian Ruoss, Françoise Beaufays, Sean Kirmani, Dongkai Chen, Johan Schalkwyk, Jonathan Herzig, Been Kim, Josh Jacob, Damien Vincent, Adrian N Reyes, Ivana Balazevic, Léonard Hussenot, Jon Schneider, Parker Barnes, Luis Castro, Spandana Raj Babbula, Simon Green, Serkan Cabi, Nico Duduta, Danny Driess, Rich Galt, Noam Velan, Junjie Wang, Hongyang Jiao, Matthew Mauger, Du Phan, Miteyan Patel, Vlado Galić, Jerry Chang, Eyal Marcus, Matt Harvey, Julian Salazar, Elahe Dabir, Suraj Satishkumar Sheth, Amol Mandhane, Hanie Sedghi, Jeremiah Willcock, Amir Zandieh, Shruthi Prabhakara, Aida Amini, Antoine Miech, Victor Stone, Massimo Nicosia, Paul Niemczyk, Ying Xiao, Lucy Kim, Sławek Kwasiborski, Vikas Verma, Ada Maksutaj Oflazer, Christoph Hirnschall, Peter Sung, Lu Liu, Richard Everett, Michiel Bakker, Ágoston Weisz, Yufei Wang, Vivek Sampathkumar, Uri Shaham, Bibo Xu, Yasemin Altun, Mingqiu Wang, Takaaki Saeki, Guanjie Chen, Emanuel
Taropa, Shanthal Vasanth, Sophia Austin, Lu Huang, Goran Petrovic, Qingyun Dou, Daniel Golovin, Grigory Rozhdestvenskiy, Allie Culp, Will Wu, Motoki Sano, Divya Jain, Julia Proskurnia, Sébastien Cevey, Alejandro Cruzado Ruiz, Piyush Patil, Mahdi Mirzazadeh, Eric Ni, Javier Snaider, Lijie Fan, Alexandre Fréchette, AJ Pierigiovanni, Shariq Iqbal, Kenton Lee, Claudio Fantacci, Jinwei Xing, Lisa Wang, Alex Irpan, David Raposo, Yi Luan, Zhuoyuan Chen, Harish Ganapathy, Kevin Hui, Jiazhong Nie, Isabelle Guyon, Heming Ge, Roopali Vij, Hui Zheng, Dayeong Lee, Alfonso Castaño, Khuslen Baatarsukh, Gabriel Ibagon, Alexandra Chronopoulou, Nicholas FitzGerald, Shashank Viswanadha, Safeen Huda, Rivka Moroshko, Georgi Stoyanov, Prateek Kolhar, Alain Vaucher, Ishaan Watts, Adhi Kuncoro, Henryk Michalewski, Satish Kambala, Bat-Orgil Batsaikhan, Alek Andreev, Irina Jurenka, Maigo Le, Qihang Chen, Wael Al Jishi, Sarah Chakera, Zhe Chen, Aditya Kini, Vikas Yadav, Aditya Siddhant, Ilia Labzovsky, Balaji Lakshminarayanan,
Carrie Grimes Bostock, Pankil Botadra, Ankesh Anand, Colton Bishop, Sam Conway-Rahman, Mohit Agarwal, Yani Donchev, Achintya Singhal, Félix de Chaumont Quitry, Natalia Ponomareva, Nishant Agrawal, Bin Ni, Kalpesh Krishna, Masha Samsikova, John Karro, Yilun Du, Tamara von Glehn, Caden Lu, Christopher A. Choquette-Choo, Zhen Qin, Tingnan Zhang, Sicheng Li, Divya Tyam, Swaroop Mishra, Wing Lowe, Colin Ji, Weiyi Wang, Manaal Faruqui, Ambrose Slone, Valentin Dalibard, Arunachalam Narayanaswamy, John Lambert, Pierre-Antoine Manzagol, Dan Karliner, Andrew Bolt, Ivan Lobov, Aditya Kusupati, Chang Ye, Xuan Yang, Heiga Zen, Nelson George, Mukul Bhutani, Olivier Lacombe, Robert Riachi, Gagan Bansal, Rachel Soh, Yue Gao, Yang Yu, Adams Yu, Emily Nottage, Tania Rojas-Esponda, James Noraky, Manish Gupta, Ragha Kotikalapudi, Jichuan Chang, Sanja Deur, Dan Graur, Alex Mossin, Erin Farnese, Ricardo Figueira, Alexandre Moufarek, Austin Huang, Patrik Zochbauer, Ben Ingram, Tongzhou Chen, Zelin Wu, Adrià Puigdomènech,
Leland Rechis, Da Yu, Sri Gayatri Sundara Padmanabhan, Rui Zhu, Chu ling Ko, Andrea Banino, Samira Daruki, Aarush Selvan, Dhruva Bhaswar, Daniel Hernandez Diaz, Chen Su, Salvatore Scellato, Jennifer Brennan, Woohyun Han, Grace Chung, Priyanka Agrawal, Urvashi Khandelwal, Khe Chai Sim, Morgane Lustman, Sam Ritter, Kelvin Guu, Jiawei Xia, Prateek Jain, Emma Wang, Tyrone Hill, Mirko Rossini, Marija Kostelac, Tautvydas Misiunas, Amit Sabne, Kyuyeun Kim, Ahmet Iscen, Congchao Wang, José Leal, Ashwin Sreevatsa, Utku Evci, Manfred Warmuth, Saket Joshi, Daniel Suo, James Lottes, Garrett Honke, Brendan Jou, Stefani Karp, Jieru Hu, Himanshu Sahni, Adrien Ali Taïga, William Kong, Samrat Ghosh, Renshen Wang, Jay Pavagadhi, Natalie Axelsson, Nikolai Grigorev, Patrick Siegler, Rebecca Lin, Guohui Wang, Emilio Parisotto, Sharath Maddineni, Krishan Subudhi, Eyal Ben-David, Elena Pochernina, Orgad Keller, Thi Avrahami, Zhe Yuan, Pulkit Mehta, Jialu Liu, Sherry Yang, Wendy Kan, Katherine Lee, Tom Funkhouser, Derek Cheng,
Hongzhi Shi, Archit Sharma, Joe Kelley, Matan Eyal, Yury Malkov, Corentin Tallec, Yuval Bahat, Shen Yan, Xintian, Wu, David Lindner, Chengda Wu, Avi Caciularu, Xiyang Luo, Rodolphe Jenatton, Tim Zaman, Yingying Bi, Ilya Kornakov, Ganesh Mallya, Daisuke Ikeda, Itay Karo, Anima Singh, Colin Evans, Praneeth Netrapalli, Vincent Nallatamby, Isaac Tian, Yannis Assael, Vikas Raunak, Victor Carbune, Ioana Bica, Lior Madmoni, Dee Cattle, Snchit Grover, Krishna Somandepalli, Sid Lall, Amelio Vázquez-Reina, Riccardo Patana, Jiaqi Mu, Pranav Talluri, Maggie Tran, Rajeev Aggarwal, RJ Skerry-Ryan, Jun Xu, Mike Burrows, Xiaoyue Pan, Edouard Yvinec, Di Lu, Zhiying Zhang, Duc Dung Nguyen, Hairong Mu, Gabriel Barcik, Helen Ran, Lauren Beltrone, Krzysztof Choromanski, Dia Kharrat, Samuel Albanie, Sean Purser-haskell, David Bieber, Carrie Zhang, Jing Wang, Tom Hudson, Zhiyuan Zhang, Han Fu, Johannes Mauerer, Mohammad Hossein Bateni, AJ Maschinot, Bing Wang, Muye Zhu, Arjun Pillai, Tobias Weyand, Shuang Liu, Oscar Akerlund,
Fred Bertsch, Vittal Premachandran, Alicia Jin, Vincent Roulet, Peter de Boursac, Shubham Mittal, Ndaba Ndebele, Georgi Karadzhov, Sahra Ghalebikesabi, Ricky Liang, Allen Wu, Yale Cong, Nimesh Ghelani, Sumeet Singh, Bahar Fatemi, Warren, Chen, Charles Kwong, Alexey Kolganov, Steve Li, Richard Song, Chenkai Kuang, Sobhan Miryoosefi, Dale Webster, James Wendt, Arkadiusz Socala, Guolong Su, Artur Mendonça, Abhinav Gupta, Xiaowei Li, Tomy Tsai, Qiong, Hu, Kai Kang, Angie Chen, Sertan Girgin, Yongqin Xian, Andrew Lee, Nolan Ramsden, Leslie Baker, Madeleine Clare Elish, Varvara Krayvanova, Rishabh Joshi, Jiri Simsa, Yao-Yuan Yang, Piotr Ambroszczyk, Dipankar Ghosh, Arjun Kar, Yuan Shangguan, Yumeya Yamamori, Yaroslav Akulov, Andy Brock, Haotian Tang, Siddharth Vashishtha, Rich Munoz, Andreas Steiner, Kalyan Andra, Daniel Eppens, Qixuan Feng, Hayato Kobayashi, Sasha Goldshtein, Mona El Mahdy, Xin Wang, Jilei, Wang, Richard Killam, Tom Kwiatkowski, Kavya Kopparapu, Serena Zhan, Chao Jia, Alexei Bendebury, Sheryl
Luo, Adrià Recasens, Timothy Knight, Jing Chen, Mohak Patel, YaGuang Li, Ben Withbroe, Dean Weesner, Kush Bhatia, Jie Ren, Danielle Eisenbud, Ebrahim Songhori, Yanhua Sun, Travis Choma, Tasos Kementsietsidis, Lucas Manning, Brian Roark, Wael Farhan, Jie Feng, Susheel Tatineni, James Cobon-Kerr, Yunjie Li, Lisa Anne Hendricks, Isaac Noble, Chris Breaux, Nate Kushman, Liqian Peng, Fuzhao Xue, Taylor Tobin, Jamie Rogers, Josh Lipschultz, Chris Alberti, Alexey Vlaskin, Mostafa Dehghani, Roshan Sharma, Tris Warkentin, Chen-Yu Lee, Benigno Uria, Da-Cheng Juan, Angad Chandorkar, Hila Sheftel, Ruibo Liu, Elnaz Davoodi, Borja De Balle Pigem, Kedar Dhamdhere, David Ross, Jonathan Hoech, Mahdis Mahdieh, Li Liu, Qiujia Li, Liam McCafferty, Chenxi Liu, Markus Mircea, Yunting Song, Omkar Savant, Alaa Saade, Colin Cherry, Vincent Hellendoorn, Siddharth Goyal, Paul Pucciarelli, David Vilar Torres, Zohar Yahav, Hyo Lee, Lars Lowe Sjoesund, Christo Kirov, Bo Chang, Deepanway Ghoshal, Lu Li, Gilles Baechler, Sébastien
Pereira, Tara Sainath, Anudhyan Boral, Dominik Grewe, Afief Halumi, Nguyet Minh Phu, Tianxiao Shen, Marco Tulio Ribeiro, Dhriti Varma, Alex Kaskasoli, Vlad Feinberg, Navneet Potti, Jarrod Kahn, Matheus Wisniewski, Shakir Mohamed, Arnar Mar Hrafnkelsson, Bobak Shahriari, Jean-Baptiste Lespiau, Lisa Patel, Legg Yeung, Tom Paine, Lantao Mei, Alex Ramirez, Rakesh Shivanna, Li Zhong, Josh Woodward, Guilherme Tubone, Samira Khan, Heng Chen, Elizabeth Nielsen, Catalin Ionescu, Utsav Prabhu, Mingcen Gao, Qingze Wang, Sean Augenstein, Neesha Subramaniam, Jason Chang, Fotis Iliopoulos, Jiaming Luo, Myriam Khan, Weicheng Kuo, Denis Teplyashin, Florence Perot, Logan Kilpatrick, Amir Globerson, Hongkun Yu, Anfal Siddiqui, Nick Sukhanov, Arun Kandoor, Umang Gupta, Marco Andreetto, Moran Ambar, Donnie Kim, Paweł Wesołowski, Sarah Perrin, Ben Limonchik, Wei Fan, Jim Stephan, Ian Stewart-Binks, Ryan Kappedal, Tong He, Sarah Cogan, Romina Datta, Tong Zhou, Jiayu Ye, Leandro Kieliger, Ana Ramalho, Kyle Kastner, Fabian
Mentzer, Wei-Jen Ko, Arun Suggala, Tianhao Zhou, Shiraz Butt, Hana Strejček, Lior Belenki, Subhashini Venugopalan, Mingyang Ling, Evgenii Eltyshev, Yunxiao Deng, Geza Kovacs, Mukund Raghavachari, Hanjun Dai, Tal Schuster, Steven Schwarcz, Richard Nguyen, Arthur Nguyen, Gavin Buttimore, Shrestha Basu Mallick, Sudeep Gandhe, Seth Benjamin, Michal Jastrzebski, Le Yan, Sugato Basu, Chris Apps, Isabel Edkins, James Allingham, Immanuel Odisho, Tomas Kocisky, Jewel Zhao, Linting Xue, Apoorv Reddy, Chrysovalantis Anastasiou, Aviel Atias, Sam Redmond, Kieran Milan, Nicolas Heess, Herman Schmit, Allan Dafoe, Daniel Andor, Tynan Gangwani, Anca Dragan, Sheng Zhang, Ashyana Kachra, Gang Wu, Siyang Xue, Kevin Aydin, Siqi Liu, Yuxiang Zhou, Mahan Malihi, Austin Wu, Siddharth Gopal, Candice Schumann, Peter Stys, Alek Wang, Mirek Olšák, Dangyi Liu, Christian Schallhart, Yiran Mao, Demetra Brady, Hao Xu, Tomas Mery, Chawin Sitawarin, Siva Velusamy, Tom Cobley, Alex Zhai, Christian Walder, Nitzan Katz, Ganesh Jawahar,
Chinmay Kulkarni, Antoine Yang, Adam Paszke, Yinan Wang, Bogdan Damoc, Zalán Borsos, Ray Smith, Jinning Li, Mansi Gupta, Andrei Kapishnikov, Sushant Prakash, Florian Luisier, Rishabh Agarwal, Will Grathwohl, Kuangyuan Chen, Kehang Han, Nikhil Mehta, Andrew Over, Shekoofeh Azizi, Lei Meng, Niccolò Dal Santo, Kelvin Zheng, Jane Shapiro, Igor Petrovski, Jeffrey Hui, Amin Ghafouri, Jasper Snoek, James Qin, Mandy Jordan, Caitlin Sikora, Jonathan Malmaud, Yuheng Kuang, Aga Świetlik, Ruoxin Sang, Chongyang Shi, Leon Li, Andrew Rosenberg, Shubin Zhao, Andy Crawford, Jan-Thorsten Peter, Yun Lei, Xavier Garcia, Long Le, Todd Wang, Julien Amelot, Dave Orr, Praneeth Kacham, Dana Alon, Gladys Tyen, Abhinav Arora, James Lyon, Alex Kurakin, Mimi Ly, Theo Guidroz, Zhipeng Yan, Rina Panigrahy, Pingmei Xu, Thais Kagohara, Yong Cheng, Eric Noland, Jinhyuk Lee, Jonathan Lee, Cathy Yip, Maria Wang, Efrat Nehoran, Alexander Bykovsky, Zhihao Shan, Ankit Bhagatwala, Chaochao Yan, Jie Tan, Guillermo Garrido, Dan Ethier, Nate
Hurley, Grace Vesom, Xu Chen, Siyuan Qiao, Abhishek Nayyar, Julian Walker, Paramjit Sandhu, Mihaela Rosca, Danny Swisher, Mikhail Dektiarev, Josh Dillon, George-Cristian Muraru, Manuel Tragut, Artiom Myaskovsky, David Reid, Marko Velic, Owen Xiao, Jasmine George, Mark Brand, Jing Li, Wenhao Yu, Shane Gu, Xiang Deng, François-Xavier Aubet, Soheil Hassas Yeganeh, Fred Alcober, Celine Smith, Trevor Cohn, Kay McKinney, Michael Tschannen, Ramesh Sampath, Gowoon Cheon, Liangchen Luo, Luyang Liu, Jordi Orbay, Hui Peng, Gabriela Botea, Xiaofan Zhang, Charles Yoon, Cesar Magalhaes, Paweł Stradomski, Ian Mackinnon, Steven Hemingray, Kumaran Venkatesan, Rhys May, Jaeyoun Kim, Alex Druinsky, Jingchen Ye, Zheng Xu, Terry Huang, Jad Al Abdallah, Adil Dostmohamed, Rachana Fellinger, Tsendsuren Munkhdalai, Akanksha Maurya, Peter Garst, Yin Zhang, Maxim Krikun, Simon Bucher, Aditya Srikanth Veerubhotla, Yaxin Liu, Sheng Li, Nishesh Gupta, Jakub Adamek, Hanwen Chen, Bernett Orlando, Aleksandr Zaks, Joost van Amersfoort,
Josh Camp, Hui Wan, HyunJeong Choe, Zhichun Wu, Kate Olszewska, Weiren Yu, Archita Vadali, Martin Scholz, Daniel De Freitas, Jason Lin, Amy Hua, Xin Liu, Frank Ding, Yichao Zhou, Boone Severson, Katerina Tsihlas, Samuel Yang, Tammo Spalink, Varun Yerram, Helena Pankov, Rory Blevins, Ben Vargas, Sarthak Jauhari, Matt Miecnikowski, Ming Zhang, Sandeep Kumar, Clement Farabet, Charline Le Lan, Sebastian Flennerhag, Yonatan Bitton, Ada Ma, Arthur Bražinskas, Eli Collins, Niharika Ahuja, Sneha Kudugunta, Anna Bortsova, Minh Giang, Wanzheng Zhu, Ed Chi, Scott Lundberg, Alexey Stern, Subha Puttagunta, Jing Xiong, Xiao Wu, Yash Pande, Amit Jhindal, Daniel Murphy, Jon Clark, Marc Brockschmidt, Maxine Deines, Kevin R. McKee, Dan Bahir, Jiajun Shen, Minh Truong, Daniel McDuff, Andrea Gesmundo, Edouard Rosseel, Bowen Liang, Ken Caluwaerts, Jessica Hamrick, Joseph Kready, Mary Cassin, Rishikesh Ingale, Li Lao, Scott Pollom, Yifan Ding, Wei He, Lizzetth Bellot, Joana Iljazi, Ramya Sree Boppana, Shan Han, Tara Thompson,
Amr Khalifa, Anna Bulanova, Blagoj Mitrevski, Bo Pang, Emma Cooney, Tian Shi, Rey Coaguila, Tamar Yakar, Marc’aurelio Ranzato, Nikola Momchev, Chris Rawles, Zachary Charles, Young Maeng, Yuan Zhang, Rishabh Bansal, Xiaokai Zhao, Brian Albert, Yuan Yuan, Sudheendra Vijayanarasimhan, Roy Hirsch, Vinay Ramasesh, Kiran Vodrahalli, Xingyu Wang, Arushi Gupta, DJ Strouse, Jianmo Ni, Roma Patel, Gabe Taubman, Zhouyuan Huo, Dero Gharibian, Marianne Monteiro, Hoi Lam, Shobha Vasudevan, Aditi Chaudhary, Isabela Albuquerque, Kilol Gupta, Sebastian Riedel, Chaitra Hegde, Avraham Ruderman, András György, Marcus Wainwright, Ashwin Chaugule, Burcu Karagol Ayan, Tomer Levinboim, Sam Shleifer, Yogesh Kalley, Vahab Mirrokni, Abhishek Rao, Prabakar Radhakrishnan, Jay Hartford, Jialin Wu, Zhenhai Zhu, Francesco Bertolini, Hao Xiong, Nicolas Serrano, Hamish Tomlinson, Myle Ott, Yifan Chang, Mark Graham, Jian Li, Marco Liang, Xiangzhu Long, Sebastian Borgeaud, Yanif Ahmad, Alex Grills, Diana Mincu, Martin Izzard, Yuan Liu,
Jinyu Xie, Louis O’Bryan, Sameera Ponda, Simon Tong, Michelle Liu, Dan Malkin, Khalid Salama, Yuankai Chen, Rohan Anil, Anand Rao, Rigel Swavely, Misha Bilenko, Nina Anderson, Tat Tan, Jing Xie, Xing Wu, Lijun Yu, Oriol Vinyals, Andrey Ryabtsev, Rumen Dangovski, Kate Baumli, Daniel Keysers, Christian Wright, Zoe Ashwood, Betty Chan, Artem Shtefan, Yaohui Guo, Ankur Bapna, Radu Soricut, Steven Pecht, Sabela Ramos, Rui Wang, Jiahao Cai, Trieu Trinh, Paul Barham, Linda Friso, Eli Stickgold, Xiangzhuo Ding, Siamak Shakeri, Diego Ardila, Eleftheria Briakou, Phil Culliton, Adam Raveret, Jingyu Cui, David Saxton, Subhrajit Roy, Javad Azizi, Pengcheng Yin, Lucia Loher, Andrew Bunner, Min Choi, Faruk Ahmed, Eric Li, Yin Li, Shengyang Dai, Michael Elabd, Sriram Ganapathy, Shivani Agrawal, Yiqing Hua, Paige Kunkle, Sujeevan Rajayogam, Arun Ahuja, Arthur Conmy, Alex Vasiloff, Parker Beak, Christopher Yew, Jayaram Mudigonda, Bartek Wydrowski, Jon Blanton, Zhengdong Wang, Yann Dauphin, Zhuo Xu, Martin Polacek, Xi Chen,
Hexiang Hu, Pauline Sho, Markus Kunesch, Mehdi Hafezi Manshadi, Eliza Rutherford, Bo Li, Sissie Hsiao, Iain Barr, Alex Tudor, Matija Kecman, Arsha Nagrani, Vladimir Pchelin, Martin Sundermeyer, Aishwarya P S, Abhijit Karmarkar, Yi Gao, Grishma Chole, Olivier Bachem, Isabel Gao, Arturo BC, Matt Dibb, Mauro Verzetti, Felix Hernandez-Campos, Yana Lunts, Matthew Johnson, Julia Di Trapani, Raphael Koster, Idan Brusilovsky, Binbin Xiong, Megha Mohabey, Han Ke, Joe Zou, Tea Sabolić, Víctor Campos, John Palowitch, Alex Morris, Linhai Qiu, Pranavaraj Ponnuramu, Fangtao Li, Vivek Sharma, Kiranbir Sodhia, Kaan Tekelioglu, Aleksandr Chuklin, Madhavi Yenugula, Erika Gemzer, Theofilos Strinopoulos, Sam El-Husseini, Huiyu Wang, Yan Zhong, Edouard Leurent, Paul Natsev, Weijun Wang, Dre Mahaarachchi, Tao Zhu, Songyou Peng, Sami Alabed, Cheng-Chun Lee, Anthony Brohan, Arthur Szlam, GS Oh, Anton Kovsharov, Jenny Lee, Renee Wong, Megan Barnes, Gregory Thornton, Felix Gimeno, Omer Levy, Martin Sevenich, Melvin Johnson,
Jonathan Mallinson, Robert Dadashi, Ziyue Wang, Qingchun Ren, Preethi Lahoti, Arka Dhar, Josh Feldman, Dan Zheng, Thatcher Ulrich, Liviu Panait, Michiel Blokzijl, Cip Baetu, Josip Matak, Jitendra Harlalka, Maulik Shah, Tal Marian, Daniel von Dincklage, Cosmo Du, Ruy Ley-Wild, Bethanie Brownfield, Max Schumacher, Yury Stuken, Shadi Noghabi, Sonal Gupta, Xiaoqi Ren, Eric Malmi, Felix Weissenberger, Blanca Huergo, Maria Bauza, Thomas Lampe, Arthur Douillard, Mojtaba Seyedhosseini, Roy Frostig, Zoubin Ghahramani, Kelvin Nguyen, Kashyap Krishnakumar, Chengxi Ye, Rahul Gupta, Alireza Nazari, Robert Geirhos, Pete Shaw, Ahmed Eleryan, Dima Damen, Jennimaria Palomaki, Ted Xiao, Qiyin Wu, Quan Yuan, Phoenix Meadowlark, Matthew Bilotti, Raymond Lin, Mukund Sridhar, Yannick Schroecker, Da-Woon Chung, Jincheng Luo, Trevor Strohman, Tianlin Liu, Anne Zheng, Jesse Emond, Wei Wang, Andrew Lampinen, Toshiyuki Fukuzawa, Folawiyo Campbell-Ajala, Monica Roy, James Lee-Thorp, Lily Wang, Iftekhar Naim, Tony, Nguy ên, Guy
Bensky, Aditya Gupta, Dominika Rogozińska, Justin Fu, Thanumalayan Sankaranarayana Pillai, Petar Veličković, Shahar Drath, Philipp Neubeck, Vaibhav Tulsyan, Arseniy Klimovskiy, Don Metzler, Sage Stevens, Angel Yeh, Junwei Yuan, Tianhe Yu, Kelvin Zhang, Alec Go, Vincent Tsang, Ying Xu, Andy Wan, Isaac Galatzer-Levy, Sam Sobell, Abodunrinwa Toki, Elizabeth Salesky, Wenlei Zhou, Diego Antognini, Sholto Douglas, Shimu Wu, Adam Lelkes, Frank Kim, Paul Cavallaro, Ana Salazar, Yuchi Liu, James Besley, Tiziana Refice, Yiling Jia, Zhang Li, Michal Sokolik, Arvind Kannan, Jon Simon, Jo Chick, Avia Aharon, Meet Gandhi, Mayank Daswani, Keyvan Amiri, Vighnesh Birodkar, Abe Ittycheriah, Peter Grabowski, Oscar Chang, Charles Sutton, Zhixin, Lai, Umesh Telang, Susie Sargsyan, Tao Jiang, Raphael Hoffmann, Nicole Brichtova, Matteo Hessel, Jonathan Halcrow, Sammy Jerome, Geoff Brown, Alex Tomala, Elena Buchatskaya, Dian Yu, Sachit Menon, Pol Moreno, Yuguo Liao, Vicky Zayats, Luming Tang, SQ Mah, Ashish Shenoy, Alex
Siegman, Majid Hadian, Okwan Kwon, Tao Tu, Nima Khajehnouri, Ryan Foley, Parisa Haghani, Zhongru Wu, Vaishakh Keshava, Khyatti Gupta, Tony Bruguier, Rui Yao, Danny Karmon, Luisa Zintgraf, Zhicheng Wang, Enrique Piqueras, Junehyuk Jung, Jenny Brennan, Diego Machado, Marissa Giustina, MH Tessler, Kamyu Lee, Qiao Zhang, Joss Moore, Kaspar Daugaard, Alexander Frömmgen, Jennifer Beattie, Fred Zhang, Daniel Kasenberg, Ty Geri, Danfeng Qin, Gaurav Singh Tomar, Tom Ouyang, Tianli Yu, Luowei Zhou, Rajiv Mathews, Andy Davis, Yaoyiran Li, Jai Gupta, Damion Yates, Linda Deng, Elizabeth Kemp, Ga-Young Joung, Sergei Vassilvitskii, Mandy Guo, Pallavi LV, Dave Dopson, Sami Lachgar, Lara McConnaughey, Himadri Choudhury, Dragos Dena, Aaron Cohen, Joshua Ainslie, Sergey Levi, Parthasarathy Gopavarapu, Polina Zablotskaia, Hugo Vallet, Sanaz Bahargam, Xiaodan Tang, Nenad Tomasev, Ethan Dyer, Daniel Balle, Hongrae Lee, William Bono, Jorge Gonzalez Mendez, Vadim Zubov, Shentao Yang, Ivor Rendulic, Yanyan Zheng, Andrew Hogue,
Golan Pundak, Ralph Leith, Avishkar Bhoopchand, Michael Han, Mislav Žanić, Tom Schaul, Manolis Delakis, Tejas Iyer, Guanyu Wang, Harman Singh, Abdelrahman Abdelhamed, Tara Thomas, Siddhartha Brahma, Hilal Dib, Naveen Kumar, Wenxuan Zhou, Liang Bai, Pushkar Mishra, Jiao Sun, Valentin Anklin, Roykrong Sukkerd, Lauren Agubuzu, Anton Briukhov, Anmol Gulati, Maximilian Sieb, Fabio Pardo, Sara Nasso, Junquan Chen, Kexin Zhu, Tiberiu Sosea, Alex Goldin, Keith Rush, Spurthi Amba Hombaiah, Andreas Noever, Allan Zhou, Sam Haves, Mary Phuong, Jake Ades, Yi ting Chen, Lin Yang, Joseph Pagadora, Stan Bileschi, Victor Cotruta, Rachel Saputro, Arijit Pramanik, Sean Ammirati, Dan Garrette, Kevin Villela, Tim Blyth, Canfer Akbulut, Neha Jha, Alban Rrustemi, Arissa Wongpanich, Chirag Nagpal, Yonghui Wu, Morgane Rivière, Sergey Kishchenko, Pranesh Srinivasan, Alice Chen, Animesh Sinha, Trang Pham, Bill Jia, Tom Hennigan, Anton Bakalov, Nithya Attaluri, Drew Garmon, Daniel Rodriguez, Dawid Wegner, Wenhao Jia, Evan Senter,
Noah Fiedel, Denis Petek, Yuchuan Liu, Cassidy Hardin, Harshal Tushar Lehri, Joao Carreira, Sara Smoot, Marcel Prasetya, Nami Akazawa, Anca Stefanoiu, Chia-Hua Ho, Anelia Angelova, Kate Lin, Min Kim, Charles Chen, Marcin Sieniek, Alice Li, Tongfei Guo, Sorin Baltateanu, Pouya Tafti, Michael Wunder, Nadav Olmert, Divyansh Shukla, Jingwei Shen, Neel Kovelamudi, Balaji Venkatraman, Seth Neel, Romal Thoppilan, Jerome Connor, Frederik Benzing, Axel Stjerngren, Golnaz Ghiasi, Alex Polozov, Joshua Howland, Theophane Weber, Justin Chiu, Ganesh Poomal Girirajan, Andreas Terzis, Pidong Wang, Fangda Li, Yoav Ben Shalom, Dinesh Tewari, Matthew Denton, Roee Aharoni, Norbert Kalb, Heri Zhao, Junlin Zhang, Angelos Filos, Matthew Rahtz, Lalit Jain, Connie Fan, Vitor Rodrigues, Ruth Wang, Richard Shin, Jacob Austin, Roman Ring, Mariella Sanchez-Vargas, Mehadi Hassen, Ido Kessler, Uri Alon, Gufeng Zhang, Wenhu Chen, Yenai Ma, Xiance Si, Le Hou, Azalia Mirhoseini, Marc Wilson, Geoff Bacon, Becca Roelofs, Lei Shu, Gautam
Vasudevan, Jonas Adler, Artur Dwornik, Tayfun Terzi, Matt Lawlor, Harry Askham, Mike Bernico, Xuanyi Dong, Chris Hidey, Kevin Kilgour, Gaël Liu, Surya Bhupatiraju, Luke Leonhard, Siqi Zuo, Partha Talukdar, Qing Wei, Aliaksei Severyn, Vít Listík, Jong Lee, Aditya Tripathi, SK Park, Yossi Matias, Hao Liu, Alex Ruiz, Rajesh Jayaram, Jackson Tolins, Pierre Marcenac, Yiming Wang, Bryan Seybold, Henry Prior, Deepak Sharma, Jack Weber, Mikhail Sirotenko, Yunhsuan Sung, Dayou Du, Ellie Pavlick, Stefan Zinke, Markus Freitag, Max Dylla, Montse Gonzalez Arenas, Natan Potikha, Omer Goldman, Connie Tao, Rachita Chhaparia, Maria Voitovich, Pawan Dogra, Andrija Ražnatović, Zak Tsai, Chong You, Oleaser Johnson, George Tucker, Chenjie Gu, Jae Yoo, Maryam Majzoubi, Valentin Gabeur, Bahram Raad, Rocky Rhodes, Kashyap Kolipaka, Heidi Howard, Geta Sampemane, Benny Li, Chulayuth Asawaroengchai, Duy Nguyen, Chiyuan Zhang, Timothee Cour, Xinxin Yu, Zhao Fu, Joe Jiang, Po-Sen Huang, Gabriela Surita, Iñaki Iturrate, Yael
Karov, Michael Collins, Martin Baeuml, Fabian Fuchs, Shilpa Shetty, Swaroop Ramaswamy, Sayna Ebrahimi, Qiuchen Guo, Jeremy Shar, Gabe Barth-Maron, Sravanti Addepalli, Bryan Richter, Chin-Yi Cheng, Eugénie Rives, Fei Zheng, Johannes Griesser, Nishanth Dikkala, Yoel Zeldes, Ilkin Safarli, Dipanjan Das, Himanshu Srivastava, Sadh MNM Khan, Xin Li, Aditya Pandey, Larisa Markeeva, Dan Belov, Qiqi Yan, Mikołaj Rybiński, Tao Chen, Megha Nawhal, Michael Quinn, Vineetha Govindaraj, Sarah York, Reed Roberts, Roopal Garg, Namrata Godbole, Jake Abernethy, Anil Das, Lam Nguyen Thiet, Jonathan Tompson, John Nham, Neera Vats, Ben Caine, Wesley Helmholz, Francesco Pongetti, Yeongil Ko, James An, Clara Huiyi Hu, Yu-Cheng Ling, Julia Pawar, Robert Leland, Keisuke Kinoshita, Waleed Khawaja, Marco Selvi, Eugene Ie, Danila Sinopalnikov, Lev Proleev, Nilesh Tripuraneni, Michele Bevilacqua, Seungji Lee, Clayton Sanford, Dan Suh, Dustin Tran, Jeff Dean, Simon Baumgartner, Jens Heitkaemper, Sagar Gubbi, Kristina Toutanova,
Yichong Xu, Chandu Thekkath, Keran Rong, Palak Jain, Annie Xie, Yan Virin, Yang Li, Lubo Litchev, Richard Powell, Tarun Bharti, Adam Kraft, Nan Hua, Marissa Ikonomidis, Ayal Hitron, Sanjiv Kumar, Loic Matthey, Sophie Bridgers, Lauren Lax, Ishaan Malhi, Ondrej Skopek, Ashish Gupta, Jiawei Cao, Mitchelle Rasquinha, Siim Põder, Wojciech Stokowiec, Nicholas Roth, Guowang Li, Michaël Sander, Joshua Kessinger, Vihan Jain, Edward Loper, Wonpyo Park, Michal Yarom, Liqun Cheng, Guru Guruganesh, Kanishka Rao, Yan Li, Catarina Barros, Mikhail Sushkov, Chun-Sung Ferng, Rohin Shah, Ophir Aharoni, Ravin Kumar, Tim McConnell, Peiran Li, Chen Wang, Fernando Pereira, Craig Swanson, Fayaz Jamil, Yan Xiong, Anitha Vijayakumar, Prakash Shroff, Kedar Soparkar, Jindong Gu, Livio Baldini Soares, Eric Wang, Kushal Majmundar, Aurora Wei, Kai Bailey, Nora Kassner, Chizu Kawamoto, Goran Žužić, Victor Gomes, Abhirut Gupta, Michael Guzman, Ishita Dasgupta, Xinyi Bai, Zhufeng Pan, Francesco Piccinno, Hadas Natalie Vogel, Octavio
Ponce, Adrian Hutter, Paul Chang, Pan-Pan Jiang, Ionel Gog, Vlad Ionescu, James Manyika, Fabian Pedregosa, Harry Ragan, Zach Behrman, Ryan Mullins, Coline Devin, Aroonalok Pyne, Swapnil Gawde, Martin Chadwick, Yiming Gu, Sasan Tavakkol, Andy Twigg, Naman Goyal, Ndidi Elue, Anna Goldie, Srinivasan Venkatachary, Hongliang Fei, Ziqiang Feng, Marvin Ritter, Isabel Leal, Sudeep Dasari, Pei Sun, Alif Raditya Rochman, Brendan O’Donoghue, Yuchen Liu, Jim Sproch, Kai Chen, Natalie Clay, Slav Petrov, Sailesh Sidhwani, Ioana Mihailescu, Alex Panagopoulos, AJ Piergiovanni, Yunfei Bai, George Powell, Deep Karkhanis, Trevor Yacovone, Petr Mitrichev, Joe Kovac, Dave Uthus, Amir Yazdanbakhsh, David Amos, Steven Zheng, Bing Zhang, Jin Miao, Bhuvana Ramabhadran, Soroush Radpour, Shantanu Thakoor, Josh Newlan, Oran Lang, Orion Jankowski, Shikhar Bharadwaj, Jean-Michel Sarr, Shereen Ashraf, Sneha Mondal, Jun Yan, Ankit Singh Rawat, Sarmishta Velury, Greg Kochanski, Tom Eccles, Franz Och, Abhanshu Sharma, Ethan Mahintorabi,
Alex Gurney, Carrie Muir, Vered Cohen, Saksham Thakur, Adam Bloniarz, Asier Mujika, Alexander Pritzel, Paul Caron, Altaf Rahman, Fiona Lang, Yasumasa Onoe, Petar Sirkovic, Jay Hoover, Ying Jian, Pablo Duque, Arun Narayanan, David Soergel, Alex Haig, Loren Maggiore, Shyamal Buch, Josef Dean, Ilya Figotin, Igor Karpov, Shaleen Gupta, Denny Zhou, Muhuan Huang, Ashwin Vaswani, Christopher Semturs, Kaushik Shivakumar, Yu Watanabe, Vinodh Kumar Rajendran, Eva Lu, Yanhan Hou, Wenting Ye, Shikhar Vashishth, Nana Nti, Vytenis Sakenas, Darren Ni, Doug DeCarlo, Michael Bendersky, Sumit Bagri, Nacho Cano, Elijah Peake, Simon Tokumine, Varun Godbole, Carlos Guía, Tanya Lando, Vittorio Selo, Seher Ellis, Danny Tarlow, Daniel Gillick, Alessandro Epasto, Siddhartha Reddy Jonnalagadda, Meng Wei, Meiyan Xie, Ankur Taly, Michela Paganini, Mukund Sundararajan, Daniel Toyama, Ting Yu, Dessie Petrova, Aneesh Pappu, Rohan Agrawal, Senaka Buthpitiya, Justin Frye, Thomas Buschmann, Remi Crocker, Marco Tagliasacchi, Mengchao Wang,
Da Huang, Sagi Perel, Brian Wieder, Hideto Kazawa, Weiyue Wang, Jeremy Cole, Himanshu Gupta, Ben Golan, Seojin Bang, Nitish Kulkarni, Ken Franko, Casper Liu, Doug Reid, Sid Dalmia, Jay Whang, Kevin Cen, Prasha Sundaram, Johan Ferret, Berivan Isik, Lucian Ionita, Guan Sun, Anna Shekhawat, Muqthar Mohammad, Philip Pham, Ronny Huang, Karthik Raman, Xingyi Zhou, Ross Mcilroy, Austin Myers, Sheng Peng, Jacob Scott, Paul Covington, Sofia Erell, Pratik Joshi, João Gabriel Oliveira, Natasha Noy, Tajwar Nasir, Jake Walker, Vera Axelrod, Tim Dozat, Pu Han, Chun-Te Chu, Eugene Weinstein, Anand Shukla, Shreyas Chandrakaladharan, Petra Poklukar, Bonnie Li, Ye Jin, Prem Eruvbetine, Steven Hansen, Avigail Dabush, Alon Jacovi, Samrat Phatale, Chen Zhu, Steven Baker, Mo Shomrat, Yang Xiao, Jean Pouget-Abadie, Mingyang Zhang, Fanny Wei, Yang Song, Helen King, Yiling Huang, Yun Zhu, Ruoxi Sun, Juliana Vicente Franco, Chu-Cheng Lin, Sho Arora, Hui, Li, Vivian Xia, Luke Vilnis, Mariano Schain, Kaiz Alarakyia, Laurel Prince,
Aaron Phillips, Caleb Habtegebriel, Luyao Xu, Huan Gui, Santiago Ontanon, Lora Aroyo, Karan Gill, Peggy Lu, Yash Katariya, Dhruv Madeka, Shankar Krishnan, Shubha Srinivas Raghvendra, James Freedman, Yi Tay, Gaurav Menghani, Peter Choy, Nishita Shetty, Dan Abolafia, Doron Kukliansky, Edward Chou, Jared Lichtarge, Ken Burke, Ben Coleman, Dee Guo, Larry Jin, Indro Bhattacharya, Victoria Langston, Yiming Li, Suyog Kotecha, Alex Yakubovich, Xinyun Chen, Petre Petrov, Tolly Powell, Yanzhang He, Corbin Quick, Kanav Garg, Dawsen Hwang, Yang Lu, Srinadh Bhojanapalli, Kristian Kjems, Ramin Mehran, Aaron Archer, Hado van Hasselt, Ashwin Balakrishna, JK Kearns, Meiqi Guo, Jason Riesa, Mikita Sazanovich, Xu Gao, Chris Sauer, Chengrun Yang, XiangHai Sheng, Thomas Jimma, Wouter Van Gansbeke, Vitaly Nikolaev, Wei Wei, Katie Millican, Ruizhe Zhao, Justin Snyder, Levent Bolelli, Maura O’Brien, Shawn Xu, Fei Xia, Wentao Yuan, Arvind Neelakantan, David Barker, Sachin Yadav, Hannah Kirkwood, Farooq Ahmad, Joel Wee, Jordan
Grimstad, Boyu Wang, Matthew Wiethoff, Shane Settle, Miaosen Wang, Charles Blundell, Jingjing Chen, Chris Duvarney, Grace Hu, Olaf Ronneberger, Alex Lee, Yuanzhen Li, Abhishek Chakladar, Alena Butryna, Georgios Evangelopoulos, Guillaume Desjardins, Jonni Kanerva, Henry Wang, Averi Nowak, Nick Li, Alyssa Loo, Art Khurshudov, Laurent El Shafey, Nagabhushan Baddi, Karel Lenc, Yasaman Razeghi, Tom Lieber, Amer Sinha, Xiao Ma, Yao Su, James Huang, Asahi Ushio, Hanna Klimczak-Plucińska, Kareem Mohamed, JD Chen, Simon Osindero, Stav Ginzburg, Lampros Lamprou, Vasilisa Bashlovkina, Duc-Hieu Tran, Ali Khodaei, Ankit Anand, Yixian Di, Ramy Eskander, Manish Reddy Vuyyuru, Jasmine Liu, Aishwarya Kamath, Roman Goldenberg, Mathias Bellaiche, Juliette Pluto, Bill Rosgen, Hassan Mansoor, William Wong, Suhas Ganesh, Eric Bailey, Scott Baird, Dan Deutsch, Jinoo Baek, Xuhui Jia, Chansoo Lee, Abe Friesen, Nathaniel Braun, Kate Lee, Amayika Panda, Steven M. Hernandez, Duncan Williams, Jianqiao Liu, Ethan Liang, Arnaud Autef,
Emily Pitler, Deepali Jain, Phoebe Kirk, Oskar Bunyan, Jaume Sanchez Elias, Tongxin Yin, Machel Reid, Aedan Pope, Nikita Putikhin, Bidisha Samanta, Sergio Guadarrama, Dahun Kim, Simon Rowe, Marcella Valentine, Geng Yan, Alex Salcianu, David Silver, Gan Song, Richa Singh, Shuai Ye, Hannah DeBalsi, Majd Al Merey, Eran Ofek, Albert Webson, Shibl Mourad, Ashwin Kakarla, Silvio Lattanzi, Nick Roy, Evgeny Sluzhaev, Christina Butterfield, Alessio Tonioni, Nathan Waters, Sudhindra Kopalle, Jason Chase, James Cohan, Girish Ramchandra Rao, Robert Berry, Michael Voznesensky, Shuguang Hu, Kristen Chiafullo, Sharat Chikkerur, George Scrivener, Ivy Zheng, Jeremy Wiesner, Wolfgang Macherey, Timothy Lillicrap, Fei Liu, Brian Walker, David Welling, Elinor Davies, Yangsibo Huang, Lijie Ren, Nir Shabat, Alessandro Agostini, Mariko Iinuma, Dustin Zelle, Rohit Sathyanarayana, Andrea D’olimpio, Morgan Redshaw, Matt Ginsberg, Ashwin Murthy, Mark Geller, Tatiana Matejovicova, Ayan Chakrabarti, Ryan Julian, Christine Chan, Qiong
Hu, Daniel Jarrett, Manu Agarwal, Jeshwanth Challagundla, Tao Li, Sandeep Tata, Wen Ding, Maya Meng, Zhuyun Dai, Giulia Vezzani, Shefali Garg, Jannis Bulian, Mary Jasarevic, Honglong Cai, Harish Rajamani, Adam Santoro, Florian Hartmann, Chen Liang, Bartek Perz, Apoorv Jindal, Fan Bu, Sungyong Seo, Ryan Poplin, Adrian Goedeckemeyer, Badih Ghazi, Nikhil Khadke, Leon Liu, Kevin Mather, Mingda Zhang, Ali Shah, Alex Chen, Jinliang Wei, Keshav Shivam, Yuan Cao, Donghyun Cho, Angelo Scorza Scarpati, Michael Moffitt, Clara Barbu, Ivan Jurin, Ming-Wei Chang, Hongbin Liu, Hao Zheng, Shachi Dave, Christine Kaeser-Chen, Xiaobin Yu, Alvin Abdagic, Lucas Gonzalez, Yanping Huang, Peilin Zhong, Cordelia Schmid, Bryce Petrini, Alex Wertheim, Jifan Zhu, Hoang Nguyen, Kaiyang Ji, Yanqi Zhou, Tao Zhou, Fangxiaoyu Feng, Regev Cohen, David Rim, Shubham Milind Phal, Petko Georgiev, Ariel Brand, Yue Ma, Wei Li, Somit Gupta, Chao Wang, Pavel Dubov, Jean Tarbouriech, Kingshuk Majumder, Huijian Li, Norman Rink, Apurv Suman, Yang Guo,
Yinghao Sun, Arun Nair, Xiaowei Xu, Mohamed Elhawaty, Rodrigo Cabrera, Guangxing Han, Julian Eisenschlos, Junwen Bai, Yuqi Li, Yamini Bansal, Thibault Sellam, Mina Khan, Hung Nguyen, Justin Mao-Jones, Nikos Parotsidis, Jake Marcus, Cindy Fan, Roland Zimmermann, Yony Kochinski, Laura Graesser, Feryal Behbahani, Alvaro Caceres, Michael Riley, Patrick Kane, Sandra Lefdal, Rob Willoughby, Paul Vicol, Lun Wang, Shujian Zhang, Ashleah Gill, Yu Liang, Gautam Prasad, Soroosh Mariooryad, Mehran Kazemi, Zifeng Wang, Kritika Muralidharan, Paul Voigtlaender, Jeffrey Zhao, Huanjie Zhou, Nina D’Souza, Aditi Mavalankar, Séb Arnold, Nick Young, Obaid Sarvana, Chace Lee, Milad Nasr, Tingting Zou, Seokhwan Kim, Lukas Haas, Kaushal Patel, Neslihan Bulut, David Parkinson, Courtney Biles, Dmitry Kalashnikov, Chi Ming To, Aviral Kumar, Jessica Austin, Alex Greve, Lei Zhang, Megha Goel, Yeqing Li, Sergey Yaroshenko, Max Chang, Abhishek Jindal, Geoff Clark, Hagai Taitelbaum, Dale Johnson, Ofir Roval, Jeongwoo Ko, Anhad Mohananey,
Christian Schuler, Shenil Dodhia, Ruichao Li, Kazuki Osawa, Claire Cui, Peng Xu, Rushin Shah, Tao Huang, Ela Gruzewska, Nathan Clement, Mudit Verma, Olcan Sercinoglu, Hai Qian, Viral Shah, Masa Yamaguchi, Abhinit Modi, Takahiro Kosakai, Thomas Strohmann, Junhao Zeng, Beliz Gunel, Jun Qian, Austin Tarango, Krzysztof Jastrzębski, Robert David, Jyn Shan, Parker Schuh, Kunal Lad, Willi Gierke, Mukundan Madhavan, Xinyi Chen, Mark Kurzeja, Rebeca Santamaria-Fernandez, Dawn Chen, Alexandra Cordell, Yuri Chervonyi, Frankie Garcia, Nithish Kannen, Vincent Perot, Nan Ding, Shlomi Cohen-Ganor, Victor Lavrenko, Junru Wu, Georgie Evans, Cicero Nogueira dos Santos, Madhavi Sewak, Ashley Brown, Andrew Hard, Joan Puigcerver, Zeyu Zheng, Yizhong Liang, Evgeny Gladchenko, Reeve Ingle, Uri First, Pierre Sermanet, Charlotte Magister, Mihajlo Velimirović, Sashank Reddi, Susanna Ricco, Eirikur Agustsson, Hartwig Adam, Nir Levine, David Gaddy, Dan Holtmann-Rice, Xuanhui Wang, Ashutosh Sathe, Abhijit Guha Roy, Blaž Bratanič,
Alen Carin, Harsh Mehta, Silvano Bonacina, Nicola De Cao, Mara Finkelstein, Verena Rieser, Xinyi Wu, Florent Altché, Dylan Scandinaro, Li Li, Nino Vieillard, Nikhil Sethi, Garrett Tanzer, Zhi Xing, Shibo Wang, Parul Bhatia, Gui Citovsky, Thomas Anthony, Sharon Lin, Tianze Shi, Shoshana Jakobovits, Gena Gibson, Raj Apte, Lisa Lee, Mingqing Chen, Arunkumar Byravan, Petros Maniatis, Kellie Webster, Andrew Dai, Pu-Chin Chen, Jiaqi Pan, Asya Fadeeva, Zach Gleicher, Thang Luong, and Niket Kumar Bhumihar.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025.
URL [https://arxiv.org/abs/2507.06261](https://arxiv.org/abs/2507.06261).
- DeepSeek-AI et al. (2025)
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang
Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang
Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan.
Deepseek-v3 technical report, 2025.
URL [https://arxiv.org/abs/2412.19437](https://arxiv.org/abs/2412.19437).
- Dehghani et al. (2023)
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.
Scaling vision transformers to 22 billion parameters.
In *International Conference on Machine Learning*, pp. 7480–7512. PMLR, 2023.
- Dettmers & Zettlemoyer (2023)
Tim Dettmers and Luke Zettlemoyer.
The case for 4-bit precision: k-bit inference scaling laws.
In *International Conference on Machine Learning*, pp. 7750–7774. PMLR, 2023.
- Dohmatob et al. (2024)
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe.
A tale of tails: Model collapse as a change of scaling laws, 2024.
URL [https://arxiv.org/abs/2402.07043](https://arxiv.org/abs/2402.07043).
- Dominguez-Olmedo et al. (2024)
Ricardo Dominguez-Olmedo, Florian E. Dorner, and Moritz Hardt.
Training on the test task confounds evaluation and emergence, 2024.
URL [https://arxiv.org/abs/2407.07890](https://arxiv.org/abs/2407.07890).
- Everett et al. (2024)
Katie E Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, et al.
Scaling exponents across parameterizations and optimizers.
In *International Conference on Machine Learning*, pp. 12666–12700. PMLR, 2024.
- Frankle & Carbin (2019)
Jonathan Frankle and Michael Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural networks.
In *International Conference on Learning Representations*, 2019.
- Gadre et al. (2024)
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al.
Language models scale reliably with over-training and on downstream tasks.
*arXiv preprint arXiv:2403.08540*, 2024.
- Gema et al. (2025)
Aryo Pradipta Gema, Alexander Hägele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, and Ethan Perez.
Inverse scaling in test-time compute, 2025.
URL [https://arxiv.org/abs/2507.14417](https://arxiv.org/abs/2507.14417).
- Gerstgrasser et al. (2024)
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo.
Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, 2024.
URL [https://arxiv.org/abs/2404.01413](https://arxiv.org/abs/2404.01413).
- Ghorbani et al. (2021)
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry.
Scaling laws for neural machine translation.
In *International Conference on Learning Representations*, 2021.
- Gordon et al. (2021)
Mitchell A Gordon, Kevin Duh, and Jared Kaplan.
Data and parameter scaling laws for neural machine translation.
In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 5915–5922, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.emnlp-main.478.
URL [https://aclanthology.org/2021.emnlp-main.478](https://aclanthology.org/2021.emnlp-main.478).
- Henighan et al. (2020)
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al.
Scaling laws for autoregressive generative modeling.
*arXiv preprint arXiv:2010.14701*, 2020.
- Hernandez et al. (2021)
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish.
Scaling laws for transfer, 2021.
URL [https://arxiv.org/abs/2102.01293](https://arxiv.org/abs/2102.01293).
- Hernandez et al. (2022)
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al.
Scaling laws and interpretability of learning from repeated data.
*arXiv preprint arXiv:2205.10487*, 2022.
- Hestness et al. (2017)
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.
Deep learning scaling is predictable, empirically, 2017.
URL [https://arxiv.org/abs/1712.00409](https://arxiv.org/abs/1712.00409).
- Hoffmann et al. (2022)
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre.
An empirical analysis of compute-optimal large language model training.
In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), *Advances in Neural Information Processing Systems*, volume 35, pp. 30016–30030. Curran Associates, Inc., 2022.
URL [https://proceedings.neurips.cc/paper_files/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf).
- Howe et al. (2025)
Nikolaus H. R. Howe, Ian R. McKenzie, Oskar John Hollinsworth, Michał Zając, Tom Tseng, Aaron David Tucker, Pierre-Luc Bacon, and Adam Gleave.
Scaling trends in language model robustness.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=tNGdLEL4R0](https://openreview.net/forum?id=tNGdLEL4R0).
- Hu et al. (2024)
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun.
Predicting emergent abilities with infinite resolution evaluation, 2024.
URL [https://arxiv.org/abs/2310.03262](https://arxiv.org/abs/2310.03262).
- Hughes et al. (2024)
John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma.
Best-of-n jailbreaking, 2024.
URL [https://arxiv.org/abs/2412.03556](https://arxiv.org/abs/2412.03556).
- Hutter (2021)
Marcus Hutter.
Learning curve theory, 2021.
URL [https://arxiv.org/abs/2102.04074](https://arxiv.org/abs/2102.04074).
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Jiang et al. (2024)
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo.
Investigating data contamination for pre-training language models, 2024.
URL [https://arxiv.org/abs/2401.06059](https://arxiv.org/abs/2401.06059).
- Kalajdzievski (2024)
Damjan Kalajdzievski.
Scaling laws for forgetting when fine-tuning large language models, 2024.
URL [https://arxiv.org/abs/2401.05605](https://arxiv.org/abs/2401.05605).
- Kaplan et al. (2020)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models.
*arXiv preprint arXiv:2001.08361*, 2020.
- Kazdan et al. (2024)
Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, and Sanmi Koyejo.
Collapse or thrive? perils and promises of synthetic data in a self-generating world, 2024.
URL [https://arxiv.org/abs/2410.16713](https://arxiv.org/abs/2410.16713).
- Kimi et al. (2025)
Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu
Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang, Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang, Zhen Yang, Zhilin Yang, Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu,
Weiyu Zhuang, and Xinxing Zu.
Kimi k2: Open agentic intelligence, 2025.
URL [https://arxiv.org/abs/2507.20534](https://arxiv.org/abs/2507.20534).
- Kudugunta et al. (2023)
Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, Prateek Jain, et al.
Matformer: Nested transformer for elastic inference.
*arXiv preprint arXiv:2310.07707*, 2023.
- Kumar et al. (2024)
Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan.
Scaling laws for precision.
*arXiv preprint arXiv:2411.04330*, 2024.
- Kwok et al. (2025)
Jacky Kwok, Christopher Agia, Rohan Sinha, Matt Foutter, Shulu Li, Ion Stoica, Azalia Mirhoseini, and Marco Pavone.
Robomonkey: Scaling test-time sampling and verification for vision-language-action models.
*arXiv preprint arXiv:2506.17811*, 2025.
- Levi (2025)
Noam Itzhak Levi.
A simple model of inference scaling laws.
In *Forty-second International Conference on Machine Learning*, 2025.
- Liew et al. (2025)
Seng Pei Liew, Takuya Kato, and Sho Takase.
Scaling laws for upcycling mixture-of-experts language models.
*arXiv preprint arXiv:2502.03009*, 2025.
- Lin et al. (2024)
Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee.
Scaling laws in linear regression: Compute, parameters, and data.
*arXiv preprint arXiv:2406.08466*, 2024.
- Ludziejewski et al. (2025)
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski, Maciej Stefaniak, Michał Krutul, Jan Małaśnicki, Marek Cygan, Piotr Sankowski, Kamil Adamczewski, Piotr Miłoś, et al.
Joint moe scaling laws: Mixture of experts can be memory efficient.
*arXiv preprint arXiv:2502.05172*, 2025.
- Maloney et al. (2022)
Alexander Maloney, Daniel A Roberts, and James Sully.
A solvable model of neural scaling laws.
*arXiv preprint arXiv:2210.16859*, 2022.
- McKenna et al. (2025)
Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A Choquette-Choo, Badih Ghazi, Georgios Kaissis, Ravi Kumar, Ruibo Liu, et al.
Scaling laws for differentially private language models.
In *Forty-second International Conference on Machine Learning*, 2025.
- McKenzie et al. (2024)
Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman, and Ethan Perez.
Inverse scaling: When bigger isn’t better, 2024.
URL [https://arxiv.org/abs/2306.09479](https://arxiv.org/abs/2306.09479).
- Mhaskar (1996)
Hrushikesh N Mhaskar.
Neural networks for optimal approximation of smooth and analytic functions.
*Neural computation*, 8(1):164–177, 1996.
- Muennighoff et al. (2023)
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel.
Scaling data-constrained language models.
*Advances in Neural Information Processing Systems*, 36:50358–50376, 2023.
- OpenAI et al. (2024)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo
Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan
Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez,
Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia
Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.
Gpt-4 technical report, 2024.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- Paquette et al. (2024)
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington.
4+ 3 phases of compute-optimal neural scaling laws.
*arXiv preprint arXiv:2405.15074*, 2024.
- Pearce & Song (2024)
Tim Pearce and Jinyeop Song.
Reconciling kaplan and chinchilla scaling laws.
*Transactions on Machine Learning Research*, 2024.
- Pearce et al. (2025)
Tim Pearce, Tabish Rashid, David Bignell, Raluca Georgescu, Sam Devlin, and Katja Hofmann.
Scaling laws for pre-training agents and world models.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=HHwGfLOKxq](https://openreview.net/forum?id=HHwGfLOKxq).
- Pinkus (1999)
Allan Pinkus.
Approximation theory of the mlp model in neural networks.
*Acta numerica*, 8:143–195, 1999.
- Porian et al. (2024)
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon.
Resolving discrepancies in compute-optimal scaling of language models.
In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), *Advances in Neural Information Processing Systems*, volume 37, pp. 100535–100570. Curran Associates, Inc., 2024.
URL [https://proceedings.neurips.cc/paper_files/paper/2024/file/b6341525cd84f3be0ef203e4d7cd8556-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/b6341525cd84f3be0ef203e4d7cd8556-Paper-Conference.pdf).
- Press & Wolf (2017)
Ofir Press and Lior Wolf.
Using the output embedding to improve language models.
In *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*, pp. 157–163, 2017.
- Qin et al. (2025)
Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R Fung, et al.
Scaling laws of synthetic data for language models.
*arXiv preprint arXiv:2503.19551*, 2025.
- Qiu et al. (2025)
Shikai Qiu, Lechao Xiao, Andrew Gordon Wilson, Jeffrey Pennington, and Atish Agarwala.
Scaling collapse reveals universal dynamics in compute-optimally trained neural networks.
In *Forty-second International Conference on Machine Learning*, 2025.
- Qwen et al. (2025)
Qwen, An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report, 2025.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Rae et al. (2022)
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones,
James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.
Scaling language models: Methods, analysis & insights from training gopher, 2022.
URL [https://arxiv.org/abs/2112.11446](https://arxiv.org/abs/2112.11446).
- Roberts et al. (2022)
Daniel A Roberts, Sho Yaida, and Boris Hanin.
*The principles of deep learning theory*, volume 46.
Cambridge University Press Cambridge, MA, USA, 2022.
- Rosenfeld et al. (2020)
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit.
A constructive prediction of the generalization error across scales.
In *International Conference on Learning Representations*, 2020.
- Rosenfeld et al. (2021)
Jonathan S Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit.
On the predictability of pruning across scales.
In Marina Meila and Tong Zhang (eds.), *Proceedings of the 38th International Conference on Machine Learning*, volume 139 of *Proceedings of Machine Learning Research*, pp. 9075–9083. PMLR, 18–24 Jul 2021.
URL [https://proceedings.mlr.press/v139/rosenfeld21a.html](https://proceedings.mlr.press/v139/rosenfeld21a.html).
- Sadhukhan et al. (2025)
Ranajoy Sadhukhan, Zhuoming Chen, Haizhong Zheng, Yang Zhou, Emma Strubell, and Beidi Chen.
Kinetics: Rethinking test-time scaling laws, 2025.
URL [https://arxiv.org/abs/2506.05333](https://arxiv.org/abs/2506.05333).
- Sardana et al. (2024)
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle.
Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.
In *International Conference on Machine Learning*, pp. 43445–43460. PMLR, 2024.
- Schaeffer (2023)
Rylan Schaeffer.
Pretraining on the test set is all you need, 2023.
URL [https://arxiv.org/abs/2309.08632](https://arxiv.org/abs/2309.08632).
- Schaeffer et al. (2023)
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo.
Are emergent abilities of large language models a mirage?
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
- Schaeffer et al. (2024)
Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo.
Why has predicting downstream capabilities of frontier ai models with scale remained elusive?, 2024.
URL [https://arxiv.org/abs/2406.04391](https://arxiv.org/abs/2406.04391).
- Schaeffer et al. (2025)
Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, and Sanmi Koyejo.
How do large language monkeys get their power (laws)?
In *Forty-second International Conference on Machine Learning*, 2025.
- Sharma & Kaplan (2022)
Utkarsh Sharma and Jared Kaplan.
Scaling laws from the data manifold dimension.
*Journal of Machine Learning Research*, 23(9):1–34, 2022.
- Shukor et al. (2025)
Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby.
Scaling laws for native multimodal models, 2025.
URL [https://arxiv.org/abs/2504.07951](https://arxiv.org/abs/2504.07951).
- Snell et al. (2024a)
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
Scaling llm test-time compute optimally can be more effective than scaling model parameters.
*arXiv preprint arXiv:2408.03314*, 2024a.
- Snell et al. (2024b)
Charlie Snell, Eric Wallace, Dan Klein, and Sergey Levine.
Predicting emergent capabilities by finetuning, 2024b.
URL [https://arxiv.org/abs/2411.16035](https://arxiv.org/abs/2411.16035).
- Spigler et al. (2020)
Stefano Spigler, Mario Geiger, and Matthieu Wyart.
Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm.
*Journal of Statistical Mechanics: Theory and Experiment*, 2020(12):124001, December 2020.
ISSN 1742-5468.
doi: 10.1088/1742-5468/abc61d.
URL [http://dx.doi.org/10.1088/1742-5468/abc61d](http://dx.doi.org/10.1088/1742-5468/abc61d).
- Srivastava et al. (2023)
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan
Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang,
Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan
Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna
Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel
Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen
Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
URL [https://arxiv.org/abs/2206.04615](https://arxiv.org/abs/2206.04615).
- Sun et al. (2025)
Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, Yu Cheng, Yangyu Tao, Zhanhui Kang, Chengzhong Xu, Di Wang, and Jie Jiang.
Scaling laws for floating point quantization training, 2025.
URL [https://arxiv.org/abs/2501.02423](https://arxiv.org/abs/2501.02423).
- Tao et al. (2024)
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong.
Scaling laws with vocabulary: Larger models deserve larger vocabularies.
*arXiv preprint arXiv:2407.13623*, 2024.
- Tay et al. (2021)
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler.
Scale efficiently: Insights from pre-training and fine-tuning transformers.
*arXiv preprint arXiv:2109.10686*, 2021.
- Wei et al. (2022)
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.
Emergent abilities of large language models, 2022.
URL [https://arxiv.org/abs/2206.07682](https://arxiv.org/abs/2206.07682).
- Wu & Lo (2024)
Tung-Yu Wu and Pei-Yu Lo.
U-shaped and inverted-u scaling behind emergent abilities of large language models, 2024.
URL [https://arxiv.org/abs/2410.01692](https://arxiv.org/abs/2410.01692).
- Wu et al. (2024)
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang.
Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2024.
URL [https://arxiv.org/abs/2408.00724](https://arxiv.org/abs/2408.00724).
- Xiong et al. (2023)
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.
Effective long-context scaling of foundation models, 2023.
URL [https://arxiv.org/abs/2309.16039](https://arxiv.org/abs/2309.16039).
- Zhai et al. (2022)
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.
Scaling vision transformers.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 12104–12113, 2022.
- Zhang et al. (2024)
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat.
When scaling meets llm finetuning: The effect of data, model and finetuning method, 2024.
URL [https://arxiv.org/abs/2402.17193](https://arxiv.org/abs/2402.17193).
- Zhang (2023)
Susan Zhang.
After ignoring the details in all these "lets-fit-a-cloud-of-points-to-a-single-line" papers (all likely wrong when you really extrapolate), @stephenroller finally convinced me to work through the math in the chinchilla paper and as expected, this was a doozy., January 2023.
URL [https://x.com/suchenzang/status/1616752482226671620](https://x.com/suchenzang/status/1616752482226671620).
- Zhipu-AI et al. (2025)
Zhipu-AI, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang, Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng, Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi,
Junli Gao, Ke Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang, Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zihan Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang.
Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025.
URL [https://arxiv.org/abs/2508.06471](https://arxiv.org/abs/2508.06471).

## Appendix A Language Model Usage

Language models were used by the authors to aid or polish the writing of the paper.
Authors take full responsibility for the content.

## Appendix B Chinchilla’s Architectural Hyperparameters and Model Parameters

Below, we include Hoffmann et al. (2022)’s Table A9 listing model architectural hyperparameters alongside the reported model parameters. We augment the table with the standard formula model parameters (Eqn. [1](https://arxiv.org/html/2509.23963v1#S2.E1)) and the best fit formula model parameters (Eqn. [3](https://arxiv.org/html/2509.23963v1#S2.E3)).

**Table 2: Chinchilla Language Models. We copy Chinchilla’s Table A9 listing the model parameters and model architectural hyperparameters of all models used in the Chinchilla fitting processes. Parameters specified in millions (${1}\mathrm{e}{6}$).**
| Table A9 from Hoffmann et al. (2022) | Our Contribution |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| d_model | ffw_size | kv_size | n_heads | n_layers | n_vocab | Chinchilla’s<br>Reported<br>Model<br>Parameters (M) | Best Fit<br>Formula’s<br>Model<br>Parameters (M) | Standard<br>Formula’s<br>Model<br>Parameters (M) |
| 512 | 2048 | 64 | 8 | 8 | 32168 | 44 | 44 | 42 |
| 576 | 2304 | 64 | 9 | 9 | 32168 | 57 | 57 | 54 |
| 640 | 2560 | 64 | 10 | 10 | 32168 | 74 | 74 | 70 |
| 640 | 2560 | 64 | 10 | 13 | 32168 | 90 | 90 | 84 |
| 640 | 2560 | 64 | 10 | 16 | 32168 | 106 | 106 | 99 |
| 768 | 3072 | 64 | 12 | 12 | 32168 | 117 | 117 | 110 |
| 768 | 3072 | 64 | 12 | 15 | 32168 | 140 | 140 | 131 |
| 768 | 3072 | 64 | 12 | 18 | 32168 | 163 | 163 | 152 |
| 896 | 3584 | 64 | 14 | 14 | 32168 | 175 | 175 | 164 |
| 896 | 3584 | 64 | 14 | 16 | 32168 | 196 | 196 | 183 |
| 896 | 3584 | 64 | 14 | 18 | 32168 | 217 | 217 | 202 |
| 1024 | 4096 | 64 | 16 | 16 | 32168 | 251 | 251 | 234 |
| 1024 | 4096 | 64 | 16 | 18 | 32168 | 278 | 278 | 259 |
| 1024 | 4096 | 64 | 16 | 20 | 32168 | 306 | 306 | 285 |
| 1280 | 5120 | 128 | 10 | 18 | 32168 | 425 | 425 | 395 |
| 1280 | 5120 | 128 | 10 | 21 | 32168 | 489 | 488 | 454 |
| 1408 | 5632 | 128 | 11 | 18 | 32168 | 509 | 509 | 474 |
| 1280 | 5120 | 128 | 10 | 24 | 32168 | 552 | 552 | 513 |
| 1408 | 5632 | 128 | 11 | 21 | 32168 | 587 | 587 | 545 |
| 1536 | 6144 | 128 | 12 | 19 | 32168 | 632 | 632 | 587 |
| 1408 | 5632 | 128 | 11 | 25 | 32168 | 664 | 690 | 640 |
| 1536 | 6144 | 128 | 12 | 22 | 32168 | 724 | 724 | 672 |
| 1536 | 6144 | 128 | 12 | 23 | 32168 | 816 | 755 | 701 |
| 1792 | 7168 | 128 | 14 | 20 | 32168 | 893 | 893 | 828 |
| 1792 | 7168 | 128 | 14 | 22 | 32168 | 1018 | 976 | 905 |
| 1792 | 7168 | 128 | 14 | 26 | 32168 | 1143 | 1143 | 1060 |
| 2048 | 8192 | 128 | 16 | 20 | 32168 | 1266 | 1156 | 1073 |
| 2176 | 8704 | 128 | 17 | 22 | 32168 | 1424 | 1424 | 1320 |
| 2048 | 8192 | 128 | 16 | 25 | 32168 | 1429 | 1429 | 1324 |
| 2048 | 8192 | 128 | 16 | 28 | 32168 | 1593 | 1593 | 1475 |
| 2176 | 8704 | 128 | 17 | 25 | 32168 | 1609 | 1609 | 1490 |
| 2304 | 9216 | 128 | 18 | 24 | 32168 | 1731 | 1730 | 1603 |
| 2176 | 8704 | 128 | 17 | 28 | 32168 | 1794 | 1794 | 1661 |
| 2304 | 9216 | 128 | 18 | 26 | 32168 | 2007 | 1868 | 1730 |
| 2304 | 9216 | 128 | 18 | 32 | 32168 | 2283 | 2282 | 2113 |
| 2560 | 10240 | 128 | 20 | 26 | 32168 | 2298 | 2297 | 2127 |
| 2560 | 10240 | 128 | 20 | 30 | 32168 | 2639 | 2638 | 2442 |
| 2560 | 10240 | 128 | 20 | 34 | 32168 | 2980 | 2979 | 2756 |
| 2688 | 10752 | 128 | 22 | 36 | 32168 | 3530 | 3530 | 3257 |
| 2816 | 11264 | 128 | 22 | 36 | 32168 | 3802 | 3802 | 3516 |
| 2944 | 11776 | 128 | 22 | 36 | 32168 | 4084 | 4083 | 3785 |
| 3072 | 12288 | 128 | 24 | 36 | 32168 | 4516 | 4515 | 4176 |
| 3584 | 14336 | 128 | 28 | 40 | 32168 | 6796 | 6795 | 6281 |
| 4096 | 16384 | 128 | 32 | 42 | 32168 | 9293 | 9292 | 8587 |
| 4352 | 17408 | 128 | 32 | 47 | 32168 | 11452 | 11450 | 10613 |
| 4608 | 18432 | 128 | 36 | 44 | 32168 | 12295 | 12294 | 11360 |
| 4608 | 18432 | 128 | 32 | 47 | 32168 | 12569 | 12568 | 11680 |
| 4864 | 19456 | 128 | 36 | 47 | 32168 | 13775 | 14319 | 13266 |
| 4992 | 19968 | 128 | 32 | 49 | 32168 | 14940 | 14939 | 13937 |
| 5120 | 20480 | 128 | 40 | 47 | 32168 | 16183 | 16182 | 14950 |

## Appendix C Theoretical Analysis

Here, we provide a detailed analysis of the empirical results obtained in the main text from a theoretical perspective. We begin by repeating the derivation of the baseline compute-optimal scaling for the number of tokens as a function of model parameters, and continue to systematically work through the perturbations discussed in [section˜3](https://arxiv.org/html/2509.23963v1#S3).

### C.1 Baseline Compute-Optimal Scaling Derivation

The Chinchilla scaling law for pretraining loss $L$ as a function of non-embedding model parameters $N$ and number of training tokens $D$ is given by [eq.˜4](https://arxiv.org/html/2509.23963v1#S2.E4), which we repeat here

$$ $L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}},$ (10) $$

where $E$ is the irreducible loss, and $(A,\alpha)$ and $(B,\beta)$ are parameters for the model size and data scaling terms, respectively.

The training compute budget $C$ is approximately proportional to the product of model size and training data, $C\approx cND$, where $c>0$ is some constant factor. This allows us to express the number of training tokens as a function of compute and model size: $D=C/(cN)$. Substituting this into the loss function yields the loss for a fixed compute budget

$$ $L(N,C)=E+AN^{-\alpha}+B\left(\frac{C}{cN}\right)^{-\beta}=E+AN^{-\alpha}+B(c^{\beta}C^{-\beta})N^{\beta}.$ (11) $$

The optimal model size $N_{\text{opt}}$ that minimizes the loss for a fixed compute budget $C$ is simply found by differentiating [eq.˜11](https://arxiv.org/html/2509.23963v1#A3.E11) with respect to $N$ and setting the derivative to zero

$$ $\frac{\partial L}{\partial N}=-\alpha AN^{-(\alpha+1)}+\beta B(c^{\beta}C^{-\beta})N^{\beta-1}=0.$ (12) $$

Rearranging this equation reveals the optimal trade-off:

$$ $\alpha AN_{\text{opt}}^{-(\alpha+1)}=\beta B(c^{\beta}C^{-\beta})N_{\text{opt}}^{\beta-1}.$ (13) $$

Solving for $N_{\text{opt}}$:

$$ $N_{\text{opt}}^{\alpha+\beta}=\frac{\alpha A}{\beta Bc^{\beta}}C^{\beta}\implies N_{\text{opt}}=\left(\frac{\alpha A}{\beta Bc^{\beta}}\right)^{\frac{1}{\alpha+\beta}}C^{\frac{\beta}{\alpha+\beta}}$ (14) $$

The compute-optimal tokens-per-parameter ratio is $R_{\text{opt}}=D_{\text{opt}}/N_{\text{opt}}$. Using $D_{\text{opt}}=C/(cN_{\text{opt}})$, we find $R_{\text{opt}}=C/(cN_{\text{opt}}^{2})$. Substituting our expression for $N_{\text{opt}}$ results in

$$ $R_{\text{opt}}=\frac{C}{c}\left[\left(\frac{\alpha A}{\beta Bc^{\beta}}\right)^{\frac{1}{\alpha+\beta}}C^{\frac{\beta}{\alpha+\beta}}\right]^{-2}=\frac{C}{c}\left(\frac{\beta Bc^{\beta}}{\alpha A}\right)^{\frac{2}{\alpha+\beta}}C^{\frac{-2\beta}{\alpha+\beta}}.$ (15) $$

This simplifies to the final form, which shows the ratio’s dependence on the compute budget $C$ as

$$ $\frac{D_{\text{opt}}}{N_{\text{opt}}}=K\cdot C^{\frac{\alpha-\beta}{\alpha+\beta}},$ (16) $$

where $K=\frac{1}{c}\left(\frac{\beta Bc^{\beta}}{\alpha A}\right)^{\frac{2}{\alpha+\beta}}$ is a constant. The key insight from the Chinchilla works is that empirically, one finds that $\alpha\approx\beta$, making the exponent on $C$ approximately zero and the optimal ratio nearly constant. In mathematical terms, it leads to the relation

$$ $\frac{D_{\text{opt}}}{N_{\text{opt}}}\approx K=\left(\frac{B}{A}\right)^{\frac{1}{\alpha}}.$ (17) $$

If we want to recover the $20:1$ ratio of training tokens to number of parameters we expect to find that $B\approx 2.85A$ for $\alpha\approx 0.35$.

### C.2 Analysis of Perturbations

We now analyze how systematic errors in model parameter counts affect the fitted scaling parameters $(\hat{A},\hat{\alpha})$ and the resulting optimal ratio. Let $N$ be the true parameter count and $\tilde{N}$ be the perturbed (incorrect) count used for fitting. The fitting process minimizes the error between the model $L(\tilde{N},D)=\hat{E}+\hat{A}\tilde{N}^{-\hat{\alpha}}+\hat{B}D^{-\hat{\beta}}$ and the observed losses. For most types of perturbations we consider, since the data $D$ is unaffected, we assume $\hat{B}\approx B$ and $\hat{\beta}\approx\beta$. We will break this assumption when necessary.

#### C.2.1 Multiplicative Constant Perturbation

Assume the reported parameters are a constant multiple of the true parameters: $\tilde{N}=c_{m}\cdot N$. The model size term in the loss is modified to

$$ $\hat{A}\tilde{N}^{-\hat{\alpha}}=\hat{A}(c_{m}N)^{-\hat{\alpha}}=(\hat{A}c_{m}^{-\hat{\alpha}})N^{-\hat{\alpha}}.$ (18) $$

For [eq.˜18](https://arxiv.org/html/2509.23963v1#A3.E18) to match the true term $AN^{-\alpha}$, the fitting procedure will ideally find parameters such that:

- •
$\hat{\alpha}\approx\alpha$
- •
$\hat{A}c_{m}^{-\hat{\alpha}}\approx A\implies\hat{A}\approx Ac_{m}^{\alpha}$

The exponent in the optimal ratio ([16](https://arxiv.org/html/2509.23963v1#A3.E16)) becomes $(\hat{\alpha}-\beta)/(\hat{\alpha}+\beta)\approx(\alpha-\beta)/(\alpha+\beta)$.

Conclusion: A multiplicative error does not change the exponent governing the trend of the optimal ratio. The flat relationship with compute budget is preserved. However, the constant prefactor $K$ is shifted by a factor of $(c_{m}^{\alpha})^{-2/(\alpha+\beta)}=c_{m}^{-2\alpha/(\alpha+\beta)}\approx c_{m}^{-1}$, which shifts the entire line up or down on a log-log plot. This is shown in Fig. [4](https://arxiv.org/html/2509.23963v1#S3.F4) (top row) and Fig. [5](https://arxiv.org/html/2509.23963v1#S3.F5) (top left).

#### C.2.2 Additive Constant Perturbation

Assume an additive error, e.g., from including/excluding embeddings: $\tilde{N}=N+c_{a}$. The loss term $\hat{A}(N+c_{a})^{-\hat{\alpha}}$ is no longer a pure power law in $N$.

We examine the process of fitting the perturbed model $g(N;\hat{A},\hat{\alpha})=\hat{A}(N+c_{a})^{-\hat{\alpha}}$ to the true function $f(N)=AN^{-\alpha}$ by minimizing the Mean Squared Error (MSE) in log-space.

The objective is to find $(\hat{A},\hat{\alpha})$ that minimize $\sum_{i}[\log f(N_{i})-\log g(N_{i})]^{2}$.
The core of the problem lies in approximating the term $\log(N+c_{a})$. For the regime where $N\gg|c_{a}|$, which applies to the larger models in the study, we can analyze the local behavior of the function.

##### Effect on the Scaling Exponent α ^ \hat{\alpha} :

The most critical parameter in a power law is its exponent, which corresponds to the slope in a log-log plot. The true slope is constant

$$ $\frac{d(\log f(N))}{d(\log N)}=-\alpha$ (19) $$

For the perturbed function, the effective slope is not constant and depends on $N$ as

$$ $\displaystyle\text{Effective Slope}(N)$ $\displaystyle=\frac{d(\log g(N))}{d(\log N)}=-\hat{\alpha}\frac{d(\log(N+c_{a}))}{d(\log N)}=-\hat{\alpha}\left(\frac{N}{N+c_{a}}\right)$ (20) $\displaystyle\implies\hat{\alpha}=\alpha/\left(\frac{N}{N+c_{a}}\right).$ $$

The fitting procedure must select a single exponent $\hat{\alpha}$ that best represents this varying slope over the range of data. To match the true average slope of $-\alpha$, the fitted $\hat{\alpha}$ must compensate for the factor $N/(N+c_{a})$:

- •
When $c_{a}>0$, the factor $N/(N+c_{a})<1$. To achieve the target slope, the fitting process must select an exponent $\hat{\alpha}>\alpha$.
- •
When $c_{a}<0$, the factor $N/(N+c_{a})>1$ (for $N>|c_{a}|$). To compensate, the fitting process must select an exponent $\hat{\alpha}<\alpha$.

This provides a direct analytical explanation for the observed behavior of $\hat{\alpha}$ in Figure 4, which is smaller than the true $\alpha$ for $c_{a}<0$ and increases approximately linearly for $c_{a}>0$.

##### Effect on the Prefactor A ^ \hat{A} :

Once the optimal $\hat{\alpha}$ is determined, the prefactor $\hat{A}$ is chosen to minimize the remaining offset. We can approximate this by enforcing that the functions match at some effective "pivot" point $N_{0}$ that is characteristic of the dataset.

$$ $f(N_{0})\approx g(N_{0})\implies AN_{0}^{-\alpha}\approx\hat{A}(N_{0}+c_{a})^{-\hat{\alpha}}.$ (21) $$

Solving for $\hat{A}$ gives

$$ $\hat{A}\approx A\cdot N_{0}^{-\alpha}\cdot(N_{0}+c_{a})^{\hat{\alpha}}=A\left(\frac{N_{0}+c_{a}}{N_{0}}\right)^{\hat{\alpha}}\left(N_{0}\right)^{\hat{\alpha}-\alpha}.$ (22) $$

Assuming for simplicity that the pivot point is chosen such that the $N_{0}^{\hat{\alpha}-\alpha}$ term is of order one, we focus on the dominant term

$$ $\hat{A}\approx A\left(1+\frac{c_{a}}{N_{0}}\right)^{\hat{\alpha}}.$ (23) $$

This relationship explains the rapid growth of $\hat{A}$. Since we have already established that $\hat{\alpha}$ itself increases with $c$, the prefactor $\hat{A}$ grows due to two compounding effects: an increase in the base $(1+c_{a}/N_{0})$ and an increase in the exponent $\hat{\alpha}$. This leads to the exponential-like growth observed empirically in [fig.˜4](https://arxiv.org/html/2509.23963v1#S3.F4).

#### C.2.3 Systematic Bias Perturbation

Assume a bias where the error itself scales with model size, modeled as $\tilde{N}=\mu_{\text{geo}}(N/\mu_{\text{geo}})^{s}$, where $\mu_{\text{geo}}$ is the geometric mean of the true parameter counts and $s$ is the bias factor. The model size term becomes

$$ $\hat{A}\tilde{N}^{-\hat{\alpha}}=\hat{A}\left(\mu_{\text{geo}}^{1-s}N^{s}\right)^{-\hat{\alpha}}=\left(\hat{A}\mu_{\text{geo}}^{-(1-s)\hat{\alpha}}\right)N^{-s\hat{\alpha}}$ (24) $$

To match the true term $AN^{-\alpha}$, the exponent and the constant term must satisfy the relations

$$ $\hat{\alpha}=\frac{\alpha}{s},\quad\hat{A}=\mu_{\text{geo}}^{\frac{\alpha(1-s)}{s}}A.$ (25) $$

The fitted exponent is inversely proportional to the bias factor $s$, which is verified empirically in [section˜3.3](https://arxiv.org/html/2509.23963v1#S3.SS3). The exponent in the optimal ratio is now $(\alpha/s-\beta)/(\alpha/s+\beta)$.

Conclusion: A systematic bias also breaks the $\hat{\alpha}\approx\beta$ condition, unless $s=1$.

- •
If $s<1$ (inflating larger models relative to smaller ones), then $\hat{\alpha}>\alpha\approx\beta$. The exponent on $C$ becomes positive, and the optimal ratio increases with compute.
- •
If $s>1$ (shrinking larger models relative to smaller ones), then $\hat{\alpha}<\alpha\approx\beta$. The exponent on $C$ becomes negative, and the optimal ratio *decreases* with compute.

This perturbation also qualitatively changes the optimal scaling strategy, with the direction of the change depending on the nature of the bias, as seen in [fig.˜5](https://arxiv.org/html/2509.23963v1#S3.F5) (bottom left).

## Appendix D Related Work

##### Scaling Laws in Neural (Language) Models

While initial research on scaling laws in neural models began decades ago (Barkai et al., 1993; Mhaskar, 1996; Pinkus, 1999), advances in scaling large language models brought such interest into renewed focus (Hestness et al., 2017; Kaplan et al., 2020; Brown et al., 2020), causing an explosion of research.
For a non-exhaustive list, theoretical understanding of scaling laws has advanced substantially (Spigler et al., 2020; Bousquet et al., 2020; Hutter, 2021; Sharma & Kaplan, 2022; Maloney et al., 2022; Roberts et al., 2022; Bahri et al., 2024; Paquette et al., 2024; Atanasov et al., 2024; Bordelon et al., 2024a; b; Lin et al., 2024; Brill, 2024), complemented by empirical studies (Rosenfeld et al., 2020; Henighan et al., 2020; Gordon et al., 2021; Tay et al., 2021; Ghorbani et al., 2021; Zhai et al., 2022; Alabdulmohsin et al., 2022; Dehghani et al., 2023; Bachmann et al., 2023; Everett et al., 2024; Qiu et al., 2025).

Additional research has also studied how scaling interacts with specific considerations such as efficient inference (Sardana et al., 2024; Bian et al., 2025), transfer (Hernandez et al., 2021; Barnett, 2024), data quality and diversity (Chen et al., 2025; Hernandez et al., 2022; Muennighoff et al., 2023; Qin et al., 2025; Shukor et al., 2025), overtraining (Gadre et al., 2024), quantization and precision (Dettmers & Zettlemoyer, 2023; Sun et al., 2025; Kumar et al., 2024), differential privacy (McKenna et al., 2025), distillation (Busbridge et al., 2025), model architecture (Clark et al., 2022; Kudugunta et al., 2023; Abnar et al., 2025; Ludziejewski et al., 2025; Liew et al., 2025), context length (Xiong et al., 2023; Agarwal et al., 2024; Arora et al., 2024), vocabulary size (Tao et al., 2024), robustness to jailbreaking (Howe et al., 2025; Anil et al., 2024; Hughes et al., 2024), pruning (Rosenfeld et al., 2021), multimodality (Aghajanyan et al., 2023; Cherti et al., 2023), fine-tuning (Kalajdzievski, 2024; Zhang et al., 2024) and agents and world models (Pearce et al., 2025).

Recent work has also highlighted novel scaling phenomena such as inverse scaling (McKenzie et al., 2024; Gema et al., 2025), emergent capabilities (Srivastava et al., 2023; Wei et al., 2022; Schaeffer et al., 2023; Hu et al., 2024; Schaeffer et al., 2024; Snell et al., 2024b; Wu & Lo, 2024), and critical issues like data contamination (Schaeffer, 2023; Jiang et al., 2024; Dominguez-Olmedo et al., 2024) and model-data feedback loops (Dohmatob et al., 2024; Gerstgrasser et al., 2024; Kazdan et al., 2024).
The advent of so-called “thinking” or “reasoning” models (Jaech et al., 2024) has sparked a new wave of interest in scaling inference compute (Brown et al., 2024; Snell et al., 2024a; Wu et al., 2024; Chen et al., 2024; Sadhukhan et al., 2025; Levi, 2025; Schaeffer et al., 2025; Kwok et al., 2025).

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 6
presentation: 3
questions: Relative to the widely cited $\sim 20:1$ Chinchilla scaling ratio, the training token counts for current open-source large language models (LLMs) such as Qwen's 1B to 32B parameter series are consistently maintained at over 1 Trillion tokens. Does this training regimen constitute excessive training (over-training), and, crucially, is it an appropriate scale for exploring current scaling laws? Specifically, what guidance does this observed practice offer regarding the determination of optimal hyperparameters?
soundness: 3
strengths: A significant strength of this work lies in its compelling research perspective, which undertakes a rigorous analysis of a previously existing, yet imprecisely defined, theory. The experimental design is thorough, providing robust validation for the conclusions drawn. Furthermore, the manuscript is exceptionally well-written, demonstrating a high degree of clarity and fluency.
summary: This paper investigates the definitional ambiguity surrounding model parameters within the Chinchilla study. The analysis demonstrates that while the relative error among the three distinct parameter computation methods is substantial , the resulting impact on the final, compute-optimal data-to-parameter ratio is found to be negligible.
weaknesses: * Objectively, the research findings of this paper offer a comparatively limited marginal contribution. 
* Furthermore, despite the highly insightful nature of the original Chinchilla work, its practical application value within the industry remains constrained.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Can the authors point to exact data they are using to conduct their scaling law fits?
2. Can the authors provide any evidence that scaling law fits they obtain are of sufficient quality, eg by conducting MSE/prediction error measurements on held out points?
3. Can the authors elaborate on studies like FarSeer https://arxiv.org/abs/2506.10972 that indicate that Chinchilla style compute optimal TTP advice might be entirely off for the larger compute scales? Would that affect the message of the current work?
soundness: 2
strengths: * The paper addresses an important issue: how much does misspecification of model parameter counts might affect the validity of derived scaling laws, focusing on Chinchilla case as the one still very much relevant for practitioners.
* Three alternate parameter interpretations is a convincing study showing the insensitivity of the scaling law fits to moderate (~15 %) parameter disagreements.
* The perturbation framework containing the four types of perturbation (multiplicative, additive, systematic bias, log-normal noise) covers realistic modes of parameter‐miscounting (e.g., whether embeddings/attention/FFN are fully counted, rounding, etc).
* The paper is clearly written: it explains well the motivation, methodology and results.
summary: In spirit of number of previous works, the paper revisits the Chinchilla scaling law study, noting that model parameter counts used in Chinchilla’s analysis are ambiguous, which leads to question how the perturbations in model param counts would impact important conclusions from the study like the claimed token to parameter (TTP) ratio (heuristics of ~20 tokens-per-parameter) in which many practitioners arguably still trust. 

To answer this question, authors first investigate three interpretations how Chinchilla study may have determined the model params - i) via Table A9 from original study (original reported params) ii) via using standard formula for transformer arch param numbers (standard params) and iii) via a best fit formula used by authors that comes close to reported params (best fit). Authors find that despite relative error of up to 15.2\% in param numbers between those sets, the parametric fit L(N,D) and the resulting $D_{opt}/N_{opt} (C)$ fits do not differ substantially. 

Further, authors conduct a perturbation study to test how sensitive the scaling law fits are to various errors in model param estimation. Authors distort the parameter counts of the original Chinchilla’s experiments in four structured ways: multiplicative scaling, additive offsets, systematic geometrical bias, log-normal noise. They then re-fit the parametric loss L(N,D) and examine how the implied compute-optimal TTP shifts. The reported finding is that for multiplicative and random‐noise perturbations the TTP fit remains rather stable, while for additive or systematic bias perturbations the TTP fit slope can change significantly. Overall, the authors draw the conclusion that Chinchilla’s core result — estimates for compute-optimal TTP ratio $D_{opt}/N_{opt}$ — is robust to various potential errors in parameter counting. The authors state thus that while Chinchilla’s experimental data has ambiguities, the guidance it gives for practitioners is still valid.
weaknesses: 1. Main weakness in my opinion is the strong claim of Chinchilla conclusions eg with regard to TTP estimates being good guidance for practitioners following the conducted study. In my opinion, the paper convincingly shows that possible perturbations of model param numbers do not alter substantially scaling law fit as conducted in Chinchilla study. However, it does not make clear whether the conclusions from those fits are justifiable. Eg if we look at Fig 2, the D/N fits show high uncertainty, making especially for larger scale compute region above $10^{22}$ unclear what TTP estimate to take. It appears that even the slope of D/N fit might change from negative to positive, making a strong difference for TTP conclusions on larger scales. While this is not the deficit of the current work and is inherited from Chinchilla design, I think the statement abour relying on const 20 TTP is confusing, as the study does not deal with repairing this particular deficit of original Chinchilla work.

2. I think as the study deals with effect of model param perturbations only, one has to be again cautious to make a general statement about Chinchilla hints to be reliable - as also further elements of the study, eg dataset composition nature etc, can be in same way source of changing the important trends as model param perturbation, which is though not checked by the study.

3. There is no enough detail about the data used for the current study. It seems that authors rely on previous work done by Besiroglu et al to extract the data from original Chinchilla plots, but it is nowhere mentioned.

4. It is not clear why authors rely only on Chinchilla data to make a case for TTP recommendation testing. Eg, Llama 3 reports also offer numerous datapoints that were used to fit scaling laws and it would be good to understand whether TTP estimation there leads to same or different results as in Chinchilla, which would provide for practitioners more consistency if both were aligned in their recommendations derived from scaling law fits.

5. It seems there is not a clear validation of scaling law fits, eg using held out points, to make sure that obtained fits actually retain the sufficient quality. This was also not performed in original Chinchilla study, validation there was arguably done by using three different approaches and observing similar trends. Here authors use only Approach 3, parametric loss fit, and there should be a way to show that obtained fits are actually to be trusted.

6. I think some relevant citations are missing. Eg FarSeer https://arxiv.org/abs/2506.10972 or https://arxiv.org/abs/2410.11840

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 3
questions: See above.
soundness: 2
strengths: The paper is well written and is focused on one problem. Robustness of the scaling laws is an important problem as it affects its predictive power.
summary: The paper analyses Chinchilla scaling law's robustness w.r.t. the model parameters. It does two experiments: (1) reran the Chinchilla fit with two different interpretations of model parameter counts, besides the Chinchilla one; (2) introduce synthetic error / perturbation to model parameter counts to test for the robustness of the Chinchilla fit. The paper found that in both experiments, the Chinchilla scaling law does not meaningfully change and Chinchilla scaling law are most sensitive to additive or systematic errors, but overall, Chinchilla’s key results withstand sizable perturbations.
weaknesses: 1. While robustness of Chinchilla scaling law is an important problem, I believe the paper would be greatly strengthened it is more based on real-world use cases of the Chinchilla scaling law, so that findings can guide practice. Currently, it is more theoretical studies. For example,

1-1: the first experiment consider three possible model parameters counts: Chinchilla's reported model parameters, best fit, and standard formula. I feel instead of best fit, the choice of model parameters could be based on how people actually count model parameters, for example according Kaplan et al, Hoffmann et al, whether embedding weights are included, whether efficient implementation of operators that reduces parameters are considered.

1-2: The second experiment is based on synthetic perturbations of model parameters, while such robustness understanding is helpful, I believe it would be made more practically helpful if the analysis is based on realistic variation in how people count parameters.

2. I am curious and hope the authors could elaborate in revision, what previous works have also analysed the model parameters count aspect of the neural scaling law (Kaplan, Chinchilla, or others), what are their findings, and how this work compares to them and finds anything novel. For example, Porian et al also discusses different model parameters counts in detail in their table 2 and Appendix B in Appendix, conducts experiments with two different kinds of models counts and compare them in Figure 1 and 7, and found "limited difference" in scaling laws. Elaborating this point in paper would help readers better position this work in context. I note that while line 48 ~ 53 mentions existing works, the discussion do not more specifically touch on experiments & findings related to model counts and robustness. 

- Porian, Tomer, et al. "Resolving discrepancies in compute-optimal scaling of language models." Advances in Neural Information Processing Systems 37 (2024): 100535-100570.

3. I encourage the authors to describe more details how existing works count parameters, like in Kaplan et al, Hoffman et all, and other works. This helps readers understand where the descripency is in the real world. Appendix B in Porian et al would be a good example. 

4. I wonder how exactly Hoffman et al calculates mode counts? I understand this information might be missing from the original paper. But I also read from Appendix B in Porian et al that "Hoffmann et al. [25] account for both linear and attention layers in their
FLOP computation essentially using Neff in their first two estimation approaches. However, their third approach appears to ignore the attention FLOPs and also count the embeddings parameters,i.e., setting N′ = N + dv." It would be very helpful if the authors understand how Hoffman calculates model parameters and also describe in main paper. And then, the best fit model count will not be necessary.

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 3
questions: - What data was used for the fits? What was number of points and $N\times D$ combinations?
- Are the wide confidence intervals in chinchilla due to systematic errors (i.e. noise in the model parameters estimation)? 
- How the predictions of Chinchilla change with perturbations? E.g. how the $L(N,D)$ extrapolation changes with perturbation on larger compute budgets?
soundness: 3
strengths: - Analyzed the ambiguity in Chinchilla's model parameters count.
- Systematic model parameters perturbations like additive term, don't affect main Chinchilla claims - constant token-to-parameter ratio.
- The paper is well written and concise.
summary: In this paper authors perform a sensitivity analysis of "Chinchilla" [1] scaling laws of LLMs (approach 3 - parametric $L(N,D)$ fit), that prescribed a compute-optimal token-to-parameters (TTP) ratio being constant. The authors perturbed model parameter count $N$ in 4 systematic ways (multiplicative, additive, systematic bias, log-normal noise) and re-fitted Chinchilla scaling laws on the perturbed data. They find out that Chinchilla's constant optimal TTP ratio holds to some extent with both multiplicative and with added noise and in general Chinchilla's prescriptions are reliable.

1. Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).
weaknesses: ### Weaknesses

- The main weakness of this work is the lack significance. In the abstract and the introduction, authors lay out the motivations behind this paper: can Chinchilla's prescribed token-to-parameter ratio be trusted? Unfortunately, the paper answers this question only partially: if we have a systematic error in counting model parameters, the compute optimal TTP will likely be constant. However, the paper doesn't give any insight whether Chinchilla's initial $L(N,D)$ functional form, from which the optimal TTP ratio is derived from, is reliable in the first place. Handful of recent works looked into other functional forms and/or approaches that prescribe other TTP ratios [1,2,3]. Despite, the findings presented in the paper being valid and important, I don't think the analysis is deep enough and the paper's insights add to our understanding of compute optimal scaling of LLMs.

### References

1. McLeish, Sean, et al. "Gemstones: A model suite for multi-faceted scaling laws." arXiv preprint arXiv:2502.06857 (2025).
2. Li, Houyi, et al. "Farseer: A Refined Scaling Law in Large Language Models." arXiv preprint arXiv:2506.10972 (2025).
3. Hu, Shengding, et al. "Minicpm: Unveiling the potential of small language models with scalable training strategies." _arXiv preprint arXiv:2404.06395_ (2024).


Decision: Reject