Unified Neural Network Scaling Laws and Scale-time Equivalence

## Abstract

Abstract As neural networks continue to grow in size but datasets might not, it is vital to understand how much performance improvement can be expected: is it more important to scale network size or data volume? Thus, neural network scaling laws, which characterize how test error varies with network size and data volume, have become increasingly important. However, existing scaling laws are often applicable only in limited regimes and often do not incorporate or predict well-known phenomena such as double descent. Here, we present a novel theoretical characterization of how three factors — model size, training time, and data volume — interact to determine the performance of deep neural networks. We first establish a theoretical and empirical equivalence between scaling the size of a neural network and increasing its training time proportionally. Scale-time equivalence challenges the current practice, wherein large models are trained for small durations, and suggests that smaller models trained over extended periods could match their efficacy. It also leads to a novel method for predicting the performance of large-scale networks from small-scale networks trained for extended epochs, and vice versa. We next combine scale-time equivalence with a linear model analysis of double descent to obtain a unified theoretical scaling law, which we confirm with experiments across vision benchmarks and network architectures. These laws explain several previously unexplained phenomena: reduced data requirements for generalization in larger models, heightened sensitivity to label noise in overparameterized models, and instances where increasing model scale does not necessarily enhance performance. Our findings hold significant implications for the practical deployment of neural networks, offering a more accessible and efficient path to training and fine-tuning large models.

## 1 Introduction

Progress in artificial intelligence (AI) has relied heavily on the dramatic growth in the size of models and datasets. An active area of research focuses on understanding how test error decreases with increases in model and data size. This work has led to the development of scaling laws which posit that test error decreases as a power law with both. However, several theoretical aspects remain unclear. One significant gap is understanding how test error and the existing scaling laws change as the training time is varied  (Kaplan et al., ; Bahri et al., ; Rosenfeld et al., ; Sharma & Kaplan, ).

The practical relevance of this question is clear: under a fixed compute budget, what is the optimal balance between scaling the model size and dataset volume, and what is the right amount of training for a given data volume? This is particularly relevant in the context of large language models (LLMs), which are often trained for a single epoch, raising questions about the potential efficacy of training smaller models for longer (more epochs).

Furthermore, current scaling laws do not account for other well-known phenomena in learning, such as double descent (Belkin et al., ), in which model performance exhibits non-monotonic changes with respect to training data volume, model size, and training time. In particular, double descent theory predicts that test error should increase rapidly at the interpolation threshold, the point at which the model interpolates the training set (Nakkiran et al., ; Advani & Saxe, ). Like scaling laws, current theories of double descent leave several empirical phenomena unexplained: past explanations of double descent require it to occur, but empirically double descent is often not observed; it is unclear whether the interpolation threshold should grow or shrink with model size; prior theory does not explain why models in the infinite-parameter limit sometimes perform worse than their finite-parameter counterparts.

We seek the simplest possible unified framework in which to understand learning with respect to model size, data volume, and training time. In doing so, we aim to capture the essential scaling properties of learning in deep neural networks. Our results unify double descent with scaling laws and help to understand when models are sufficiently large for effective performance, how double descent is affected by varying training time and model size, and the variability and shape of loss curves across different problems.

The contributions of this paper are multi-fold:

- •
We theoretically and empirically demonstrate that scaling the size of a neural network is functionally equivalent to increasing its training time by a proportional factor.
- •
Leveraging this insight, we 1) predict the performance of large-scale networks using small-scale networks trained for many epochs and 2) predict the performance of networks trained for many epochs using the performance of large networks trained for one epoch.
- •
Using scale-time equivalence, we propose a unified scaling law for deep neural networks that provides a new explanation for parameter-wise double descent: double descent occurs when small models, which effectively train slower than larger models, acquire noisy data features.
- •
Through experiments conducted on standard vision benchmarks across multiple network architectures, we validate that our model explains several previously unexplained phenomena, including 1) the reduced data requirement for generalization in larger models, 2) the large impact of label noise on overparameterized models, 3) why error of overparameterized models often increases with scale.

## 2 Related work

### 2.1 Scaling laws

Neural network scaling laws describe how generalization error scales with data and model size. A number of works have observed power-law scaling with respect to data and model size (Kaplan et al., ; Rosenfeld et al., ; Clark et al., ), which has been explained theoretically (Bahri et al., ; Sharma & Kaplan, ; Hutter, ).

However, other work demonstrates that model scale may not be sufficient to predict model performance (Tay et al., ), and casts doubt on power laws as the best model of error rate scaling (Alabdulmohsin et al., ; Bansal et al., ; Mahmood et al., ). Moreover, scaling with respect to training time, holding data volume fixed, remains poorly understood. These observations highlight the need for a more general framework that can predict model performance under many settings.

### 2.2 Double descent

Double descent is an empirically observed phenomenon in which generalization error of machine learning models with respect to training data volume, model size, and training time exhibits an initial decrease, followed by a brief, sharp increase followed by a final decrease (Belkin et al., ; Nakkiran et al., ). Double descent with respect to model and data size has been theoretically understood as occurring due to a high degree of overfitting at the interpolation threshold, the point at which the model size is just sufficient to interpolate the training data (Adlam & Pennington, ; D’Ascoli et al., ; Belkin et al., ; Advani & Saxe, ). Typically, this work uses tools from random matrix theory to explain double descent for random feature models, in which linear regression maps a random, fixed feature pool to the desired output (Simon et al., ; Atanasov et al., ; Adlam et al., ; Bordelon et al., ; Maloney et al., ; Mei & Montanari, ).

However, these models do not explain double descent behavior in terms of training time. Originally, Nakkiran et al. (; ) hypothesized that training models for longer results in an effective increase in model size, thus allowing time-wise double descent to be explained in the same way as scale-wise double descent. Recent works have provided an alternate explanation: time-wise double descent occurs due to data features being learned at varying scales (Pezeshki et al., ; Heckel & Yilmaz, ; Stephenson & Lee, ). Error rises as models overfit to quickly-learned noisy features, but then falls as models more slowly learn signal features. In this work, we unify this explanation of time-wise double descent with the traditional account of double descent.

## 3 Scale-time equivalence in neural networks

In this section, we demonstrate that model size and training time may be traded off with each other. This result is consistent with and generalizes prior results demonstrating that models learn functions of increasing complexity over time (Nakkiran et al., ). We establish the result theoretically in a simplified model and validate it empirically in neural networks across several datasets and architectures.

### 3.1 Random subspace model

Following the lines of prior double descent analyses in random feature models, we construct a random subspace model to demonstrate scale-time equivalence theoretically. Consider a large $P$ dimensional model with parameters $\beta$, such that the function represented by the model depends only on a low-dimensional linear projection of $\beta$. This is reasonable for neural networks: it is well known that neural networks trained by stochastic gradient descent tend towards flat minima of their loss landscapes, thus revealing many redundant dimensions in the network (i.e. only a low-dimensional subspace affects the network output). Specifically, we denote the low-dimensional projection as $\alpha\in\mathbb{R}^{r}$ where $r<P$ which is constructed as:

$$ $\alpha=K\beta$ (1) $$

where $K\in\mathbb{R}^{r\times P}$ is a fixed projection matrix.

Assume that we can only control a random $p$-dimensional ($P>p>r$) linear subspace of the large model. This may again be a reasonable assumption for model classes such as neural networks: smaller neural networks can naturally be viewed as linear subspaces of larger neural networks that contain them architecturally (i.e. have a larger width and depth) (Hall & Li, ). We denote the parameters of our controllable $p$ dimensional subspace model as $\theta\in\mathbb{R}^{p}$, where:

$$ $\beta=R\theta+\beta_{0}$ (2) $$

for a random matrix $R\in\mathbb{R}^{P\times p}$ with elements drawn iid from a unit Gaussian and with $\beta_{0}\in\mathbb{R}^{P}$ fixed. We will represent time $t$ with subscript $t$ (i.e. $\beta_{0}$ denotes $\beta$ at time 0 0). Observe that any $p$ dimensional affine subspace of $\mathbb{R}^{P}$ may be represented in the form above for some choice of $R$ and $\beta_{0}$.

Under these assumptions, we can show that under gradient descent, increasing the scale $p$ of the model is equivalent to increasing training time:

###### Theorem 1 .

We denote the loss as a function of $\alpha$: $L\in\mathbb{R}^{r}\to\mathbb{R}$. Suppose $L$ has Lipschitz constant $l$ and its second derivative has Lipschitz constant $h$. Suppose that continuous time gradient flow is applied to $\theta$ with learning rate $\eta$ from initialization $\theta=0$. We denote $\alpha_{t}=K(R\theta_{t}+\beta_{0})$ where $\theta_{t}$ are the parameters at time $t$. Denote $A_{t}\in\mathbb{R}^{r}$ as the solution to:

$$ $\dot{A}_{t}=-\eta KK^{T}\nabla L(A_{t})$ (3) $$

with initial condition $A_{0}=K\beta_{0}$. Note that $A_{t}$ does not depend on $p$. Then, with probability $1-\epsilon$:

$$ $||\alpha_{t}-A_{pt}||\leq\frac{l\sqrt{||K||_{F}^{4}+||KK^{T}||_{F}^{2}}}{h \sqrt{p\epsilon}||KK^{T}||}(e^{\eta pth||KK^{T}||}-1)$ (4) $$

See Appendix A for a proof. The theorem implies closeness between the function implemented by the network, represented by $\alpha_{t}$, and another quantity $A_{pt}$ which only depends on the product $pt$. In other words, the learned model only depends on the product of the number of parameters $p$ and time $t$ (up to some error). Thus, we may interpret the product $pt$ as representing the distance of a model along the training trajectory; larger $pt$ implies more training progress. Increasing the number of parameters by a scale factor is equivalent to increasing the training time by the same scale factor and vice versa: scale is equivalent to time. Intuitively, this is because each parameter allows the function to learn at a fixed rate; thus, adding more parameters linearly increases the effective learning rate.

The result also reveals when such a scale-time equivalence cannot be made. The error bound implies that when training progress $pt$ is fixed, as $p$ grows, scale and time become increasingly equivalent (the bound approaches zero). The equivalence breaks down for small $p$ since here, the randomly chosen subspace of the model may or may not align well with $K$; as $p$ grows, the amount of alignment becomes less stochastic. Moreover, as training progress $pt$ grows, the bound grows exponentially because small perturbations to the model early in the training trajectory lead to exponentially larger changes later in the trajectory. Finally, we highlight that our result holds under standard neural network parameterizations in which the gradient of the model output with respect to each parameter does not scale with $p$; in other parameterizations such as Neural Tangent Kernel (Jacot et al., ), we may expect a different form of scale-time equivalence.

We first validate our prediction in a simple linear model in which a varying fraction of the model parameters are controllable. See Appendix C for details. In Figure 1, we find lines of 1:1 proportionality between model scale and the number of iterations required to reach a fixed loss level, validating our theory.

Figure: Figure 1: Proportional trade-off between model scale and training time: testing the prediction on a linear model. Red lines indicate tradeoff curves between number of training iterations and model size. Curves are computed by, for each model size, measuring the minimum amount of training time necessary to achieve different loss levels. Different curves indicate different performance thresholds; darker lines indicate a smaller error threshold. Margins indicate standard errors over $5$ trials. Grey dashed lines represent 1:1 proportionality between scale and training iterations.
Refer to caption: extracted/5841354/plots/tradeoff_toy.png

Figure: (a) MNIST, CNN
Refer to caption: extracted/5841354/plots/tradeoff_mnist_cnn_sgd_0.0.png

###### Theorem 1 .

### 3.2 Empirical validation in neural networks

We next turn to examine whether scale-time equivalence is present empirically in neural networks. We conduct experiments on MNIST (Deng, ), CIFAR-10 (Krizhevsky, ), and SVHN (Goodfellow et al., ) training a 7-layer convolutional neural network (CNN) and a 6-layer multilayer perception (MLP) with stochastic gradient descent (SGD). To assess scale-time equivalence, we measure the minimum amount of training time required to achieve non-zero generalization under various network widths and by varying the dataset size by subsampling. Scale-time equivalence predicts that wider networks will require less time to generalize in a systematically predictable way. See Appendix C for further experimental details.

As observed in Figure 2, in all settings, we see a clear tradeoff curve between scale and training time: increasing scale by a fixed factor is nearly equivalent to reducing training time by another fixed factor. Importantly, the scale here is set as the effective number of network parameters, defined as the maximum number of training points that can be fit by the network, not the absolute number of parameters. We set the effective parameter count as the cube root of the number of parameters. Appendix B provides a heuristic argument for this scaling rate.

Under this choice of scale, we find a systematic and predictable relationship between scale and training time, demonstrating that scale-time equivalence can be empirically observed. We also note that this phenomenon has been observed in prior literature (although not quantitatively characterized); for instance, Nakkiran et al. () find that patterns of double descent are similar with respect to training epochs and network size. We emphasize that these results are limited to gradient descent (the setting of our theory). With Adam optimization (Kingma & Ba, ), the number of epochs for generalization first decreases with scale, then increases (see Appendix D Figure 7). We hypothesize that since Adam has an adaptive learning rate, it is using a smaller effective learning rate for very large networks. As learning rate shrinks, more epochs are needed to generalize, leading to the observed results.

Figure: Figure 3: By scale-time equivalence, small models trained for long times predict performance of large models trained for small times and vice versa: test of prediction. Predicted and true test and train error of a CNN (top row) and MLP (bottom row) trained on MNIST. Column 1: predicting the performance of larger models over a few epochs by training smaller models for up to $100$ epochs. Column 2: predicting performance of smaller models over many epochs by training larger models for $1$ epoch. We use scale-time equivalence to predict the equivalent scale or number of epochs for each prediction. Margins indicate standard errors over 5 trials.
Refer to caption: extracted/5841354/plots/prederrorvsscale_mnist_cnn.png

## 4 Predicting optimal network scale and training time

Scale-time equivalence suggests it should be possible to predict the performance of large models by training small models for many epochs and vice versa. This allows us to predict the optimal network scale and training time for a given dataset and base architecture.

On benchmark datasets and architectures, we conduct two experiments: 1) predict performance under varying model scales from a small network trained for long training times, 2) predict performance over long training times by using larger networks trained for just 1 epoch. See Appendix C for experimental details and Figures 10 and 11 for full results.

Figure 3 illustrates that scale-time equivalence can indeed be used to extrapolate performance on large scales and training times. Predictions of test and train performance under large model scales are particularly close to the true performance; notably, we can closely predict the scale at which generalization starts to occur. However, there is a small discrepancy between predicted and actual performance; we believe this can be corrected with dataset and model-specific tuning of the scale-time trade-off curve. Nevertheless, our findings reveal that scale-time equivalence can be used to predict optimal network scale and training time.

## 5 A unified view of double descent w.r.t. training time, model scale and training set size

We next leverage scale-time equivalence to obtain a more-unified understanding of the phenomenon of double descent, with respect to training time, parameter count and training set size.

Figure: (a) Loss trajectories
Refer to caption: extracted/5841354/plots/toy_vs_p_noisy.png

### 5.1 Error scaling over time in a linear model

Following the approach of Pezeshki et al. (); Heckel & Yilmaz (); Stephenson & Lee (); Schaeffer et al. (), we first present a simple linear model that explains double descent with respect to time.

Consider a linear student-teacher setting in which training set outputs $Y\in\mathbb{R}^{n}$ are constructed as:

$$ $Y=Xw+\varepsilon$ (5) $$

where training data $X\in\mathbb{R}^{n\times m}$, noise $\varepsilon\in\mathbb{R}^{n}$, $w\in\mathbb{R}^{m}$ is the unknown true model, $n$ is the number of training points and $m$ is the model dimensionality. We assume $w$ is drawn independently from $\varepsilon$. Then, the parameters $\theta_{t}$ of a linear model learned after $t$ time of gradient flow on mean squared error (with learning rate $\eta$) can be expressed as:

$$ $\theta_{t}=X^{\dagger}(I-e^{-\eta XX^{T}t})Y$ (6) $$

The resulting prediction error $x^{T}\theta_{t}-x^{T}w$ on a test point $x$ can be expressed as:

$$ $x^{T}[X^{\dagger}(I-e^{-\eta XX^{T}t})X-I]w+x^{T}X^{\dagger}(I-e^{-\eta XX^{T} t})\varepsilon$ (7) $$

We then use the singular value decomposition $U\Sigma V^{T}$ of $X$ to express the prediction error:

$$ $x^{T}V[\Sigma^{\dagger}(I-e^{-\eta\Sigma\Sigma^{T}t})\Sigma-I]V^{T}w+x^{T}V \Sigma^{\dagger}(I-e^{-\eta\Sigma\Sigma^{T}t})V^{T}\varepsilon\\ =\sum_{i=1}^{m}-(x^{T}V)_{i}(V^{T}w)_{i}e^{-\eta\sigma_{i}^{2}t}+(x^{T}V)_{i}( V^{T}\varepsilon)_{i}\frac{1-e^{-\eta\sigma_{i}^{2}t}}{\sigma_{i}}$ (8) $$

where $\sigma_{i}$ are the singular values of $X$, and we denote $\sigma_{i}=0$ for $i>n$ when $n<m$ (in this case, $\frac{1-e^{-\eta\sigma_{i}^{2}t}}{\sigma_{i}}$ denotes 0 0). Using the independence of $w$ and $\varepsilon$, we finally may simply express the expected squared prediction error as:

$$ $\mathbb{E}[(x^{T}w-x^{T}\theta_{t})^{2}]\\ =\mathbb{E}[(\sum_{i=1}^{m}S_{i}e^{-\eta\sigma_{i}^{2}t})^{2}]+\mathbb{E}[( \sum_{i=1}^{m}N_{i}\frac{1-e^{-\eta\sigma_{i}^{2}t}}{\sigma_{i}})^{2}]$ (9) $$

where $S_{i}=-(x^{T}V)_{i}(V^{T}w)_{i}$, $N_{i}=(x^{T}V)_{i}(V^{T}\varepsilon)_{i}$. The first, signal term captures how well $w$ can be learned in the absence of noise.
In the underparameterized regime ($n>m$), this term approaches 0 0 as $t\to\infty$: without noise, the model can be learned perfectly. Observe that in general, the prediction error is not predicted to decay directly as a power law with $t$: instead, it decays or grows following a combination of exponential curves (though a combination of exponential decays at different rates can mimic a power law (Reed & Hughes, )).

The second, noise term initially starts at 0 0 and grows over time. Notably, the size of the noise term is largest near the interpolation threshold (when $m\approx n$) since the smallest singular values $\sigma_{i}$ will take small, non-zero values. As $n$ or $m$ grows larger than the other (distance from the interpolation threshold increases), the size of the noise term decreases. However, the noise term’s magnitude monotonically increases with $t$.

For any given singular component of $X$, if noise components are large relative to signal components (i.e. $|N_{i}|>|S_{i}|$), we expect that error will increase at the corresponding timescale, around $t=\frac{1}{\eta\sigma_{i}^{2}}$ and vice versa. Thus, if small singular value components are noisy, error will increase later during training, while if large singular value components are noisy, error will increase early during training. Double-descent occurs when noisy components are acquired first (increase in error) followed by signal components (decrease in error). Figure 4(a) illustrates how different acquisition rates of noise vs. signal components may yield different loss trajectories, with some corresponding to double descent. We may also view these trends geometrically in the parameter space; Figure 4(b) shows that if signal and noise correspond to orthogonal parameter dimensions, then double descent (orange curve) corresponds to a setting in which noise dimensions are learned rapidly before signal dimensions. The optimal training time depends on which point in the parameter trajectory is closest to the true optimum and may occur either at the end of training or at an intermediate point.

Figure: Figure 5: Larger models require less data to interpolate: test of prediction. Test and train error of CNN models trained on MNIST under varying levels of data. Different curves indicate different model scales; darker colors indicate larger models. Margins indicate standard errors over 5 trials.
Refer to caption: extracted/5841354/plots/errorvsdata_varyscale_mnist_cnn.png

### 5.2 Early noise acquisition can explain parameter-wise double descent

Next, we hypothesize that due to scale-time equivalence, parameter-wise double descent occurs due to the same mechanism as time-wise double descent. Namely, in settings where noise is acquired before signal, smaller scale models (which require effectively larger training time by scale-time equivalence) fail to acquire signal.

More precisely, by combining Equation 9 and scale-time equivalence, we propose the following scaling law in terms of the number of parameters $p$ and time $t$ by substituting $t$ with $pt$:

$$ $\textit{error}^{2}=\mathbb{E}[(\sum_{i=1}^{\infty}S_{i}e^{-\eta\sigma_{i}^{2} pt})^{2}]+\mathbb{E}[(\sum_{i=1}^{\infty}N_{i}\frac{1-e^{-\eta\sigma_{i}^{2}pt }}{\sigma_{i}})^{2}]$ (10) $$

where $\sigma_{i}$ depends on the number of training points $n$ and $S_{i}$ and $N_{i}$ are random variables determining the strength of signal and noise respectively. This scaling law simultaneously explains double descent in terms of both $p$ and $t$. Moreover, it can explain double descent in terms of data volume $n$ as well: if $n$ is set such that there are several small non-zero values of $\sigma_{i}$, the noise term becomes amplified. This explanation for double descent in $n$ follows prior literature (Advani & Saxe, ).

The explanation for parameter-wise double descent differs from conventional wisdom in which double descent occurs due to the same reason as double descent in $n$: namely, when the number of model parameters is close to $n$, the model is highly sensitive to noisy directions in the training data (corresponding to small $\sigma_{i}$ in a linear model), thus severely overfitting. How can we distinguish this hypothesis from ours? We propose three tests to separate the two hypotheses.

### 5.3 Less data required for generalization with model scale

The interpolation threshold can be defined as the point when the number of data points equals the effective complexity of a model (Nakkiran et al., ). Thus, for any model, the location of the interpolation threshold with respect to data volume is the model’s effective capacity: higher capacity models have a rightward-shifted interpolation threshold with respect to data volume. Conventional double descent theory argues that this point corresponds to a sharp decrease in the test set error as the amount of data grows. Starting from zero data, we would therefore expect that larger models (which have a larger effective capacity), would experience a sharp decrease in test set error at higher data volumes: larger models require more data to generalize. By contrast, under our explanation, as model size grows, the amount of data needed to generalize decreases: since model size corresponds to training time, generalization occurs more easily with larger models (equivalently, more training time).

To test this, we conduct experiments on benchmark datasets and architectures. Figure 5
reveals that larger models indeed require less data to generalize, thus supporting our hypothesis. (See Appendix C for experimental details and Figure 8 for full results.) Indeed, the training set errors decrease with data volume which is at odds with conventional double descent theory in which more data is always harder to interpolate. Fundamentally, this is because conventional double descent theory typically assumes complete training convergence, which does not explain phenomena under the practically relevant setting of a fixed training budget. In summary, the increased ease of generalization with model scale does not neatly fit into standard theories of double descent, but readily fits in our explanation.

Figure: (a) MNIST, MLP
Refer to caption: extracted/5841354/plots/errorvstime_mnist_mlp.png

### 5.4 Persistent vs local effects of noise on error curves

Another key distinction is how the two explanations behave under varying levels of noise. Under the conventional explanation, as the noise ($|N_{i}|$) increases, the error increases mostly locally around $p\approx n$, near the interpolation threshold. This is because in Equation 9, the noise coefficients $N_{i}$ multiply terms that are largest near the interpolation threshold and decrease as either data volume or model scale grow larger than the other (Schaeffer et al., ). In contrast, under our hypothesis, as the noise increases, the error monotonically grows with both model scale and time (see Equation 10). In other words, when the noise level changes, our hypothesis predicts a global performance change with respect to model scale while the conventional explanation predicts a primarily local change.

We conduct experiments on benchmark datasets and architectures to validate this. We add label noise to the training set and evaluate performance as a function of epochs, model scale and data size. Appendix C includes experimental details. Noise leads to an increase in error in all cases, Figure 6
(See Appendix D for additional plots), with a persistent or growing error with training time (epochs) and model scale. However, it leads to only localized increases in error in terms of the dataset size: in the case of MNIST and SVHN, for instance, there is nearly no change in error at the highest data volume.

### 5.5 Shape of loss curve: upturn with model size

A final key distinction between the two explanations is their prediction of the shape of the loss curve. Under the conventional explanation of double descent, past the interpolation threshold, increasing model size always improves performance. However, under our hypothesis, increasing model size may in certain cases worsen performance, if signal features are learned before noisy features. In Figure  6, we find that CIFAR-10 and SVHN indeed exhibit a U-shaped error curve with respect to model scale such that larger models do worse. This departure from a conventional double descent curve is consistent with our model.

## 6 Discussion

In this work, we demonstrate that model scale and training time can be traded off with each other. This enables us to re-frame parameter-wise double descent as occurring due to the same mechanism as epoch-wise double descent, which occurs due to the early acquisition of noise features during training. This framing of parameter-wise double descent has a number of surprising implications unexplained by standard explanations of double descent: 1) generalization requires less data with a larger model, 2) label noise significantly increases test error even for highly overparameterized models, 3) increasing model scale for overparameterized models need not always improve performance.

How can we reconcile our findings with conventional theory and experimental findings on double descent and scaling laws? Note that in past work, the presence of double descent with respect to model scale is often dependent on the choice of dataset, model, and whether label noise is added (Nakkiran et al., ). While past explanations of parameter-wise double descent necessitate that it must occur, our explanation is more flexible: double descent need not occur if noisy features are acquired later in training than signal features. Indeed, our theory explains why local increases in test error in terms of model scale are often relatively modest in contrast to the sharp spike predicted by conventional theory (Belkin et al., ). Thus, our explanation is more consistent with the variability of double descent observed in the literature.

Regarding scaling laws, our predicted scaling law in Equation 10 is more flexible than the power law scalings predicted in prior literature; indeed, with the proper settings of signal and noise parameters $S_{i}$ and $N_{i}$, we may recover power-law scalings of error with respect to time, model scale and data volume. However, our approach retains the flexibility to explain error scalings in settings where a power law does not explain empirical error trends, as we see in our experimental results.

Given the parametric flexibility of our scaling law, how can we use it to predict performance as model scale or training time increases? We demonstrate that by scale-time equivalence, performance under varying training times can be used to predict performance under varying model scales and vice versa. Our approach eliminates the need for strong parametric assumptions in the form of scaling law to make extrapolation predictions. This is particularly useful in cases where error increases with scale for overparameterized models since our approach can be used to predict an optimal model size. On the other hand, we require empirically evaluating the performance of models (under either a smaller scale or lower training time). We believe our approach can be valuable to practitioners who have the flexibility to run some limited empirical small-scale experiments before full-scale training.

Finally, our results suggest that smaller models trained for a longer time may behave as well as larger models, which we empirically observe on vision benchmarks. This is particularly important in the age of LLMs, where very large models are trained for a small number of epochs (often just one). Although we do not consider the language domain in this work, we believe it is a critical future direction to apply our findings. If smaller models trained for longer could achieve similar performance to large models, this would have the potential to make LLM training and fine-tuning vastly more accessible to users. We hope future work can explore this exciting possibility.

## Appendix A Proof of Theorem 1

First, observe that gradient descent applied to $\theta$ corresponds to:

$$ $\dot{\theta}=-\eta R^{T}K^{T}\nabla L(\alpha)$ (11) $$

This yields a change in $\alpha$ of:

$$ $\dot{\alpha}=-\eta KRR^{T}K^{T}\nabla L(\alpha)$ (12) $$

#### Properties of K ⁢ R ⁢ R T ⁢ K T 𝐾 𝑅 superscript 𝑅 𝑇 superscript 𝐾 𝑇 KRR^{T}K^{T} italic_K italic_R italic_R start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_K start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT

First, we investigate the properties of the $r\times r$ random matrix $KRR^{T}K^{T}$. Since the elements of $R$ are drawn iid from a unit Gaussian, $\mathbb{E}[RR^{T}]=pI$. Thus,

$$ $\mathbb{E}[KRR^{T}K^{T}]=pKK^{T}$ (13) $$

Next, consider the element at the $i$th row and $j$th column of $KRR^{T}K^{T}$. This may be expressed as:

$$ $K_{i,:}RR^{T}K^{T}_{:,j}=\sum_{l}K_{i,:}R_{:,l}R^{T}_{l,:}K^{T}_{:,j}=\sum_{l} (K_{i,:}R_{:,l})(K_{j,:}R_{:,l})$ (14) $$

where we express different columns of $R$ as $R_{:,l}$. Note that each term in the summand is independent from one another since $R$ has iid elements. Observe that $K_{i,:}R_{:,l}=\sum_{k}K_{i,k}R_{k,l}$ and each term $K_{i,k}R_{k,l}$ is an independent mean zero Gaussian with variance $K^{2}_{i,k}$. Thus, $K_{i,:}R_{:,l}$ is a mean zero Gaussian with variance $K_{i,:}K^{T}_{:,i}$. Moreover, since the expectation of $K_{i,:}RR^{T}K^{T}_{:,j}$ is $pK_{i,:}K^{T}_{:,j}$, $K_{i,:}R_{:,l}$ and $K_{j,:}R_{:,l}$ are jointly Gaussian with covariance $K_{i,:}K^{T}_{:,j}$.

Now, consider the expectation of $(K_{i,:}RR^{T}K^{T}_{:,j})^{2}$:

$$ $\mathbb{E}[(K_{i,:}RR^{T}K^{T}_{:,j})^{2}]=\mathbb{E}[\sum_{l}(K_{i,:}R_{:,l}) (K_{j,:}R_{:,l})\sum_{l^{\prime}}(K_{i,:}R_{:,l^{\prime}})(K_{j,:}R_{:,l^{ \prime}})]$ (15) $$

since $(K_{i,:}R_{:,l})(K_{j,:}R_{:,l})$ and $(K_{i,:}R_{:,l^{\prime}})(K_{j,:}R_{:,l^{\prime}})$ are independent for $l\neq l^{\prime}$:

$$ $\mathbb{E}[(K_{i,:}RR^{T}K^{T}_{:,j})^{2}]=\mathbb{E}[\sum_{l}(K_{i,:}R_{:,l}) ^{2}(K_{j,:}R_{:,l})^{2}]+p(p-1)\mathbb{E}[(K_{i,:}R_{:,l})(K_{j,:}R_{:,l})]^{2}$ (16) $$

$\mathbb{E}[(K_{i,:}R_{:,l})(K_{j,:}R_{:,l})]$ is simply $K_{i,:}K_{:,j}^{T}$. Since $K_{i,:}R_{:,l}$ and $K_{j,:}R_{:,l}$ are jointly Gaussian, we may reparameterize them as:

$$ $K_{i,:}R_{:,l}=az_{i}$ (17) $$

$$ $K_{j,:}R_{:,l}=bz_{i}+cz_{j}$ (18) $$

where $z_{i}$ and $z_{j}$ are independent unit Gaussians, $a=\sqrt{K_{i,:}K^{T}_{:,i}}$, $b=\frac{K_{i,:}K_{:,j}^{T}}{a}$, $c=\sqrt{K_{j,:}K^{T}_{:,j}-b^{2}}$. Then, $(K_{i,:}R_{:,l})^{2}(K_{j,:}R_{:,l})^{2}$ may be expressed as:

$$ $(K_{i,:}R_{:,l})^{2}(K_{j,:}R_{:,l})^{2}=(az_{i})^{2}(bz_{i}+cz_{j})^{2}=a^{2} z_{i}^{2}(bz_{i}+cz_{j})^{2}=a^{2}b^{2}z_{i}^{4}+2a^{2}bcz_{i}^{3}z_{j}+a^{2}c ^{2}z_{i}^{2}z_{j}^{2}$ (19) $$

Taking the expectation:

$$ $\mathbb{E}\left[a^{2}b^{2}z_{i}^{4}+2a^{2}bcz_{i}^{3}z_{j}+a^{2}c^{2}z_{i}^{2} z_{j}^{2}\right]=a^{2}b^{2}\mathbb{E}[z_{i}^{4}]+2a^{2}bc\mathbb{E}[z_{i}^{3}z _{j}]+a^{2}c^{2}\mathbb{E}[z_{i}^{2}z_{j}^{2}]$ (20) $$

Using the moments of unit Gaussians, we have $\mathbb{E}[z_{i}^{4}]=3$, $\mathbb{E}[z_{i}^{3}z_{j}]=0$, $\mathbb{E}[z_{i}^{2}z_{j}^{2}]=1$. Thus, we have

$$ $\mathbb{E}[(K_{i,:}R_{:,l})^{2}(K_{j,:}R_{:,l})^{2}]=3a^{2}b^{2}+a^{2}c^{2}=a^ {2}(3b^{2}+c^{2})\\ =a^{2}(K_{j,:}K^{T}_{:,j}+2\frac{(K_{i,:}K_{:,j}^{T})^{2}}{a^{2}})=K_{i,:}K^{T }_{:,i}K_{j,:}K^{T}_{:,j}+2(K_{i,:}K_{:,j}^{T})^{2}$ (21) $$

Finally, we may express the expectation of $(K_{i,:}RR^{T}K^{T}_{:,j})^{2}$ as:

$$ $\mathbb{E}[(K_{i,:}RR^{T}K^{T}_{:,j})^{2}]=p(K_{i,:}K^{T}_{:,i}K_{j,:}K^{T}_{: ,j}+2(K_{i,:}K_{:,j}^{T})^{2})+p(p-1)(K_{i,:}K_{:,j}^{T})^{2}$ (22) $$

#### Decomposition into expectation and noise

We now decompose $KRR^{T}K^{T}$ into an expectation term and a mean-zero noise term:

$$ $KRR^{T}K^{T}=pKK^{T}+N$ (23) $$

where $N\in\mathbb{R}^{r\times r}$ is a mean-zero matrix. Note that each element of $N$ has variance:

$$ $\mathbb{E}[N_{i,j}^{2}]=\mathbb{E}[(K_{i,:}RR^{T}K^{T}_{:,j})^{2}]-p^{2}(K_{i, :}K^{T}_{:,j})^{2}=p(K_{i,:}K^{T}_{:,i}K_{j,:}K^{T}_{:,j}+(K_{i,:}K_{:,j}^{T}) ^{2})$ (24) $$

Thus, the squared Frobenius norm of $N$ has expectation:

$$ $\mathbb{E}[||N||_{F}^{2}]=\sum_{i,j}\mathbb{E}[N_{i,j}^{2}]=p\sum_{i,j}K_{i,:} K^{T}_{:,i}K_{j,:}K^{T}_{:,j}+(K_{i,:}K_{:,j}^{T})^{2}=p(||K||_{F}^{4}+||KK^{T }||_{F}^{2})$ (25) $$

We express the dynamics of $\alpha$ as:

$$ $\dot{\alpha}=-\eta pKK^{T}\nabla L(\alpha)-\eta N\nabla L(\alpha)$ (26) $$

Now suppose we have two copies of $\alpha$: one copy ($\alpha^{(1)}$) with noise-free dynamics, and a second copy with noise:

$$ $\dot{\alpha}^{(1)}=-\eta pKK^{T}\nabla L(\alpha^{(1)})$ (27) $$

$$ $\dot{\alpha}^{(2)}=-\eta pKK^{T}\nabla L(\alpha^{(2)})-\eta N\nabla L(\alpha^{ (2)})$ (28) $$

We define the discrepancy between them as $\delta=\alpha^{(2)}-\alpha^{(1)}$, which has dynamics:

$$ $\dot{\delta}=-\eta pKK^{T}[\nabla L(\alpha^{(2)})-\nabla L(\alpha^{(1)})]-\eta N \nabla L(\alpha^{(2)})$ (29) $$

Consider the rate of change of the $\ell_{2}$ norm of $\delta$:

$$ $\frac{d}{dt}||\delta||\leq||\dot{\delta}||\leq\eta p||KK^{T}[\nabla L(\alpha^{ (2)})-\nabla L(\alpha^{(1)})]||+\eta||N\nabla L(\alpha^{(2)})||$ (30) $$

Using the Lipschitz bounds on $\nabla L$ and $L$:

$$ $\frac{d}{dt}||\delta||\leq\eta ph||KK^{T}||||\delta||+\eta||N||l$ (31) $$

where the matrix norms in the expression denote $\ell_{2}$ operator norm.

Now, returning to $||N||_{F}^{2}$, note that by Markov’s inequality, with probability $1-\epsilon$:

$$ $||N||_{F}^{2}\leq\frac{p}{\epsilon}(||K||_{F}^{4}+||KK^{T}||_{F}^{2})$ (32) $$

Using the fact that $||N||\leq||N||_{F}$, with probability $1-\epsilon$:

$$ $\frac{d}{dt}||\delta||\leq\eta ph||KK^{T}||||\delta||+\eta\frac{\sqrt{p}}{ \sqrt{\epsilon}}\sqrt{||K||_{F}^{4}+||KK^{T}||_{F}^{2}}l$ (33) $$

This is a differential inequality in $||\delta||$. Observe that $||\delta||$ takes its maximum possible trajectory at equality. Assuming $||\delta||=0$ at time $t=0$, this differential inequality may be solved by simply solving the equality case and setting the solution as the upper bound on $\delta_{t}$ (with subscript denoting time $t$):

$$ $||\delta_{t}||\leq\frac{\eta\frac{\sqrt{p}}{\sqrt{\epsilon}}\sqrt{||K||_{F}^{4 }+||KK^{T}||_{F}^{2}}l}{\eta ph||KK^{T}||}(e^{\eta ph||KK^{T}||t}-1)=\frac{ \sqrt{||K||_{F}^{4}+||KK^{T}||_{F}^{2}}l}{\sqrt{p\epsilon}h||KK^{T}||}(e^{\eta pth ||KK^{T}||}-1)$ (34) $$

#### Solving the noise-free dynamics

Now, we consider the noise-free dynamics:

$$ $\dot{\alpha}^{(1)}=-\eta pKK^{T}\nabla L(\alpha^{(1)})$ (35) $$

Note that $A_{pt}$ solves the dynamical equation for $\alpha^{(1)}$ by the chain rule:

$$ $\frac{d}{dt}A_{pt}=p(-\eta KK^{T}\nabla L(A_{pt}))=-\eta pKK^{T}\nabla L(A_{pt})$ (36) $$

Thus, we may simply express $\alpha_{t}^{(1)}$ as $A_{pt}$.

#### Final result

Combining and reexpressing our previous results, we may write that with probability $1-\epsilon$:

$$ $||\alpha^{(2)}_{t}-A_{pt}||=||\alpha^{(2)}_{t}-\alpha^{(1)}_{t}||=||\delta_{t} ||\leq\frac{l\sqrt{||K||_{F}^{4}+||KK^{T}||_{F}^{2}}}{h\sqrt{p\epsilon}||KK^{T }||}(e^{\eta pth||KK^{T}||}-1)$ (37) $$

Thus, the true dynamics deviate from $A_{pt}$ by a bounded amount.

## Appendix B Measuring Effective Parameter Count

In this section, we aim to provide a justification for why the cube root of the absolute number of parameters of a network is a good proxy for the effective number of parameters. Finding the effective number of parameters requires defining the parameter count at which the interpolation threshold occurs; this number is the effective parameter count. Unfortunately, in most models, training data is not perfectly interpolated by the model; thus defining the interpolation threshold location exactly is nontrivial. Instead, we identify a property of the interpolation threshold in the training error of linear models, then identify at which parameter count this property also holds in nonlinear models.

#### Training error in linear models

Suppose we are provided a training set $X\in\mathbb{R}^{n\times p}$ and corresponding training labels $Y\in\mathbb{R}^{n}$ where $n$ is the number of data points. Assume that $X$ has elements drawn uniformly from a unit Gaussian. We aim to find a parameter $\theta\in\mathbb{R}^{p}$ such that:

$$ $Y\approx X\theta$ (38) $$

The minimum norm solution minimizing the mean squared error is:

$$ $\theta=X^{\dagger}Y$ (39) $$

where $\dagger$ denotes pseudoinverse. Denoting the predicted training labels as $\hat{Y}$, the mean squared error is then:

$$ $\frac{1}{n}||Y-\hat{Y}||_{2}^{2}=\frac{1}{n}||Y-XX^{\dagger}Y||_{2}^{2}=\frac{ 1}{n}Y^{T}(I-XX^{\dagger})Y$ (40) $$

Finally, assuming that $Y_{i}^{2}=1$ for all $i$, we may write the mean training error as:

$$ $\frac{1}{n}||Y-\hat{Y}||_{2}^{2}=\max(1-\frac{p}{n},0)$ (41) $$

Thus, for a linear model, the training error decreases linearly at rate $-\frac{1}{n}$ with respect to $p$ before the interpolation threshold ($p=n$), and then is zero after the interpolation threshold. We extract a key property around the interpolation threshold from the linear model: the training error is $\frac{1}{n}$ at $p=n-1=O(n)$. Note that for any $\alpha<1$, the training error is $\frac{1}{n^{\alpha}}$ before $O(n)$.

#### Power law training rate decay

Next, we consider power-law decays of training rate error and characterize which power laws are consistent with the interpolation threshold properties outlined above. Consider a power law decay of the mean training error of $\frac{n^{\alpha}}{p^{\beta}}$. Note that in order to satisfy condition (2), we must have:

$$ $\frac{1}{n}=O(\frac{n^{\alpha}}{n^{\beta}})$ (42) $$

where we set $p=O(n)$. Thus, $\beta-\alpha=1$. Power-law decays of the form $\frac{n^{\alpha}}{p^{\alpha+1}}$ are consistent with the interpolation threshold property.

#### Error scaling in neural networks

Next, we turn to model training error scaling in neural networks. We make the following heuristic argument: to fit $n$ training points, a network of width $m$ needs to encode $O(mn)$ numbers corresponding to $m+1$ numbers for each training point (to represent the features and labels of each training point). On the other hand, the network has $O(m^{2})$ parameters since its intermediate layer weights have $m^{2}$ parameters. We expect that scaling both the network’s capacity of $O(m^{2})$ and the required capacity of $O(mn)$ at the same rate will not change the training error. Thus, we expect the training error to be a function of $\frac{mn}{m^{2}}=\frac{n}{m}$.

To find which function of $\frac{n}{m}$ models the training error, we introduce another argument. We assume that the network can be approximated as an ensemble of $O(m^{2})$ submodels, each with $O(1)$ parameters. Suppose the model output is the average of the outputs of the submodels. Then, assuming that the distribution of the submodel outputs on any given training point has variance $O(1)$ over different model initializations, the variance of the ensemble output is $O(\frac{1}{m^{2}})$. Thus, over model initialization, the model output at any given training point will center around a mean value with deviations on the order of $O(\frac{1}{m})$. Assuming that as the network capacity goes to $\infty$, the training error goes to 0 0, the mean value of the predicted output on a training point must be the true value; thus, the predicted output differs from the true output by $O(\frac{1}{m})$. This corresponds to a mean squared error on the training points scaling as $O(\frac{1}{m^{2}})$. Finally, to get the dependence on $n$, we use the observation that the error must depend on only the fraction $\frac{n}{m}$. Thus, the mean squared training error scales as $O(\frac{n^{2}}{m^{2}})$.

Finally, we know by the argument above that power law error rate scaling of the form $\frac{n^{\alpha}}{p^{\alpha+1}}$ are consistent with the interpolation threshold property, where $p$ is the effective number of parameters. Equating this with the neural network scaling result of $O(\frac{n^{2}}{m^{3}})$, we must have that $\alpha=2$, yielding:

$$ $O(\frac{n^{2}}{p^{3}})=O(\frac{n^{2}}{m^{2}})$ (43) $$

Equating the denominators, we have $p^{3}=m^{2}$, or $p=m^{2/3}$. In other words, the effective number of parameters is $m^{2/3}$. Since the absolute number of parameters in the network is $m^{2}$, the effective number of parameters is the cube root of the absolute parameter count.

## Appendix C Experimental Details

### C.1 Model training

We used three benchmark datasets: CIFAR-10, MNIST, and SVHN. Each dataset was subject to preprocessing involving standard normalization. For CIFAR-10 and SVHN, the normalization was performed using means and standard deviations of (0.5, 0.5, 0.5). For MNIST, the normalization used a mean and standard deviation of 0.5.

Two types of neural network architectures were evaluated:

- •
Convolutional Neural Network (CNN): Our CNN architecture consisted of six ReLU activated convolutional layers, with kernel sizes (3, 3, 3, 3, 3, 2 for CIFAR-10/SVHN or 1 for MNIST), strides (2, 1, 2, 1, 1, 1), and number of filters (5s, 10s, 20s, 40s, 80s) where s is a width parameter. This was followed by a fully connected layer.
- •
Multilayer Perceptron (MLP): The MLP architecture comprised six fully connected layers with hidden layer width 10s where s is a scale parameter.

The width parameter was set to values of 1, 2, 5, 10, 20, 50, and 100. When we refer to ”model scale”, we quantify this as the cube root of the number of network parameters rather than the value of the width parameter. We tested two learning rates (0.001 and 0.01) in combination with two optimizers (Adam and SGD respectively). Label noise was introduced at levels of 0.0 (no noise) and 0.2 (20% noise) to evaluate the robustness of the models. The number of training samples was varied among 100, 200, 500, 1000, 2000, 5000, 10000, 20000, and 50000. The samples were randomly selected from their respective base training sets. A constant batch size of 32 and 100 training epochs were used across all experiments. Mean Squared Error (MSE) was used as the loss function. Model performance was assessed using Mean Squared Error (MSE) on both training and test sets. For reproducibility, we set a manual seed for the PyTorch random number generator. Five different seeds (101, 102, 103, 104, 105) were used to assess the variance in results due to random initialization.

### C.2 Scale-time tradeoff validation

#### Linear model

We construct a loss function as:

$$ $L(\theta)=||K(R\theta+\beta_{0})-\alpha^{*}||^{2}$ (44) $$

where $\alpha^{*}$ is a target value of $\alpha$. $K$, $R$, $\beta_{0}$ and $\alpha^{*}$ are all independently sampled from unit Gaussians, and $\theta$ is initialized from a unit Gaussian. We set $P=1000$ and $r=3$. We train $\theta$ via gradient descent with learning rate $10^{-6}$ and evaluate the number of iterations required to reach various values of the loss for varying values of $p$. This setup corresponds to a linear classifier trained with gradient descent on mean squared error using a training batch of $k=3$ points where only $p$ of the $P=1000$ parameters are controllable.

#### Neural networks

To evaluate the scale time tradeoff, for each model, we compute the minimum number of epochs necessary to achieve a test set MSE below $0.09$.

### C.3 Shifting interpolation threshold

Experimental results are shown for networks trained with SGD.

### C.4 Impact of noise and shape of loss curve

Experimental results are shown for networks trained with Adam; since Adam trains faster, this allows us to examine training trends at later points in training.

### C.5 Optimal scale predictions

To predict error over scale using error over time or vice versa, we treat the quantity $pt$ as a predictor of performance, where $p$ is the effective number of network parameters. Thus, if we wish to know the error of a network at $(p_{1},t_{1})$ and we know the error of the network for all $p_{0}$, then we may predict the error at $(p_{1},t_{1})$ as the error at $(p_{0},(\frac{p_{1}}{p_{0}})t_{1})$.

### C.6 Computing Infrastructure

Experiments were run on a computing cluster with GPUs ranging in memory size from 11 GB to 80 GB.

## Appendix D Additional Experiments

Figure: (a) MNIST, CNN
Refer to caption: extracted/5841354/plots/tradeoff_mnist_cnn_adam_0.0.png

Figure: (a) MNIST, CNN
Refer to caption: extracted/5841354/plots/errorvsdata_varyscale_mnist_cnn.png

Figure: (a) MNIST, CNN
Refer to caption: extracted/5841354/plots/errorvstime_mnist_cnn.png

Figure: (a) MNIST, CNN
Refer to caption: extracted/5841354/plots/prederrorvsscale_mnist_cnn.png

Figure: (a) MNIST, CNN
Refer to caption: extracted/5841354/plots/prederrorvstime_mnist_cnn.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 1
strengths: 1. The idea of functional equivalence between models at different scales is potentially very attractive for modern practices of neural network training, where the (training) compute-optimal trade-offs have to be balanced with the potential benefits of having a smaller model at inference time. 

2. The idea of formalizing the relationship between training time and scale is also interesting from a theoretical perspective. In gradient descent, training time has been linked to various forms of explicit regularization, and theoretically investigating its connection to model scale is especially interesting in the context of neural scaling laws.
summary: The paper put forward the hypothesis that increasing the training time model is in direct relationship with the model size: a small model trained for a long time is functionally equivalent to a larger model trained for a shorter time. The authors devise a simple theoretical model where this behavior can be analytically shown. They also experimentally show how this can be empirically verified in neural network training, and how their theory fits in the double descent phenomenon. The experimental results are achieved with MLPs and CNNs on computer vision benchmarks such as SVHN, CIFAR-10 and MNIST.
weaknesses: I think that despite the captivating and well-posed questions, the paper fails to deliver on several aspects. Namely:

1. The authors claim to theoretically demonstrate that scaling the size of a neural network is equivalent to increasing its training time. The connection between the model studied here and neural networks is unclear/misleading, and some design choices are not thoroughly explained. The theoretical model has the unusual structure of equation 2, which has no connection to theoretical models for neural networks. Also, the model is linear. Also, why are $r,p,P$ all different? Why couldn’t you only have a large dimensional ambient space and a constant-sized low-rank subspace? 

2. I would have appreciated it if the connection to neural networks were toned down, and the proposed model motivated in its own right. I do not understand why the authors haven't studied more established settings, where for instance the double descent phenomenon has been shown, or where scaling laws have been tractably derived. Starting from there, it would be nice to understand which (and why) additional assumptions have to be made. With this respect, why haven’t the authors approached the problem from more standard settings in neural network theory (e.g. teacher-student [1] or random feature models [2])?

3. The concept of *minimal capacity to fit the data to a certain loss level* is crucial here, but never satisfiably addressed. In fact, an underlying assumption that is never made by the authors is that the model can fit the data at any scale (only stating “Consider a large $P$ dimensional model”). Otherwise, it would not be possible for a small model to match the performances of a large model, no matter how long the small model has been trained (e.g. imagine training a model with very small width on complicated datasets). 

4. The definition of **model scale** as $N^{1/3}$ ($N$ is the number of parameters) put forward by the authors is a heuristic, which is not convincing enough given its importance in the experiments on neural networks. More concretely, the authors define the model scale as "the maximum number of training points that can be fit by the network", which is set to $N^{1/3}$. The number of training points that can be fit by a model is a well-studied and hard problem. For arbitrary data points, the complexity of a function class determines how many data points can be fit. In general, it depends on the data's complexity and the model's flexibility. I find that the cube root of the number of parameters is arbitrary and ignores the data dependence. As the authors detail in the “experimental details” Section in the appendix, *“The width parameter was set to values of 1, 2, 5, 10, 20, 50, and 100. When we refer to ”model scale”, we quantify this as the cube root of the number of network parameters rather than the value of the width parameter.”* In the current form, the scale-time equivariance narrative heavily relies on the cubic root assumption, and the heuristic argument provided is insufficient considering its importance in the paper’s narrative. 

5. In section 5.3, it is assumed that larger models have a smaller time to interpolation threshold. However, in the experiment of Figure 5, it appears that interpolation has not happened yet, as there are no U-shaped curves, which are the footprint of double descent. It would be nice to reproduce some of the results on double descent both in model size and training time to better substantiate some of the claims. In this sense, we would expect the interpolation threshold (peak of double descent curve) to vary with model size and training time predictably according to the scale-time equivariance theory. 

6. The authors only test two learning rates, which makes it impossible to establish whether the chosen learning rate is optimal. Also, usually the optimal learning rate shifts with scale, and this is not addressed by the authors. 

7. It is well-known that there is a correspondence between training time and regularization strength (e.g. [3]). The proposed theory put forward the interesting hypothesis of a similar connection between training time and model scale. I would say that mentioning this line of work would better help this paper’s positioning in the current literature. 

8. In lines 367-269 a few articles “the” missing: If noise components →  if the noise components, etc..


[1] A Dynamical Model of Neural Scaling Laws

[2] A Solvable Model of Neural Scaling Laws

[3] A Continuous-Time View of Early Stopping for Least Squares Regression (https://proceedings.mlr.press/v89/ali19a)

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: S3.1: I think this section in general is not very clear and could benefit from additional explanations and better arguments for the applicability to neural networks.

L130: Neural networks with SGD often converge to flat minima when trained with a high learning rate. However, I am not sure the projection K would be fixed throughout training nor that a gradient flow model can capture the same behavior.

L139: Here you give an example of how a small neural network can be seen as a subnetwork of a larger model. This would likely imply setting various weights to zero or one for identity and so on. This would likely not be a random matrix that looks anything like R.

L152: Why can you assume that eta is not a function of p? In actual neural network training the learning rate often varies with the width and depth of the network being trained (see e.g. various muP literature).

L209: What do you mean by “minimum amount of training time required to achieve non-zero generalization”?

L244: The cube root argument should also be justified in the main body along with its applicability to neural networks. The model this is derived for seems a lot closer to the previous theoretical model than neural networks so only applying it to neural networks feels odd.

L252:  Why not tune your learning rates in general? As mentioned before, the learning rate often varies with the network width and depth. Using a fixed rate seems unprincipled.

S4: This section is very light on details. Overall the paper feels a bit crammed, trying to cover more material than can be done well on 10 pages.
soundness: 2
strengths: - An interesting theoretical take on scaling and double descent.
- Mostly well written, clear figures.
summary: This paper analyzes a simple proxy model for neural network optimization, gradient flow on a random feature model. It theoretically shows that under certain conditions, particularly on the complexity of the objective function and Lipschitz continuity of the gradients and second derivative, the speed of convergence depends on the product of the model size and time. The authors interpret this result as an equivalence of time and scale in optimization. The paper experimentally explores whether this relation holds for very small scale MLPs and conv-nets on simple vision tasks like MNIST and CIFAR. They then use time-scale equivalence to offer additional insights into double descent in terms of training time, parameter count and dataset size.
weaknesses: - The claims made in the paper seem too strong. The abstract mentions theoretical characterization of deep neural networks multiple times but the paper only theoretically analyzes simple proxy models. It also mentions large scale neural networks but all experiments in the paper are performed on small MNIST / SVHN / CIFAR networks. The same goes for the claims in the contribution list in the introduction.
- The paper does not sufficiently discuss its limitations (e.g. in terms of the weaknesses mentioned here).
- The neural network experiments are done on very small scale vision tasks which is typically not where neural scaling laws are valuable.
- The applicability of the random feature model analysis to neural networks is not sufficiently justified.
- Some parts of the paper are not very clear (see below).

**Edit: The authors have promised changes that I believe would sufficiently address these weaknesses. Conditioned on those changes I no longer have significant concerns and have raised my score.**

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: What are the empirical details used to run Figure 4?

Conceptually, I am unsure of the claim in line 505: “model scale and training time can be traded off with each other.” Suppose I take an overparameterized model with $p$ parameters that is able to interpolate the data in a fixed number of training steps $t$. Based on this claim, I should be able to make $p$ much smaller, up to the point of making the model underparameterized, if I increase $t$ proportionally - however, in this case, my model will not be able to train to convergence. 

If the authors are open to discussion and to provide more details on their setup, I am willing to increase my score.
soundness: 2
strengths: The paper studies an interesting phenomenon, relevant to practitioners and in line with the current trend in literature of scaling laws: can we train smaller models on more data without sacrificing performance? The authors also test their claims empirically on multiple datasets and architectures.
summary: The paper aims to build a theoretical model for the scaling laws of neural networks as a function of model size, training time and data volume. The focus of the paper is to show the existence of “scale-time equivalence”, meaning that in the training of a neural network, one can trade off parameters for training smaller models on more data. The authors also analyse double descent in the framing of their linear model, proposing that epoch-wise double descent happens when small models learn noisy features first from the data.
weaknesses: To begin with, in Section 3.1, the authors introduce a random subspace model as a toy setting to study their scaling argument. Could the authors explain why this model is a good approximation for what happens in the case of a neural network, as well as provide citations to prior works using a similar model (i.e. in line 127)? For example where does the input data get used in this model? What is the loss used to train model? It would help for the authors to expand with more details the setup they are working with.

My biggest critique however is that it seems to me that the bound that the authors get in Equation (4) is vacuous, for the following reason. Naively, the rhs of (4) seems to be dominated by $e^{pt}$ term. Thus, if we increase the number of parameters (or the training time, or both), the distance between $\alpha_t$ and $A_{pt}$ is upper bounded by a quantity that grows exponentially fast. Could the authors explain more what they mean in this case? Following the proof, I do not understand where $A_{pt}$ appears from in Eq. (36). Could the authors please elaborate?

It would be very helpful if the authors could provide more information with respect to the empirics. Several plots tend to be vague i.e. “darker lines indicate a smaller error threshold” - what is the actual value?, “darker lines indicate more data” - same question.

Continuing to Section 4, Figure 3 Column 2 seems to have incomplete traces for the “predicted” (dashed) lines. Could the authors explain what happened in this case?

In Section 5, the analysis of the linear model seems very similar to the analysis done in the seminal work of Advani and Saxe. While the authors rightfully cite this work, it is not clear to me what their analysis provides on top of the existing work.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Apart from the points in weakness, here are my additional questions.

* In Figure 2, how is the batch size chosen? Would different choices of batch size affect the result?

* How is eq 10 derived from eq 9? My understanding is that M can be interpreted as the model size in linear models. Thus, eq 10 corresponds to the case $M\to\infty$ and it is unclear what the parameter size $p$ is.
soundness: 2
strengths: The paper provides novel explanations to double descent and other unexplained phenomena based on analysis of linear models.
summary: This work presents theoretical and empirical characterizations of how the model size, training time and data size affect the performance of neural networks. The authors first establish a scale-time equivalence in a linear model trained with gradient flow. They then conduct simulations on MNIST to verify their finding. Moreover,  the authors combine the scale-time equivalence with an analysis of the linear model to provide new interpretations of previously unexplained phenomena in double descent. Simulation results on CNN and MLP corroborate their interpretations.
weaknesses: The theoretical derivations in this work are limited to simple linear models. It is unclear to what extent these observations hold in large models and other optimization settings.

* For example, Theorem 1 assumes the model is trained via gradient flow (which is close to full-batch GD), while in practice it is recommended to use one-pass SGD/Adam. Theorem 1 seems to lack a characterization of how the sample size correlates with the training time. This differs from the setting under which the authors conduct the simulations (mini-batch SGD). 

* Also, the scaling laws (eq 9,10) are derived for linear models. However, the discussions in section 5.2-5.5 are for more general models, and are heuristically built upon these equations and some implicit assumptions on the signal and noise components. There lack precise and rigorous characterizations of the test error for nonlinear models/neural networks.

* Lack a discussion of more recent works on explaining neural scaling laws: e.g., [1,2]. In particular, [1] studied the scaling laws in linear model trained via one-pass sgd. I wonder how results in [1] reconcile with Theorem 1 in your work.

* I would appreciate an explicit discussion of the limitations of this work.





[1]. Lin, L., Wu, J., Kakade, S. M., Bartlett, P. L., & Lee, J. D. (2024). Scaling Laws in Linear Regression: Compute, Parameters, and Data. arXiv preprint arXiv:2406.08466.

[2]. Paquette, E., Paquette, C., Xiao, L., & Pennington, J. (2024). 4+ 3 Phases of Compute-Optimal Neural Scaling Laws. arXiv preprint arXiv:2405.15074.

Meta-review: This paper explores the concept of "scale-time equivalence," suggesting that smaller neural networks trained for extended periods can match the performance of larger models trained briefly, supported by a theoretical linear model and experiments on small-scale vision tasks. The work offers a potentially valuable perspective on training efficiency and suggests a novel explanation for double descent, improving our understanding of model scaling. However, the theoretical analysis is limited to a simplistic linear model that is weakly connected to real neural networks, the experimental validation relies on small-scale models and datasets, and the core assumption of scale-time equivalence hinges on a questionable definition of model scale, ultimately raising concerns about the paper's generalizability and practical applicability. I do not recommend acceptance at this time.

Decision: Reject