Unified Neural Network Scaling Laws and Scale-time Equivalence

Authors: Akhilan Boopathy, Ila R Fiete
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: generalization, neural network, scaling law, double descent

Abstract:
As neural networks continue to grow in size but datasets might not, it is vital to understand how much performance improvement can be expected: is it more important to scale network size or data volume? Thus, neural network scaling laws, which characterize how test error varies with network size and data volume, have become increasingly important. However, existing scaling laws are often applicable only in limited regimes and often do not incorporate or predict well-known phenomena such as double descent. Here, we present a novel theoretical characterization of how three factors --- model size, training time, and data volume --- interact to determine the performance of deep neural networks. We first establish a theoretical and empirical equivalence between scaling the size of a neural network and increasing its training time proportionally. Scale-time equivalence challenges the current practice, wherein large models are trained for small durations, and suggests that smaller models trained over extended periods could match their efficacy. It also leads to a novel method for predicting the performance of large-scale networks from small-scale networks trained for extended epochs, and vice versa. We next combine scale-time equivalence with a linear model analysis of double descent to obtain a unified theoretical scaling law, which we confirm with experiments across vision benchmarks and network architectures. These laws explain several previously unexplained phenomena: reduced data requirements for generalization in larger models, heightened sensitivity to label noise in overparameterized models, and instances where increasing model scale does not necessarily enhance performance. Our findings hold significant implications for the practical deployment of neural networks, offering a more accessible and efficient path to training and fine-tuning large models.

Decision: reject
Reviewer scores: 3, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730075937006

thread_type:
official_review

<thread_object>
creator: reviewer_w5rn

<content>
summary:
This work presents theoretical and empirical characterizations of how the model size, training time and data size affect the performance of neural networks. The authors first establish a scale-time equivalence in a linear model trained with gradient flow. They then conduct simulations on MNIST to verify their finding. Moreover,  the authors combine the scale-time equivalence with an analysis of the linear model to provide new interpretations of previously unexplained phenomena in double descent. Simulation results on CNN and MLP corroborate their interpretations.

strengths:
The paper provides novel explanations to double descent and other unexplained phenomena based on analysis of linear models.

weaknesses:
The theoretical derivations in this work are limited to simple linear models. It is unclear to what extent these observations hold in large models and other optimization settings.

* For example, Theorem 1 assumes the model is trained via gradient flow (which is close to full-batch GD), while in practice it is recommended to use one-pass SGD/Adam. Theorem 1 seems to lack a characterization of how the sample size correlates with the training time. This differs from the setting under which the authors conduct the simulations (mini-batch SGD). 

* Also, the scaling laws (eq 9,10) are derived for linear models. However, the discussions in section 5.2-5.5 are for more general models, and are heuristically built upon these equations and some implicit assumptions on the signal and noise components. There lack precise and rigorous characterizations of the test error for nonlinear models/neural networks.

* Lack a discussion of more recent works on explaining neural scaling laws: e.g., [1,2]. In particular, [1] studied the scaling laws in linear model trained via one-pass sgd. I wonder how results in [1] reconcile with Theorem 1 in your work.

* I would appreciate an explicit discussion of the limitations of this work.





[1]. Lin, L., Wu, J., Kakade, S. M., Bartlett, P. L., & Lee, J. D. (2024). Scaling Laws in Linear Regression: Compute, Parameters, and Data. arXiv preprint arXiv:2406.08466.

[2]. Paquette, E., Paquette, C., Xiao, L., & Pennington, J. (2024). 4+ 3 Phases of Compute-Optimal Neural Scaling Laws. arXiv preprint arXiv:2405.15074.

questions:
Apart from the points in weakness, here are my additional questions.

* In Figure 2, how is the batch size chosen? Would different choices of batch size affect the result?

* How is eq 10 derived from eq 9? My understanding is that M can be interpreted as the model size in linear models. Thus, eq 10 corresponds to the case $M\to\infty$ and it is unclear what the parameter size $p$ is.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730588544288

thread_type:
official_review

<thread_object>
creator: reviewer_wzg2

<content>
summary:
The paper aims to build a theoretical model for the scaling laws of neural networks as a function of model size, training time and data volume. The focus of the paper is to show the existence of “scale-time equivalence”, meaning that in the training of a neural network, one can trade off parameters for training smaller models on more data. The authors also analyse double descent in the framing of their linear model, proposing that epoch-wise double descent happens when small models learn noisy features first from the data.

strengths:
The paper studies an interesting phenomenon, relevant to practitioners and in line with the current trend in literature of scaling laws: can we train smaller models on more data without sacrificing performance? The authors also test their claims empirically on multiple datasets and architectures.

weaknesses:
To begin with, in Section 3.1, the authors introduce a random subspace model as a toy setting to study their scaling argument. Could the authors explain why this model is a good approximation for what happens in the case of a neural network, as well as provide citations to prior works using a similar model (i.e. in line 127)? For example where does the input data get used in this model? What is the loss used to train model? It would help for the authors to expand with more details the setup they are working with.

My biggest critique however is that it seems to me that the bound that the authors get in Equation (4) is vacuous, for the following reason. Naively, the rhs of (4) seems to be dominated by $e^{pt}$ term. Thus, if we increase the number of parameters (or the training time, or both), the distance between $\alpha_t$ and $A_{pt}$ is upper bounded by a quantity that grows exponentially fast. Could the authors explain more what they mean in this case? Following the proof, I do not understand where $A_{pt}$ appears from in Eq. (36). Could the authors please elaborate?

It would be very helpful if the authors could provide more information with respect to the empirics. Several plots tend to be vague i.e. “darker lines indicate a smaller error threshold” - what is the actual value?, “darker lines indicate more data” - same question.

Continuing to Section 4, Figure 3 Column 2 seems to have incomplete traces for the “predicted” (dashed) lines. Could the authors explain what happened in this case?

In Section 5, the analysis of the linear model seems very similar to the analysis done in the seminal work of Advani and Saxe. While the authors rightfully cite this work, it is not clear to me what their analysis provides on top of the existing work.

questions:
What are the empirical details used to run Figure 4?

Conceptually, I am unsure of the claim in line 505: “model scale and training time can be traded off with each other.” Suppose I take an overparameterized model with $p$ parameters that is able to interpolate the data in a fixed number of training steps $t$. Based on this claim, I should be able to make $p$ much smaller, up to the point of making the model underparameterized, if I increase $t$ proportionally - however, in this case, my model will not be able to train to convergence. 

If the authors are open to discussion and to provide more details on their setup, I am willing to increase my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643271388

thread_type:
official_review

<thread_object>
creator: reviewer_qkw1

<content>
summary:
This paper analyzes a simple proxy model for neural network optimization, gradient flow on a random feature model. It theoretically shows that under certain conditions, particularly on the complexity of the objective function and Lipschitz continuity of the gradients and second derivative, the speed of convergence depends on the product of the model size and time. The authors interpret this result as an equivalence of time and scale in optimization. The paper experimentally explores whether this relation holds for very small scale MLPs and conv-nets on simple vision tasks like MNIST and CIFAR. They then use time-scale equivalence to offer additional insights into double descent in terms of training time, parameter count and dataset size.

strengths:
- An interesting theoretical take on scaling and double descent.
- Mostly well written, clear figures.

weaknesses:
- The claims made in the paper seem too strong. The abstract mentions theoretical characterization of deep neural networks multiple times but the paper only theoretically analyzes simple proxy models. It also mentions large scale neural networks but all experiments in the paper are performed on small MNIST / SVHN / CIFAR networks. The same goes for the claims in the contribution list in the introduction.
- The paper does not sufficiently discuss its limitations (e.g. in terms of the weaknesses mentioned here).
- The neural network experiments are done on very small scale vision tasks which is typically not where neural scaling laws are valuable.
- The applicability of the random feature model analysis to neural networks is not sufficiently justified.
- Some parts of the paper are not very clear (see below).

**Edit: The authors have promised changes that I believe would sufficiently address these weaknesses. Conditioned on those changes I no longer have significant concerns and have raised my score.**

questions:
S3.1: I think this section in general is not very clear and could benefit from additional explanations and better arguments for the applicability to neural networks.

L130: Neural networks with SGD often converge to flat minima when trained with a high learning rate. However, I am not sure the projection K would be fixed throughout training nor that a gradient flow model can capture the same behavior.

L139: Here you give an example of how a small neural network can be seen as a subnetwork of a larger model. This would likely imply setting various weights to zero or one for identity and so on. This would likely not be a random matrix that looks anything like R.

L152: Why can you assume that eta is not a function of p? In actual neural network training the learning rate often varies with the width and depth of the network being trained (see e.g. various muP literature).

L209: What do you mean by “minimum amount of training time required to achieve non-zero generalization”?

L244: The cube root argument should also be justified in the main body along with its applicability to neural networks. The model this is derived for seems a lot closer to the previous theoretical model than neural networks so only applying it to neural networks feels odd.

L252:  Why not tune your learning rates in general? As mentioned before, the learning rate often varies with the network width and depth. Using a fixed rate seems unprincipled.

S4: This section is very light on details. Overall the paper feels a bit crammed, trying to cover more material than can be done well on 10 pages.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734162404

thread_type:
official_review

<thread_object>
creator: reviewer_s73m

<content>
summary:
The paper put forward the hypothesis that increasing the training time model is in direct relationship with the model size: a small model trained for a long time is functionally equivalent to a larger model trained for a shorter time. The authors devise a simple theoretical model where this behavior can be analytically shown. They also experimentally show how this can be empirically verified in neural network training, and how their theory fits in the double descent phenomenon. The experimental results are achieved with MLPs and CNNs on computer vision benchmarks such as SVHN, CIFAR-10 and MNIST.

strengths:
1. The idea of functional equivalence between models at different scales is potentially very attractive for modern practices of neural network training, where the (training) compute-optimal trade-offs have to be balanced with the potential benefits of having a smaller model at inference time. 

2. The idea of formalizing the relationship between training time and scale is also interesting from a theoretical perspective. In gradient descent, training time has been linked to various forms of explicit regularization, and theoretically investigating its connection to model scale is especially interesting in the context of neural scaling laws.

weaknesses:
I think that despite the captivating and well-posed questions, the paper fails to deliver on several aspects. Namely:

1. The authors claim to theoretically demonstrate that scaling the size of a neural network is equivalent to increasing its training time. The connection between the model studied here and neural networks is unclear/misleading, and some design choices are not thoroughly explained. The theoretical model has the unusual structure of equation 2, which has no connection to theoretical models for neural networks. Also, the model is linear. Also, why are $r,p,P$ all different? Why couldn’t you only have a large dimensional ambient space and a constant-sized low-rank subspace? 

2. I would have appreciated it if the connection to neural networks were toned down, and the proposed model motivated in its own right. I do not understand why the authors haven't studied more established settings, where for instance the double descent phenomenon has been shown, or where scaling laws have been tractably derived. Starting from there, it would be nice to understand which (and why) additional assumptions have to be made. With this respect, why haven’t the authors approached the problem from more standard settings in neural network theory (e.g. teacher-student [1] or random feature models [2])?

3. The concept of *minimal capacity to fit the data to a certain loss level* is crucial here, but never satisfiably addressed. In fact, an underlying assumption that is never made by the authors is that the model can fit the data at any scale (only stating “Consider a large $P$ dimensional model”). Otherwise, it would not be possible for a small model to match the performances of a large model, no matter how long the small model has been trained (e.g. imagine training a model with very small width on complicated datasets). 

4. The definition of **model scale** as $N^{1/3}$ ($N$ is the number of parameters) put forward by the authors is a heuristic, which is not convincing enough given its importance in the experiments on neural networks. More concretely, the authors define the model scale as "the maximum number of training points that can be fit by the network", which is set to $N^{1/3}$. The number of training points that can be fit by a model is a well-studied and hard problem. For arbitrary data points, the complexity of a function class determines how many data points can be fit. In general, it depends on the data's complexity and the model's flexibility. I find that the cube root of the number of parameters is arbitrary and ignores the data dependence. As the authors detail in the “experimental details” Section in the appendix, *“The width parameter was set to values of 1, 2, 5, 10, 20, 50, and 100. When we refer to ”model scale”, we quantify this as the cube root of the number of network parameters rather than the value of the width parameter.”* In the current form, the scale-time equivariance narrative heavily relies on the cubic root assumption, and the heuristic argument provided is insufficient considering its importance in the paper’s narrative. 

5. In section 5.3, it is assumed that larger models have a smaller time to interpolation threshold. However, in the experiment of Figure 5, it appears that interpolation has not happened yet, as there are no U-shaped curves, which are the footprint of double descent. It would be nice to reproduce some of the results on double descent both in model size and training time to better substantiate some of the claims. In this sense, we would expect the interpolation threshold (peak of double descent curve) to vary with model size and training time predictably according to the scale-time equivariance theory. 

6. The authors only test two learning rates, which makes it impossible to establish whether the chosen learning rate is optimal. Also, usually the optimal learning rate shifts with scale, and this is not addressed by the authors. 

7. It is well-known that there is a correspondence between training time and regularization strength (e.g. [3]). The proposed theory put forward the interesting hypothesis of a similar connection between training time and model scale. I would say that mentioning this line of work would better help this paper’s positioning in the current literature. 

8. In lines 367-269 a few articles “the” missing: If noise components →  if the noise components, etc..


[1] A Dynamical Model of Neural Scaling Laws

[2] A Solvable Model of Neural Scaling Laws

[3] A Continuous-Time View of Early Stopping for Least Squares Regression (https://proceedings.mlr.press/v89/ali19a)

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732310150100

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their careful reading of our manuscript and for their constructive feedback. We appreciate the opportunity to clarify the key aspects of our work, particularly regarding our theoretical model and its applicability to neural networks.

**Clarifying the Theoretical Model and Its Applicability**

A central concern raised by the reviewers is the perceived disconnect between our theoretical model and neural networks, with some suggesting that our model is limited to linear settings or lacks relevance to neural networks.

We would like to clarify that our theoretical model is designed to capture essential characteristics of neural network training, including nonlinearity and possible overparameterization. Specifically:

- General Nonlinear Loss Function: In our model, the loss function $L(\alpha)$ can be any differentiable function, potentially non-convex and arising from a deep neural network. The parameters $\alpha = K\beta$ represent the function implemented by the network, where $K$ is a fixed projection matrix. This setup is intended to abstract the idea that the network's output depends on certain directions in parameter space.

- Random Subspace of Parameters: The assumption that we control a random $p$-dimensional subspace of the parameters reflects the practical scenario where we increase the width or depth of a neural network. In neural networks, expanding the architecture introduces new parameters in random directions (due to random initialization). Thus, our model captures the effect of increasing model size on training dynamics.

- Scale-Time Equivalence: Our main theoretical result demonstrates that scaling the size of the parameter subspace ($p$) is equivalent to increasing training time ($t$), under gradient flow dynamics. This result holds for general differentiable loss functions and does not rely on linearity. It reveals a fundamental trade-off between model size and training time that is applicable to neural networks trained with gradient-based methods.

**Why Not Use Standard Theoretical Models?**

Some reviewers suggested using more standard theoretical models, such as random feature models or teacher-student setups. While these models are valuable for certain analyses, we deliberately chose a different approach for the following reasons:

- Generality: Our model is intended to be more general and less reliant on specific architectural assumptions. For instance, random feature models are linear models in the parameters, thus not capturing the nonlinearities of neural networks. Teacher-student setups often also operate under the assumption of a particular architecture. By not restricting ourselves to particular models, we aim to derive insights that are broadly applicable across different neural network architectures. 

- Focus on Scale-Time Trade-off: Traditional models may not readily capture the specific relationship between model scale and training time that we are investigating. Our model is specifically designed to analyze this trade-off in a principled way.

We appreciate the reviewers' concerns and believe that they stem from misunderstandings that we can address by clarifying our theoretical model and its relevance to neural networks. Our work provides a novel perspective on the interplay between model scale and training time, offering insights that are both theoretically sound and practically relevant.

We are happy to further clarify any of these points as we believe they are central to understanding our submission.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores the concept of "scale-time equivalence," suggesting that smaller neural networks trained for extended periods can match the performance of larger models trained briefly, supported by a theoretical linear model and experiments on small-scale vision tasks. The work offers a potentially valuable perspective on training efficiency and suggests a novel explanation for double descent, improving our understanding of model scaling. However, the theoretical analysis is limited to a simplistic linear model that is weakly connected to real neural networks, the experimental validation relies on small-scale models and datasets, and the core assumption of scale-time equivalence hinges on a questionable definition of model scale, ultimately raising concerns about the paper's generalizability and practical applicability. I do not recommend acceptance at this time.