Explaining Scaling Laws of Neural Network Generalization
Authors: Yasaman Bahri
Ethan Dyer
Jared Kaplan
Jaehoon Lee
Utkarsh Sharma
Venue: ICLR 2022 Submitted (2022)
Keywords: scaling laws
neural networks
generalization
overparameterized models
underparameterized models

Abstract: The test loss of well-trained neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents: super-classing image tasks does not change exponents, while changing input distribution (via changing datasets or adding noise) has a strong effect. We further explore the effect of architecture aspect ratio on scaling exponents.

Under review as a conference paper at ICLR 2022
EXPLAINING SCALING LAWS OF NEURAL NETWORK GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
The test loss of well-trained neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose theory that explains and connects these scaling laws. We identify variance- limited and resolution- -imetted scaling behavior for both dataset and model size, for total of four scaling regimes. The variance- limited scaling follows simply from the existence of well-behaved infinite data or infinite width limit, while the resolution-
regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution- limited scaling exponents related by duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents: super-classing image tasks does not change exponents, while changing input distribution (via changing datasets or adding noise) has a strong effect. We further explore the effect of architecture aspect ratio on scaling exponents.
SCALING LAWS FOR NEURAL NETWORKS
For a large variety of models and datasets, neural network performance has been empirically observed to scale as a power-law with model size and dataset size (Hestness et al., 2017 Kaplan et al. 2020 Rosenfeld et al., 2020b, Henighan et al. 2020). These exponents determine how quickly performance improves with more data and larger models. We would like understand why these power-laws emerge, and what features of the data and models determine the values of the power law exponents.
In this work, we present theoretical framework for understanding scaling laws in trained neural networks. We identify four related scaling regimes with respect to the number of model parameters P and the dataset size D. With respect to each of D, P, there is both a variance-limited regime and rexolution-limited regime.
Cariance-LLmiteen Regime In the limit of infinite data or arbitrarily wide model, some aspects of neural network training simplify. Specifically, if we fix one of D, P and study scaling with respect to the other parameter as it becomes arbitrarily large, then the difference between the finite test loss and its limiting value scales as 1/x, i.e. as a pawer-law with exponent 1, with x = or /P x width in deep networks and x = D or P in linear models.
Resolution-Limiiee Regime In this regime, one of D or P is effectively infinite, and we study scaling as the other parameter increases. In this case, a variety works have empirically observed power-law scalings 1/x²,, typically with 0 < a < 1 for both x or D. We derive exponents in this regime precisely in the setting of random feature models (c.f. next section). Empirically, we find that our theoretical predictions for exponents hold in pretrained, fine-tuned models even though these lie outside theoretical setting.

---

Under review as a conference paper at ICLR 2022
Variance-limited Theery
Resolution-limited
107
12.7
20
10P
Predtctions
12
Dalauet Stae
12.6
20-4
10
0p: (CKN)
00:
qu: 1.0
4p:1.
1p: 1.00
oo: 1.12
Sy: 1.0
40: 1500
10
up:
0.37
0.50
12.5
Teacher
10'
30'
10.
10
10
10t
3.0
0.2
0.4
0.6
0.8
3.0
Dataset sin (D)
Datasel
Interpolating Training Paints
d-dimensoons
Resolution-lmitee
Varianco-limited limited Theory
10P
10
20-
100
20-2
3.5
Love
10-1
de: (NSE)
Ry. 1.23
30
ay: 0:46
ouc 0.24
aut 0.40
10-
dec 1.20
4%
2700
10
10
22
(a)
(b)
Figure (a) Four scaling regimes Here we exhibit the four regimes we focus on this work. (top-left, bottom-right) Variance-limnn limited scaling of under- paddt-parameetrrzeedd models with dataset size and over-parameterized models with number parameters (width) exhibit universal scaling (app = ow = 1) independent the architecture or underlying dataset. (top-right, bottom-left) Resolation-limited wver-parameterized models with dataset or under parameterized models with model size exhibit scaling with exponents that depend on the details of the data distribution. These four regimes also found in random feature (Figure 2h) and pretrained models (see supplement). (b) Resolution
models interpolate the data manifold Linear interpolation between training points in four-dimensional input space (top). We show teacher model and four student models, each trained on different sized datasets. In all cases teacher and student approximately agree on the training endpoints, but as the training set size increases they increasingly match everywhere. (bottom) We show 4/o versus the data manifold dimension (input dimension for tcacher-student models, intrinsic dimension for standard datasets). We find that the studdet models follow 4/ap (dark dashed line), while relationship for a four layer CNN (solid) and WRN (hollow) on standard datasets is less clear.
For more general nonlinear models, we propose a refinement of naive bounds into estimates via expansions that hold asymptotically. 'hese rely on the idea that additional data (in the infinite model-size limit) or added model parameters (in the infinite data limit) are used by the model to carve up the data manifold into smaller components. For smooth manifolds, loss, and network, the test loss will depend on the linear size of sub-region, while it is the d-dimensional sub-region volume that scales inversely with P or D, giving rise to 1/d To test this empirically, we make measurements of the resolution- -sslutiion-imiieed exponents in neural networks and intrinsic dimension of the data manifold, shown in Figure bb
Explicit Derivation We derive the scaling laws for these four regimes explicitly in the setting of random feature teacher-student models, which also applies to neural networks the large width limit. This setting allows us to solve for the test error directly terms of the feature covariance (kernel). The scaling of the test loss then follows from the asymptotic decay of the spectrum of the covariance matrix. For generic continuous kernels on d-dimensional manifold, we further relate this the dimension of the data manifold.
Summary of Contributions:
1. We propose four scaling regimes for neural networks. The variance-limited and resolution- limited regimes originate from different mechanisms, which we identify. To our knowledge,
A visualization of this successively better approximation with dataset size is shown in Figure for models trained predict data generated hy a random fully-connected network.
2

---

Under review as a conference paper at ICLR 2022
this categorization has not been previously exhibited. We provide empirical support for all four regimes in deep networks on standard datasets.
2. We derive the poriance-limited regime under simple yet general assumptions (Theorem D
3. We present a hypothesis for resolution-limited scaling through refinement of naive bounds (Theo- rems 2 3 for general nonlinear models. We empirically test the dependence of the estimates on intrinsic dimension of the data manifold for deep networks on standard datasets (Figure
In the setting of random feature teacher-student networks, we derive both periance-limitee and resolution-limited scaling exponents exactly. the latter case, we relate this the spectral decay of kernels. We identify novel duality that exists between model and dataset size scaling.
5. We empirically investigate predictions from the random features setting in pretrained, fine-tuned models standard datasets and find they give excellent agreement.
6. We study the dependence of the scaling exponent changes in architecture and data, finding that (i) changing the input distribution via switching datasets and (ii) the addition of noise have strong effects on the exponent, while (ii) changing the target task via superclassing does not.
There have been a number of recent works demonstrating empirical scaling laws (Hestness et al. 2017 Kaplan et al.) 2020) Rosenfeld et al.) 2020b] Henighan et al.) 2020 Rosenfeld et al. 2020a) in deep neural networks, including scaling laws with model size, dataset size, compute, and other observables such as mutual information and pruning. Some precursors (Ahmad & Tesauro 199T) can be found in earlier literature. Recently, scaling laws have also played a significant role in motivating work on the largest models that have yet been developed
Related Works:
1989 Cohn & Tesauro
(Brown et al.
2020 Fedus al., 202T
There has been comparatively little work theoretical ideas (Sharma & Kaplan
2020 Bisla et al. 2021) that match and explain empirical findings in generic deep neural networks. In the particular case of large width, deep neural networks behave as random feature models (Neal 1994 Lee et al.
2018; Matthews et al. 20188] Jacot et al. 2018 Lee et al. 2019 Dyer & Gur-Ari 2020), and known
results on the loss scaling of kernel methods can be applied (Spigler et al. 2020, Bordelon et al.
2020). Though not in the original, Bordelon et al. (2020) analyze resolution-limiter dataset size
scaling for power-law spectra later versions.
During the completion of this work, Hutter (2021) presented a specific solvable model of learning exhibiting non-trivial power-law scaling for power-law (Zipf) distributed features. This does not directly relate to the setups studied in this work, present bounds that supersede our results. Concurrent to our work, Bisla et al. (2021) presented a derivation of the resolution- limited scaling with dataset size, also stemming from nearest neighbor distance scaling on data manifolds. However, they do not discuss requirements on model versus dataset size or how this scaling behavior fits into other asymptotic scaling regimes.
In the -immiee regime, scaling laws in the context of random feature models (Rahimi Recht| 2008 Hastie al., 2019 d'Ascoli et al. 2020), deep linear models Advani & Saxe 2017
Advani et al. 2020), one-hidden-llyer networks (Mei & Montanari
2019 Adlam & Pennington
2020a b). and wide neural networks treated Gaussian processes
trained in the NTK regime
(Lee et al.) 2019 Dyer & Gur-Ari 2020 Andreassen & Dyer
2020
Geiger et al. 2020) have been 2020 to motivate particular ansatz
studied. In particular, this behavior was used in (Kaplan et al.
for simultaneous scaling with data and model size. The resolution limited analysis can perhaps be viewed as an attempt to quantify the ideal-world generalization error of Nakkiran et al. (202T).
This work makes use of classic results connecting the spectrum of smooth kernel to the geometry
is defined over (Weyl 1912 Reade, 1983 Kühn 1987] Ferreira Menegatto 2009) and on the scaling of iteratively refined approximations to smooth manifolds (Stein 1999. Bickel al. 2007
de Laat 201
3

---

Under review as a conference paper at ICLR 2022
2 FOUR SCALING REGIMES
Throughout this work we will be interested in how the average test loss L(D,P) depends on the dataset size D and the number of model parameters P. Unless otherwise noted, I. denotes the test loss averaged over initialization of the parameters and draws of size D training set. Some of our results only pertain directly to the scaling with width x vP, but we expect many of the intuitions apply more generally. We use the notation ap, ap,, and aw to indicate scaling exponents with respect to dataset size, parameter count, and width. All proof's appear in the supplement.
2.1 VARIANCE-LIMITEED EXPONENTS
In the limit of large D the outputs of an appropriately trained network approach limiting form with corrections which scale as D- Similarly, recent work shows that wide networks have a smooth large P limit (Jacot et al. 2018). where fluctuations scale as 1/VP. P. If the loss is sufficiently smooth then its value will approach the asymptotic loss with corrections proportional to the variance, (1/ (L/Don or //) In Theorem 1 we present sufficient conditions on the loss to ensure this variance dominated scaling. We note, these conditions are satisfied by mean squared error and cross entropy loss, though conjecture the result holds even more generally.
Theorem 1.
Let E(f) be the test loss as a function of network output, (L = (((f))), and ler fr be the network output after T training steps, thought of as a random variable over weight initialization, draws of the training dataset, and optimization seed. Further let fr be concentrating with E[(ír E[frl) O (e) vk 2 2. If is finite degree polynomial, or has bounded second derivative, or is 2-Hölder, then E [((TT]] - E (E []]] =(e).
Dataset scaling Consider a neural network, and its associated training loss TArain (0). For every value of the weights, the training loss, thought of random variable over draws training set of size D, concentrates around the population loss, with variance which scales as O (D-1). If the optimization procedure is sufficiently smooth, the trained weights, network output, and higher
moments, will approach their infinite D values, ED (fr ED [fr)) = O O (D-1). Here, the subseript D on the expectation indicates an average over draws of the training set. This scaling
together with Theorem gives the variance limited scaling of loss with dataset size.
This concentration result with respect dataset size has appeared for linear models in Rahimi & Recht (208) and for single hidden layer networks with high-dimensional input data in Mei & Montanari (2019) ; Adlam & Pennington (2020a b). In the supplement we prove this for GD and SGD with polynomial loss as well as present informal arguments more generally. Additionally, we present examples violating the smoothness assumption and exhibiting different scaling.
Large Width Scaling We can make very similar argument in the w - limit. It has been shown that the predictions from an infinitely wide network, either under Bayesian inference Neal 1994 Lee et al. 2018), when trained via gradient descent (Jacot et al., 2018 Lee et al.) 2019) approach limiting distribution at large width equivalent to linear model. Furthermore, corrections to the infinite width behavior are controlled by the variance of the full model around the linear model predictions. This variance (and higher moments) have been shown to scale as 1/w (Dyer & Gur-Ari
2020 Yaida 2020 Andreassen & Dyer 2020), EE,, (fr - Ew If implies the loss will differ from w oo limit by a term proportional 1/w.
O (w Theorem then
We note that there has also been work studying the combined large depth and large width limit, where Hanin & Nical (2020) found well-defined infinite size limit with controlled fluctuations. In any such context where the model predictions concentrate, we expect the loss to scale with the variance of
the model output. In the case of linear models, studied below, the variance is O(P-1) rather than (VP),, and we see the associated variance scaling in this case.
4

---

Under review as a conference paper at ICLR 2022
2.2 RISOLUTION-LIMTERR EXPONENTS
In this section we consider training and data drawn uniformly from a compact d-dimensional manifold, Ma, and targets given by smooth function y F(x) on this manifold.
Para-marametrrized Dataset Scaling
Consider the double limit of an over- over- weer-paaamtterized model with large training set size, P> D 1. We further consider well-trained models, i.e. models that interpolate all training data. The goal is to understand LD). we assume that the learned model f is sufficiently smooth, then the dependence of the loss on D can be bounded in terms of the dimension of the data manifold Ma.
Informally, if our train and test data are drawn i.i.d. from the same manifold, then the distance from test point to the closest training data point decreases as add more and more training data points. In particular, this distance scales ( (D-11/d) (Levina & Bickel 2005). Furthermore, if f. F are both sufficiently smooth, they cannot differ too much over this distance. If in addition the loss function, L, is smooth function vanishing when F, we have I. O(D-1/d) This is summarized in the following theorem.
Theorem 2. Let L(F), f and F be Lipschitz with constants Kt, KG, and KF. Further let D be training dataset of size D sampled i..d from Ma and let f(x) = F(x), Vx E DD then L(D) = O (Kymax(Is K ( D-1//)
Parameter Scaling
We will again assume that F varies smoothly on an underlying compact d-dimensional manifold Ma- We can obtain bound on L(P) if we imagine that f approximates as a piecewise function with roughly P regions (see Sharma & Kaplan (0220). Here, we instead make use the argument from the over-parameterized, resolution-limited regime above. we construct sufficiently smooth estimator for F by interpolating among P' randomly chosen points from the (arbitrarily large) training set, then by the argument above the loss will be bounded by O(P-1/d)
Theorem 3. Let L(f). f and F be Lipschitz with constants KL, Ky, and Ky. Further let f(z) F(x) for P points sampled ii.d from Ma then T(P) O (Kymax(K, 5x) P-1//)
From Bounds Estimates
Theorems 2 and are phrased bounds, but we expect the stronger statement that these bounds also generically serve estimates, that eg L(D) IDD ((D==/d) for e > 2, and similarly for parameter scaling. If we assume that FF and are analytic functions Ma and that the loss function L(F is analytic in f - F and minimized at f = FF, then the loss at a given test input, Tiea, can be expanded around the nearest training point, æerii L(xxea) = 2m= am (Zuain) (Irest - Eeriin " where the first term is of finite order n 2 2 because the loss vanishes at the training point. As the typical distance between nearest neighbor points scales as D-1/d on a d-dimensional manifold, the loss will be dominated by the leading term, T. < D-n/d at large D. Note that if the model provides an accurate piecewise linear approximation, we will generically find n 2
2.3 EXPLICIT REALIZATION IN LINEAR MODELS
In the proceeding sections we have conjectured typical case scaling relations for model's test loss. We have further given intuitive arguments for this behavior which relied on smoothness assumptions on the loss and training procedure. In this section, we provide a concrete realization of all four scaling regimes within the context of linear models. Of particular interest is the resolution-limited regime, where the scaling of the loss a consequence the linear model kernel spectrum the scaling of over-parameterizee models with dataset size and inder-parametert consequence of classic result, originally due to Weyl (1912), bounding the spectrum of sufficiently smooth kernel functions by the dimension of the manifold they act on.
models with parameters is
For simplicity we have used very compressed notation for multi-tensor contractions in higher order terms.
5

---

Under review as a conference paper at ICLR 2022
Variance
Resolution on on-iiition-imiited
Karnel spectrum
10
10
102
10
30
10P
10*
10-1
10
Loss
30->
02: 1.01
02: 1.02
0y: 1.01
1.01
10-
10
4p:0.35 0030.34
02: 0.43
02: 0.76
02: 0.67
10-4
9x: 0.42
dx. 0.34
0c: 0.60
0u: 9.55
30
1.01
1:00
0.52
0.85
10
0.50
0.71
10
1.00
10-5
122
1.0
10'
10"
18'
10P
10P
sout size
Datssct (D)
Dstasct size (D)
10a
Resclution-liitted
10
Varfance-limited
Fit exponents
10
10
1.2
10a
10
108
1.0
30
0.30.34
0.00.04
0.79.7
0.70.7
Low-
10-1
10-2
1:14
1.14
1:15
1.15
0.6
0.8
10
0.92
10-3
1:14
10
0.4
10
10'
10
10r
10
107
30
10
0.4
0.6
0.8
1.0
1.2
Par ammter count
Paramater count
(a)
(b)
Figure 2: (a) Random feature models exhibit all four scaling regimes Here we consider linear teacher- student models with random features trained with MSE loss to convergence. We see both variance-limited scaling (top-left, bottom-right) and resolution-limited scaling (top-right, bottom-left). Data is varied by downsampling MNIST by the specified pool size. (b) Duality and spectra in random feature models Here we show the relation between the decay of the kernel spectra, ax, and the scaling of the loss with number of data points, ap, and with number of parameters, ap (top) The spectra of random FC kernels on pooled MNIST (bottom) appear well described by power law decay. The theoretical relation op = op ax is given by the black dashed line.
Linear predictors serve as a model system for learning. Such models are used frequently in practice when more expressive models are unnecessary or infeasible (McCullagh & Nelder 1989 Rifkin &
(oppert/2007 Hastie et al. 2009) and also serve as an instructive test bed to study training dynam-
ies (Advani et al. 2020 Goh 2017 [Hastie et al. 2019 Nakkiran 2019 Grosse 2021). Furthermore,
in the large width limit, randomly initialized neural networks become Gaussian Processes Neal
1994 Lee et al. 2018 Matthews al. 2018 Novak al. 2019
Garriga-Alonso al.
2019 Yang Huang et al.
2019) and in the low-learning rate regime (Lee al. 2019
Lewkowycz al.
2020
2020) neural networks train as linear models at infinite width (Jacot et al. 2018 Lee et al. 2019
Chizat et al. 2019).
Here we discuss linear models in general terms, though the results immediately hold for the special cases of wide neural networks. In this section we focus on teacher-studens models with weights initialized to zero and trained with mean squared error (MSE) loss to their global optimum.
We consider linear teacher, F, and student f. F(x) E was Fa (x), [(x)=E[ (xf Here {FM) } are a (potentially infinite) pool of features and the teacher weights, war are taken to be normal distributed, w (O, 1/S). The student model is built out of a subset of the teacher features. To vary the number of parameters in this simple model, we construct P features, fy=1.....r P. by introducing projector P onto a P-dimensional subspace of the teacher features, fn = Em Far.
We train by sampling a training set of size D and minimizing the MSE loss, Luxin = DD Ed (f(x.) - F(x,))² We are interested in the test loss averaged over draws of our teacher and training dataset. The infinite data test loss, L(P) == limp L(D,P) P). takes the form.
L(P) = 1 Tr C CPT (PCPT) -1 pc]
(1)
25
Here we have introduced the feature-feature second moment-matris C E. [F())
If the teacher and student features had the same span, this would vanish, but due the mismatch the loss is non-zero. On the other hand, if we keep finite number of training points, but allow the

---

Under review as a conference paper at ICLR 2022
student to use all of the teacher features, the test loss, L(D) = limp-s L(D, P), takes the form,
L(D) = - -(x))-))(())
Here, K(x, .z)) is the data-data second moment matrix, IC indicates restricting one argument to the D training points, while K. indicates restricting both. This test loss vanishes as the number of training points becomes infinite but is non-zero for finite training size.
We present a full derivation of these expressions in the supplement. In the remainder of this section, we explore the scaling of the test loss with dataset and model size.
2.3.1 VARIANCE-LIMITED EXPONENTS
To derive the limiting expressions I and 2 for the loss one makes use of the fact that the sample expectation of the second moment matrix over the finite dataset, and finite feature set is close the full covariance, 1 >>= a=1 F(x,) (zza) C 5C, 1 fx) K. 8K, with the fluctuations P
satisfying D _
O(D-1) and Ep [ō = O(P-1), where expectations are taken over draws of a dataset of size D and over feature sets. Using these expansions yields the variance- limited scaling, L(D. P) - L(P) O(D-1), L(D, L(D) = O(P-1) in the under-parameterized and over-parameterized settings respectively.
In Figure 2 we see evidence these scaling relations for features built from randomly initialized ReLU networks on pooled MNIST independent of the pool size. In the supplement we provide an in-depth derivation this behavior and expressions for the leading contributions L(D, L(P) and L(D, P) L(D).
2.3.2 RESOLUTION- LIMITED EXPONENTS
We now would like to analyze the scaling behavior of our linear model the iooluttio-limitee regimes, that is the scaling with P when 1 & < D and the scaling with D when 1 < D < P. In these cases, the scaling controlled by the shared spectrum of C or K. This spectrum often well described by power-law, where eigenvalues X satisfy A -. See Figure 2p for example spectra on pooled MNIST. In this case, will argue that the losses also obey a power law scaling, with the exponents controlled by the spectral decay factor, 1 1+ ag.
L(D) x D D=OK, L(P) x P-
(3)
In other words, in this setting, or aD ag- This is supported empirically in Figure 2p. We then argue that when the kernel function, IC is sufficiently smooth manifold of dimension d, x d-1, thus realizing the more general resolution-limics picture described above.
From spectra to scaling laws for the loss To be concrete let us focus the pee-pparaeterrized loss. If introduce the notation e; for the eigenvectors of C and é; for the eignvectors of I DD F(x.) FT(a), the loss becomes,
D
S L(D)
D Σ 22))) jml
M 11
(4)
2
Before discussing the general asymptotic behavior of we can gain some intuition by considering the case of large ag In this case, Ej A Ej (see e.g. Loukas (017)). we can simplify to,
L(D) Σ 1+ax
a ox O(D O(D==x-)) ~x-1)
(5)
More generally in the supplement, following Bordelon et 2020 replica theory methods derive, L(D) D-ak and L(P) x ak without requiring the large
Canatar et al. (202I). we use
ok limit.
7

---

Under review as a conference paper at ICLR 2022
Super-classed CIFAR-100
Neass
Corrupted CIFAR-10
stddev
100
0.200
0.150
10
10a
0.125
Loos
Loss
0.100
0.075
0p.0.39
10.00..
0g-0.39
dp 0p-0.37
0.050
0.30.39 0.39
Rp m0.40
20
m0.46
9p m0.
0.025
10-3
0.
5 10"2
dg
0.000
10
Dotaset 92e (D)
Datasett size (D)
Figure 3: Effect data distribution on scaling exponents For CIFAR-I00 superclassed to N classes (left), we find that the number of target classes does not have visihle effect the scaling exponent. (right) For CIFAR-I0 with the addition of Gaussian noise to inputs, we the strength of noise has a strong effect on performance scaling with dataset size. All models are WRN-28 28-10.
discussed simple argument that resolution- limited exponents 1/d, where d is the dimension of the data manifold. Our goal now is to explain how this connects with the linearized models and kernels discussed above: how does the spectrum of eigenvalues of kernel relate to the dimension of the data manifold?
Data Manifolds and Kernels In Section 2.2
The key point is that sufficiently smooth kernels must have an eigenvalue spectrum with a bounded tail. Specifically, a C' kernel on dd-iimensional space must have eigenvalues An S ,ाा, (Kühn) 1987). In the generic case where the covariance matrices we have discussed can be interpreted as kernels manifold, and they have spectra saturating the bound, linearized models will inherit scaling exponents given by the dimension of the manifold.
As simple example, consider d-torus. In this case we study the Fourier series de- composition, and examine the case of kernel K(x - y). This must take the form K En, lan, sin(ng (r y)) + bu, cos(n) (x - y))], where n, = (M1, .na....,,,a)) are integer indices, and On,, bn, are the overall Fourier coefficients. To guarantee that K is a C function, we must have 1 any, bn, I nd4: where nd = N indexes the number of an, in decreasing order. But this means that this simple case, the tail eigenvalues of the kernel must be bounded by Nitva as N > oc.
2.4 DUALITY
We argued above that for kernels with pure power law spectra, the asymptotic scaling of the under- parameterized loss with respect to model size and the over parameterized loss with respect to dataset size share a common exponent. In the linear setup at hand, the relation between the under- parameterized parameter dependence and over-parameterized dataset dependence is even stronger. The under-paree parameterized and over- pee-paaraeterizee losses are directly related by exchanging the projection onto random features with the projection onto random training points. Note, sample-wise double descent observed in Nakkiran (2019) concrete realization of this duality for simple data distribution. In the supplement, we present examples exhibiting the duality of the loss dependence on model and dataset size outside of the asymptotic regime.
3 EXPERIMENTS
3.1 DEEP TEACHER-STUDENT MODELS
Our theory can be tested very directly in the teacher-student framework, in which a teacher deep neural network generates synthetic data used train student network. Here, is possible to generate unlimited training samples and, crucially, controllably tune the dimension of the data manifold. We
8

---

Under review as a conference paper at ICLR 2022
accomplish the latter by scanning over the dimension of the inputs the teacher. We have found that when scanning over both model size and dataset size, the interpolation exponents closely match the prediction of 4/d. The dataset size scaling is shown in Figure while model size scaling experiments appear in the supplement and have previously been observed in Sharma & Kaplan (2020).
3.2 VARIANCF-LIMITED SCALING IN THE WILD
Variance-limited scaling, (Section 2.1 be universally observed in real datasets. Figure (top-left, nottom-right measures the cerannc--lmieeed dataset scaling exponent op and width scaling exponent ow. In both cases, we find striking agreement with the theoretically predicted values aD, ow = 1 across a variety of dataset, network architecture, stochastic batch size and loss type. Our testbed includes deep fully-connected and convolutional networks with Relu or Erf nonlinearities and MSE or softmax- crommax-cros-cmmee entrryy losses. The supplement contains experimental details.
3.3 RESOLUTION- SCALING IN THE WILD
In addition to teacher-student models, we explored resolution-limited scaling behavior in the context of standard classification datasets. Wide ResNet (WRN) models (Zagoruyko & Komodakis 2016) were trained for fixed number of steps with cosine decay. In Figure Db we also include data from a four hidden layer CNN detailed in the supplement. As detailed above, we find dataset dependent scaling behavior in this context.
further investigated the effect of the data distribution on the resolution-limited exponent, ap, by tuning the number of target classes and input noise (Figure 3}. To probe the effect of the number of classes, we constructed tasks derived from CIFAR-100 by grouping classes into broader semantic categories. We found that performance depends on the number of categories, but ap is insensitive to this number. In contrast, the addition of Gaussian noise had more pronounced effect on ap- This suggest a picture in which the network learns to model the input data manifold, independent of the classification task, consistent with observations in Nakkiran & Bansal (020);; Grathwohl et al. (220)).
We also explored the effect of aspect ratio on dataset scaling, finding that the exponent magnitude increases with width up a critical width, while the dependence on depth is milder (see supplement).
DISCUSSION
We have presented a framework for categorizing neural scaling laws, along with derivations that help to explain their very general origins. Crucially, our predictions agree with empirical findings in settings which have often proven challenging for theory - deep neural networks on real datasets. The variance-scaling regime yields, for smooth test losses, a universal prediction of ap = 1 (for D > P) and aw = 1 (for w > D). The resolution issolution-lmited regime yields exponents whose numerical value is variable and data and model dependent.
There are many intriguing directions for future work; amongst these, we highlight in particular. The invariance of the dataset scaling exponent to superclassing (Figure suggests that deep networks may be largely learning properties of the input data manifold akin unsupervised learning rather than significant task-specific structure, which may shed light on the versatility of learned deep network representations for different downstream tasks. This begs to be explored further.
Limitations One limitation is that our theoretical results are asymptotic, while experiments are performed with finite models and datasets. This is apparent in the resolution-limites regime which requires a hierarchy (D > P or P a breakdown of the predicted scaling behavior as D (P) become large and the hierarchy is lost. Furthermore in the iooluuiionlimitee regime for deep networks, our thcoretical tools rely on positing the existence of data manifold. A precise definition of the data manifold, however, is lacking forcing us use imperfect proxies, such as nearest neighbor distances of final embedding layers.
> D). In Figures I and 2 top-right (bottom-left), we see

---

Under review as a conference paper at ICLR 2022
Ethics Statement Work on scaling laws provides an opportunity for discussion how to define and measure progress in machine learning. The values of exponents allow to estimate expected gains that come from increases in scale of dataset, model, and compute. Applying similar considerations to other metrics (i.e. transfer, bias, robustness) in principle allows one to quantify whether and how models are improving or degrading with scale and at what environmental or computational cost. On the other hand, one may require that truly non-trivial progress machine learning be progress that occurs modulo scale: namely, improvements in performance across different tasks that are not simple extrapolations of existing behavior. And perhaps the right combinations algorithmic, model, and dataset improvements
lead emergent behavior new scales. Large language models such as GPT-3 (Fig. 1.2 in Brown et al. (2200) have exhibited this in the context of few-shot learning. We hope our work spurs further research in understanding and controlling neural scaling laws.
REFERENCES
Ben Adlam and Jeffrey Pennington. The Neural Tangent Kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pp- 74-84. PMLR, 2020a.
Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine ane-grained bias- variance decomposition. Advances Neural Information Processing Systems, 33, 2020b.
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error neural networks. arXiv preprint arXiv:1710.03667, 2017.
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics general- ization error neural networks. Neural Networks, 132:428 446, 2020.
Subutai Ahmad and Gerald Tesauro. Scaling and generalization in neural networks: a case study. In Advances in neural information processing systems, pp. 160-168, 1989.
Alnur Ali, J Zico Kolter, and Ryan Tibshirani. continuous-tinn time view of early stopping for least squares regression. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1370-1378. 2019.
Anders Andreassen and Ethan Dyer. Asymptotics wide convolutional neural networks. arxiv preprint arXiv:2008. 0,07055000.08675 2020.
Peter Bickel, Bo Li, et al. Local polynomial regression on unknown manifolds. In Complex datasets and inverse problems, pp. 177-186. Institute of Mathematical Statistics, 2007.
Devansh Bisla, Apoorva Nandini Saridena, and Anna Choromanska. A theoretical-empiriee approach to estimating sample complexity of dnns. arXiv preprint arXiv:2105.0186) 2021.
Blake Bordelon, Abdulkadir Canatar, and Cengiz. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pp. 1024-1034. PMLR, 2020.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miin and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http::/githus: com.googge/jiax
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005.144 14165, 2020.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model align- ment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1):1-12, 2021.
10

---

Under review as a conference paper at ICLR 2022
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, pp. 2937-2947, 2019.
Omry Cohen, Or Malka, and Zohar Ringel. Learning curves for deep neural networks: gaussian field theory perspective. arXiv preprint 1rXXvv:906.053.00 2019.
David Cohn and Gerald Tesauro. Can neural networks do better than the vapnik-chervonenkis bounds? In Advances Neural Information Processing Systems, pp. 911-917, 1991.
David de Laat. Approximating manifolds by meshes: asymptotic bounds in higher codimension. Master's Thesis, University of Groningen, Groningen, 2011.
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feymman diagrams. In International Conference on Learning Representations, 2020. URL /tt/hhssss///ppeeeeeew.eeeeee/er
id= gFFNKDD
Stéphane d' Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning, pp. 2280-2290. PMLR, 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
JC Ferreira and VA Menegatto. Eigenvalues of integral operators defined by smooth positive definite kernels. Integral Equations and Operator Theory, 64(1):61-81) 2009.
Adrià Garriga-Alonso, Laurence Aitchison, and Carl Edward Rasmussen. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representations, 2019.
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2)003401,, 2020.
Federica Gerace, Bruno Loureiro, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. General- isation error learning with random features and the hidden manifold model. In International Conference on Machine Learning, pp. 3452-3462. PMLR, 2020.
Gabriel Goh. Why momentum really works. Distill, 2017. doi: 10.2391 1/d911///iilll00000 URL
http::/ pub/20177111711111 wttpb////0/mmmmetummm
Will Grathwohl, Kuan-Chieh Wang, Joern Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat
like one. In International Conference on Learning Representations, 2020. URL https
//openreview htt..eeeee/oommiiitlceeeeet.eE
Roger Grosse. University of Toronto CSC2541 winter 2021 neural net training dynamics, lecture notes, 2021. URL //wp::://wwww/ cs twronto.oooooooo http://www...ooooooooooooooooo//
2021
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. In International Conference on Learning Representations, 2020. URL httl:tttss//opereeee/
net/foruue ftnt/fooum??icc Baorrummid-SSUndT4K4ko
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903 1903.0850000..... 2019.
11

---

Under review as a conference paper at ICLR 2022
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van 7ee. Flax: A neural network library and ecosystem for JAX, 2020. URL
http://github ctm www.gooog///.alaaa.oooo
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman. Dario Amodei, and Sam Mc- Candlish. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint
2017.
Wei Huang, Weitao Du, Richard Yi Da Xu, and Chunrui Liu. Implicit bias of deep linear networks in the large learning rate phase. arXiv preprint arXiv:2011.1257, 2020.
Marcus Hutter. Learning curve theory. arXiv preprint arXiv:2 2021.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel: Convergence and generalization in neural networks. In Advances Neural Information Processing Systems, 2018. 88
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.0886 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint arXiv:1912.11370, 6(2):8, 2019.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2661-2671, 2019.
Thomas Kühn. Eigenvalues of integral operators with smooth positive definite kernels. Archiv der Mathematik, 49(6):525-53) 534,, 1987.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Sam Schoenholz, Jeffrey Pennington, and Jascha Sohl-dickstein. Deep neural networks as Gaussian processes. International Conference on Learning Representations, 2018.
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. In Advances in Neural Information Processing Systems, 2019.
Jachoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 33, 2020.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In Advances in neural information processing systems, pp. 777-784, 2005.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022. 2020.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 /00.:039....0.8... 2016.
12

---

Under review as a conference paper at ICLR 2022
Andreas Loukas. How close are the eigenvectors of the sample and actual covariance matrices? In International Conference Machine Learning, pp- 2228-2237 PMLR, 2017.
Dörthe Malzahn and Manfred Opper. Learning curves for gaussian processes regression: A framework for good approximations. Advances in neural information processing systems, pp. 273-279, 2001.
Dörthe Malzahn and Manfred Opper. A variational approach to learning curves. In T. Dietterich, S. Becker, and Z. Ghahramani (eds.), Advances in Neural Information Processing Systems, vol- ume 14, pp. 463-469 MIT Press, 2002. URL httpp:://pprocceeinggggssss
cc/
Dörthe Malzahn and Manfred Opper. Learning curves and bootstrap estimates for inference with gaussian processes: A statistical mechanics study. Complexity, 8(4):57-63, 2003.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference Learning Representations, 2018.
McCullagh and John A Nelder. Generalized Linear Models, volume 37. CRC Press, 1989.
Song Mei and Andrea Montanari. The generalization error random features regression: Precise asymptotics and double descent curve. arXiv preprint vrXiv:1908.053:0 2019.
Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent. arXiv preprint arXiv:1912.07222,, 2019.
Preetum Nakkiran and Yamini Bansal. Distributional generalization: new kind of generalization. arXiv preprint arXiv:2009.08092, 2020.
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. International Conference on Learning Representa- tions, 2021. /tL/pttpp://opeeeevviiee http:/rwwwwweeeeeeeeoooodddeeeee
Radford M. Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, Dept. of Computer Science, 1994.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abolañia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2019.
Roman Novak, Lechao Xiao, Jiri Hron, Jachoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein. and Samuel S. Schoenholz. Neural Tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020. URL htttt:://gtiiie
com/
Sroolle/neura-taaree
Giorgio Parisi. A sequence of approximated solutions the SK model for spin glasses. Journal of Physies A: Mathematical and General, 13(4):1115, 1980.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026-8037, 2019.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: replacing minimization with randomization learning. Nips, pp. 1313-1320. Citeseer, 2008
JB Reade, Eigenvalues of positive definite kernels. SIAM Journal on Mathematical Analysis, 14(1): 152-157, 1983.
Ryan M Rifkin and Ross A Lippert. Notes regularized least squares, 2007.
13

---

Under review as a conference paper at ICLR 2022
Sam Ritchie, Ambrose Slone, and Vinay Ramasesh. Caliban: Docker-based job manager for reproducible workflows. Journal of Open Source Software, 5(53):2403, 2020. doi: 10.21 1052010///sss.
02403. URL htttps::/do..org//// 20000/2001111//1 02403
Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit. On the predictability of pruning across scales. arXiv preprint arXiv:2006.1062/ 2020a.
Jonathan Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. constructive prediction of the generalization error across scales. International Conference on Leurning Representations, 2020b.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. International Conference on Learning Representations, 2017.
Vaishaal Shankar, Alex Chengyu Fang, Wenshuo Guo, Sara Fridovich-Kan Keil, Ludwig Schmidt, Jonathan Relen-Kelley, and Benjamin Recht. Neural kernels without tangents. In International Conference on Machine Learning, 2020.
Utkarsh Sharma and Jared Kaplan. neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802 2020.
Peter Sollich. Learning curves for gaussian processes. In Proceedings of the 11th International Conference on Neural Information Processing Systems, pp. 344-350, 1998.
Peter Sollich and Anason Halees. Learning curves for gaussian process regression: Approximations and bounds. Neural computation, 14 (4(6))1393-144 2002.
Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernll methods: empirical data versus teacher-studer paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124001 2020.
Michael L Stein. Interpolation of Spatial Data: Some Theory for Kriging. Springer Science & Business Media, 1999.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6114, PMLR, 2019.
Matthew Urry and Peter Sollich. Replica theory for learning curves for gaussian processes on random graphs. Journal of Physics A: Mathematical and Theoretical, 45(42):425005, 2012.
Hermann Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differentialgle- ichungen (mit einer anwendung auf die theorie der hohlraumstrahlung). Mathematische Annalen, 71(4):441-473 1912.
Christopher KI Williams and Francesco Vivarelli. Upper and lower bounds on the learning curve for gaussian processes. Machine Leurning, 40(1):77-102, 2000.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning. 2018
Sho Yaida. Non-Gaussial processes and neural networks at finite widths. In Mathematical and Scientific Machine Learning Conference, 2020.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv: 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
14

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL SETUP
Figure 1 (top-left)
Experiments utilize relatively small models, with the number of trainable parameteters P ~ ((100), trained with full-batch gradient descent (GD) and small learning rate on datasets of size D >> P. Each data point in the figure represents an average over subsets of size D sampled from the full dataset. Experiments are done using Neural Tangents (Novak et al.] (2220) based on JAX (Bradbury et al. 2018). All experiment except denoted as (CNN), use 3-layer, width-8 fully-connected networks. CNN architecture used is Myrtle-5 network (Shankar et al. 2020) with 8 channels. Relu activation function with critical initialization (Schoenholz al.) 2017 Lee et al. 2018 Xiao al.) 2018) was used. Unless specified softmax-cross-entron loss was used. We performed full-batch gradient descent update for all dataset sizes without 1.2 regularization. 20 different training data sampling seeds were averaged for each point. For fully-connected networks, input pooling of size 4 was performed for CIFAR- 10/100 dataset and pooling of size 2 was performed for MNIST and Fashion-MNIST dataset. This was to reduce number of parameters in the input layer (# of pixels x width) which can be quite large even for small width networks.
Figure 1 (top-right) All experiments were performed using a Flax (Heek et al., 2020) implementation of Wide ResNet 28- 10 ZZagoruyko & Komodakis 2016), and performed using the Caliban experiment manager (Ritchie al. 2020). Models were trained for 78125 total steps with cosine learning rate
decay (Loshchilov & Hutter 2016) and an augmentation policy consisting of random flips and crops.
We report final loss, though we found no qualitative difference between using final loss, best loss,
final accuracy or best accuracy (see Figure ST)
Figure 1 (bottom-left) The setup was identical to Figure (top-right) except that the model consid- ered was a depth 10 residual network with varying width.
Figure 1 (boftom-right)
Experiments are done using Neural Tangents. All experiments use 100 training samples and two- hidden layer fully-connected networks of varying width (ranging from w 64 to W 11, 585) with Relu nonlinearities unless specified as Erf. Full-batch gradient descent and cross-entropy loss were used unless specified as MSE, and the figure shows curves from a random assortment of training times ranging from 100 to 500 steps (equivalently, epochs). Training was done with learning rates small enough so as to avoid catapult dynamics (Lewkowycz et al. 2020) and no T.2 regularization; in such a setting, the infinite-width learning dynamics is known to be equivalent to that of linearized models (Lee et al. 2019). Consequently, for each random initialization of the parameters, the test loss of the width linearized model was additionally computed in the identical training setting. This value approximates the limiting behavior L(xx) known theoretically and is subtracted off from the final test loss of the (nonlinear) neural network before averaging over 50 random initializations to yield each of the individual data points in the figure.
A.1 DEEP TEACHER-STUDDER MODELS
The teacher-student scaling with dataset size (figure S2 was performed with fully-connected teacher and student networks with two hidden layers and widths 96 and 192, respectively, using PyTorch (Paszke et al. 2019). The inputs were random vectors sampled uniformly from a hypercube of dimension d = 2, 3. ... 9. To mitigate noise, we ran the experiment on eight different random seeds, fixing the random seed for the teacher and student as we scanned over dataset sizes. We also used a fixed test dataset, and a fixed training set, which was sub-sampled for the experiments with smaller D. The student networks were trained using MSE loss and Adam optimizer with a maximum learning rate of 10 a cosine learning rate decay, and batch size of 64, and 40, 000 steps of training. The test losses were measured with early stopping. We combine test losses from different random seeds by averaging the logarithm of the loss from each seed.

---

Under review as a conference paper at ICLR 2022
CIFAR-10
CIFAR-100
final loss best loss
final lass
best loss
final error
final errer
10
best error
best error
10
10a
10a
104
10a
10a
104
SVHN
FashionMNIST
final loss best loss
final loss
best loss
final error
final error
best error
best error
A
10-1.
10a
10
10a
10a
10
Figure S1: Alternate metrics and stopping conditions We find similar scaling behavior for both the loss and error, for final best (early stopped) metrics.
In our experiments, always use inputs that are uniformly sampled from d-dimensional hypercube, following the setup Sharma & Kaplan 2020). They also utilized several intrisic dimension (ID) estimation methods and found the estimates were close to the input dimension, latter for comparisons. For the dataset size scans used randomly initialized teachers with width 96, and students with width 192. We found similar results with other network sizes.
simply the
The final scaling exponents and input dimensions are show in the bottom Figure We used the same experiments for the top of that figure, interpolating the behavior of both teacher and set of students between two fixed training points. The students only differed by the size of their training sets, but had the same random seeds and were trained in the same way. In that figure the input space dimension was four.
Finally, also used similar setup study variance- limited exponents and scaling. In that case used much smaller models, with 16-dimensional hidden layers, and a correspondingly larger learning rate. We then studied scaling with D again, with results pictured in Figure Th.
A.2 CNN ARCHITECTURE FOR RESOLUTION-LIMTTR SCALING
Figure Dp includes data from CNN architectures trained on image datasets. The architectures are summarized in Table We used Adam optimizer for training, with cross-entropy loss. Each network was trained for long enough achieve either a clear minimum or a plateau in test loss. Specifically, CIFAR1O, MNIST and Fashion MNIST were trained for 50 epochs, CIFARIO0 was trained for 100 epochs and SVHN was trained for 10 epochs. The default Keras training parameters were used. In case of S SVHN included the additional images training data. We averaged (in log space)
2

---

Under review as a conference paper at ICLR 2022
Teacher/Student Detaset Size
10
10
10-
223
Figure S2: This figure shows scaling trends of MSE loss with dataset size for teacher/student models. The
exponents extracted from these fits and their associated input-space dimensionalities are shown in Figure
Layer
Width
Layer
Width
CNN window (3, 3)
50
CNN window (3,3) 3)
50
2D Max Pooling (2,
2D Max Pooling (2, 2)
CNN window (3, 3)
100
CNN window (3,3)
100
2D Max Pooling (2,
2D Max Pooling (2, 2)
CNN window (3, 3)
100
CNN window (3,
200
Dense
64
Dense
256
Dense
10
Dense
100
Layer
Width
CNN window (3, 3)
64
2D Max Pooling (2, 2)
CNN window (3, 3)
64
2D Max Pooling (2, 2)
Dense
128
Dense
10
Table 1: CNN architectures for CIFAR10, MNIST, Fashion MNIST (left), CIFAR100 (center) and SVHN (right)
20 runs for CIFARI00 and CIFARIO, 16 runs for MNIST, 12 runs for Fashion MNIST, and 5 runs for SVHN. The results of these experiments are shown in Figure [S3
The measurement of input-space dimensionality for these experiments was done using the nearest- neighbour algorithm, described in detail Appendix B and Sharma & Kaplan (2020). We used and 4 nearest neighbors and averaged over the three.
A.3
TEACHER-STUDERS EXPERIMENT FOR SCALING OF LOSS WITH MODEL SIZE
We replicated the teacher-student setup in Sharma & Kaplan 2020) to demonstrate the scaling of loss with model size. The resulting variation -4/ap with input-space dimensionality is shown in figure S4 In our implementation we averaged (in log space) over iterations, with a fixed, randomly generated teacher.
B EFFECT OF ASPECT RATIO ON SCALING EXPONENTS
We trained Wide ResNet architectures of various widths and depths on CIFAR-10 10 accross dataset sizes. We found that the effect of depth on dataset scaling was mild for the range studied, while the effect of width impacted the scaling behavior up until a saturating width, after which the scaling behavior fixed. See Figure S5

---

Under review as a conference paper at ICLR 2022
66.99
2202
10
**
3x 10
Outand Size
Dutedd
MNIST: scaling DatasetSion
CPARLO:: Loss scaling with Dalisee
3.30
5x 12
164
1x
Text
SVIIN Loss scaling with Batauet Sie
3324
Figure $3: This figure shows scaling trends of CE loss with dataset size for various image datasets. The exponents extracted from these fits and their associated input-space dimensionalities are shown in Figure [
Teacher/Student Model Size Exponents
4/zx
14
12
Dimension
Figure S4: This figure shows variation ap with the input-space dimension. exponent or is scaling exponent of loss with model size for teacher-student setup.
4

---

Under review as a conference paper at ICLR 2022
CIFAR-10 varying width (d=28)
Width factor
CIFAR-10 varying depth (k=10)
Depth
1ge
28
Loss
Lots
0gi 0:42
0.50
0.48
0.54
0p: 0.55
16
0gl 0.58
0gl 0.58
0.58
10a
10
10a
10
10
Dataset size (D)
Dataset size (D)
Figure S5: Effect of aspect ratio dataset scaling We find that for WRN-d-k trained CIFAR-IO, varying depth from 10 to 40 has relatively mild effect on scaling behavior, while varying the width multiplier, k, from 12 has noticeable effect, up until saturating width.
C PROOF OF THEOREM
We now prove Theorem repeated below for convenience.
Theorem 1. Let E(J) be the test loss function of nelwork oulput, (L = [((s)), and let fr be the network output after T training steps, thought of as random variable over weight initialization, draws of the training dataset, and optimization seed. Further let fr be concentrating with E[(fr [[ffl)] = O (c) vk 2 2. If is a finite degree polynomial, or has bounded second derivative, or 2- 2-Hölder, then E [((fr)] - (E [fr]) (c))
Proof. Case 1 - finite degree polynomial: In this case, we can write,
{(fr) - E(E (fr) Σ ((k) (E())) (S- E((r)
(SI)
k!
where K is the polynomial degree and g(k) is the k-th derivative of e. Taking the expectation of S1 and using the moment scaling proves the result.
Case 2 bounded second derivative: The quadratic mean value theorem states that for any fr, there exists a such that,
((fr) {(E [fr]) -fr Efr)
- (c) (fr (fr--E(TT)) 2
(S2)
Taking the expectation of (S2) and using the fact that ('(() is bounded yields the desired result.
Case - 2-Hölder: Lastly. the loss being 2-Hölder means we may write,
((fr) ((E [fr]) s (e(fr) - {(E [fri] s Ke (fr E 2
(S3)
Again, taking the expectation of this inequality completes the proof.
A note on loss variance Theorem 1 concerns the mean loss, however we would also like to understand if this scaling holds for typical instances. This can be understood by examining how the variance of the loss or altetnatively how E [| (fr) - (E [fr]]]] scales.
For Case 3 2-Hölder loss, we can rerun the argument of Theorem E (fr) - E (B [fr])]] (c).
using
S3
to yield
For Cases and 2, we can attempt to apply the same argument in the proof. This almost works. In particular, using Hölder's inequality, B[([r E[fr)) O (e) vk 2 2 implies E[[ír [E[rr]*
5

---

Under review as a conference paper at ICLR 2022
O (c) vk 2. Taking the absolute value and expectation of (11)
S2 then gives
E Ble (BIfr)] le (E (E(fr
E E[fr II (c).
(S4)
In general, the above assumptions on and fi imply only that E llfr - E [fr]l] = O (Vε) and thus typical instances of the loss will exhibit less dramatic scaling with than the mean. If we further assume, however, that fr on average has been trained such as to be sufficiently close to a local minimum of the loss, such that (e' (IE [fr]]] O (VE), then typical instances will also obey the O (e) scaling.
D VARIANCE-LIMITED DATASET SCALING
In this section, we expand on our discussion of the guriance-limited dataset scaling, L(D) limp- L(D) (D-1). We first explain some intuition for why this behavior might be expected for sufficiently smooth loss. We then derive it explicitly for losses that are polynomial the weights. Finally, we present non non-shooth examples where the scaling can be violated either by having unbounded loss, or first derivative.
D.1 INTUITION
At a high level, the intuition is as follows. For any fixed value of weights, 9, the training loss with D training points (thought of as a random variable over draws of the dataset), T [f) concentrates around the population loss Lgpp [8], with variance that scales as O (D-1).
Our optimization procedure can be thought of as a map from initial weights and training loss to final weights Op: (0o, L [0] + 0r. If this map is sufficiently smooth for instance satisfying the assumptions of Theorem or well approximated by Taylor series about all Ep [ [O.]] then the output, dr, will also concentrate around its infinite D limit with variance scaling as O (D- Finally, if the population loss is also sufficiently smooth, the test loss for model trained on D data points averaged over draws of the dataset, L(D) = Ep pop [Or]]. satisfies L(D) - limp L(D) = 0 (D-1). We now walk through this in a little more detail.
Early time We can follow this intuition bit more explicitly for the first few steps of gradient descent. As the training loss at initialization, Lurin 10),, is sample average over D i..d draws, it concentrates around the population loss [0] with variance O (D-1). As a result, the initial alai will also concentrate with O (D-1) variance and will the weights at time 1, gradient, 9u 08 0, 0o 1/90. The training loss at time step 1, is then given by
Tamai [0: Taman 10g 90)
(S5)
If Lusin is sufficiently smooth around 0g - ED [go]. then we get that Lusin [0]] concentrates around [0 with O (D-1) variance. We can keep bouneing back and forth between gradient (or
equivalently weights) and training loss for any number of steps T which does not scale with D.
Plugging this final 0r into the population loss and taking the expectation over draws of the training
set, L(D) = ED [ [0 If Lpup is also sufficiently smooth, this yields L(D) limp- L(D) (D-1)..
Here we have used the term sufficiently smooth. A sufficient set of criteria are given in Theorem I however this is likely too restrictive. Indeed, any set of train and population loss for which Taylor series (or asymptomatic series with optimal truncation) give an (D (D-1) error around the training points Ep [0.=o..T] will have this behavior.
Local minimum The above intuition relied on training for a number of steps that was fixed as D is taken large. Here we present some alternative intuition for the variance-limitee scaling at late times, as training approaches local minimum in the loss. For simplicity we discuss a one-dimensional loss.
Consider local minimum, 0*, of the population loss. As D is taken large, with high probability, the training loss will have local minimum, 0*, such that 10" 0*1 O (D-1). One way to this,

---

Under review as a conference paper at ICLR 2022
is to note that for generic local minimum the first derivative changes sign, i.e. we can find 01, 82 such that 01 < 8" < 02 and either [0) < O, /pp [pop10]] [02] > or / [02] < 0, pop [01] > 0. To be concrete let's ss focus on the first case (the argument will be identical in either case). As D becomes large, the probability that the training loss at 0, and 02 differs significantly from the population loss
approaches zero. This can be seen from Markov's inequality, where, P (1 T [0] > a) S
Varp (Fimin (0])
or more dramatically from Hoeffding 's inequality (assuming bounded Lirain Lop
a
lying in an interval of size I)
P (1 Lop [0]| > a) S 2e 7DPaa
(S6)
Here to have non-vanishing probability as take D large, Lii (01) and [20 must be closer than O (D-1). If 01 and 0g are taken to be O (D"), then Linin must change sign, indicating an extremum of L however can do even better. If we assume Liain is Lipshitz about 0* then can still ensure sign change even if |01 - 1, A02 - 8 = O O (D-'). Using concentration of L [0] ensures the extremum is local minimum. For non- non-generic minimum (i.e. vanishing first derivatives) we can apply the same arguments to higher order derivatives (assuming they exist) of Lpop- Thus for a local minimum of Lopp with high probability Luain will have a corresponding minimum within a distance O (D-1)
If we now consider an initialization procedure, Oo. and training procedure such that training converges to the local minimum of the training loss, 0", and that the population loss is sufficiently smooth about 8 (c.g. Lipshitz), then Ep Luain - [0]]] ]] Ep EDpLuain [0 Lopp [0+10 I] 1 [0"]] The first term vanishes, while the second is O(D-1). If we further assume that this happens on average over choices of 6g then we expect L(D) lim L(D) = O (D (D-1).
At first blush it may be surprising that the variance- limited scaling holds even for mini-batch training. Indeed this case, there is batch noise that comes at a much higher scale than any variance due to the finite training set size. Indeed, the effect of mini-batching changes the final test loss, however if we fix the SGD procedure or average over SGD seeds, as we take D large, we can still ask how the training loss for model trained under SGD training set of size D differs from that for model trained under SGD
SGD
infinite training set.
To see this, we fist consider averaging over minibatches of size B, but where points are drawn i.i.d. with replacement. If we denote the batch at step t by B, and the average over independent draws of this batch by EB [.. then note we can translate moments with respect to batch draws with empirical averages over the entire training set. Explicitly, consider ca and da potentially correlated, but each drawn i.i.d. within a batch. We have that,
1
En
Σ
Co
Σ
ca
B aEB.
D
Eg
B
Σ aEB
B A'EB
Σ
de'
=
1
B
D
Σ a=1
D
ca
(
D
1
D do
+
BDD
11
D Σ Cada
a'=1
(S7)
This procedure means, after taking average draws of SGD batch, rather than thinking about function of mini-batch averages, we can equivalently consider modified function, with explicit dependence on the batch size, but that is only function of empirical means over the training set. We can thus recycle the above intuition for the scaling of smooth functions of empirical means.
The above relied on independently drawing every sample from every batch. At the other extreme, we can consider drawing batches without shuffling and increasing training set size by B datapoints at a time, so as to keep the initial set of batches in an epoch fixed. In this case, the first deviation in training between a dataset of size D and one of size D + B happens at the last batch in the first epoch after processing 7 datapoints.
7

---

Under review as a conference paper at ICLR 2022
As extreme example, consider the case where D > BT. In this only take T steps, the loss is constant for all D >> BT and so limp L(D;T: ; B) L(BT;T; B) and thus L(D > BT) - limp-+20 L(D) = 0 (and in particular is trivially O (D-1)).
D.2 POLYNOMIAL LOSS
Before discussing neural network training review the concentration behavior of polynomials of sample means.
Lemma Let z = = 10) - empirical means, over i.i.d draws of d and let e denote the distributional mean. Further, let X ( (g) (g be monomial in the sample means. Then X concentrates with moments O (D-
ED
(D-1)
(S8)
Here, En [o] denotes the average over independent draws of samples.
Proof. To establish this we can proceed by direct computation.
[xx (c (c()))......6)))))))
DD
(S9)
--1))=
n ED [XP]
n-p
p=U
Each term in the sum can be computed using
ED [X]] = ED
po .... (c(J)
(c(U)pp
Σ
ED
(0) c
... ..)
c(J)
Kc)
(0)
pkg
co)
(1)
pphy
a
(J)
(1)
D(P Dipoi kc)
ED
(0)
a pkg
(0)
(1)
(1)
(J)
(al' fa
(D-1)
D(D - D(r Dio k:)
Ki - 1)) -1))
(d.01) pho
(c(1)) phi ... (..())))
pk.
+0
= -((20))
(10)) P
(1)
+ +
Plugging this into
S9 establishes the lemma.
In the above, we use the multi-indes notation (a!2) for the collection of indices on the and the notation {ax * a3
(i) for the subset of terms in the sum where all indices take different values.
Lemma immediately implics that the mean of polynomials of a(0) concentrate around their infinite data limit.
ED (0 (60h (60)60) e0)) ((),,..(k)))
= -O
(S10)
for g E Pk (2 (())) [(K]].
With this out of the way, we can proceed analysing the scaling of trained neural networks. Here consider the simplified setting where the network map, and loss & evaluated on each training example, X. (Ia, yo), are polynomial of degree J and K in the weights, 0,.
K
f(x) = Σ
(1) MiPa.... (z)0p, 0ps ... - ((xa) = Σ
(0) 1102-14 (x.)0p, ... 0.. (S11)
1=1
8

---

Under review as a conference paper at ICLR 2022
The training loss can then be written as,
K
Lrain
pN Pp2-pp..0 ... 0u pa pi
Σ 1 (xa)
(S12)
D
Here we have used the convention that the repeated weight indices Hy are summed over.
Gradient Descent As result of the gradient descent weight update, Oc+1 0g
Dhain the 08
weights at time T are polynomial of degree (K- 1) in the z!i)
[c00) a(1) o(K)
Or IIT
(S13)
The coefficients of this polynomial depend the initial weights, Og- Plugging these weights back into the network output, have that the network funetion time T is again polynomial in c(1) now with degree (K 1)²
(T(x) E PIe [c0)) 2 [60]]
(S14)
Thus, again using Lemma fr concentrates with variance ODD-1)
ED [
- ED [frl) - =0
(S15)
and by Theorem I the loss will obey they variance-limited scaling.
Stochastic Gradient Descent We now consider the same setup of polynomial loss, but now trained via stochastic gradient descent (SGD). We consider SGD batches drawn i.i.d. with replacement and are interested in the test loss averaged over SGD draws, with fixed batch size, B.
We proceed by proving the following lemma, which allows us to reuse a similar argument to the GD case.
Lemma 2. Let z(tit) = EAEB, Co for i = O...J be mini-hatch averages, over B i.i.d. draws of Co Further, let X = (a jho (alliti)) ... ( (o(Jitalykk be monomial in the mini-batch means.
Then EB [X] E PEiLo ( . (t where d(0) are empirical means over the full training set i.i.d random variables as in Lemma and EB [o] denotes the expectation over draws of SGD batches of size B.
Proof. Expectations over draws of batches at different time steps are independent. Thus, WLOG, we can consider := to t mini-batch sums.
f.y. We can again proceed by direct computation, expanding the
Eg [X]
EB
Σ
(0)
(1)
BE. bs
(0)
(1)
(1)
Upp
(al
(S16)
(8) take the same different values.
To proceed, must keep track of terms in the sum where the all a!9 are different, the expectation over batch draws fully factorizes. More generally (S16) can be decomposed
sum over products.
One way of keeping track of the index combinatorics is introduce set of graphs, т, where each graph E r has ko vertices of type 0, k vertices of type I..., and ki vertices of type J (one vertex (6) index). Any pair of vertices may have zero one edge between them. For any set of for each d three vertices, V1, U2, and U3 with edges (wi, U2) and (v2, 03) there must also be an edge (V1, The set I consists of all possible ways of connecting these vertices consistent with these rules.

---

Under review as a conference paper at ICLR 2022
For each graph, 7/, we denote connected components by and denote the number of vertices of type i within the connected component a by m With this we can write the sum,
(166
as
Eg
X
Σ (
EB
Σ
(0)
m
(0)
m
(2)
ca
(J)
m (
OET
(0)
m
m
Σ (B)
(
(1)
(c()))
(S17)
D
yET
OET
Σ (B)
II
dmm (0) ml
(1))
ver
0ET
Here (B) is a combinatoric factor associated to each graph, not relevant for the argument. The mo (6) take on values 0 ki, the multi-index, can take IT= (k: 1) different values, which re-index to d( , (kc+)((( Mcanwhile, the degree of (S17) in d(t) is bounded by the number of total vertices in each graph, i.e. Ei kg This establishes the lemma.
For a polynomial loss of degree K, the mini-batch training loss at each time step takes the form
20:0)
Mipa...us us On Op2 ... -0p c(ii))
1
Σ 10)
(xa)
(S18)
B
a=EB.
(1.01 00
The update rule, 0,+1 = 02
ensures that 0r is polynomial of degree (K - 1)7 in the
((8:0) a(6:1) ((:.T)
Or E 1(-1) ((010)
((0:1)
((02T) a(1.0) g(1:))
((122)
z(K:0) ((Ka) (((K)),{(Ka))
(K:2))
(S19)
and consequently, denoting the test loss evaluated at 0r by L(0r].
[ E Pk(K-1yr (o(0:)) z(u:1) ((03)) 2(1:0) a(1:1)
2(:3)) 20K x(K=1)
(KT)
(S20)
Using Lemma 2 the expectation of L (rr] over draws of SGD batches is given by
Es [[[or]] E PK(K-1)"
[ ( KK (K -))=*]]]
(S21)
Finally, denoting Ep [E [[] by L(D; B) and applying Lemma I gives
L(D; B) lim L(D; B) O (D-1).
(S22)
D-+00
D.3 NON-SMOOTH EXAMPLES
Here we present two worked examples where non- non-bounded or non-smooth loss leads to violations of the variance dominated scaling. In example one, the system obeys the variance dominated scaling at early times, but exhibits different behavior for times larger than the dataset size. In the second example, the system violates the variance dominated scaling even for two gradient descent steps, as a result of an unbounded derivative the loss
Example - unbounded loss at late times Consider a dataset with two varieties of data points, drawn with probabilities a and a, and one-dimensional quadratic losses, l (concave up) and lz (concave down), on these two varieties.
1 92 (2) = 2 8² (0) = 2
(823)
We thank Anonymous for suggesting these types of examples.
10

---

Under review as a conference paper at ICLR 2022
If, in a slight abuse of notation, further denote the training loss sample with nı points of type I and D --n, n1 points of type two by lyz and the population loss at given value of the weight by pppp
we have
nh 1 2))00 Lppp 2 2 8²
mm
2 02
(S24)
For this example take 1/2. In this case, the minimum of the population loss is at zero, while the minimum of the training loss can be at zero, or at too depending whether the training sample has n1 greater than or less than D/2. We can thus create a situation where at late training times, Or does not concentrate around the minimum of the population loss.
As we work through this example explicitly, we will see the following. (i) A mismatch larger than O (D-1) between the population minimum and the minimum found by training on a sample set of size D requires times T larger than constant multiple of D. (ii) The quantity we study throughout this work is the difference between the infinite data limit of the test loss, and the finite data value, L(D) lim p D.) The minimum of the infinite data limit of the test loss is not the same as the minimum of the population loss, min p L(D) + ming Lppp In this example one diverges, while the other is finite. In particular this example evades the scaling result by L(D) for times larger than D having diverging limit.
Explicitly, we study the evolution of the model under gradient flow.
i = -2 ( Or e- 2(8 11)
-2
(S25)
The test loss averaged draws of the dataset is given by
L(D:T) = En, 2a-2
1 0
" (a a (a-1) 2)) (1 (1-a (1---))) 08
(S26)
If we consider this loss at large D and fixed T we get
L(D:T) = 4(a t)T aa-2) 0g | 8T -x)
+ O (D-2))
(S27)
D
and thus I(D;T) -imp-px T(D:; T) O (D-1) expected.
If on the other hand we consider taking T >>>>> D we have
L(D:T > D) e 2T a (a-
11 2 (1 (1- (1-a)
(S28)
the limit Tim L(D;T> > D) diverges.
Lastly, we note that if we take T BD with B < [log(1 - a2)l22 we can approach the large D limit with non non-generic, tuneable exponential convergence.
Example 2 - unbounded derivative Again, consider a two variety setup, this case with equal probabilities and per sample losses,
} 2 + We will consider different values of a > 0. The train loss and population loss are then,
(0)
1 00 (2)
02
1
1010
(S29)
2a
2a
1 1/m m1
Ex 82 + D
10
Lpop = - 62
(S30)
2
2
We consider a model initialized to 0u = and trained for two steps of gradient descent with learning rate
9t = 0g +
n1
0,10, a-
0.1 = 0,
(S31)
D
2
11

---

Under review as a conference paper at ICLR 2022
Two update steps gives
0/
n1
(S32)
D
2
The test loss given by the population loss evaluated at 02 averaged over test set draws.
T.(D) = E,g
0
D
R1
1
22
2
D
2
1==0
D 22 x- 1)²² V 2m
1
+0
(DD))
(S33)
2
T (a D D D0
2 V
Here we have approximated the binomial distribution at large D with a normal distribution using Stirling's approximation.
Note that 2 then L(D) limp- L(D) = -
i.. the finite sample loss approaches the infinite data loss with the predicted variance-limited scaling. For 0 < a
get different
scaling controlled by a. Note that the gradient, expression (S3I) is singular at the origin for precisely in this range.
In summary, this example achieves a different scaling exponent through diverging gradient.
E PROOF OF THEOREMS 2 AND 3
In this section we detail the proof Theorems and The key observation to make use of the fact that nearest neighbor distances for points sampled i.i.d. from d-dimensional manifold have mean ED., || - ~|| = O (D 1/d), where is the nearest neighbor of and the expectation the mean over data-points and draws of the dataset e.g. (Levina & Bickel 2005).
The theorem statements copied for convenience. In the main, in abuse of notation, used L(J) to indicate the value of the test loss as function of the network f. and L(D) to indicate the test loss averaged over the population, draws of the dataset, model initializations and training. To be more explicit below, we will use the notation {(f(x) to indicate the test loss for single network evaluated at single test point.
Theorem 2. Let ((J"), and F be Lipschitz with constants KL, K and K and ((F) = O. Further let D be a training dataset of size D sampled i.i.d from Ma and let fxx) F(x), Vx E D then L(D)- = O (Kimax(K K D-1D/
Proof. Consider a network trained on a particular draw of the training data. For each training point, x, & denote the neighboring training data point. Then by the above Lipschitz assump- tions and the vanishing of the loss on the true target, we have {(f()) S K2 -x)) < KL (Ky + Kx) Iz 2). With this, the average test loss is bounded as
L(D) S K2 (K3 + ED || = (
(S34)
In the last equality, we used the above mentioned scaling of nearest neighbor distances.
Theorem 3. Let ((f), f and F be Lipschitz with constants KL, Ky, and Ky. Further let f(x) F(x) for P points sampled ii.d from Ma then I. ((P) = O KKmmx(KK, P-1dd
Proof. Denote by the P points, 2, for which f(2) F(2). For each test point let & denote the closest point in P. argmin, (|x - =I). Adopting this notation, the result follows by the same argument Theorem 2
12

---

Under review as a conference paper at ICLR 2022
F RANDOM FEATURE MODELS
Here we present random feature models in more detail. We begin by reviewing exact expressions for the loss. We then go onto derive its asymptotic properties. We again consider training model f(x) =
where fr are drawn from some larger pool of features, {FM }. [u(x)) -E P (x))
Note, if (#) form complete set of functions over the data distribution, than any target function, ((z). can be expressed as y = -M= WMr (z). The extra constraint in a teacher-student model is specifying the distribution of the WM. The variance-limited scaling goes through with without the seacher-student assumption, however it is crucial for analysing the variance-limited behavior.
As in Section 2.3 we consider models with weights initialized to zero and trained to convergence with mean squared error loss.
D Σ
Lirain
(((aa)
- ya)²
(S35)
2D a=1
The data and feature second moments play central role our analysis. We introduce the notation,
C E. (m)) č= 1- F(E.) C CPPCP" Ö = =PPCI.
(S36)
x(x, 1 - R= k= Dam
<(x,,')= ( k= K
Pinin
Here the seript notation indicates the full feature space while the block letters restricted to the student features. The bar represents restriction the training dataset. We will also indicate kernels with one index in the training set as K((x) == K(x,, (.=1...)) and ((x)= == K(x, I......D. After this notation spree, the test loss can be written for under- paddr--aramettrriee models, P S. D as
L(D, En (Tr (C + -C''' -
(S37)
and for over- pee-parraeterizedd models (at the unique minimum found by GD, or projected Newton's method),
L(D, 2,b,, -
(S38)
Here the expectation Ed [o] is expectation with respect to iid draws of a dataset of size D from the input distribution, while E,, [+] is an ordinary expectation over the input distribution. Note, expression (S37) is also valid for wer-parameterizee models and (S38) is valid for under-parameterizer models if the inverses are replaces with the Moore-Penrose pseudo-inverse. Also note, the two expressions can be related by echanging the projections onto finite features with the projection onto the training dataset and the sums of teacher features with the expectation over the data manifold. This realizes the duality between dataset and features discussed above.
F.1 ASYMPTOTIC EXPRESSIONS
We are interested in S37 and $38 in the limits of large P and D.
Variance-limited scaling We begin with the mnder-parameterizee case. In the limit of lots of data the sample estimate of the feature feature second moment matrix, C, approaches the true second moment matrix, C. Explicitly, if we define the difference, 8C by C = C + SC. We have
ED [8C] = 0
1
ED [8C N. 8CM.N.1 = D
EE FM (x) (x) (z) Ca,,N, CMaNa) (S39)
ED |6CM, N, ... 6CM,NNl 0 Vn > 2.
13

---

Under review as a conference paper at ICLR 2022
The key takeaway from (S39) is that the dependence D is manifest.
Using these expressions in S37 yields.
L)D,P)= = 25
Tr (C CP" Pc)
P
+ 2DS
Σ
NiNN
5MMM M2
(P") NiNo (c="pppp""
MM C.M.
NiN2
M1.2N1.2=1
-2 -2 MrM (p' N,N2
+
(S40)
Here have introduced the notation, Tm, N₂ MrN; As above, defining
Ez [FM, Ao
L(P) == lim P)
25 - c-pc
(S41)
D-+oc
that though I(D,P) I(P) is somewhat cumbersome quantity to compute, involving the average of quartic tensor over the data distribution, its dependence D is simple.
For the over- werr-prameterized case, we can similarly expand (S38) using K = KC With fluctuations
satisfying,
Ep [0K] = 0
Ep [8K &Kazbbl = p [Ep [f (Za, (F Kaybh Kasbe)
(S42)
Ep [8Kayas 5Ka,aa,
0 (P-2) vn > 2.
This gives the expansion
L(D,)) 2 En,D [((I,)) -(x)) + +O(P-1),
(S43)
and
1 L(D) = 2 EgDD ((x))
(S44)
Resolution-limited scaling We now move onto studying the parameter scaling of L(P) and dataset scaling of L(D). We explicitly analyse the dataset scaling of L(D), with the parameter scaling following via the dataset parameter duality.
Much work has been devoted to evaluating the expression,
S44
Williams & Vivarelli 2000
Malzahn & Opper] 2002 Sollich & Halees 2002). One approach is to use the replica trick tool originating in the study of disordered systems which computes the expectation of logarithm of random variable via simpler moment contributions and analyticity assumption (Parisi 1980). The
replica trick has long history technique study the generalization properties of kernel methods
(Sollich 1998 Malzahn & Opper 2001 2003 Urry & Sollich 2012 Cohen et al.) 2009 Gerace
et al.) 2020; Bordelon et 2020). We will most closely follow the work of Canatar et al. (2021)
who use the replica method to derive an expression for the test loss of linear feature models in terms
of the eigenvalues of the kernel C and w, the coefficient vector of the target labels in terms of the model features.
n22 (K + DA,)
A
L(D)
Σ
1-7 7
2
(S45)
R R= Σ KA:
DX?
7 7=E (k+ (K+
R R DA2
This is the ridge-less, noise-free limit of equation (4) of Canatar al. (021). Here we analyze the asymptotic behavior of these expressions for eigenvalues satisfying power-law decay, (-(1-ax) and for targets coming from teacher-student setup, NO 1/S).
14

---

Under review as a conference paper at ICLR 2022
Fixed Low Regularization
Tuned Regularization
10-4
10a
10
6x10
1o08
10
4x10
3x10
10
2x10
10
size (P, Solld).
ID, Dashed)
size (P. Solld), Dataset (D, Dashed)
Figure S6: Duality between dataset size vs feature number in pretrained features Using pre- trained embedding features of EfficientNet-BB (Tan & Lc 2019) for different levels of regularization, we see that loss as function of dataset size or loss as function of the feature dimension track each other both for small regularization (left) and for tuned regularization (right). Note that regularization strength with trained-feature kernels can be mapped to inverse training time (Ali et al. 2019 Lec al. 2020p. Thus (left) corresponds to long training time and exhibits double descent behavior, while (right) corresponds optimal early stopping.
To begin, we note that teacher-student models in the limit of many features, the overlap coefficients are equal to the teacher weights, up to a rotation 4; WDaswas- As we are choosing an isotropic Gaussian initialization, we insensitive this rotation and, in particular, E. [ي]]] 1/S. See
Figure 58 for empirical support of the average constancy of w; for the teacher-student setting and contrast with realistic labels.
With this simplification, we now compute the asymptotic scaling of (S45) by approximating the sums with integrals and expanding the resulting expressions in large D. We use the identities:
10 du (Ke Dr (I+a)) = =
x (1+aa)
n2 Ita
+ a
D
2F
m,
(1 + a)T
n + Ita
1+a
1+a
2F b,, -1) xxy" " + By +...,
(S46)
Here Fi is the hypergeometric function and the second line gives its asymptotic form at large y. constant which does not effect the asymptotic scaling.
Using these relations yields
K x
7 x Do, and L(D) x D-ak,
(S47)
as promised. Here we have dropped sub-leading terms at large D. Scaling behavior for parameter scaling L(P) follow via the dataset parameter duality.
F.2 DUALITY BEYOND ASYMPTOTICS
Expressions dataset even without taking any asymptotic limits. We thus expect the dependence of test loss parameter count and dataset size to be related quite generally in linear feature models. See Section G for further details.
S37 and S38 are related by changing projections onto finite feature set, and finite
G LEARNED FEATURES
In this section, we consider linear models with features coming from pretrained neural networks. Such features are useful for transfer learning applications (e.g. Kornblith al. (2019); Kolesnikoy take pretrained embedding features from EfficientNet-B5
al. 2019)). In Figures 56 and S7
15

---

Under review as a conference paper at ICLR 2022
Pett wet
Pettas avdll
Teind
Tolael See
Randecanimbad Thiery
Varance briee Theary ovn
Brckiknn Therry
Verkrestniod
betise w.ak5
bxit wxbsa
bethy
Swall and
6x10-
b- w-ur
2xte
17
Fosuue
Pesture Me1
Fosture 25
Figure S7: Four scaling regimes exhibited by pretrained embedding features Using pretrained embedding features of EfficientNet-B5 (Tan & Le 2019) for fixed low regularization (left) and tuned regularization (right), we can identify four regimes of scaling using real CIFAR-10 labels.
model (Tan & Le 2019) using TF hut The EfficientNet model is pretrained using the ImageNet dataset with input image size of (456, 456). To extract features for the (32, 32) CIFAR-10 images, we use bilinear resizing. We then train linear classifier top of the penultimate pretrained features. To explore the effect feature size, P. and dataset size D, we randomly subset the feature dimension and training dataset size and average over 5 random seeds. Prediction on test points are obtained as kernel ridge regression problem with linear kernel. We note that the regularization ridge parameter can be mapped to an inverse carly-stopping time (Ali et al. 2019; Lee et al., (020) of a corresponding ridgeless model trained via gradient descent. Inference with low regularization parameter denotes training for long time while tuned regularization parameter is equivalent to optimal early stopping.
In Figure S7 we evidence of all four scaling regimes for low regularization (left four) and optimal regularization (right four). We speculate that the deviation from the predicted variance-limite exponent = an = 1 for the case of fixed low regularization (late time) is possibly due to the double descent resonance at D = P which interferes with the power law fit.
In Figure S6 we observe the duality between dataset size D (solid) and feature size (dashed) - the loss as a function of the number of features is identical to the loss as function of dataset size for both the optimal loss (tuned regularization) or late time loss (low regularization).
In Figure S8
we also compare properties of random features (using the infinite-width limit) and learned features from trained WRN 28-10 models. We note that teacher-student models, where the feature class matches the target function and ordinary, fully trained models real data (Figure have significantly larger exponents than models with fixed features and realistic targets.
The measured W; the coefficient of the task labels under the i-th feature constant as function index i for all teacher-student settings. However for real targets, W; are only constant for the well-performing Myrtle 10 and WRN trained features (last two columns).
S45) are approximately
whtn:://www.ttesoffooo.mhhhh
16

---

Under review as a conference paper at ICLR 2022
FC
CNN-EC
Myrtle-10
WRN pretrained
2x1
Rial to 0.05
15:-0: 0.20
Real a. 0.07
TSt no 0.23
20
10-3
T5
0.41
20
6x10-
Resl: 5.15
4x16-
10112
15.44
2.52
1x10-2
10-4
Rea 0.25
10'
Daraett
Datenet
Dalase:
Dataset aze
0.28
1.31
10
10m
10-7
20
10
10
12a
1.5
107
1.1
1.7
151
107
1.7
22-2
10-2
10:
10-4
60
1.54
10-1
+6
204
1-21
10-5
30
182
K6
0.07
4:40
125
101
10
101
125
201
127
101
107
12
107
207
307
10
107
107
Figure S8: Loss on the teacher targets scale better than real targets for both untrained and trained features The first three columns infinite width kernels while the last column is kernel built out of features from the penultimate layer of pretrained WRN 28-18 10 models on CIFAR-10. 10. The first row is the loss as function of dataset size D for teacher-student targets vs real targets. The observed dataset scaling exponent is denoted in the legend. The second row is the normalized partial sum of kernel eigenvalues. The partial sum's 's scaling exponent is measured to capture the effect of the finite dataset size when empirical Og close to zero. The third row shows W; for teacher-student and real target compared against the kernel eigenvalue decay. We see the seacher-student W; are approximately constant.
17


Reviews:
[{"review_id": "rHsGi2rCGzj", "reviewer": "Reviewer_TzFb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The decomposition of Variance-Limited Regime and Resolution-Limited Regime is somehow interesting. However, the reviewer find out it's another kind of bias-variance decomposition. The variance limited regime happens when the model size is constrained, if the rademencher complexity is bounded, then you have the 1/sqrt(data number) convergence, it's the variance in the statistics setting. The  Resolution-Limited Regime in the paper is the data is infinite and then the rate is the approximation rate.\n\n\nFor the Variance-Limited Regime, the statement is not clear. As I assume, the paper assumes the neural network lies in the  neural tangent kernel regime. Otherwise, the network weights are correlated, you can't do any concentration on width.  For the dataset scaling, the author claims \"For every value of the weights, the training loss, thought of as a random variable over draws of a training set of size D, concentrates around the population loss, with a variance which scales as O(1/D)\". This is wrong and is taught in statistical learning 101 course. The learned network is not independent with the data, so you can't use the concentration inequality, that's why the empirical process technique is introduced.(i.e. a uniform bound of all possible neural network is needed here (foundations of machine learning, Theorem 2.13 and chapter 3) .) Also the rate is not O(1/D) but is O(1/sqrt(D)) due to the  central limit theorem/Chernoff bound.\n\nFor the Resolution-Limited Regime, the 1/d scaling law is actually not surprising. The 1/d scaling law is actually the same as the non-parametric statistics [6]  and there is a line of research using a neural network to achieve this rate[1-5]. The bound the paper achieve is the same as the approximation rate obtained by [1-4]. They achieved N^{-s/d} approximation error for the s-holder function, and is standard and well-known in the literature. For the kernel setting, the nonparametric rate is from the Eigen decay  is also well known, see references in [5].\n\nIn section 2.3.1,the L(D,P)-L(D) should be O(1/sqrt{P}) but not O(1/P) due to central limit theorem/Chernoff bound,  similar errors appear many times in the paper as I think. In the case of linear models **with l2 loss**, I agree the variance will become O(1/P), but it's not the case for general model.\n\nThe paper is somehow interesting, however, the reviewer discovered that the author has little knowledge in basic statistics, as for example bias-variance decomposition, parametric rate (1/sqrt{N} variance fluctuation), and non-parametric rate (1/d scaling law). The notation in the paper is not well-defined and most of the \"theorems\" don't state the full set of assumptions of all theoretical results and even don't state the rigorous setting of the theorems. This doesn't seem like a rigorous paper but like a blog post. (For an example, in section 2.1 the expectation is an expectation on a different variable but the author use the same notation.)\n\n\n[1] Schmidt-Hieber J. Nonparametric regression using deep neural networks with ReLU activation function[J]. The Annals of Statistics, 2020, 48(4): 1875-1897.\n\n[2] Farrell M H, Liang T, Misra S. Deep neural networks for estimation and inference[J]. Econometrica, 2021, 89(1): 181-213.\n\n[3] Suzuki T, Nitanda A. Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space[J]. arXiv preprint arXiv:1910.12799, 2019.\n\n[4] Chen M, Jiang H, Liao W, et al. Efficient approximation of deep relu networks for functions on low dimensional manifolds[J]. Advances in neural information processing systems, 2019, 32: 8174-8184.\n\n[5] Nitanda A, Suzuki T. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime[J]. arXiv preprint arXiv:2006.12297, 2020.\n\n[6] Tsybakov A B. Introduction to Nonparametric Estimation[J]. Springer series in statistics, 2009: I-XII,1-214.", "summary_of_the_paper": "This paper considered Variance-Limited Regime and Resolution-Limited Regime to explain the explored neural scaling law. In the Variance-Limited Regim, one fixed one of the D and P and in the Resolution-Limited Regime, one of the D and P is effectively infinite. The paper aims to provide a parametric rate scaling law in the Variance-Limited Regime  and a non-parametric scaling law in the Resolution-Limited Regime.", "summary_of_the_review": "The paper is not well-written and theoretical results are not well-stated. The paper also igonres standard results in the statistical results, bias-variance decomposition vs the four regimes and the non-parametric rates. The reviewer also find out that the author lack of knowledge in basic statistical machine learning theory. Reviewer suggests the author write down the model considered (random feature, neural tangent kernel regime NN?) rigorously and state all the notation and theorem in detail. \nThe empirical experiment in the paper is interesting and meets the bar of a top conference, however the \"theory\" part is confusing, not novel and even something can be wrong.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SazFeBPghb0", "reviewer": "Reviewer_78Lv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a phenomenon called scaling laws of test loss for neural networks. In a nutshell, the test loss decreases by following a power law curve (exponent) with respect to width or data set size, while the other being fixed.  Specifically, the variance limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by\npositing that models are effectively resolving a smooth data manifold. The proofs are for linear models (random feature) and pretrained models. Authors also observe several empirical relationships between datasets and scaling exponents: super-classing image tasks does not change exponents, while changing\ninput distribution (via changing datasets or adding noise) has a strong effect.\n\nOverall, the paper studies a very interesting phenomenon. It's quite interesting to see such power law with respect to width and data set sizes with the other being fixed. Such good combination of interesting phenomenon with theoretical explanation is a great fit for ICLR. Hence, an accept has been recommended. \n\nHowever, there are several concerns remaining.\n\n1. It's counter-intuitive to see the accuracy follows power law for width. In the limit, it becomes the kernel regime, and we all know NTK is different from neural network. In this case, there must be finite width which perform better than infinite width? Then this is a counter-example to your phenomenon?\n\n2. What's the practical different between variance limited and resolution limited? These two terms are bit confusing.\n\n3. The data set size is reasonbale, but does not consider if we add more rare examples? If rare examples are of measure 0, then it's not reflected in your theory, but can make a big difference in practice?\n\n4. In some cases, I find smaller network perform better (say width 32), larger width cannot work. How do you explain this?", "summary_of_the_paper": "The paper proposes a phenomenon called scaling laws of test loss for neural networks. In a nutshell, the test loss decreases by following a power law curve (exponent) with respect to width or data set size, while the other being fixed.  Specifically, the variance limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by\npositing that models are effectively resolving a smooth data manifold. The proofs are for linear models (random feature) and pretrained models. Authors also observe several empirical relationships between datasets and scaling exponents: super-classing image tasks does not change exponents, while changing\ninput distribution (via changing datasets or adding noise) has a strong effect.", "summary_of_the_review": "Overall, the paper studies a very interesting phenomenon. It's quite interesting to see such power law with respect to width and data set sizes with the other being fixed. Such good combination of interesting phenomenon with theoretical explanation is a great fit for ICLR. Hence, an accept has been recommended.  However, authors need to clarify a few concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "AWkMACvGMPY", "reviewer": "Reviewer_f3X7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Neural scaling laws have become a popular topic in machine learning, so the theoretical understanding, which this submission attempts to address, is an important research problem. However, I have the following concerns:  \n\n**(i)** Many important prior works (related to the results presented in this submission) were not cited. The authors should discuss the difference from and improvement upon these papers. In particular, \n1. If we do not take optimization into account, then the so-called \"scaling law\" of neural network has already been extensively studied in the nonparametric literature. For example, [Schmidt-Hieber] and [Suzuki] derived the learning rates of deep neural network in fitting certain target functions; such analysis is usually divided into two parts, the estimation error, which scales like $D^{-\\alpha}$, and the approximation error, which scales with $P^{-\\beta}$, where the exponents $\\alpha, \\beta$ depend on the problem dimensionality and smoothness of the target function. \n2. If we restrict ourselves to linear/kernel models, then power-law generalization error rates are fairly standard and well-known results (e.g., see [Caponnetto and De Vito]). Moreover, for neural networks in the kernel regime, the role of optimization (i.e., the number of training steps) can also be incorporated in the analysis, as in [Nitanda and Suzuki]. \n3. For the random features model, the \"dual\" relation between the model width and training set size has been rigorously studied in [Ghorbani et al.] and [Mei et al.]. Note that [Ghorbani et al.] has been publicly available for more than 2 years, so I don't think the current submission can be treated as concurrent work. \n\n- Caponnetto and De Vito, FOCM 2007. https://link.springer.com/article/10.1007/s10208-006-0196-8. \n- Schmidt-Hieber, Annals of Statistics 2020. https://arxiv.org/pdf/1708.06633.pdf. \n- Suzuki, ICLR 2018. https://arxiv.org/pdf/1810.08033.pdf.\n- Ghorbani et al., Annals of Statistics 2021.  https://arxiv.org/pdf/1904.12191.pdf.\n- Suzuki and Nitanda, ICLR 2021. https://arxiv.org/pdf/2006.12297.pdf.\n- Mei et al., 2021. https://arxiv.org/pdf/2101.10588.pdf.\n\n**(ii)** The theoretical results are somewhat underwhelming, and some discussions lack rigor. \n1. It is not clear whether the $O(1/D)$-fluctuation is relevant in practical neural network training, because it omits other dependencies. For example, if we naively couple the finite-sample (stochastic) gradient descent with the population counterpart, then without additional assumptions, the difference between the two trajectories can blow up exponentially in time. Hence this bound would be uninformative beyond the early phase of training. Moreover, this does not provide any insight on the scaling law of training speed.  \n2. The smoothness parameters in Theorem 2 and 3 are treated as constants independent to $D$ or $P$. Why wouldn't the Lipschitz constant of the learned interpolator depend on $D$, especially in the noisy setting? \n3. Many findings in Section 2.3 and 2.4 have little to do with actual neural network. For example, while a wide model can be \"frozen\" in the kernel regime under certain initialization, it is not clear why a trained network in the $D\\gg P$ regime would resemble a random features model with similar spectral properties. \n4. For the replica computation, the authors should elaborate on the difference from the following papers, both of which derived power-law decay of generalization error for kernel regression.\n- Spigler et al., J. Stat. Mech. 2020. https://arxiv.org/pdf/1905.10843.pdf.\n- Bordelon et al., ICML 2020. https://arxiv.org/pdf/2002.02561.pdf.\n\n**Additional Comments**\n\n- For Section 2.3.2, the authors should comment on when one can expect such power-law decay of the kernel spectra to hold true. Intuitively this may not happen when $D\\asymp d$ (for some notion of intrinsic dimensionality $d$) -- in this regime the kernel eigenvalues are usually bounded away from 0. ", "summary_of_the_paper": "This paper characterized the scaling of generalization error with respect to the number of training samples $D$ and parameters $P$ for certain smooth estimators in two regimes: the variance-limited regime and the resolution-limited regime; this divide is somewhat analogous to the parametric vs. nonparametric estimation. The authors also argued that the large-width and large-sample limit have similar scaling behavior. The findings are supported by empirical evidence. ", "summary_of_the_review": "I evaluated this submission primarily as a theory paper; in my opinion the current theoretical contribution is below the acceptance bar, for reasons listed above. I am happy to adjust my score if the authors can address my concerns. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the scaling laws of neural networks with respect to the number of training samples $D$ and parameters $P$ for some estimators in two regimes: the variance-limited regime and the resolution-limited regime. The theoretical results are supported by some numerical experiments.

Unfortunately, the paper has several critical issues, in particular, in its novelty and technical correctness. 
1. The theoretical analyses lack much of their rigor. The assumptions and problem setups are not precisely introduced. Accordingly, the statement of each theorem is presented in an inaccurate way. Moreover, some theoretical consequences contain technical flaws (e.g., $1/P$ should be replaced by $1/\sqrt{P}$ without an appropriate assumption on the loss function such as strong convexity and smoothness).  
2. Many of the presented results are already known in the literatures. It is unfortunate that the authors did no cite relevant existing literatures and did not discuss its novelty compared with the existing work. 

For those reasons, this paper lacks its novelty and the quality of the paper is not sufficient to be accepted.  
I recommend the authors to thoroughly survey the literature of the statistical learning theory from classic nonparametric regression analyses to recent advances on overparameterization.
Decision: Reject