Year: 2021

Abstract:
The test loss of well-trained neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variancelimited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolutionlimited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents: super-classing image tasks does not change exponents, while changing input distribution (via changing datasets or adding noise) has a strong effect. We further explore the effect of architecture aspect ratio on scaling exponents. 1 SCALING LAWS FOR NEURAL NETWORKS For a large variety of models and datasets, neural network performance has been empirically observed to scale as a power-law with model size and dataset size (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2020b; Henighan et al., 2020). These exponents determine how quickly performance improves with more data and larger models. We would like to understand why these power-laws emerge, and what features of the data and models determine the values of the power law exponents. In this work, we present a theoretical framework for understanding scaling laws in trained neural networks. We identify four related scaling regimes with respect to the number of model parameters P and the dataset size D. With respect to each of D, P , there is both a variance-limited regime and a resolution-limited regime. Variance-Limited Regime In the limit of infinite data or an arbitrarily wide model, some aspects of neural network training simplify. Specifically, if we fix one of D,P and study scaling with respect to the other parameter as it becomes arbitrarily large, then the difference between the finite test loss and its limiting value scales as 1/x, i.e. as a power-law with exponent 1, with x = D or √ P ∝ width in deep networks and x = D or P in linear models. Resolution-Limited Regime In this regime, one of D or P is effectively infinite, and we study scaling as the other parameter increases. In this case, a variety of works have empirically observed power-law scalings 1/x, typically with 0 < α < 1 for both x = P or D. We derive exponents in this regime precisely in the setting of random feature models (c.f. next section). Empirically, we find that our theoretical predictions for exponents hold in pretrained, fine-tuned models even though these lie outside our theoretical setting.

heading: 1 SCALING LAWS FOR NEURAL NETWORKS
text: For a large variety of models and datasets, neural network performance has been empirically observed to scale as a power-law with model size and dataset size (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2020b; Henighan et al., 2020). These exponents determine how quickly performance improves with more data and larger models. We would like to understand why these power-laws emerge, and what features of the data and models determine the values of the power law exponents.
In this work, we present a theoretical framework for understanding scaling laws in trained neural networks. We identify four related scaling regimes with respect to the number of model parameters P and the dataset size D. With respect to each of D, P , there is both a variance-limited regime and a resolution-limited regime.
Variance-Limited Regime In the limit of infinite data or an arbitrarily wide model, some aspects of neural network training simplify. Specifically, if we fix one of D,P and study scaling with respect to the other parameter as it becomes arbitrarily large, then the difference between the finite test loss and its limiting value scales as 1/x, i.e. as a power-law with exponent 1, with x = D or √ P ∝ width in deep networks and x = D or P in linear models.
Resolution-Limited Regime In this regime, one of D or P is effectively infinite, and we study scaling as the other parameter increases. In this case, a variety of works have empirically observed power-law scalings 1/xα, typically with 0 < α < 1 for both x = P or D. We derive exponents in this regime precisely in the setting of random feature models (c.f. next section). Empirically, we find that our theoretical predictions for exponents hold in pretrained, fine-tuned models even though these lie outside our theoretical setting.
For more general nonlinear models, we propose a refinement of naive bounds into estimates via expansions that hold asymptotically. These rely on the idea that additional data (in the infinite model-size limit) or added model parameters (in the infinite data limit) are used by the model to carve up the data manifold into smaller components. For smooth manifolds, loss, and network, the test loss will depend on the linear size of a sub-region, while it is the d-dimensional sub-region volume that scales inversely with P or D, giving rise to α ∝ 1/d.1 To test this empirically, we make measurements of the resolution-limited exponents in neural networks and intrinsic dimension of the data manifold, shown in Figure 1b.
Explicit Derivation We derive the scaling laws for these four regimes explicitly in the setting of random feature teacher-student models, which also applies to neural networks in the large width limit. This setting allows us to solve for the test error directly in terms of the feature covariance (kernel). The scaling of the test loss then follows from the asymptotic decay of the spectrum of the covariance matrix. For generic continuous kernels on a d-dimensional manifold, we can further relate this to the dimension of the data manifold.

heading: Summary of Contributions:
text: 1. We propose four scaling regimes for neural networks. The variance-limited and resolutionlimited regimes originate from different mechanisms, which we identify. To our knowledge,
1A visualization of this successively better approximation with dataset size is shown in Figure 1b for models trained to predict data generated by a random fully-connected network.
this categorization has not been previously exhibited. We provide empirical support for all four regimes in deep networks on standard datasets.
2. We derive the variance-limited regime under simple yet general assumptions (Theorem 1).
3. We present a hypothesis for resolution-limited scaling through refinement of naive bounds (Theorems 2, 3), for general nonlinear models. We empirically test the dependence of the estimates on intrinsic dimension of the data manifold for deep networks on standard datasets (Figure 1b).
4. In the setting of random feature teacher-student networks, we derive both variance-limited and resolution-limited scaling exponents exactly. In the latter case, we relate this to the spectral decay of kernels. We identify a novel duality that exists between model and dataset size scaling.
5. We empirically investigate predictions from the random features setting in pretrained, fine-tuned models on standard datasets and find they give excellent agreement.
6. We study the dependence of the scaling exponent on changes in architecture and data, finding that (i) changing the input distribution via switching datasets and (ii) the addition of noise have strong effects on the exponent, while (iii) changing the target task via superclassing does not.
Related Works: There have been a number of recent works demonstrating empirical scaling laws (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2020b; Henighan et al., 2020; Rosenfeld et al., 2020a) in deep neural networks, including scaling laws with model size, dataset size, compute, and other observables such as mutual information and pruning. Some precursors (Ahmad & Tesauro, 1989; Cohn & Tesauro, 1991) can be found in earlier literature. Recently, scaling laws have also played a significant role in motivating work on the largest models that have yet been developed (Brown et al., 2020; Fedus et al., 2021).
There has been comparatively little work on theoretical ideas (Sharma & Kaplan, 2020; Bisla et al., 2021) that match and explain empirical findings in generic deep neural networks. In the particular case of large width, deep neural networks behave as random feature models (Neal, 1994; Lee et al., 2018; Matthews et al., 2018; Jacot et al., 2018; Lee et al., 2019; Dyer & Gur-Ari, 2020), and known results on the loss scaling of kernel methods can be applied (Spigler et al., 2020; Bordelon et al., 2020). Though not in the original, Bordelon et al. (2020) analyze resolution-limited dataset size scaling for power-law spectra in later versions.
During the completion of this work, Hutter (2021) presented a specific solvable model of learning exhibiting non-trivial power-law scaling for power-law (Zipf) distributed features. This does not directly relate to the setups studied in this work, or present bounds that supersede our results. Concurrent to our work, Bisla et al. (2021) presented a derivation of the resolution-limited scaling with dataset size, also stemming from nearest neighbor distance scaling on data manifolds. However, they do not discuss requirements on model versus dataset size or how this scaling behavior fits into other asymptotic scaling regimes.
In the variance-limited regime, scaling laws in the context of random feature models (Rahimi & Recht, 2008; Hastie et al., 2019; d’Ascoli et al., 2020), deep linear models (Advani & Saxe, 2017; Advani et al., 2020), one-hidden-layer networks (Mei & Montanari, 2019; Adlam & Pennington, 2020a;b), and wide neural networks treated as Gaussian processes or trained in the NTK regime (Lee et al., 2019; Dyer & Gur-Ari, 2020; Andreassen & Dyer, 2020; Geiger et al., 2020) have been studied. In particular, this behavior was used in (Kaplan et al., 2020) to motivate a particular ansatz for simultaneous scaling with data and model size. The resolution-limited analysis can perhaps be viewed as an attempt to quantify the ideal-world generalization error of Nakkiran et al. (2021).
This work makes use of classic results connecting the spectrum of a smooth kernel to the geometry it is defined over (Weyl, 1912; Reade, 1983; Kühn, 1987; Ferreira & Menegatto, 2009) and on the scaling of iteratively refined approximations to smooth manifolds (Stein, 1999; Bickel et al., 2007; de Laat, 2011).

heading: 2 FOUR SCALING REGIMES
text: Throughout this work we will be interested in how the average test loss L(D,P ) depends on the dataset size D and the number of model parameters P . Unless otherwise noted, L denotes the test loss averaged over initialization of the parameters and draws of a size D training set. Some of our results only pertain directly to the scaling with width w ∝ √ P , but we expect many of the intuitions apply more generally. We use the notation αD, αP , and αW to indicate scaling exponents with respect to dataset size, parameter count, and width. All proofs appear in the supplement.

heading: 2.1 VARIANCE-LIMITED EXPONENTS
text: In the limit of large D the outputs of an appropriately trained network approach a limiting form with corrections which scale as D−1. Similarly, recent work shows that wide networks have a smooth large P limit (Jacot et al., 2018), where fluctuations scale as 1/ √ P . If the loss is sufficiently smooth then its value will approach the asymptotic loss with corrections proportional to the variance, (1/D or 1/ √ P ). In Theorem 1 we present sufficient conditions on the loss to ensure this variance dominated scaling. We note, these conditions are satisfied by mean squared error and cross entropy loss, though we conjecture the result holds even more generally.
Theorem 1. Let `(f) be the test loss as a function of network output, (L = E [`(f)]), and let fT be the network output after T training steps, thought of as a random variable over weight initialization, draws of the training dataset, and optimization seed. Further let fT be concentrating with E[(fT − E[fT ])k] = O ( )∀k ≥ 2. If ` is a finite degree polynomial, or has bounded second derivative, or is 2-Hölder, then E [`(fT )]− ` (E [fT ]) = O( ).
Dataset scaling Consider a neural network, and its associated training loss Ltrain(θ). For every value of the weights, the training loss, thought of as a random variable over draws of a training set of size D, concentrates around the population loss, with a variance which scales as O ( D−1 ) . If the optimization procedure is sufficiently smooth, the trained weights, network output, and higher moments, will approach their infinite D values, ED [ (fT − ED [fT ])k ] = O ( D−1 ) . Here, the
subscript D on the expectation indicates an average over draws of the training set. This scaling together with Theorem 1 gives the variance limited scaling of loss with dataset size.
This concentration result with respect to dataset size has appeared for linear models in Rahimi & Recht (2008) and for single hidden layer networks with high-dimensional input data in Mei & Montanari (2019); Adlam & Pennington (2020a;b). In the supplement we prove this for GD and SGD with polynomial loss as well as present informal arguments more generally. Additionally, we present examples violating the smoothness assumption and exhibiting different scaling.
Large Width Scaling We can make a very similar argument in the w → ∞ limit. It has been shown that the predictions from an infinitely wide network, either under Bayesian inference (Neal, 1994; Lee et al., 2018), or when trained via gradient descent (Jacot et al., 2018; Lee et al., 2019) approach a limiting distribution at large width equivalent to a linear model. Furthermore, corrections to the infinite width behavior are controlled by the variance of the full model around the linear model predictions. This variance (and higher moments) have been shown to scale as 1/w (Dyer & Gur-Ari, 2020; Yaida, 2020; Andreassen & Dyer, 2020), Ew [ (fT − Ew [fT ])k ] = O ( w−1 ) . Theorem 1 then
implies the loss will differ from its w =∞ limit by a term proportional to 1/w.
We note that there has also been work studying the combined large depth and large width limit, where Hanin & Nica (2020) found a well-defined infinite size limit with controlled fluctuations. In any such context where the model predictions concentrate, we expect the loss to scale with the variance of the model output. In the case of linear models, studied below, the variance is O(P−1) rather than O( √ P ), and we see the associated variance scaling in this case.

heading: 2.2 RESOLUTION-LIMITED EXPONENTS
text: In this section we consider training and test data drawn uniformly from a compact d-dimensional manifold, x ∈Md, and targets given by some smooth function y = F(x) on this manifold.

heading: Over-Parameterized Dataset Scaling
text: Consider the double limit of an over-parameterized model with large training set size, P D 1. We further consider well-trained models, i.e. models that interpolate all training data. The goal is to understand L(D). If we assume that the learned model f is sufficiently smooth, then the dependence of the loss on D can be bounded in terms of the dimension of the data manifoldMd.
Informally, if our train and test data are drawn i.i.d. from the same manifold, then the distance from a test point to the closest training data point decreases as we add more and more training data points. In particular, this distance scales as O(D−1/d) (Levina & Bickel, 2005). Furthermore, if f , F are both sufficiently smooth, they cannot differ too much over this distance. If in addition the loss function, L, is a smooth function vanishing when f = F , we have L = O(D−1/d). This is summarized in the following theorem.
Theorem 2. Let L(f), f and F be Lipschitz with constants KL, Kf , and KF . Further let D be a training dataset of size D sampled i.i.d from Md and let f(x) = F(x), ∀x ∈ D then L(D) = O ( KLmax(Kf ,KF )D−1/d ) .

heading: Under-Parameterized Parameter Scaling
text: We will again assume that F varies smoothly on an underlying compact d-dimensional manifold Md. We can obtain a bound on L(P ) if we imagine that f approximates F as a piecewise function with roughly P regions (see Sharma & Kaplan (2020)). Here, we instead make use of the argument from the over-parameterized, resolution-limited regime above. If we construct a sufficiently smooth estimator for F by interpolating among P randomly chosen points from the (arbitrarily large) training set, then by the argument above the loss will be bounded by O(P−1/d). Theorem 3. Let L(f), f and F be Lipschitz with constants KL, Kf , and KF . Further let f(x) = F(x) for P points sampled i.i.d fromMd then L(P ) = O ( KLmax(Kf ,KF )P−1/d ) .

heading: From Bounds to Estimates
text: Theorems 2 and 3 are phrased as bounds, but we expect the stronger statement that these bounds also generically serve as estimates, so that eg L(D) = Ω(D−c/d) for c ≥ 2, and similarly for parameter scaling. If we assume that F and f are analytic functions onMd and that the loss function L(f,F) is analytic in f −F and minimized at f = F , then the loss at a given test input, xtest, can be expanded around the nearest training point, x̂train, L(xtest) = ∑∞ m=n≥2 am(x̂train)(xtest − x̂train)m,2 where the first term is of finite order n ≥ 2 because the loss vanishes at the training point. As the typical distance between nearest neighbor points scales as D−1/d on a d-dimensional manifold, the loss will be dominated by the leading term, L ∝ D−n/d, at large D. Note that if the model provides an accurate piecewise linear approximation, we will generically find n ≥ 4.

heading: 2.3 EXPLICIT REALIZATION IN LINEAR MODELS
text: In the proceeding sections we have conjectured typical case scaling relations for a model’s test loss. We have further given intuitive arguments for this behavior which relied on smoothness assumptions on the loss and training procedure. In this section, we provide a concrete realization of all four scaling regimes within the context of linear models. Of particular interest is the resolution-limited regime, where the scaling of the loss is a consequence of the linear model kernel spectrum – the scaling of over-parameterized models with dataset size and under-parameterized models with parameters is a consequence of a classic result, originally due to Weyl (1912), bounding the spectrum of sufficiently smooth kernel functions by the dimension of the manifold they act on.
2For simplicity we have used a very compressed notation for multi-tensor contractions in higher order terms.

heading: 102 103 104
text: Linear predictors serve as a model system for learning. Such models are used frequently in practice when more expressive models are unnecessary or infeasible (McCullagh & Nelder, 1989; Rifkin & Lippert, 2007; Hastie et al., 2009) and also serve as an instructive test bed to study training dynamics (Advani et al., 2020; Goh, 2017; Hastie et al., 2019; Nakkiran, 2019; Grosse, 2021). Furthermore, in the large width limit, randomly initialized neural networks become Gaussian Processes (Neal, 1994; Lee et al., 2018; Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019; Yang, 2019), and in the low-learning rate regime (Lee et al., 2019; Lewkowycz et al., 2020; Huang et al., 2020) neural networks train as linear models at infinite width (Jacot et al., 2018; Lee et al., 2019; Chizat et al., 2019).
Here we discuss linear models in general terms, though the results immediately hold for the special cases of wide neural networks. In this section we focus on teacher-student models with weights initialized to zero and trained with mean squared error (MSE) loss to their global optimum.
We consider a linear teacher, F , and student f , F (x) = ∑S M=1 ωMFM (x), f(x) = ∑P µ=1 θµfµ(x) . Here {FM} are a (potentially infinite) pool of features and the teacher weights, ωM are taken to be normal distributed, ω ∼ N (0, 1/S). The student model is built out of a subset of the teacher features. To vary the number of parameters in this simple model, we construct P features, fµ=1,...,P , by introducing a projector P onto a P -dimensional subspace of the teacher features, fµ = ∑ M PµMFM .
We train by sampling a training set of size D and minimizing the MSE loss, Ltrain = 1 2D ∑D a=1 (f(xa)− F (xa))
2. We are interested in the test loss averaged over draws of our teacher and training dataset. The infinite data test loss, L(P ) := limD→∞ L(D,P ), takes the form.
L(P ) = 1 2S Tr [ C − CPT ( PCPT )−1 PC] . (1) Here we have introduced the feature-feature second moment-matrix, C = Ex [ F (x)FT (x) ] .
If the teacher and student features had the same span, this would vanish, but due to the mismatch the loss is non-zero. On the other hand, if we keep a finite number of training points, but allow the
student to use all of the teacher features, the test loss, L(D) := limP→S L(D,P ), takes the form,
L(D) = 1 2 Ex [ K(x, x)− ~K(x)K̄−1 ~K(x) ] . (2)
Here, K(x, x′) is the data-data second moment matrix, ~K indicates restricting one argument to the D training points, while K̄ indicates restricting both. This test loss vanishes as the number of training points becomes infinite but is non-zero for finite training size.
We present a full derivation of these expressions in the supplement. In the remainder of this section, we explore the scaling of the test loss with dataset and model size.

heading: 2.3.1 VARIANCE-LIMITED EXPONENTS
text: To derive the limiting expressions (1) and (2) for the loss one makes use of the fact that the sample expectation of the second moment matrix over the finite dataset, and finite feature set is close to the full covariance, 1D ∑D a=1 F (xa)F T (xa) = C+ δC, 1P f T (x)f(x′),= K+ δK, with the fluctuations
satisfying ED [ δC2 ] = O(D−1) and EP [ δK2 ] = O(P−1), where expectations are taken over draws of a dataset of size D and over feature sets. Using these expansions yields the variance-limited scaling, L(D,P )−L(P ) = O(D−1), L(D,P )−L(D) = O(P−1) in the under-parameterized and over-parameterized settings respectively.
In Figure 2a we see evidence of these scaling relations for features built from randomly initialized ReLU networks on pooled MNIST independent of the pool size. In the supplement we provide an in-depth derivation of this behavior and expressions for the leading contributions to L(D,P )− L(P ) and L(D,P )− L(D).

heading: 2.3.2 RESOLUTION-LIMITED EXPONENTS
text: We now would like to analyze the scaling behavior of our linear model in the resolution-limited regimes, that is the scaling with P when 1 P D and the scaling with D when 1 D P . In these cases, the scaling is controlled by the shared spectrum of C or K. This spectrum is often well described by a power-law, where eigenvalues λi satisfy λi = 1i1+αK . See Figure 2b for example spectra on pooled MNIST. In this case, we will argue that the losses also obey a power law scaling, with the exponents controlled by the spectral decay factor, 1 + αK .
L(D) ∝ D−αK , L(P ) ∝ P−αK . (3)
In other words, in this setting, αP = αD = αK . This is supported empirically in Figure 2b. We then argue that when the kernel function, K is sufficiently smooth on a manifold of dimension d, αK ∝ d−1, thus realizing the more general resolution-limited picture described above.
From spectra to scaling laws for the loss To be concrete let us focus on the over-parameterized loss. If we introduce the notation ei for the eigenvectors of C and ēi for the eignvectors of 1 D ∑D a=1 F (xa)F T (xa), the loss becomes,
L(D) = 1
2 S∑ i=1 λi(1− D∑ j=1 (ei · ēj)2) . (4)
Before discussing the general asymptotic behavior of (4), we can gain some intuition by considering the case of large αK . In this case, ēj ≈ ej (see e.g. Loukas (2017)), we can simplify (4) to,
L(D) ∝ ∞∑ D+1 1 i1+αK = αKD −αK +O(D−αK−1) . (5)
More generally in the supplement, following Bordelon et al. (2020); Canatar et al. (2021), we use replica theory methods to derive, L(D) ∝ D−αK and L(P ) ∝ P−αK , without requiring the large αK limit.
Data Manifolds and Kernels In Section 2.2, we discussed a simple argument that resolutionlimited exponents α ∝ 1/d, where d is the dimension of the data manifold. Our goal now is to explain how this connects with the linearized models and kernels discussed above: how does the spectrum of eigenvalues of a kernel relate to the dimension of the data manifold?
The key point is that sufficiently smooth kernels must have an eigenvalue spectrum with a bounded tail. Specifically, a Ct kernel on a d-dimensional space must have eigenvalues λn . 1n1+t/d (Kühn, 1987). In the generic case where the covariance matrices we have discussed can be interpreted as kernels on a manifold, and they have spectra saturating the bound, linearized models will inherit scaling exponents given by the dimension of the manifold.
As a simple example, consider a d-torus. In this case we can study the Fourier series decomposition, and examine the case of a kernel K(x − y). This must take the form K =∑ nI
[anI sin(nI · (x− y)) + bnI cos(nI · (x− y))], where nI = (n1, · · · , nd) are integer indices, and anI , bnI are the overall Fourier coefficients. To guarantee that K is a C
t function, we must have anI , bnI . 1 nd+t
where nd = N indexes the number of anI in decreasing order. But this means that in this simple case, the tail eigenvalues of the kernel must be bounded by 1
N1+t/d as N →∞.

heading: 2.4 DUALITY
text: We argued above that for kernels with pure power law spectra, the asymptotic scaling of the underparameterized loss with respect to model size and the over-parameterized loss with respect to dataset size share a common exponent. In the linear setup at hand, the relation between the underparameterized parameter dependence and over-parameterized dataset dependence is even stronger. The under-parameterized and over-parameterized losses are directly related by exchanging the projection onto random features with the projection onto random training points. Note, sample-wise double descent observed in Nakkiran (2019) is a concrete realization of this duality for a simple data distribution. In the supplement, we present examples exhibiting the duality of the loss dependence on model and dataset size outside of the asymptotic regime.

heading: 3 EXPERIMENTS

heading: 3.1 DEEP TEACHER-STUDENT MODELS
text: Our theory can be tested very directly in the teacher-student framework, in which a teacher deep neural network generates synthetic data used to train a student network. Here, it is possible to generate unlimited training samples and, crucially, controllably tune the dimension of the data manifold. We
accomplish the latter by scanning over the dimension of the inputs to the teacher. We have found that when scanning over both model size and dataset size, the interpolation exponents closely match the prediction of 4/d. The dataset size scaling is shown in Figure 1, while model size scaling experiments appear in the supplement and have previously been observed in Sharma & Kaplan (2020).

heading: 3.2 VARIANCE-LIMITED SCALING IN THE WILD
text: Variance-limited scaling, (Section 2.1), can be universally observed in real datasets. Figure 1a (top-left, bottom-right) measures the variance-limited dataset scaling exponent αD and width scaling exponent αW . In both cases, we find striking agreement with the theoretically predicted values αD, αW = 1 across a variety of dataset, network architecture, stochastic batch size and loss type. Our testbed includes deep fully-connected and convolutional networks with Relu or Erf nonlinearities and MSE or softmax-cross-entropy losses. The supplement contains experimental details.

heading: 3.3 RESOLUTION-LIMITED SCALING IN THE WILD
text: In addition to teacher-student models, we explored resolution-limited scaling behavior in the context of standard classification datasets. Wide ResNet (WRN) models (Zagoruyko & Komodakis, 2016) were trained for a fixed number of steps with cosine decay. In Figure 1b we also include data from a four hidden layer CNN detailed in the supplement. As detailed above, we find dataset dependent scaling behavior in this context.
We further investigated the effect of the data distribution on the resolution-limited exponent, αD, by tuning the number of target classes and input noise (Figure 3). To probe the effect of the number of classes, we constructed tasks derived from CIFAR-100 by grouping classes into broader semantic categories. We found that performance depends on the number of categories, but αD is insensitive to this number. In contrast, the addition of Gaussian noise had a more pronounced effect on αD. This suggest a picture in which the network learns to model the input data manifold, independent of the classification task, consistent with observations in Nakkiran & Bansal (2020); Grathwohl et al. (2020).
We also explored the effect of aspect ratio on dataset scaling, finding that the exponent magnitude increases with width up to a critical width, while the dependence on depth is milder (see supplement).

heading: 4 DISCUSSION
text: We have presented a framework for categorizing neural scaling laws, along with derivations that help to explain their very general origins. Crucially, our predictions agree with empirical findings in settings which have often proven challenging for theory – deep neural networks on real datasets. The variance-scaling regime yields, for smooth test losses, a universal prediction of αD = 1 (for D P ) and αW = 1 (for w D). The resolution-limited regime yields exponents whose numerical value is variable and data and model dependent.
There are many intriguing directions for future work; amongst these, we highlight one in particular. The invariance of the dataset scaling exponent to superclassing (Figure 3) suggests that deep networks may be largely learning properties of the input data manifold – akin to unsupervised learning – rather than significant task-specific structure, which may shed light on the versatility of learned deep network representations for different downstream tasks. This begs to be explored further.
Limitations One limitation is that our theoretical results are asymptotic, while experiments are performed with finite models and datasets. This is apparent in the resolution-limited regime which requires a hierarchy (D P or P D). In Figures 1a and 2a top-right (bottom-left), we see a breakdown of the predicted scaling behavior as D (P ) become large and the hierarchy is lost. Furthermore in the resolution-limited regime for deep networks, our theoretical tools rely on positing the existence of a data manifold. A precise definition of the data manifold, however, is lacking forcing us to use imperfect proxies, such as nearest neighbor distances of final embedding layers.
Ethics Statement Work on scaling laws provides an opportunity for discussion on how to define and measure progress in machine learning. The values of exponents allow us to estimate expected gains that come from increases in scale of dataset, model, and compute. Applying similar considerations to other metrics (i.e. transfer, bias, robustness) in principle allows one to quantify whether and how models are improving or degrading with scale and at what environmental or computational cost. On the other hand, one may require that truly non-trivial progress in machine learning be progress that occurs modulo scale: namely, improvements in performance across different tasks that are not simple extrapolations of existing behavior. And perhaps the right combinations of algorithmic, model, and dataset improvements can lead to emergent behavior at new scales. Large language models such as GPT-3 (Fig. 1.2 in Brown et al. (2020)) have exhibited this in the context of few-shot learning. We hope our work spurs further research in understanding and controlling neural scaling laws.

heading: A EXPERIMENTAL SETUP
text: Figure 1 (top-left)
Experiments utilize relatively small models, with the number of trainable parameteters P ∼ O(1000), trained with full-batch gradient descent (GD) and small learning rate on datasets of sizeD P . Each data point in the figure represents an average over subsets of size D sampled from the full dataset. Experiments are done using Neural Tangents (Novak et al., 2020) based on JAX (Bradbury et al., 2018). All experiment except denoted as (CNN), use 3-layer, width-8 fully-connected networks. CNN architecture used is Myrtle-5 network (Shankar et al., 2020) with 8 channels. Relu activation function with critical initialization (Schoenholz et al., 2017; Lee et al., 2018; Xiao et al., 2018) was used. Unless specified softmax-cross-entropy loss was used. We performed full-batch gradient descent update for all dataset sizes without L2 regularization. 20 different training data sampling seeds were averaged for each point. For fully-connected networks, input pooling of size 4 was performed for CIFAR-10/100 dataset and pooling of size 2 was performed for MNIST and Fashion-MNIST dataset. This was to reduce number of parameters in the input layer (# of pixels × width) which can be quite large even for small width networks.
Figure 1 (top-right) All experiments were performed using a Flax (Heek et al., 2020) implementation of Wide ResNet 28-10 (Zagoruyko & Komodakis, 2016), and performed using the Caliban experiment manager (Ritchie et al., 2020). Models were trained for 78125 total steps with a cosine learning rate decay (Loshchilov & Hutter, 2016) and an augmentation policy consisting of random flips and crops. We report final loss, though we found no qualitative difference between using final loss, best loss, final accuracy or best accuracy (see Figure S1).
Figure 1 (bottom-left) The setup was identical to Figure 1 (top-right) except that the model considered was a depth 10 residual network with varying width.
Figure 1 (bottom-right)
Experiments are done using Neural Tangents. All experiments use 100 training samples and twohidden layer fully-connected networks of varying width (ranging from w = 64 to W = 11, 585) with Relu nonlinearities unless specified as Erf. Full-batch gradient descent and cross-entropy loss were used unless specified as MSE, and the figure shows curves from a random assortment of training times ranging from 100 to 500 steps (equivalently, epochs). Training was done with learning rates small enough so as to avoid catapult dynamics (Lewkowycz et al., 2020) and no L2 regularization; in such a setting, the infinite-width learning dynamics is known to be equivalent to that of linearized models (Lee et al., 2019). Consequently, for each random initialization of the parameters, the test loss of the finite-width linearized model was additionally computed in the identical training setting. This value approximates the limiting behavior L(∞) known theoretically and is subtracted off from the final test loss of the (nonlinear) neural network before averaging over 50 random initializations to yield each of the individual data points in the figure.

heading: A.1 DEEP TEACHER-STUDENT MODELS
text: The teacher-student scaling with dataset size (figure S2) was performed with fully-connected teacher and student networks with two hidden layers and widths 96 and 192, respectively, using PyTorch (Paszke et al., 2019). The inputs were random vectors sampled uniformly from a hypercube of dimension d = 2, 3, · · · , 9. To mitigate noise, we ran the experiment on eight different random seeds, fixing the random seed for the teacher and student as we scanned over dataset sizes. We also used a fixed test dataset, and a fixed training set, which was sub-sampled for the experiments with smaller D. The student networks were trained using MSE loss and Adam optimizer with a maximum learning rate of 3× 10−3, a cosine learning rate decay, and a batch size of 64, and 40, 000 steps of training. The test losses were measured with early stopping. We combine test losses from different random seeds by averaging the logarithm of the loss from each seed.
102 103 104
10 1
100
CIFAR-10 final loss best loss final error best error
102 103 104
100
CIFAR-100 final loss best loss final error best error
103 104 105
10 1
SVHN final loss best loss final error best error
103 104
10 1
FashionMNIST final loss best loss final error best error
Figure S1: Alternate metrics and stopping conditions We find similar scaling behavior for both the loss and error, and for final and best (early stopped) metrics.
In our experiments, we always use inputs that are uniformly sampled from a d-dimensional hypercube, following the setup of Sharma & Kaplan (2020). They also utilized several intrisic dimension (ID) estimation methods and found the estimates were close to the input dimension, so we simply use the latter for comparisons. For the dataset size scans we used randomly initialized teachers with width 96, and students with width 192. We found similar results with other network sizes.
The final scaling exponents and input dimensions are show in the bottom of Figure 1b. We used the same experiments for the top of that figure, interpolating the behavior of both teacher and a set of students between two fixed training points. The students only differed by the size of their training sets, but had the same random seeds and were trained in the same way. In that figure the input space dimension was four.
Finally, we also used a similar setup to study variance-limited exponents and scaling. In that case we used much smaller models, with 16-dimensional hidden layers, and a correspondingly larger learning rate. We then studied scaling with D again, with results pictured in Figure 1a.

heading: A.2 CNN ARCHITECTURE FOR RESOLUTION-LIMITED SCALING
text: Figure 1b includes data from CNN architectures trained on image datasets. The architectures are summarized in Table 1. We used Adam optimizer for training, with cross-entropy loss. Each network was trained for long enough to achieve either a clear minimum or a plateau in test loss. Specifically, CIFAR10, MNIST and Fashion MNIST were trained for 50 epochs, CIFAR100 was trained for 100 epochs and SVHN was trained for 10 epochs. The default Keras training parameters were used. In case of SVHN we included the additional images as training data. We averaged (in log space) over
26 27 28 29 210 211 212 213 Dataset Size
10 9
10 8
10 7
10 6
10 5
10 4
Lo ss
Teacher/Student Dataset Size
2
3
4
5
6
7
8
9
In pu
t D im
en sio
n
Figure S2: This figure shows scaling trends of MSE loss with dataset size for teacher/student models. The exponents extracted from these fits and their associated input-space dimensionalities are shown in Figure 1.
20 runs for CIFAR100 and CIFAR10, 16 runs for MNIST, 12 runs for Fashion MNIST, and 5 runs for SVHN. The results of these experiments are shown in Figure S3.
The measurement of input-space dimensionality for these experiments was done using the nearestneighbour algorithm, described in detail in Appendix B and C in Sharma & Kaplan (2020). We used 2, 3 and 4 nearest neighbors and averaged over the three.

heading: A.3 TEACHER-STUDENT EXPERIMENT FOR SCALING OF LOSS WITH MODEL SIZE
text: We replicated the teacher-student setup in Sharma & Kaplan (2020) to demonstrate the scaling of loss with model size. The resulting variation of −4/αP with input-space dimensionality is shown in figure S4. In our implementation we averaged (in log space) over 15 iterations, with a fixed, randomly generated teacher.

heading: B EFFECT OF ASPECT RATIO ON SCALING EXPONENTS
text: We trained Wide ResNet architectures of various widths and depths on CIFAR-10 accross dataset sizes. We found that the effect of depth on dataset scaling was mild for the range studied, while the effect of width impacted the scaling behavior up until a saturating width, after which the scaling behavior fixed. See Figure S5.
102 103 104 Dataset Size
100
2 × 100
3 × 100
Te st
L os
s
CIFAR10: Loss scaling with Dataset Size D : 0.198
103 104 Dataset Size
3 × 10 1
4 × 10 1
6 × 10 1
Te st
L os
s
FashionMNIST: Loss scaling with Dataset Size D : 0.207
103 104 Dataset Size
10 1
Te st
L os
s
MNIST: Loss scaling with Dataset Size D : 0.397
103 104 Dataset Size
3 × 100
4 × 100
5 × 100
Te st
L os
s
CIFAR100: Loss scaling with Dataset Size D : 0.164
103 104 105 Dataset Size
100
Te st
L os
s
SVHN: Loss scaling with Dataset Size D : 0.242
Figure S3: This figure shows scaling trends of CE loss with dataset size for various image datasets. The exponents extracted from these fits and their associated input-space dimensionalities are shown in Figure 1.
4 6 8 10 12 Dimension
4
6
8
10
12
14
4/ P
Teacher/Student Model Size Exponents d = 4/ P
Figure S4: This figure shows the variation of αP with the input-space dimension. The exponent αP is the scaling exponent of loss with model size for teacher-student setup.
103 104 Dataset size (D)
100 Lo ss
CIFAR-10 varying width (d=28)
D: 0.42 D: 0.50 D: 0.54 D: 0.58 D: 0.58
103 104 Dataset size (D)
100
Lo ss
CIFAR-10 varying depth (k=10)
D: 0.48 D: 0.55 D: 0.58 D: 0.58
1 2
4
10
12 Width factor
10
16
28
40 Depth
Figure S5: Effect of aspect ratio on dataset scaling We find that for WRN-d-k trained on CIFAR-10, varying depth from 10 to 40 has a relatively mild effect on scaling behavior, while varying the width multiplier, k, from 1 to 12 has a more noticeable effect, up until a saturating width.

heading: C PROOF OF THEOREM 1
text: We now prove Theorem 1 repeated below for convenience.
Theorem 1. Let `(f) be the test loss as a function of network output, (L = E [`(f)]), and let fT be the network output after T training steps, thought of as a random variable over weight initialization, draws of the training dataset, and optimization seed. Further let fT be concentrating with E[(fT − E[fT ])k] = O ( )∀k ≥ 2. If ` is a finite degree polynomial, or has bounded second derivative, or is 2-Hölder, then E [`(fT )]− ` (E [fT ]) = O( ).
Proof. Case 1 – finite degree polynomial: In this case, we can write,
`(fT )− `(E [fT ]) = K∑ k=1 `(k) (E [fT ]) k! (fT − E [fT ])k , (S1)
where K is the polynomial degree and `(k) is the k-th derivative of `. Taking the expectation of (S1) and using the moment scaling proves the result.
Case 2 – bounded second derivative: The quadratic mean value theorem states that for any fT , there exists a c such that,
`(fT )− `(E [fT ]) = (fT − E [fT ]) `′(E [fT ]) + 1
2 `′′(c) (fT − E [fT ])2 . (S2)
Taking the expectation of (S2) and using the fact that f ′′(c) is bounded yields the desired result.
Case 3 – 2-Hölder: Lastly, the loss being 2-Hölder means we may write,
`(fT )− `(E [fT ]) ≤ |`(fT )− `(E [fT ])| ≤ K` (fT − E [fT ])2 . (S3)
Again, taking the expectation of this inequality completes the proof.
A note on loss variance Theorem 1 concerns the mean loss, however we would also like to understand if this scaling holds for typical instances. This can be understood by examining how the variance of the loss or altetnatively how E [|` (fT )− ` (E [fT ])|] scales.
For Case 3 – 2-Hölder loss, we can rerun the argument of Theorem 1, using (S3) to yield E [|` (fT )− ` (E [fT ])|] = O ( ).
For Cases 1 and 2, we can attempt to apply the same argument as in the proof. This almost works. In particular, using Hölder’s inequality, E[(fT − E[fT ])k] = O ( )∀k ≥ 2 implies E[|fT − E[fT ]|k] =
O ( )∀k ≥ 2. Taking the absolute value and expectation of (S1) or (S2) then gives E [|` (fT )− ` (E [fT ])|] ≤ |`′ (E [fT ])|E [|fT − E [fT ]|] +O ( ) . (S4)
In general, the above assumptions on ` and fT imply only that E [|fT − E [fT ]|] = O ( √ ) and thus typical instances of the loss will exhibit a less dramatic scaling with than the mean. If we further assume, however, that fT on average has been trained such as to be sufficiently close to a local minimum of the loss, such that |`′ (E [fT ])| = O ( √ ), then typical instances will also obey the O ( ) scaling.
D VARIANCE-LIMITED DATASET SCALING
In this section, we expand on our discussion of the variance-limited dataset scaling, L(D) − limD→∞ L(D) = O ( D−1 ) . We first explain some intuition for why this behavior might be expected for sufficiently smooth loss. We then derive it explicitly for losses that are polynomial in the weights. Finally, we present non-smooth examples where the scaling can be violated either by having unbounded loss, or first derivative.
D.1 INTUITION
At a high level, the intuition is as follows. For any fixed value of weights, θ, the training loss with D training points (thought of as a random variable over draws of the dataset), Ltrain[θ] concentrates around the population loss Lpop[θ], with variance that scales as O ( D−1 ) .
Our optimization procedure can be thought of as a map from initial weights and training loss to final weights Op : (θ0, Ltrain[θ]) → θT . If this map is sufficiently smooth – for instance satisfying the assumptions of Theorem 1 or well approximated by a Taylor series about all ED [Ltrain[θt]] – then the output, θT , will also concentrate around its infinite D limit with variance scaling as O ( D−1 ) . Finally, if the population loss is also sufficiently smooth, the test loss for a model trained on D data points averaged over draws of the dataset, L(D) = ED [Lpop[θT ]], satisfies L(D)− limD→∞ L(D) = O ( D−1 ) . We now walk through this in a little more detail.
Early time We can follow this intuition a bit more explicitly for the first few steps of gradient descent. As the training loss at initialization, Ltrain[θ0], is a sample average over D i.i.d draws, it concentrates around the population loss Lpop[θ0] with variance O ( D−1 ) . As a result, the initial
gradient, g0 = ∂Ltrain∂θ0 will also concentrate with O ( D−1 ) variance and so will the weights at time 1, θ1 = θ0 − ηg0. The training loss at time step 1, is then given by Ltrain[θ1] = Ltrain[θ0 − g0] . (S5)
If Ltrain is sufficiently smooth around θ0 − ED [g0], then we get that Ltrain[θ1] concentrates around Ltrain[θ1] with O ( D−1 ) variance. We can keep bouncing back and forth between gradient (or equivalently weights) and training loss for any number of steps T which does not scale with D. Plugging this final θT into the population loss and taking the expectation over draws of the training set, L(D) = ED [Lpop[θT ]]. If Lpop is also sufficiently smooth, this yields L(D)− limD→∞ L(D) = O ( D−1 ) .
Here we have used the term sufficiently smooth. A sufficient set of criteria are given in Theorem 1; however this is likely too restrictive. Indeed, any set of train and population loss for which a Taylor series (or asymptomatic series with optimal truncation) give an O ( D−1 ) error around the training points ED [θt=0...T ] will have this behavior.
Local minimum The above intuition relied on training for a number of steps that was fixed as D is taken large. Here we present some alternative intuition for the variance-limited scaling at late times, as training approaches a local minimum in the loss. For simplicity we discuss a one-dimensional loss.
Consider a local minimum, θ∗, of the population loss. As D is taken large, with high probability, the training loss will have a local minimum, θ̄∗, such that |θ∗ − θ̄∗| = O ( D−1 ) . One way to see this,
is to note that for a generic local minimum the first derivative changes sign, i.e. we can find θ1, θ2 such that θ1 < θ∗ < θ2 and either L′pop[θ1] < 0, L ′ pop[θ2] > 0 or L ′ pop[θ2] < 0, L ′ pop[θ1] > 0. To be concrete let’s focus on the first case (the argument will be identical in either case). As D becomes large, the probability that the training loss at θ1 and θ2 differs significantly from the population loss approaches zero. This can be seen from Markov’s inequality, where, P
(∣∣L′train[θ]− L′pop[θ]∣∣ > a) ≤ VarD(L′train[θ])
a2 , or more dramatically from Hoeffding’s inequality (assuming bounded Ltrain − Lpop lying in an interval of size I)
P (∣∣L′train[θ]− L′pop[θ]∣∣ > a) ≤ 2e− 2ID2a2 . (S6)
Here to have non-vanishing probability as we take D large, L′train[θ1] and L ′ train[θ2] must be closer than O ( D−1 ) . If θ1 and θ2 are taken to be O ( D0 ) , then L′train must change sign, indicating an extremum of Ltrain; however we can do even better. If we assume Ltrain is Lipshitz about θ∗ then we can still ensure a sign change even if |θ1−θ∗|, |θ2−θ∗| = O ( D−1 ) . Using concentration of L′′train[θ] ensures the extremum is a local minimum. For non-generic minimum (i.e. vanishing first derivatives) we can apply the same arguments to higher order derivatives (assuming they exist) of Lpop. Thus for a local minimum of Lpop, with high probability Ltrain will have a corresponding minimum within a distance O ( D−1
) If we now consider an initialization procedure, θ0, and training procedure such that training converges to the local minimum of the training loss, θ̄∗, and that the population loss is sufficiently smooth about θ∗ (e.g. Lipshitz), then ED[Ltrain[θ̄∗]−Lpop[θ∗]] = ED[Ltrain[θ̄∗]−Lpop[θ̄∗]]+ED[Lpop[θ̄∗]−Lpop[θ∗]]. The first term vanishes, while the second is O(D−1). If we further assume that this happens on average over choices of θ0 then we expect L(D)− limD→∞ L(D) = O ( D−1 ) .
SGD At first blush it may be surprising that the variance-limited scaling holds even for mini-batch training. Indeed in this case, there is batch noise that comes in at a much higher scale than any variance due to the finite training set size. Indeed, the effect of mini-batching changes the final test loss, however if we fix the SGD procedure or average over SGD seeds, as we take D large, we can still ask how the training loss for a model trained under SGD on a training set of size D differs from that for a model trained under SGD on an infinite training set.
To see this, we fist consider averaging over minibatches of size B, but where points are drawn i.i.d. with replacement. If we denote the batch at step t by Bt and the average over independent draws of this batch by EB [•], then note we can translate moments with respect to batch draws with empirical averages over the entire training set. Explicitly, consider ca and da potentially correlated, but each drawn i.i.d. within a batch. We have that,
EB
[ 1
B ∑ a∈Bt ca
] = 1
D D∑ a=1 ca
EB
[( 1
B ∑ a∈Bt ca
)( 1
B ∑ a′∈Bt da′
)] = ( 1− 1
B
)( 1
D D∑ a=1 ca
)( 1
D D∑ a′=1 da′
) + 1
B
1
D D∑ a=1 cada .
(S7)
This procedure means, after taking an average over draws of SGD batch, rather than thinking about a function of mini-batch averages, we can equivalently consider a modified function, with explicit dependence on the batch size, but that is only a function of empirical means over the training set. We can thus recycle the above intuition for the scaling of smooth functions of empirical means.
The above relied on independently drawing every sample from every batch. At the other extreme, we can consider drawing batches without shuffling and increasing training set size by B datapoints at a time, so as to keep the initial set of batches in an epoch fixed. In this case, the first deviation in training between a dataset of size D and one of size D+B happens at the last batch in the first epoch after processing D datapoints.
As an extreme example, consider the case where D > BT . In this case, as we only take T steps, the loss is constant for all D > BT and so limD→∞ L(D;T ;B) = L(BT ;T ;B) and thus L(D > BT )− limD→∞ L(D) = 0 (and in particular is trivially O ( D−1 ) ).

heading: D.2 POLYNOMIAL LOSS
text: Before discussing neural network training we review the concentration behavior of polynomials of sample means.
Lemma 1. Let c̄(i) = 1D ∑D a=1 c (i) a for i = 0 . . . J be empirical means, over D i.i.d. draws of c (i) a and let c(i) denote the distributional mean. Further, let X = (c̄(0))k0(c̄(1))k1 · · · (c̄(J))kJ be a
monomial in the sample means. Then X concentrates with moments O ( D−1 ) ,
ED [( X − (c(0))k0(c(1))k1 · · · (c(J))kJ )n] = O ( D−1 ) . (S8)
Here, ED [•] denotes the average over independent draws of D samples.
Proof. To establish this we can proceed by direct computation. ED [( X − (c(0))k0(c(1))k1 · · · (c(J))kJ )n] =
n∑ p=0 (−1)n−p ( n p ) ED [Xp] ( (c(0))k0(c(1))k1 · · · (c(J))kJ )n−p (S9) Each term in the sum can be computed using
ED [Xp] = ED [ (c̄(0))pk0(c̄(1))pk1 · · · (c̄(J))pkJ ] = 1
D(p ∑J i=0 ki) ∑ {a(i)α } ED [( c (0) a (0) 1 · · · c(0) a (0) pk0 )( c (1) a (1) 1 · · · c(1) a (1) pk1 ) · · · ( c (J) a (J) 1 · · · c(J) a (J) pkJ )]
= 1 D(p ∑J i=0 ki) ∑ {a(i)α 6=a(j)β } ED [( c (0) a (0) 1 · · · c(0) a (0) pk0 )( c (1) a (1) 1 · · · c(1) a (1) pk1 ) · · · ( c (J) a (J) 1 · · · c(J) a (J) pkJ )] +O ( D−1
) = D(D − 1) · · · (D − (p ∑J i=0 ki − 1))
D(p ∑J i=0 ki)
( c(0) )pk0 ( c(1) )pk1 · · · ( c(J) )pkJ +O ( D−1 ) = (( c(0) )k0 ( c(1) )k1 · · · ( c(J) )kJ)p +O ( D−1 ) .
Plugging this into (S9) establishes the lemma.
In the above, we use the multi-index notation {a(i)α } for the collection of indices on the ci and the notation {a(i)α 6= a(j)β } for the subset of terms in the sum where all indices take different values.
Lemma 1 immediately implies that the mean of polynomials of c̄(i) concentrate around their infinite data limit.
ED [( g ( c̄(0), c̄(1), . . . , c̄(K) ) − g ( c(0), c(1), . . . , c(K) ))n] = O ( D−1 ) , (S10)
for g ∈ PK [ c̄(0), c̄(1), . . . , c̄(K) ] .
With this out of the way, we can proceed to analysing the scaling of trained neural networks. Here we consider the simplified setting where the network map, f , and loss ` evaluated on each training example, xa = (xa, ya), are polynomial of degree J and K in the weights, θµ,
f(x) = J∑ i=1 b(i)µ1µ2...µi(x)θµ1θµ2 · · · θµi `(xa) = K∑ i=1 c(i)µ1µ2...µi(xa)θµ1θµ2 · · · θµi . (S11)
The training loss can then be written as,
Ltrain = K∑ i=1 c̄(i)µ1µ2...µiθµ1θµ2 · · · θµi , c̄ (i) = 1 D D∑ a=1 c(i)(xa) . (S12)
Here we have used the convention that the repeated weight indices µj are summed over.
Gradient Descent As a result of the gradient descent weight update, θt+1 = θt − η ∂Ltrain∂θ , the weights at time T are a polynomial of degree (K − 1)T in the c̄(i).
θT ∈ P(K−1)T [ c̄(0), c̄(1), . . . , c̄(K) ] . (S13)
The coefficients of this polynomial depend on the initial weights, θ0. Plugging these weights back into the network output, we have that the network function at time T is again a polynomial in c̄(i), now with degree J (K − 1)T .
fT (x) ∈ PJ(K−1)T [ c̄(0), c̄(1), . . . , c̄(K) ] . (S14)
Thus, again using Lemma 1, fT concentrates with variance O(D−1). ED [ (fT − ED [fT ])2 ] = O ( D−1 ) . (S15)
and by Theorem 1 the loss will obey they variance-limited scaling.
Stochastic Gradient Descent We now consider the same setup of polynomial loss, but now trained via stochastic gradient descent (SGD). We consider SGD batches drawn i.i.d. with replacement and are interested in the test loss averaged over SGD draws, with fixed batch size, B.
We proceed by proving the following lemma, which allows us to reuse a similar argument to the GD case.
Lemma 2. Let c̃(i;t) = 1B ∑ a∈Bt c (i) a for i = 0 . . . J be mini-batch averages, over B i.i.d. draws of c (i) a . Further, let X = (c̃(0;t0))k0(c̃(1;t1))k1 · · · (c̃(J;tJ ))kJ be a monomial in the mini-batch means. Then EB [X] ∈ P∑J i=0 ki [ d̄(0), d̄(1), . . . , d̄( ∏J i=0(ki+1)−1) ] , where d̄(i) are empirical means over the full training set of i.i.d. random variables as in Lemma 1 and EB [•] denotes the expectation over draws of SGD batches of size B.
Proof. Expectations over draws of batches at different time steps are independent. Thus, WLOG, we can consider t := t0 = t1 = · · · = tJ . We can again proceed by direct computation, expanding the mini-batch sums.
EB [X] = 1 B ∑J i=0 ki EB  ∑ {a(i)α }∈Bt ( c (0) a (0) 1 · · · c(0) a (0) k0 )( c (1) a (1) 1 · · · c(1) a (1) k1 ) · · · ( c (J) a (J) 1 · · · c(J) a (J) kJ ) . (S16)
To proceed, we must keep track of terms in the sum where the a(i)α take the same or different values. If all a(i)α are different, the expectation over batch draws fully factorizes. More generally (S16) can be decomposed as a sum over products.
One way of keeping track of the index combinatorics is to introduce a set of graphs, Γ, where each graph γ ∈ Γ has k0 vertices of type 0, k1 vertices of type 1, . . . , and kJ vertices of type J (one vertex for each a(i)α index). Any pair of vertices may have zero or one edge between them. For any set of three vertices, v1, v2, and v3 with edges (v1, v2) and (v2, v3) there must also be an edge (v1, v3). The set Γ consists of all possible ways of connecting these vertices consistent with these rules.
For each graph, γ, we denote connected components by σ and denote the number of vertices of type i within the connected component σ by m(i)σ . With this we can write the sum, (S16) as
EB [X] = ∑ γ∈Γ Sγ(B) ∏ σ∈γ EB
[ 1
B ∑ a∈Bt ( c(0)a )m(0)σ ( c(1)a )m(1)σ · · · ( c(J)a
)m(J)σ ]
= ∑ γ∈Γ Sγ(B) ∏ σ∈γ 1 D D∑ a=1 ( c(0)a )m(0)σ ( c(1)a )m(1)σ · · · ( c(J)a )m(J)σ = ∑ γ∈Γ Sγ(B) ∏ σ∈γ d̄({m (0) σ ,m (1) σ ,...,m (J) σ }) .
(S17)
Here Sγ(B) is a combinatoric factor associated to each graph, not relevant for the argument. The m (i) σ take on values 0 to ki, so the multi-index, can take on ∏J i=1(ki + 1) different values, which we
re-index to d̄(0), d̄(1), . . . , d̄( ∏J i=1(ki+1)−1). Meanwhile, the degree of (S17) in d̄(i) is bounded by the
number of total vertices in each graph, i.e. ∑J i=0 ki. This establishes the lemma.
For a polynomial loss of degree K, the mini-batch training loss at each time step takes the form
L (t) train = K∑ i=1 c̃(i;t)µ1µ2...µiθµ1θµ2 · · · θµi , c̃ (i;t) = 1 B ∑ a=∈Bt c(i)(xa) . (S18)
The update rule, θt+1 = θt − η ∂L (t+1) train ∂θ ensures that θT is a polynomial of degree (K − 1)
T in the c̃(i;0), c̃(i;1), · · · , c̃(i;T )
θT ∈ P(K−1)T [ c̃(0;0), c̃(0;1), . . . , c̃(0;T ), c̃(1;0), c̃(1;1), . . . , c̃(1;T ), . . . , c̃(K;0), c̃(K;1), . . . , c̃(K;T ) ] ,
(S19)
and consequently, denoting the test loss evaluated at θT by L[θT ], L[θT ] ∈ PK(K−1)T [ c̃(0;0), c̃(0;1), . . . , c̃(0;T ), c̃(1;0), c̃(1;1), . . . , c̃(1;T ), . . . , c̃(K;0), c̃(K;1), . . . , c̃(K;T ) ] .
(S20)
Using Lemma 2, the expectation of L[θT ] over draws of SGD batches is given by EB [L[θT ]] ∈ PK(K−1)T [ d̄(0), . . . , d̄(K K(K−1)TK) ] . (S21)
Finally, denoting ED [EB [L[θT ]]] by L(D;B) and applying Lemma 1 gives
L(D;B)− lim D→∞
L(D;B) = O ( D−1 ) . (S22)

heading: D.3 NON-SMOOTH EXAMPLES
text: Here we present two worked examples where non-bounded or non-smooth loss leads to violations of the variance dominated scaling. In example one, the system obeys the variance dominated scaling at early times, but exhibits different behavior for times larger than the dataset size. In the second example, the system violates the variance dominated scaling even for two gradient descent steps, as a result of an unbounded derivative in the loss.3
Example 1 – unbounded loss at late times Consider a dataset with two varieties of data points, drawn with probabilities α and 1− α, and one-dimensional quadratic losses, `1 (concave up) and `2 (concave down), on these two varieties.
`1(θ) = 1
2 θ2 , `2(θ) = −
1 2 θ2 . (S23)
3We thank Anonymous for suggesting these two types of examples.
If, in a slight abuse of notation, we further denote the training loss on a sample with n1 points of type 1 and D−n1 points of type two by `n1 and the population loss at a given value of the weight by Lpop, we have
`n1 = ( n1 D − 1 2 ) θ2 , Lpop = ( α− 1 2 ) θ2 . (S24)
For this example we take α > 1/2. In this case, the minimum of the population loss is at zero, while the minimum of the training loss can be at zero, or at ±∞ depending on whether the training sample has n1 greater than or less than D/2. We can thus create a situation where at late training times, θT does not concentrate around the minimum of the population loss. As we work through this example explicitly, we will see the following. (i) A mismatch larger than O ( D−1 ) between the population minimum and the minimum found by training on a sample set of size D requires times T larger than a constant multiple of D. (ii) The quantity we study throughout this work is the difference between the infinite data limit of the test loss, and the finite data value, L(D)− limD→∞ L(D). The minimum of the infinite data limit of the test loss is not the same as the minimum of the population loss, min limD→∞ L(D) 6= minθ Lpop. In this example one diverges, while the other is finite. In particular this example evades the scaling result by L(D) for times larger than D having a diverging limit.
Explicitly, we study the evolution of the model under gradient flow. θ̇ = −2 ( n1 D − 1 2 ) θ , θT = e −2(n1D − 1 2 )T θ0 . (S25)
The test loss averaged over draws of the dataset is given by L(D;T ) = En1 [( α− 1
2
) θ2T ] = e2T ( α− 1
2
)( 1− α ( 1− e− 4TD ))D θ20 (S26)
If we consider this loss at large D and fixed T we get
L(D;T ) = e−4(α− 1 2 )T ( α− 1
2
) θ20 ( 1 +
8T 2α(1− α) D
+O ( D−2 )) , (S27)
and thus L(D;T )− limD→∞ L(D;T ) = O ( D−1 ) as expected.
If on the other hand we consider taking T D we have L(D;T D) = e2T ( α− 1
2
) (1− α)D θ20 , (S28)
the limit limD,T→∞ L(D;T D) diverges.
Lastly, we note that if we take T = βD with β < | log(1− α)|/2 we can approach the large D limit with non-generic, tuneable exponential convergence.
Example 2 – unbounded derivative Again, consider a two variety setup, this case with equal probabilities and per sample losses,
`1(θ) = 1
2 θ2 +
1
2α |θ|α , `2(θ) =
1 2 θ2 − 1 2α |θ|α . (S29)
We will consider different values of α > 0. The train loss and population loss are then,
`n1 = 1
2 θ2 +
1
α ( n1 D − 1 2 ) |θ|α , Lpop = 1 2 θ2 . (S30)
We consider a model initialized to θ0 = 1 and trained for two steps of gradient descent with learning rate 1.
gt = θt + ( n1 D − 1 2 ) θt|θt|α−2 , θt+1 = θt − gt . (S31)
Two update steps gives
θ2 = ∣∣∣∣n1D − 12 ∣∣∣∣α . (S32)
The test loss is given by the population loss evaluated at θ2 averaged over test set draws.
L(D) = En1 [ 1
2 θ22
] = 1
2D+1 D∑ n1=0 ( D n1 ) ∣∣∣∣n1D − 12 ∣∣∣∣2α
=
√ D
2π ∫ ∞ −∞ e−2D(x− 1 2 ) 2 ∣∣∣∣x− 12 ∣∣∣∣2α +O (D−1) = Γ ( α+ 12 ) 21+α √ π D−α +O ( D−1 ) .
(S33)
Here we have approximated the binomial distribution at large D with a normal distribution using Stirling’s approximation.
Note that if α ≥ 1 then L(D)− limD→∞ L(D) = O ( D−1 ) i.e. the finite sample loss approaches the infinite data loss with the predicted variance-limited scaling. For 0 < α < 1, we get a different scaling controlled by α. Note that the gradient, expression (S31) is singular at the origin for α precisely in this range.
In summary, this example achieves a different scaling exponent through a diverging gradient.

heading: E PROOF OF THEOREMS 2 AND 3
text: In this section we detail the proof of Theorems 2 and 3. The key observation is to make use of the fact that nearest neighbor distances for D points sampled i.i.d. from a d-dimensional manifold have mean ED,x [|x− x̂|] = O ( D−1/d ) , where x̂ is the nearest neighbor of x and the expectation is the mean over data-points and draws of the dataset see e.g. (Levina & Bickel, 2005).
The theorem statements are copied for convenience. In the main, in an abuse of notation, we used L(f) to indicate the value of the test loss as a function of the network f , and L(D) to indicate the test loss averaged over the population, draws of the dataset, model initializations and training. To be more explicit below, we will use the notation `(f(x)) to indicate the test loss for a single network evaluated at single test point.
Theorem 2. Let `(f), f and F be Lipschitz with constants KL, Kf , and KF and `(F) = 0. Further let D be a training dataset of size D sampled i.i.d fromMd and let f(x) = F(x), ∀x ∈ D then L(D) = O ( KLmax(Kf ,KF )D−1/d ) .
Proof. Consider a network trained on a particular draw of the training data. For each training point, x, let x̂ denote the neighboring training data point. Then by the above Lipschitz assumptions and the vanishing of the loss on the true target, we have `(f(x)) ≤ KL |f(x)−F(x)| ≤ KL (Kf +KF ) |x− x̂|. With this, the average test loss is bounded as
L(D) ≤ KL (Kf +KF )ED,x [|x− x̂|] = O ( KLmax(Kf ,KF )D−1/d ) . (S34)
In the last equality, we used the above mentioned scaling of nearest neighbor distances.
Theorem 3. Let `(f), f and F be Lipschitz with constants KL, Kf , and KF . Further let f(x) = F(x) for P points sampled i.i.d fromMd then L(P ) = O ( KLmax(Kf ,KF )P−1/d ) .
Proof. Denote by P the P points, z, for which f(z) = F(z). For each test point x let x̂ denote the closest point in P , x̂ = argminP (|x− z|). Adopting this notation, the result follows by the same argument as Theorem 2.

heading: F RANDOM FEATURE MODELS
text: Here we present random feature models in more detail. We begin by reviewing exact expressions for the loss. We then go onto derive its asymptotic properties. We again consider training a model f(x) = ∑P µ=1 θµfµ(x), where fµ are drawn from some larger pool of features, {FM},
fµ(x) = ∑S M=1 PµMFM (x).
Note, if {FM (x)} form a complete set of functions over the data distribution, than any target function, y(x), can be expressed as y = ∑S M=1 ωMFM (x). The extra constraint in a teacher-student model is specifying the distribution of the ωM . The variance-limited scaling goes through with or without the teacher-student assumption, however it is crucial for analysing the variance-limited behavior.
As in Section 2.3 we consider models with weights initialized to zero and trained to convergence with mean squared error loss.
Ltrain = 1
2D D∑ a=1 (f(xa)− ya)2 . (S35)
The data and feature second moments play a central role in our analysis. We introduce the notation,
C = Ex [ F (x)FT (x) ] , C̄ = 1
D D∑ a=1 F (xa)F T (xa) , C = PCPT , C̄ = PC̄PT .
K(x, x′) = 1 S FT (x)F (x′) , K̄ = K ∣∣∣ Dtrain , K(x, x′) = 1 P fT (x)f(x′) , K̄ = K ∣∣∣ Dtrain . (S36)
Here the script notation indicates the full feature space while the block letters are restricted to the student features. The bar represents restriction to the training dataset. We will also indicate kernels with one index in the training set as ~K(x) := K(x, xa=1...D) and ~K(x) := K(x, xa=1...D). After this notation spree, the test loss can be written for under-parameterized models, P ≤ D as
L(D,P ) = 1 2S ED [ Tr ( C + C̄PT C̄−1CC̄−1PC̄ − 2C̄PT C̄−1PC )] . (S37)
and for over-parameterized models (at the unique minimum found by GD, SGD, or projected Newton’s method),
L(D,P ) = 1
2 Ex,D
[ K(x, x) + ~K(x)T K̄−1K̄K̄−1 ~K(x)− 2 ~K(x)T K̄−1 ~K(x) ] . (S38)
Here the expectation ED [•] is an expectation with respect to iid draws of a dataset of size D from the input distribution, while Ex [•] is an ordinary expectation over the input distribution. Note, expression (S37) is also valid for over-parameterized models and (S38) is valid for under-parameterized models if the inverses are replaces with the Moore-Penrose pseudo-inverse. Also note, the two expressions can be related by echanging the projections onto finite features with the projection onto the training dataset and the sums of teacher features with the expectation over the data manifold. This realizes the duality between dataset and features discussed above.

heading: F.1 ASYMPTOTIC EXPRESSIONS
text: We are interested in (S37) and (S38) in the limits of large P and D.
Variance-limited scaling We begin with the under-parameterized case. In the limit of lots of data the sample estimate of the feature feature second moment matrix, C̄, approaches the true second moment matrix, C. Explicitly, if we define the difference, δC by C̄ = C + δC. We have
ED [δC] = 0
ED [δCM1N1δCM2N2 ] = 1
D (Ex [FM1(x)FN1(x)FM2(x)FN2(x)]− CM1N1CM2N2) ED [δCM1N1 · · · δCMnNn ] = O ( D−2 ) ∀n > 2 . (S39)
The key takeaway from (S39) is that the dependence on D is manifest.
Using these expressions in (S37) yields.
L(D,P ) = 1 2S Tr ( C − CPTC−1PC ) + 1
2DS P∑ M1,2N1,2=1 TM1N1M2N2 [ δM1M2 ( PTC−1P ) N1N2 + (C−1PC2PTC−1)M1M2C−1N1N2
−2 ( CPTC−1P ) M1M2 ( PTC−1P ) N1N2 ] +O ( D−2 ) .
(S40)
Here we have introduced the notation, TM1N1M2N2 = Ex [FM1(x)FN1(x)FM2(x)FN2(x)].
As above, defining
L(P ) := lim D→∞
L(D,P ) = 1 2S Tr ( C − CPTC−1PC ) . (S41)
we see that though L(D,P )− L(P ) is a somewhat cumbersome quantity to compute, involving the average of a quartic tensor over the data distribution, its dependence on D is simple.
For the over-parameterized case, we can similarly expand (S38) usingK = K+δK. With fluctuations satisfying,
EP [δK] = 0
EP [δKa1b1δKa2b2 ] = 1
P (EP [fµ(xa1)fµ(xb1)fµ(xa2)fµ(xb2)]−Ka1b1Ka2b2) EP [δKa1a1 · · · δKanan ] = O ( P−2 ) ∀n > 2 .
(S42)
This gives the expansion
L(D,P ) = 1
2 Ex,D
[ K(x, x)− ~K(x)T K̄−1 ~K(x) ] +O(P−1) , (S43)
and
L(D) = 1
2 Ex,D
[ K(x, x)− ~K(x)T K̄−1 ~K(x) ] . (S44)
Resolution-limited scaling We now move onto studying the parameter scaling of L(P ) and dataset scaling of L(D). We explicitly analyse the dataset scaling of L(D), with the parameter scaling following via the dataset parameter duality.
Much work has been devoted to evaluating the expression, (S44) (Williams & Vivarelli, 2000; Malzahn & Opper, 2002; Sollich & Halees, 2002). One approach is to use the replica trick – a tool originating in the study of disordered systems which computes the expectation of a logarithm of a random variable via simpler moment contributions and analyticity assumption (Parisi, 1980). The replica trick has a long history as a technique to study the generalization properties of kernel methods (Sollich, 1998; Malzahn & Opper, 2001; 2003; Urry & Sollich, 2012; Cohen et al., 2019; Gerace et al., 2020; Bordelon et al., 2020). We will most closely follow the work of Canatar et al. (2021) who use the replica method to derive an expression for the test loss of linear feature models in terms of the eigenvalues of the kernel C and ω̄, the coefficient vector of the target labels in terms of the model features.
L(D) = κ2 1− γ ∑ i
λiω̄ 2 i
(κ+Dλi) 2 ,
κ = ∑ i κλi κ+Dλi , γ = ∑ i
Dλ2i
(κ+Dλi) 2 .
(S45)
This is the ridge-less, noise-free limit of equation (4) of Canatar et al. (2021). Here we analyze the asymptotic behavior of these expressions for eigenvalues satisfying a power-law decay, λi = i−(1+αK) and for targets coming from a teacher-student setup, w ∼ N (0, 1/S).
101 102 103 Feature size (P, Solid) , Dataset size (D, Dashed)
10 1
100
101
102
Lo ss
Fixed Low Regularization
101 102 103 Feature size (P, Solid) , Dataset size (D, Dashed)
10 1
2 × 10 2
3 × 10 2
4 × 10 2
6 × 10 2
Tuned Regularization
101
102
103
P / D
Figure S6: Duality between dataset size vs feature number in pretrained features Using pretrained embedding features of EfficientNet-B5 (Tan & Le, 2019) for different levels of regularization, we see that loss as function of dataset size or loss as a function of the feature dimension track each other both for small regularization (left) and for tuned regularization (right). Note that regularization strength with trained-feature kernels can be mapped to inverse training time (Ali et al., 2019; Lee et al., 2020). Thus (left) corresponds to long training time and exhibits double descent behavior, while (right) corresponds to optimal early stopping.
To begin, we note that for teacher-student models in the limit of many features, the overlap coefficients ω̄ are equal to the teacher weights, up to a rotation ω̄i = OiMwM . As we are choosing an isotropic Gaussian initialization, we are insensitive to this rotation and, in particular, Ew [ ω̄2i ]
= 1/S. See Figure S8 for empirical support of the average constancy of ω̄i for the teacher-student setting and contrast with realistic labels.
With this simplification, we now compute the asymptotic scaling of (S45) by approximating the sums with integrals and expanding the resulting expressions in large D. We use the identities:∫ ∞
1
dx x−n(1+α)( κ+Dx−(1+α) )m = κ−m Γ
( n− 11+α ) (1 + α)Γ ( n+ α1+α ) 2F1(m,n− 1 1 + α , n+ α 1 + α , −D κ ) 2F1 (a, b, c,−y) ∝ y−a + By−b + . . . ,
(S46)
Here 2F1 is the hypergeometric function and the second line gives its asymptotic form at large y. B is a constant which does not effect the asymptotic scaling.
Using these relations yields
κ ∝ D−αK , γ ∝ D0, and L(D) ∝ D−αK , (S47)
as promised. Here we have dropped sub-leading terms at large D. Scaling behavior for parameter scaling L(P ) follow via the dataset parameter duality.

heading: F.2 DUALITY BEYOND ASYMPTOTICS
text: Expressions (S37) and (S38) are related by changing projections onto finite feature set, and finite dataset even without taking any asymptotic limits. We thus expect the dependence of test loss on parameter count and dataset size to be related quite generally in linear feature models. See Section G for further details.

heading: G LEARNED FEATURES
text: In this section, we consider linear models with features coming from pretrained neural networks. Such features are useful for transfer learning applications (e.g. Kornblith et al. (2019); Kolesnikov et al. (2019)). In Figures S6 and S7, we take pretrained embedding features from an EfficientNet-B5
101 102 103 104 Dataset size (D)
10 5 10 4 10 3 10 2 10 1 100 101 102 Lo ss - Lo ss ( )
Variance-limited: Theory D = 1 P=10, D=1.13 P=25, D=1.25 P=35, D=1.28
101 102 103 104 Dataset size (D)
10 2
10 1
100
Lo ss
Resolution-limited: Theory D = P P=1156, D=0.23 P=1587, D=0.25 P=2048, D=0.26
101 102 103 Feature size (P)
10 1
100
Lo ss
Resolution-limited: Theory P = D D=1156, P=0.25 D=1587, P=0.27 D=2048, P=0.28
101 102 103 Feature size (P)
10 3 10 2 10 1 100 101 102 Lo ss - Lo ss ( )
Variance-limited: Theory P = 1 D=10, P=1.27 D=25, P=1.35 D=35, P=1.41
Dataset size (D) 10 5
10 4
10 3
10 2
10 1
Lo ss
- Lo
ss (
)
Variance-limited: Theory D = 1
P=10, D=1.09 P=25, D=1.04 P=35, D=1.00
Dataset size (D) 10 2
Lo ss
Resolution-limited: Theory D = P
P=1156, D=0.29 P=1587, D=0.30 P=2048, D=0.30
101 102 103 Feature size (P)
2 × 10 2
3 × 10 2
4 × 10 2
6 × 10 2
Lo ss
Resolution-limited: Theory P = D D=1156, P=0.28 D=1587, P=0.28 D=2048, P=0.29
101 102 103 Feature size (P)
10 3
10 2
10 1
Lo ss
- Lo
ss (
)
Variance-limited: Theory P = 1 D=10, P=1.13 D=25, P=1.07 D=35, P=1.05
Figure S7: Four scaling regimes exhibited by pretrained embedding features Using pretrained embedding features of EfficientNet-B5 (Tan & Le, 2019) for fixed low regularization (left) and tuned regularization (right), we can identify four regimes of scaling using real CIFAR-10 labels.
model (Tan & Le, 2019) using TF hub4. The EfficientNet model is pretrained using the ImageNet dataset with input image size of (456, 456). To extract features for the (32, 32) CIFAR-10 images, we use bilinear resizing. We then train a linear classifier on top of the penultimate pretrained features. To explore the effect feature size, P , and dataset size D, we randomly subset the feature dimension and training dataset size and average over 5 random seeds. Prediction on test points are obtained as a kernel ridge regression problem with linear kernel. We note that the regularization ridge parameter can be mapped to an inverse early-stopping time (Ali et al., 2019; Lee et al., 2020) of a corresponding ridgeless model trained via gradient descent. Inference with low regularization parameter denotes training for long time while tuned regularization parameter is equivalent to optimal early stopping.
In Figure S7 we see evidence of all four scaling regimes for low regularization (left four) and optimal regularization (right four). We speculate that the deviation from the predicted variance-limited exponent αP = αD = 1 for the case of fixed low regularization (late time) is possibly due to the double descent resonance at D = P which interferes with the power law fit.
In Figure S6, we observe the duality between dataset size D (solid) and feature size P (dashed) – the loss as a function of the number of features is identical to the loss as function of dataset size for both the optimal loss (tuned regularization) or late time loss (low regularization).
In Figure S8, we also compare properties of random features (using the infinite-width limit) and learned features from trained WRN 28-10 models. We note that teacher-student models, where the feature class matches the target function and ordinary, fully trained models on real data (Figure 1a), have significantly larger exponents than models with fixed features and realistic targets.
The measured ω̄i – the coefficient of the task labels under the i-th feature (S45) are approximately constant as function of index i for all teacher-student settings. However for real targets, ω̄i are only constant for the well-performing Myrtle-10 and WRN trained features (last two columns).
4https://www.tensorflow.org/hub
101 102 103 104 Dataset size (D)
10 1Lo ss
FC TS: D = 0.20 Real: D = 0.05
101 102 103 104 Dataset size (D)
10 1
3 × 10 2 4 × 10 2 6 × 10 2
2 × 10 1 CNN-VEC
TS: D = 0.23 Real: D = 0.07
101 102 103 104 Dataset size (D)
10 4
10 3
10 2
10 1 Myrtle-10
TS: D = 0.41 Real: D = 0.15
101 102 103 104 Dataset size (D)
10 12 10 9 10 6 10 3
WRN pretrained
TS: D = 2.52 Real: D = 0.25
100 101 102 103 104 i
10 2
10 1
100
i C i i/
0
K = 0.26
100 101 102 103 104 i
10 2
10 1
100 K = 0.26
100 101 102 103 104 i
10 4 10 3 10 2 10 1 100
K = 0.46
100 101 102 103 104 i
10 14
10 10
10 6
10 2 K = 1.31
100 101 102 103 i
10 8
10 5
10 2
101
No rm
al ize
d Va
lu e
2 TS 0.03 2 Real 1.41
i
100 101 102 103 i
10 8
10 5
10 2
101
2 TS 0.03 2 Real 1.37
i
100 101 102 103 i
10 8
10 5
10 2
101
2 TS -0.07 2 Real -0.45
i
100 101 102 103 i
10 10
10 7
10 4
10 1
102
2 TS -0.07 2 Real -0.40
i
Figure S8: Loss on the teacher targets scale better than real targets for both untrained and trained features The first three columns are infinite width kernels while the last column is a kernel built out of features from the penultimate layer of pretrained WRN 28-10 models on CIFAR-10. The first row is the loss as a function of dataset size D for teacher-student targets vs real targets. The observed dataset scaling exponent is denoted in the legend. The second row is the normalized partial sum of kernel eigenvalues. The partial sum’s scaling exponent is measured to capture the effect of the finite dataset size when empirical αK is close to zero. The third row shows ω̄i for teacher-student and real target compared against the kernel eigenvalue decay. We see the teacher-student ω̄i are approximately constant.

Review summaries:
The paper proposes a phenomenon called scaling laws of test loss for neural networks. In a nutshell, the test loss decreases by following a power law curve (exponent) with respect to width or data set size, while the other being fixed.  Specifically, the variance limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. The proofs are for linear models (random feature) and pretrained models. Authors also observe several empirical relationships between datasets and scaling exponents: super-classing image tasks does not change exponents, while changing input distribution (via changing datasets or adding noise) has a strong effect.

This paper characterized the scaling of generalization error with respect to the number of training samples $D$ and parameters $P$ for certain smooth estimators in two regimes: the variance-limited regime and the resolution-limited regime; this divide is somewhat analogous to the parametric vs. nonparametric estimation. The authors also argued that the large-width and large-sample limit have similar scaling behavior. The findings are supported by empirical evidence. 

This paper considered Variance-Limited Regime and Resolution-Limited Regime to explain the explored neural scaling law. In the Variance-Limited Regim, one fixed one of the D and P and in the Resolution-Limited Regime, one of the D and P is effectively infinite. The paper aims to provide a parametric rate scaling law in the Variance-Limited Regime  and a non-parametric scaling law in the Resolution-Limited Regime.