The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes
Authors: Alexander Atanasov
Blake Bordelon
Sabarish Sainathan
Cengiz Pehlevan
Venue: ICLR 2023 poster (2023)
Keywords: Feature Learning
Neural Tangent Kernel
Scaling Laws
Deep Ensembles

Abstract: For small training set sizes $P$, the generalization error of wide neural networks is well-approximated by the error of an infinite width neural network (NN), either in the kernel or mean-field/feature-learning regime. However, after a critical sample size $P^*$, we empirically find the finite-width network generalization becomes worse than that of the infinite width network. In this work, we empirically study the transition from infinite-width behavior to this \textit{variance-limited} regime  as a function of sample size $P$ and network width $N$. We find that finite-size effects can become relevant for very small dataset sizes on the order of $P^* \sim \sqrt{N}$ for polynomial regression with ReLU networks. We discuss the source of these effects using an argument based on the variance of the NN's final neural tangent kernel (NTK). This transition can be pushed to larger $P$ by enhancing feature learning or by ensemble averaging the networks. We find that the learning curve for regression with the final NTK is an accurate approximation of the NN learning curve. Using this, we provide  a toy model which also exhibits $P^* \sim \sqrt{N}$ scaling and has $P$-dependent benefits from feature learning. 

Published as a conference paper at ICLR 2023
THE ONSET OF VARIANCE-LIMITE BEHAVIOR FOR NETWORKS IN THE LAZY AND RICH REGIMES
Alexander Atanasov" s1, Blake Bordelon" it, Sabarish Sainathan & Cengiz Pehlevan i1
$ Department of Physics
JJohn A. Paulson School of Engineering and Applied Sciences
CCetter for Brain Science
Harvard University
Cambridge, MA 02138, USA
{atanasov, blake bordelon, cpehlevan www.aaraaaa.....e edu
ABSTRACT
For small training set sizes P. the generalization error of wide neural networks is -plloopprox by the error of an infinite width neural network (NN), ei- ther in the kernel or mean- field/ffattur--lemmngg regime. However, after a critical sample size P we empirically find the finite-width network generalization be- comes worse than that of the infinite width network. In this work, we empirically study the transition from infinite-widd behavior to this variance-limited regime as a function of sample size P and network width N. We find that finite-size ef- fects can become relevant for very small dataset sizes on the order of pa ~ vN for polynomial regression with ReLU networks. We discuss the source of these effects using an argument based on the variance of the NN's final neural tangent kernel (NTK). This transition can be pushed to larger P by enhancing feature learning or by ensemble averaging the networks. We find that the learning curve for regression with the final NTK is an accurate approximation of the NN learning curve. Using this, we provide a toy model which also exhibits p VN scaling and has P-dependent benefits from feature learning.
INTRODUCTION
Deep learning systems are achieving state of the art performance on a variety of tasks (Tan & Le 2019 Hoffmann al. 2022). Exactly how their generalization is controlled by network architecture, training procedure, and task structure is still not fully understood. One promising direction for deep learning theory in recent years the infinite-width limit. Under a certain parameterization, infinite- width networks yield a kernel method known as the neural tangent kernel (NTK) (Jacot et al.) [2018] Lee et al. 2019). Kernel methods are easier to analyze, allowing for accurate prediction of the generalization performance of wide networks in this regime (Bordelon et al., 2020 Canatar et al.) 2021 Bahri et al. 202T SSmon et al.) (02T). Infinite-width networks can also operate the mean- field regime if network outputs are rescaled by a small parameter that enhances feature learning et al. 2018 Chizat et al. 2019 Geiger et al.) 2020b Yang & Hu 2020 Bordelon & Pehlevan 2022).
While width networks provide useful limiting cases for deep learning theory, real networks have finite width. Analysis at finite width is more difficult, since predictions dependent on the initialization of parameters. While several works have attempted to analyze feature evolution and kernel statistics at large but finite width (Dyer & Gur-Ari 2020. Roberts et al.] (002), the implications of finite width on generalization are not entirely clear. Specifically, it is unknown at what value of the training set size P the effects of finite width become relevant, what impact this critical P has on the learning curve, and how it is affected by feature learning.
To identify the effects of finite width and feature learning on the deviation from infinite width learn- ing curves, we empirically study neural networks trained across wide range of output scales a, widths N, and training set sizes P on the simple task of polynomial regression with a ReLU neural network. Concretely, our experiments show the following:
"These authors contributed equally.

---

Published as a conference paper at ICLR 2023
Learning curves for polynomial regression transition exhibit significant finite-wind width effects very early, around P VN. Finite- NNs at large are always outperformed by their anfinite-width counterparts. We show this gap is driven primarily by variance of the predic- tor over initializations (Geiger et al. (020a). Following prior work (Bahri et al. 2021) ) refer to this as the limited regime. We compare three d ensembling methods reduce error in this regime.
Feature-lcarning NNs show improved generalization both before and after the transition to the variance limited regime, Feature learning can be enhanced through re-scaling the output of the network by small scalar by training on more complex task (a higher-degree polynomial). We show that alignment between the final NTK and the target function on test data improves with feature learning and sample size.
We demonstrate that the learning curve for the NN is well-captured by the learning curve for kernel regression with the final empirical NTK, eNTK, as has been observed in other works VVaas et al. 2022 Geiger et al. 20206] Atanasov et al. 2021 Wei et al. 2022).
Using this correspondence between the NN and the final NTK, we provide a cursory ac- count of how fluctuations in the final NTK over random initializations are suppressed at large width N and large feature learning strength. In a toy model, we reproduce several scaling phenomena, including the P ~ vN transition and the improvements due to feature learning through an alignment effect.
We validate that these effects qualitatively persist in the realistic setting of wide ResNets Zagoruyko & Komodakis (2017) trained on CIFAR in appendix E
Overall, our results indicate that the onset of finite-widie corrections to generalization in neural net- works become relevant when the scale of the variance of kernel fluctuations becomes comparable to the bias component of the generalization error in the contribution to generalization error can be reduced both through ensemble averaging and through feature learning, which we show promotes higher alignment between the final kernel and the task. We construct model of noisy random features which reproduces the essential aspects of our obser- vations.
decomposition. The variance
1.1 RELATED WORKS
Geiger al. (2200a) analyzed the scaling of network generalization with the number of model
parameters. Since the NTK fluctuates with variance O(N ) for width N network (Dyer & Gur- Ari 2020; Roberts et al. 202T), they find that finite width networks in the lazy regime generically perform worse than their infinite width counterparts.
The scaling laws of networks over varying N and P were also studied, both empirically and theoret- ically by Bahri et al. (202T). They consider two types of learning curve scalings. First, they describe resolution-
scaling, where either training set size or width are effectively infinite and the scal- ing behavior of generalization error with the other quantity is studied. There, the scaling laws can been obtained by the theory in [Bordelon et al. (020)). Second, they analyze variance-limin finite value and the other parameter is taken infinity.
scaling
where width training set size are fixed While that work showed for any fixed P that the learning curve converges to the infinite width curve as ONN--1, these asymptotics do not predict, for fixed N, which value of P the NN learning curve begins to deviate from the infinite width theory. This is the focus of our work.
The contrast between rich and lazy networks has been empirically studied in several prior works. Depending on the structure of the task. the lazy regime can have either worse (Fort et al,, 2020) or better (Ortiz-Jiménez al.)
Geiger et al. (2020b) performance than the feature learning regime. For our setting, where the signal depends on only a small number of relevant input directions, we expect representation learning to be useful, as discussed in (Ghorbani et al., 2020) Paccolat et al., 2021b). Consequently, we posit and verify that the rich network will outperform the lazy one.
Our toy model is inspired by the literature on random feature models. Analysis of generalization for two layer networks at initialization in the limit of high dimensional data have been carried out using techniques from random matrix theory (Mei & Montanari
2022 Hu & Lu 2020 Adlam & Pennington, 2020al Dhifallah & Lu| 2020 Adlam & Pennington 2020b) and statistical mechanics
(Gerace et al.) 2020; d' Ascoli et al. 2020; d Ascoli et al.) 2020). Several of these works have
identified that when N is comparable to P. the network generalization error has contribution
2

---

Published as a conference paper at ICLR 2023
from variance over initial parameters. Further, they provide theoretical explanation of the benefit of ensembling predictions of many networks trained with different initial parameters. Recently, Ba et al. (2022) studied regression with the hidden features of a two layer network after taking one step of gradient descent, finding significant improvements to the learning curve due to feature learning. [Zavatone-Vie Veth et al.] (2022) analyzed linear regression for Bayesian deep linear networks with width N comparable to sample size P and demonstrated the advantage of training multiple layers compared only training the only last layer, finding that feature learning advantage has leading correction of scale (P/N)² ? at small P/N.
PROBLEM SETUP AND NOTATION
We consider a supervised task with dataset D = {xx y* are drawn from population distribution p(ær, y). Our experiments will focus training networks to interpolate degree k polynomials on the sphere (full details in Appendix [A]. For this task, the infinite width network learning curves can be found analytically. In particular at large P the generalization error scales as //2 (Bordelon et al. 2020). We take single output feed-forward NN fo R + I with hidden width N for each layer. We let 0 denote all trainable parameters of the network. Using NTK parameterization (lacot et al. 2018), 8). the activations for an input æ are given by
رس= of size P. The pairs of data points
D W
p(h L 2...... L,
h
zj.
(I)
VD
Here, the output of the network is fo n1 We take yp to be positively homogenous function, in our case a ReLU nonlinearity, but this is not strictly necessary (Appendix C.2). At initialization we have Wi (N, 1). Consequently, the scale of the output at initialization is O(a²). As consequence of the positive homogeneity of the network, the scale of the output is given by ot. controls the feature learning strength of given NN. Large a corresponds to lazy network while small yields rich network with feature movement. More details on how controls feature learning are given in Appendix C.I and C.2
In what follows, will denote the infinite width NTK limit of this network by NTK We will denote its finite width linearization by eN (2x x') == Eg 0g fax (2) l0=0y. and we will de- note its linearization around its final parameters oy by eNTK (xx xx) )= Ee f (x)0s fxx')) e== Following other authors (Chizat et al. 2019 Adlam & Pennington (020a). we will take the output to be fa == fo( - (a.)..). Thus, at initialization the function output is 0. We explain this choice further in Appendix A The parameters
x':==)
then trained with full-batch gradient descent on a mean squared error loss. We denote the final network function starting from initialization Og on dataset D by ((z) or f for short. The generalization error is calculated using held-out test
set and approximates the population risk (f) = (((()) - -y) "=vrp())))
EMPIRICAL RESULTS
In this section, we will study learning curves for ReLU NNs trained on polynomial regression tasks of varying degrees. We take our task to be learning 1/D and Qx is the kth gegenbauer polynomial. We will establish the following key observations, which we will set out to theoretically explain in Section 4.
(((xx)) where 3 is random vector of norm
1. Both eNTK, and sufficiently lazy networks perform strictly worse than NTKg. but the ensembled predictors approach the NTK. test error.
2. NNs in the feature learning regime of small a can outperform NTK for an intermediate range of P. Over this range, the effect of ensembling is less notable.
3. Even richly trained finite width NNs eventually perform worse than NTK, at sufficiently large P. However, these small a feature-learning networks become variance-limited at larger P than lazy networks. Once the pariance-limited regime, all networks benefit from ensembling over initializations.
4. For all networks, the transition to the variance-limited regime begins at a P that scales sub-linearly with N. For polynomial regression, we find P / NN
3

---

Published as a conference paper at ICLR 2023
These findings support our hypothesis that finite width introduces variance in eNTK, over initial- izations. which ultimately leads to variance in the learned predictor and higher generalization error. Although we primarily focus on polynomial interpolation tasks in this paper, in Appendix[Flwe pro- vide results for wide ResNets trained on CIFAR and observe that rich networks also outperform lazy ones, and that lazy ones benefit more significantly from ensembling.
3.1 FINITE WIDTH EFFECTS CAUSE THE ONSET OF A VARIANCE LIMITED REGIME
2=2, 1-3, M-1000
M-1000
10-1
10-1
Eg
Eg
NIK.
NTK.
10-1
MN 0.3
10a4
MN ensembied o-
MN. 0.0.5
MN ansembteed 0-0.5
10"
MN. 10.0
2.1.0
NN ensembied p-1.0
ensembred p-15.0
ensembred 20
10-5
eNTK,
10-4
ensembled
10a
10'
10a
10a
(a) l: generalization error
(b) k = 2 20- ensemble error
km6, Lm M=1000
ked,,
M- M-
100
100
Ega-1
630-1
onsombied, 6-
0.5.
erseebtred --0.3
NN, 1.0
10-1
ersembied, 2=1.2
10
NN. e- 10.0
ensombredd 100
MN 20
NN ensembied 2-20
Wiik (g-
<NiK ersembred (g
10:
104
10
10a
10a
10a
(c) k = 4 generalization error
(d) k = 4 20-fold ensemble error
Figure 1: Generalization errors of depth L = 3 neural networks across range of a values compared to NTK Ko The regression for NTK was calculated using the Neural Tangents package (Novak) let al. 2020 The exact scaling of NTK, is known to go asymptotically as p-2 for this task. a) Lazy networks perform strictly worse than NTKoc while rich networks can outperform an intermediate range of P before their performance is also limited. b) Ensembling 20 networks substantially improves lazy network and eNTK,, generalization, as well as asymptotic rich network generalization. This indicates that at sufficiently large P. these neural networks become limited by variance due to initialization. The error bars in a) and c) denote the variance due to both both training set and initialization. The error bars in b), d) denote the variance due to the train set.
for
In this section, we first investigate how finite width NN learning curves differ from infinite width NTK regression. In Figure|] i we show the generalization error Fz for a depth 3 network with width N = 1000 trained on a quadratic = 2 and quartic A = 4 polynomial regression task. Additional plots for other degree polynomials are provided in Appendix [F] We sweep over P to show the effect of more data on generalization, which is the main relationship we are interested in studying. For each training set size we sweep over grid of 20 random draws of the train set and 20 random network initializations. This for 400 trained networks in total at each choice of P, k; N, a. We see that a discrepancy arises at large enough P where the neural networks begin to perform worse than NTK...
We probe the source of the discrepancy between finite width NNs and NT by ensemble averaging network predictions fo(æ)) := (xx)) over E = 20 initializations fy. In Figures [b and ld, calculate the error of [D)) each trained on the same dataset. We then plot Fg (TD). This
4

---

Published as a conference paper at ICLR 2023
10g A.00
v4n/e//% Led, xed, Ne-000
N000
40
10
000
Co
6o
0.04
45-
0.00
10 log?
2.5
log
(a) lo generalization error
(b) k variance fraction
(c) 3 alignment
Figure 2: Phase plots in the P. a plane of a) The log generalization error log (f*)), b) The frac- tion of generalization error removed by ensembling (ff Kemel-task alignment "² K,y measured by ly|,Trk, where y and K evaluated on test data. We have plotted 'x' markers in a) to show the points where the NNs were trained.
ensembled error approximates the bias in a bias-variance decomposition (Appendix B Thus, any gap between (a) and (b) is driven by variance fø,D over 0.
We sharpen these observations with phase plots of NN generalization, variance and kernel align- ment over P, a, as shown in Figure 2 Figure 2, generalization for NNs in the rich regime (small a) have lower final Eg than lazy networks. As the dataset grows, the fraction of Eg due to initialization variance (that is, the fraction removed by ensembling) strictly increases (b).. We will show why this effect occurs in section 3.2 Figure 2 shows that, at any fixed P. the vari- ance lower for small a. To measure the impact of feature learning on the eNTK,, we plot its
alignment with the target function, measured as v Ky
y!"T<K for a test set of targets [y], and kernel [K] = cNTK J. Alignment of the kernel with the target function is known to be related good generalization (Canatar et al.) (02I). In Section 4 we revisit these effects in simple model which relates kernel alignment and variance reduction.
In addition to initialization variance, variance over dataset 'D contributes the total generalization error. Following (Adlam & Pennington (2200)), we discuss symmetric decomposition of the variance in Appendix B showing the contribution from dataset variance and the effects of bagging. We find that most of the variance in experiments is due to initialization.
We show several other plots the results of these studies the appendix. We show the effect of bagging (Figure 7), phase plots of different degree target funetions (Figures I0 9). phase plots over N, a (Figure and comparison network predictions against the initial and final kernel
regressors (Figures 18 19p.
3.2 FINAL NTK VARIANCE LEADS TO GENERALIZATION PLATEAU
In this section, we show how the variance over initialization can be interpreted as kernel variance in both the rich and lazy regimes. We also show how this implies a plateau for the generalization error.
To begin, we demonstrate empirically that all networks have the same generalization error as kernel regression solutions with their final eNTKs. At large a, the initial and the final kernel are already close, so this follows from earlier results Chizat al. (2019). In the rich regime, the properties of the eNTK, have been studied in several prior works. Several have empirically demonstrated that the eNTK, is good match to the final network predictor for trained network (Long) 2021 Vyas jet al.) 2022; Wei et al. 2022) while others have given conditions under which such an effect would hold true (Atanasov al. 2021 Bordelon & Pehlevan, 2022). We comment this in appendix C.4 We show in Figure 3 how the final network generalization error matches the generalization error of eNTK, As consequence, we can use eNTK, study the observed generalization behavior.
Next, we relate the variance of the final predictor fin to the corresponding infinite width network for. The finite size fluctuations of the kernel at initialization have been studied in (Dyer & Gur-Ari
2020 Hanin & Nica 2019 Roberts al. 2021). The variance of the kernel elements has been
5

---

Published as a conference paper at ICLR 2023
km2,
L-3
B-177
Rwagt
10a1
10--
Eg
ML. 0.0.3
16-1
ctTK e-0.5
10-4
eNTK a-10.0
wark e-1.
16-3
eNIK e=20.0
10
10-4
NN
(a)
EN
Eg
NTK,
(b) ENN Eg
N across N, C
Figure 3: Kernel regression with eNTK reproduces the learning curves of the NN with high fidelity. (a) Learning curves across different laziness settings a a width 1000 network. The solid black curve the infinite width network. Colored curves are the NN generalizations. Stars represent the eNTKys, ys, and lie on top of the corresponding NN learning curves. (b) The agreement of generaliza- tions between NNs and eNTK ys across different N and a. Here the colors denote different values while the dot, triangle and star markers denote networks of N = {177, 421, 1000} respectively.
L-3, 2-2
L-3, k-2
1-7177
0.5.5 5
12-
p=1.
1000
o=10.0
10
10
#NTK
f/10
19'
(a) Scaling of P1/2 with
(b) Scaling of P1/2 with N
Figure 4: Critical sample size P1/2 measures the onset of the variance limited regime as function of a at fixed N. (a) More feature learning (small a)) delays the transition to the variance limited regime. (b) P1/2 as a function of N for fixed a has roughly P1/2 ~ VN scaling.
shown to scale as 1/N. We perform the following vlas-variance decomposition: Take fer.D to be the eNTKo predictor, or sufficiently lazy network trained to interpolation on a dataset D. Then,
(()) )))
D...y )e)))),,.,,,
(*) y)²) + O(1/N.
We demonstrate this equality using a relationship between the infinite-width network and an infinite ensemble of finite-width networks derived in Appendix B There we also show that the O(/NN) term strictly positive for sufficiently large N. Thus, for lazy networks of sufficiently large N. finite width effects lead to strictly worse generalization error. The decomposition in Equation 2 continues to hold for rich networks at small a for is interpreted as the infinite-width mean field limit. In this case limit in the mean-fied freld regime. See Appendix B for details.
show that ensembles of rich networks are approximating an infinite width
3.3 FEATURE LEARNING DELAYS VARIANCE LIMITED TRANSITION
We now consider how feature learning alters the onset of the variance limited regime, and how this onset scales with N. We define the onset of the variance limited regime to take place at the value P P1/2 where over half the generalization error is due to variance over initializations. Equivalently we have E/(f")/E,(f* = 1/2. By using an interpolation method together with bisection, solve for P1/2 and plot in Figure

---

Published as a conference paper at ICLR 2023
2=1.3
e-2.0
o-10.0
10"
107
16-1
10
NR. 1-10
CONN
Avp 1 e::3.
Avp K. a-10
10-4
App 9 s=10
Avg.
e-2.0
Avg
10
10
0-10.0
AVD
101
10
10
12
107
107
122
122
102 100
107
10
(a) Ensembling Methods
(b) Reduction in Egg for Each Ensembling Technique
Figure 5: The random feature model suggests three possible types of ensembling: averaging the output function f(x, 0), averaging eNTK, (xx, æ'; f). and averaging the induced features (z 0). We analyze these ensembling methods for k task with width N 100 ReLU network. (a) While all ensembling methods improve generalization, averaging either the kernel (K) or features (v) gives a better improvement generalization than averaging the output function (f). Computing final kernels for many richly trained networks and performing regression with this averaged kernel gives the best performance.
(b) We plot the relative error of each ensembling method against the single init neural network. The gap between ensembling and the single init NN becomes evident for sufficiently large P1/2- For small a, all ensembling methods perform comparably, while for large ensembling the kernel or features gives much lower Egg than averaging the predictors.
Figure 4p shows that P1/2 scales as VN for this task. In the next section, we shall show that this scaling Is governed by the fact that Pi/2 close to the value where the infinite width network generalization curve E is equal to the variance of eNTKy In this case the quantities to compare L.g R p-2 and Var eNTK, R N-1
We can understand the delay of the variance limited transition, as well as the lower value of the final plateau using mechanistic picture similar to the effect observed in Attanasov al. (202T). In that setting, under small initialization, the kernel follows a deterministic trajectory. picking up a low rank component in the direction of the train set targets yy and then changing only in scale the network weights grow to interpolate the dataset. In their case, for initial output scale ,f, eNTK is deterministic up variance of O(a). In case, the kernel variance g2L/ /N. As a + 0 the kernel's ' trajectory becomes deterministic up to a variance term scaling with (a), which implies that the final predictor also has variance scaling as O(r).
initialization scales as
4 SIGNAL PLUS NOISE CORRELATED FEATURE MODEL
In Section 3.2 we have shown that in both the rich and lazy regimes, the generalization error of the NN is well approximated by the generalization of kernel regression solution with eNTK,, This finding motivates an analysis of the generalization of kernel machines which depend on network initialization 6u. Unlike many analyses of random feature models which specialize two layer networks and focus on high dimensional Gaussian random data (Mei & Montanari 2022 Adlam Pennington 2020a Gerace et al.] 2020: Ba et al. 2022), we propose to analyze regression with the eNTK, f for more general feature structures. This work builds on the kernel generalization theory for kernels developed with statistical mechanics (Bordelon et al.) 2020 Canatar et al.] 2021 Simon et al.) 2 Loureiro et al. 2021). We will attempt to derive approximate learning curves in terms of the ENTKy'ss signal and noise components, which provide some phenomenological explanations of the onset of the variance limited regime and the benefits of feature learning. Starting with the final NTK Ko, (xc, x²) which depends on the random initial parameters fg. we project its square root
k-1/2(x (x., (as defined in equation 32) fixed basis (z)))) orthonormal with respect to p(ar). This defines feature map
4/x (ax, 0o) f dde'
p(x')(()) d(''pp('))) x x')bb (x'). ls (1,
The kernel can reconstructed from these features K (x, x') Ex (x, Bo) (x', ,0)). The kernel interpolation problem can be solved by performing linear regression with features tb(x, Bo). Here, w((o) limx-0 argmin
Σp [w (b p 0n) ynl2 + A/we/2 The learned function

---

Published as a conference paper at ICLR 2023
f(z w((6) (zx Bu) is the minimum norm interpolator for the kernel ((x, x'; 0b) and matches the neural network learning curve as seen in Section 3.2, In general, since the rank of K is finite for a finite size network, the (x, 0o) have correlation matrix of finite rank Nz. Since the target function y does not depend on the initialization On. we decompose in terms of a fixed set of features (a)) E R.M (for example, the first M basis functions (bx } k2)) In this random feature model, one can interpret the initializationddeeedden fluctuations in K (æ, x'; 0o) as generat- ing fluctuations in the features (ar 0o) which induce fluctuations in the learned network predictor (xr fo). To illustrate the relative improvements to generalization from denoising these three differ- ent objects, in F we compare averaging the final kernel K, averaging the induced features d, and averaging network predictions f directly. For all at all ensembling methods provide improve- ments over training a single NN. However, we find that averaging the kernel directly and performing regression with this kernel exhibits the largest reduction in generalization error. Averaging features performs comparably. However, ensemble averaging network predictors does not perform as well as either of these other two methods. The gap between ensembling methods is more significant in the lazy regime (large a) and is negligible in the rich regime (small a).
4.1 TOY MODELS AND APPROXIMATE LEARNING CURVES
To gain insight into the role of feature noise, we characterize the test error associated with a Gaussian covariate model in a high dimensional limit P. M, Nu. - Do with a P/M, 17 = Nu/ /MM
1 M ass w", ff" VM s w, &b A(0o) + c.
M ΣM 0
y
N 0.
Σ,
(4)
This model was also studied by Loureiro et al. (2021) and subsumes the classic two layer random
feature models of prior works (Hu & Lu 2020 Adlam & Pennington, 2020a Mei & Montanari 2022 The expected generalization error for any distribution of A(O)) has the form
Es, Ey (o0) = =EA 1 7 M
1 1
w
A 1/2 - 1--p!// A GAE -/ A G AE
Σ M wx
G = +1A M1 A T + +Z.))
1== X + q TrGIAEMA' E,}
a
(5)
a TrG [AEM AT + Details of the calculation can be found in where a = P/M and y= (2+2)=
Appendix D We also provide experiments showing the predictive accuracy of the theory in Figure 6 In general, we do not know the induced distribution of A(f,) over disorder fg- In Appendix D.5 we compute explicit learning curves for simple toy model where A(0)) Ys entries as i.i.d. Gaussian over the random initialization Oo. A similar random feature model was recently analyzed with diagrammatic techniques by Maloney et al. (2022). In the high dimensional limit M, P. Nu do with P/M = a, N = our replica calculation demonstrates that test error is self-averaging (the same for every random instance of A) which we describe in Appendix D.5 and Figure 16
4.2 EXPLAINING FEATURE LEARNING BENEFITS AND ERROR PLATEAUS
Using this theory, can attempt explain some of the observed phenomena associated with the onset of the variance limited regime. First, we note that the kernels exhibit fluctuations over ini- tialization with variance O(1/N (I/N). either in the lazy or rich regime. In Figure 6 (a), we show learn- ing curves for networks of different widths in the lazy regime. Small width networks enter the variance limited regime earlier and have higher error. Similarly, if we alter the scale of the noise
>. of A. A in our toy model, the corresponding transition time P1/2 is smaller and the asymptotic error is higher. In Figure [6 (c), we show that our theory also predicts the onset of the variance limited regime at P1/2 ~ / N a ~ N-1. We stress that this scaling a consequence of the structure of the task. Since the target function is eigenfunction of the kernel, the infinite width error goes as 1/ p2 (Bordelon et al. 2020). Since variance scales as 1/N, bias and variance become comparable at P ~ v V.. Often, realistic tasks exhibit power law decays where Eg =o = P-8 with 2 (Spigler et al. 2020 Bahri et al. 2021). where we'd expect a transition around P1/2 N1/8
Using our model, we can also approximate the role of feature learning as enhancement in the signal correlation along task-relevant eigenfunctions. In Figure 0 (d) we plot the learning curves for net- works trained with different levels of feature learning, controlled by a. We see that feature learning leads to improvements in the learning curve both before and after onset of variance limits. In Figure
8

---

Published as a conference paper at ICLR 2023
1=2, 2-20
Toery tearniy Clase
10-4
231
291
10-22
10
MO. 2072
12
100
N' 1207
(a) Eg for different N
(b) Small N Large o²
(c) Variance Limited Transition
2--2
10-
124
16-1
10-
18-4
18
4/00
MTK.
2440
10-
MS. 2.0.5
20
MV. 5020
18-2
137
237
(d) Feature Scalings
(e) Richness sJ Amplified Me
(l) P-dependent Amplification
Figure 6: A toy model of noisy features reproduces qualitative dependence of learning curves on kernel fluctuations and feature learning.
(a) The empirical learning curves for networks of varying width N at large a. (b) Noisy kernel regression learning curve with noise Σ, = of M and A is projection matrix preserving 20-k top eigenmodes of Σv,, which was computed from the NTK for a depth 3 ReLU network. (c) This toy model reproduces the approximate scaling of the transition sample size Pl/2 /1/2 if of ~ N (d) NNs trained with varying richness a. Small a improves the early learning curve and asymptotic behavior. (e) Theory curves for kernel with amplified eigenvalue A - A + Ade for the target eigenfunction. This amplification mimics the effect of enhanced kernel alignment in the low a regime. Large amplification improves generalization performance. (f) P-dependent alignment where A'a ~ vP gives a better qualitative match to (d).
(e)- we plot the theoretical generalization for kernels with enhanced signal eigenvalue for the task eigenfunction y(æ=) Φu (4) This enhancement, based on the intuition of kernel alignment, leads to lower bias and lower asymptotic variance. However, this model does not capture the fact that feature learning advantages are small at small P and that the slopes of the learning curves are different at different a. Following the observation of Paccolat et al. (2021a) that kernel alignment can occur with scale vP, we plot the learning curves for signal enhancements that scale as vP. Though this toy model reproduces the onset of the variance limited regime P1/2 and the reduction in variance due to feature learning, our current result is not the complete story. A more refined future theory could use the structure of neural architecture to constrain the structure the A distribution.
CONCLUSION
We performed an extensive empirical study for deep ReLU NNs learning a fairly simple polynomial regression problems. For sufficiently large dataset size P, all neural networks under -erfforerormmmmm the infinite width limit, and we demonstrated that this worse performance is driven by initialization variance. We show that the onset of the variance limited regime can occur early in the learning curve with P1/2 ~ VN, but this can be delayed by enhancing feature learning. Finally, we studied a simple andom-feeture model to attempt to explain these effects and qualitatively reproduce the observed behavior, as well as quantitatively reproducing the relevant scaling relationship for P1/2- This work takes a step towards understanding scaling laws in regimes where finite-size networks undergo feature learning. This has implications for how the choice of initialization scale, neural architecture, and number networks in an ensemble can be tuned to achieve optimal performance under fixed compute and data budget.

---

Published as a conference paper at ICLR 2023
REFERENCES
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory generalization. In International Conference on Machine Learning, pp. 74-84. PMLR, 2020a.
Ben Adlam and Jeffrey Pennington. Understanding double descent requires fine-grained bias- variance decomposition. Advances neural information processing systems, 33:11022-1103, 2020b.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference Learning Reprexentations, 2021.
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptotics of feature learning: How one gradient step improves the representation. arXiv preprint arXiv:2205.0 2022.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws, 2021. httpss/::///xx org/ abs 2202.11122.. 06701
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vin- cent, and Simon Lacoste-Julien. Implicit regularization via neural feature aligmment. Interna- tional Conference on Artificial Intelligence and Statistics, pp. 2269-2277, PMLR, 2021.
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. arXiv preprint arXiv:2205. 09653,, 2022.
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Hal Daumé III and Aarti Singh (eds.). Pro- ceedings of the 37th International Conference Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 1024-1034. PMLR, 13-18 Jul 2020. URL https: //proceedings.. mlr. httpe///111///boreeoo02.aa. html
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: composable transformations of python+ numpy programs. Version 0.2, 5:14-24, 2018.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Commu- nications. 12, 2021.
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. In NeurlPS, 2019.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kemnlss based on centered alignment. The Journal of Machine Learning Research, 13(1):795-828, 2012.
Stéphane d' Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfitting: Where & why do they appear? Advances in Neural Information Processing Systems, 33:3058- 3069, 2020.
Oussama Dhifallah and Yue M Lu. A precise performance analysis of learning with random features. arXiv preprint arXiv:2008. 11904, 2020.
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. In Interna- tional Conference on Learning Representationss, 2020. URL https::/ hnttt/iiwww/eee/ee..........
Fooum??dd=SlgFvANEE
Stéphane d' Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in dou- ble descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning, pp. 2280-2290 PMLR, 2020.
10

---

Published as a conference paper at ICLR 2023
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geom- etry and the time evolution of the neural tangent kernel. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 18600-5861. Curran Associates, Inc., 2020. URL hnttps://proceestee ngs neurips.
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d' Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning, Journal of Statistical Mechanics: Theory and Experiment, 2020():023401, 2020a.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020 (11):113301, 2020b.
Federica Gerace, Bruno Loureiro, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. Gener- alisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning. pp. 3452-3462. PMLR, 2020.
Behrooz nari.
Ghorbani,
Song Mei, Theodor Misiakiewicz, and Andrea Monta- When do neural networks outperform kemel methods?
In NeurIPS,
2020.
URL bttps::/prrceed:: neueee cccppaper/222//aaaae 795855599533999959888
Abst rat html
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kemnel. In International Conference on Learning Representations, 2019.
Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de I Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. arXiv preprint arXiv:2009.07662, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnel: convergence and gen- eralization neural networks (invited paper). Proceedings of the 53rd Annual ACM SIGACT Symposium Theory of Computing, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pp. 3519- 3529. PMLR, 2019.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jascha Sohl-Dicksteii Wide neural networks of any depth evolve as linear models under gradient descent. ArXiv, abs/ 1902.06720 2019.
Philip M. Long. Properties of the after kernel. CoRR, abs/2105. 10585, 2021. URL https:
7 //axxii..rr/bbb//105....
Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Capturing the learning curves of generic features maps for realistic data sets with teacher-student model. CoRR, abs/2102.08127, 2021. URL https::/aaxxv..or
abs/2102.08122
Alexander Maloney, Daniel A. Roberts, and James Sully, A solvable model of neural scaling laws, 2022. URL https::p. / /ttp//aivvv////////v///a////.ae. iv..cvvo. org abs / /221 0. 16859
Song Mei and Andrea Montanari. The generalization enror of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
11

---

Published as a conference paper at ICLR 2023
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665-E767), 2018.
Roman Novak, Lechao Xiao, Jiri Hron, Jachoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020. URL htttts:t//ggtiiiii google/neuzal-tangeee
com/
Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can lin- earized neural networks actually say about generalization? Advances Neural Information Processing Systems, 34:8998-900 9010, 2021.
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2021 apr 2021a. doi: (.1088/1742-5448/bff111 URL https:
doiii 000 5468/abf113
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2021 (4):440044 2021b.
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory, 2021.
James B. Simon, Madeline Dickens, and Michael R. DeWeese. Neural tangent kernel eigenvalues accurately predict generalization, 2021.
Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher-student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):12401,, 2020.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-61 14. PMLR, 2019.
Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding gen- eralization in deep learning. arXiv preprint arXiv:2206. 10012, 2022.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. arXiv preprint arXiv:2203.06176, 2022.
Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks. ArXiv,
abs/2011.14522. 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2017.
Jacob A Zavatone- Veth, William L Tong, and Cengiz Pehlevan. Contrasting random and learned features in deep bayesian linear regression. arXiv preprint arXiv:2203.0057: 2022.
12

---

Published as a conference paper at ICLR 2023
A DETAILS ON EXPERIMENTS
We generated the dataset D = far y"" 1==1 by sampling x" uniformly on SP the unit sphere in RD. was then generated as Gegenbauer polynomial of degree & of ID projection of æ, y = (x(33 z). Because the scale of the output of the neural network relative to the target is a central quantity in this work, it is especially important to make sure the target is appropriately scaled to unit norm. We did this by defining the target to be // S3/V(Qx(B xz)² ==)))~---1. The denominator can be easily and accurately approximated by Monte Carlo sampling.
We used JAX (Bradbury et al. 2018) for all neural network training. We built multi-layer percep- trons (MLPs) of depth and 3. Most of the results are reported for depth 3 perceptrons, where there is separation between the width of the network N and the number of parameters N Sweeping over more depths and architectures is possible, but because of the extensive dimensionality of the hyperparameter search space, we have not yet experimented with deeper networks.
We considered MLPs with no bias terms. Since the Gegenbauer polynomials are mean zero, we do not need biases to fit the training set and generalize well. We have also verified that adding trainable biases does not change the final results in any substantial way.
As mentioned in the main text, consider the final output function to be the initial network output minus the output at initialization:
f Here, only 0 is differentiated through, while 0, is held fixed. The rationale for this choice is that without this subtraction, in the lazy limit the trained neural network output can be written as
(xr)
(6)
fö (x) Foo (x)
> (k( pasr
"w foo (x)).
This is the same as doing eNTK, regression the shifted targets y" foo (zz). At large initialization the shift fe, (x) amounts to adding random, deeeenddeonn noise the targets. By instead performing the subtraction, the lazy limit can be interpreted as a kernel regression on the targets themselves, which is preferable.
We trained this network with full batch gradient descent with learning rate r so that
A =
VoL 0),
C(D,0) 1
(8)
Σ [So( 1 P
Each network was trained to an interpolation threshold of 10-6. If network could not reach this threshold in under 30k steps, we checked if the training error was less than 10 times the generaliza- tion error. If this was not satisfied, then that run of the network was discarded.
For each fixed P. independent neural network initializations. This 20 x 20 table yields total of 400 neural networks trained on every combination of initialization and dataset choice.
generated 20 independent datasets. For each fixed N,a we generated 20
The infinite width network predictions were calculated using the Neural Tangents package (Novak al. 2020). The finite width eNTK,s were also calculated using the empirical methods Neural Tangents. They were trained to interpolation using the
_mse method. This substantially faster than training the linearized model using standard full-batch gradient descent, which we have found to take very long time for most networks. We use the same strategy for the yNTK,s.
For the experiments in the main text, have taken the input dimension to be D 10 and sweep over A = 1, 2, 3, 4. We swept over 15 values / in logspace from size 30 to size 10k, and over 6 values of N in logspace from size 30 to size 2150. We then swept over alpha values 0.1, 0.5, 1.0, 10.0, 20.0. Depending on a, NN, we tuned the learning rate 1) of the network small enough to stay close to the gradient flow limit, but allow for the interpolation threshold to be feasibly reached.
For each of the 1800 settings of P,N, a, I and each of the 400 networks, 400 cNTK,s, 400 eNTK ss and 20 NTK the generalization emror saved, as well as vector of predictions on a test set
13

---

Published as a conference paper at ICLR 2023
of 2000 points. In addition, for the neural networks saved both initial and final parameters. All are saved as lists of numpy arrays in a directory of about ITB. We plan to make the results of our experiments publicly accessible, alongside the code to generate them.
A.1 CIFAR EXPERIMENTS
We apply the same methodology of centering the network and allowing to control the degree of laziness by redefining
fo(æ) = a(f((z) - fo,
We consider the task of binary classification for CIFAR-1O. In order to allow P to become large divide the data into classes: animate and inanimate objects. We choose subsample eight classes and superclass them into two: (cat, deer, dog, horse) vs (airplane, automobile, ship, truck). Each superclass consists of 20,000 training examples and 4,000 test examples retrieved from the CIFAR- 10 dataset.
On subsets of this dataset, we train wide residual networks (ResNets) Zagoruyko & Komodakis (217)) of width 64 and block size 1 with the NTK parameterization Jacot et al. (2018) on this task using mini-batch gradient descent with batch size of 256 and MSE loss. Step sizes are governed by the Adam optimizer Kingma & Ba (2014) with initial learning rate 10-3, Every network trained for 24,000 steps, such that under nearly all settings of and dataset size the network has attained infinitesimal train loss.
We sweep a from 10-3 to 10 and P from 2 to 215 For each value of P, we randomly sample five training datasets of size P and compute ensembles of size 20. For each network in an ensemble the initialization and the order of the training data is randomly chosen independently those for the other networks.
B FINE-GRAINED BIAS-VARIANCE DECOMPOSITION
B.1 FINE GRAINED DECOMPOSITION GENERALIZATION ERROR
Let D be dataset of (x* y²) p(z, y) viewed random variable. Let fu represent the initial parameters of neural network, viewed as a random variable. In the case of no label noise, as in section 2.2.1 of Adlam & Pennington (2020b), we derive the symmetric decomposition of the generalization error terms of the variance due initialization and the variance due to the dataset. We have
Eg E) = ((fo, (x)) - -y)²) = ((V)), - >)) + EE Vara, foo.
(10)
Bias + Vo + Vo, + VD.0.
Here we have defined
B = ( - )
(11)
Vp = Ep VarD Eg, Iff , D (x)) En Varp (x),
(12)
Ve, = Ep Vare, Ep Eff h D (x) (x)]6u].
(13)
VD.a, = Ep Varo,. (v) - Ve, VzD
(14)
VD and Ve, give the components of the variance explained by variance in D, Og respectively. VD. the remaining part of the variance not explained by either of these two sources. As in the main text, (x) is the ensemble average of the trained predictors over initializations. Ep (<)) is commonly referred to as the bagged predictor. In the next subsection we study these terms empir-
ically.
B.2 EMPIRICAL STUDY OF DATASET VARIANCE
Using the network simulations, show that the bagged predictor does not have substantially lower generalization error in the regimes that we are interested in. This implies that most of the variance driving higher generalization enror is due variance over initializations. In figure 7
14

---

Published as a conference paper at ICLR 2023
make phase plots of the fraction of Eg that arises from variance due to initialization, variance datasets, and total variance for width 1000. This can be obtained by computing the ensembled predictor, the bagged predictor, and the ensembled- -agged predictor respectively.
60
52
12
Initialization variance lo
(b) Initialization variance &
(c) Initialization variance
14-
14
-0.
-12
-38
20
25
20
(d) Dataset variance hs = 2
(e) Dataset variance t:
3
(f) Dataset variance l =
1otal Mert ME L-d And RX-300
total se d ME L-3 bod Red000
laad 10 ML L-3 bnd RH0000
14
12
440
(g) Total variance l 2
(h) Total variance 3
(i) Total variance &
Figure 7: Phase plots of the fraction of the generalization error due to initialization variance, the dataset variance, and their combined contribution. The columns correspond to the tasks of polynomial regression for degree 2, and polynomials. Neural network has width 1000 and depth 3. Notice that the initialization variance dominates in the large P large regime
B.3 RELATING ENSEMBLED NETWORK GENERALIZATION TO INFINITE WIDTH GENERALIZATION
Making use of the fact that at leading order, the eNTK, (either in the rich or lazy regime) of a trained network has deddepeddent fluctuations with variance 1 1/N, one can write the kernel Gram matrices as
Ko Juv = [Koluu +
K p +O(/NN)
VN
(15)
1
KRo (x)] | = (k:
+
5kk (x)],, + O(1/N)
VN
Here, 6Ka,, 5ks, are the leading order fluctuations around the infinite width network. Because of how we have written them, their variance is ((1) with respect to N. Using perturbation theory
15

---

Published as a conference paper at ICLR 2023
Dyer& & Gur 2020), one demonstrate that these leading order terms have mean zero around their infinite-widd width limit.
The predictor for the eNTK, (or for sufficiently large neural network) for training set with target labels y is given by:
I (3r)a keo (2) _ K 0g y
f (z) + VN
1
δke (x)
K y
ko (x) K 8Ko, K O
VN
(16)
This implies (Geigerr et al.
2020a):
{ = O
(17)
Upon taking the ensemble, because of the mean zero property of the deviations, we get that
(f))))))) +
(18)
= - =
-
We can now bound the generalization error of the ensemble of networks in terms of the infinite-width generalization:
(((f), (z))o. )=) - (1f y) (156, (z)) - (x))
(19)
-2(0 (a) (x) (fù,
By equation 18 the second term yields positive contribution going O The last term can be bounded by Cauchy-Schwarz:
(x) - -))))))))) (St, (2)))o v (( u)2) (S%, (2r)),))))
E3
(20) After we enter the variance limited regime by taking P > P1/2 we get F S O(1/N) so this last term is bounded by - Consequently, the difference in generalization error between the infinite width NTK and an ensemble of lazy network or eNTKo predictors is subleading in 1 1/N compared the generalization gap, which goes as N-1
The same argument can be extended to any predictor that differs from some infinite width limit. In particular Bordelon & Pehlevan (2022) show that the fluctuations of the eNTK, in any mean field network are asymptotically mean zero with variance N The above argument then applies to the predictor obtained by ensembling networks that have learned features. This implies that in the vari- ance limited regime, ensemble averages of feature learning networks have the same generalization as the width mean field solutions up to term that decays faster than N -
C FEATURE LEARNING
C.1 CONTROLLING FEATURE LEARNING THROUGH INITIALIZATION SCALE
Given the feed-forward network defined in equation D one can see that the components of the activations satisfy (((") ((ah,)((-I)) and consequently that the output (1(4) 0(a²). Because of the way the network is parameterized, the changes in the output 88
of also scale as O(a"). This
implies that the eNTK at any given time scales
0f(x) 0f(x') = 0
Kg (x x') Σ
(21)
20
00
After appropriately rescaling learning rate to
22 we get
df (fx))
11 Σ Ko (f(())) )
(22)
16

---

Published as a conference paper at ICLR 2023
Under the assumption that o². < 1 and y" (1) so that the error term is O(1) get that the output changes in time as df /ct = O().
On the other hand, using the chain rule gradient update and the features in the prior layer, yiclding the scaling
show that the features change product of the
dh =1 ーリ VVN
1
(aVN)-
(23)
dt
OLVN
This gives us that the change in the features scales (aVN)- while the change in the output scales as O(1). Thus, for a sufficiently small, the features can move dramatically.
C.2 OUTPUT RESCALING WITHOUT RESCALING WEIGHTS
In the main text, use the scale at every layer to change the scale of the output function. This relies on the homogeneity of the activation function so that W - oW for all leads to rescaling f for. This would not work for nonhomogenous activations like @(h) tanh(h). However, following Chizat et al. (2019;; Geiger et al. (2020a), we note that we can set all weights to be (1()) and introduce the only in the definition of the neural network function
W (b()), N
W Wh 14
VD Dy
(24)
We note that all preactivations h have scale Oa for any choice of nonlinearity, but that (a(a). Several works have established that the a vN allows feature learning even as the network approaches infinite width Mei et al. (2018); Yang & Hu (2020): Bordelon & Pehlevan (2022). This known as the mean field or p-limmi.
C.3 KERNEL ALIGNMENT
In this section we comment on our choice of kernel alignment metric y' Ky
A(K)
(25)
Tr Klyl
For kernels that are diagonally dominant, such as those encountered in the experiments, this metric related another alignment metric
3/ Ky
AF(K)=
(26)
K K
Here |Krr the Frobenius norm of the Gram matrix of the kernel. This metric was extensively used in Baratin et al. (2021). The advantage of the first metric over the second is that one can quickly estimate the denominator of A(K) via Monte Carlo estimation of (u' K u=)
We use A(K₁) as measure of feature learning, as we have found that this more finely captures elements of feature learning than other related metrics. We list several metrics we tried that did not work.
One option for a pppresentaioon-lammin metric involves measuring the magnitude of the change between the initial and final kernels, K;, Ky: AK K K,Ir.
(27)
However, this is more sensitive to the raw parameter change than any task-relevant data. If one instead were to normalize the kernels to be unit norm the beginning and the end, the modified metric
K
K
AK
(28)
This metric however remains remarkably flat over the whole range of a, P, as does the centered kernel alignment (CKA) of Cortes al. (2012)
Tr KY
1 IT..
CKA(K, K2)
K CKC, C=1
(29)
KGle
P
17

---

Published as a conference paper at ICLR 2023
Here C' is the centering matrix that subtracts off the mean components of the kernel for P P kernel. This alignment metric has been shown to be useful in comparing neural representations (Kornblihh et al. 2019). For our task, however, because the signal is now-dimensional, only small set of eigenspaces the kernel align to this task. As result, the CKA, which counts all eigenspaces equally, appears to be too coarse capture the low-dimensional feature learning that is happening.
On the other hand, we find that 5) (with y given by the eNTK f evaluated on a test set) can very finely detect alignment along the task relevant directions. This produces clear signal of feature learning at small a and large as shown in Figure 2
A A(K f) can be related the centered kernel alignment between the eNTK y and the (mean zero) task kernel yy" where y is vector of draws from the population distribution p(æ:, y)-
C.4 RELATIONSHIP BETWEEN TRAINED NETWORK AND FINAL KERNEL
In general, the learned function contains contributions from the instantaneous NTKs at every point in the training. Concretely, following Atanasov al. (202I) we have the following formula for the final network predictor fx)
((x) = 10 di k( 1) exp ds)sss) y:
(30)
K ( 8) and [y), Yr. general there are contributions from earlier kernels k(z, t) for < oo and so the function f cannot always be written as linear combination of the final NTK K on training data: f En a₂ K (x, xp). However, as Vyas et (2022); Ataanaovv al. (20TT) have shown, the final predictions of the network are often well modeled by regression with the final NTK. We verify this for our task in section 3.2
where [k(x,t)], ((x, 22, t) and [K())
D GENERIC RANDOM FEATURE MODEL
D.1 SETTING UP THE PROBLEM: FEATURE DEFINITIONS
For random kernel, K(x, 0);
0), we first compute its Mercer decomposition
[ dx K(z)K((,, d)r (x) \=))
(31)
From the eigenvalues A and eigenfunctions ox. we can construct the square root
k1/2(x,, (x
Σ Aeda(
(32)
Lastly, using K1/2, we can get feature map by projecting against static basis {ba} giving
(xx(x) = / dx' p(x') ( ) /2,22
(x, x:; 0)bb (z²).
(33)
These features reproduce the kernel so that K Kxx, x'; = En ehu (z)) This can observed from the following observation
the (2)) =
Σ (ecor( UR = ((r((x))b
(34)
Σ (x( Σ
(ee)m (x') E UexUmm Σ (e)
(35)
t.m
where the last line follows from the orthogonality of Ukm and recovers K (x,x0;;))
D.2 DRCOMPOSITION OF FINITE WIDTH FEATURES
We now attempt to characterize the variance in the features over the sample distribution. We will first consider the case of a fixed realization of O, before providing a typical case analysis over random Oy For fixed initialization O, we define the following covariance matrices
Σ (z) T> M
(36)
18

---

Published as a conference paper at ICLR 2023
where 2 the truncated (but deterministic) features induced by the deterministic infinite width kernel. We will mainly be interested in the case where M - oo and where the target function can be expressed the linear combination y(x) w L (x) of these features. For example, in the case of our experiments on the sphere, tbwr could be the spherical harmonic functions. Further, in the M limit, will be able express the target features &b linear combinations of the features M
(x 0o) = A(0o) A()) E RN
(37)
The matrix A(0n) are the coefficients of the decomposition which can vary over initializations. Crucially (0o) projects to the subspace of dimension Nx where the finite width features have variance over æ. The population risk for this Og has an irreducible component
(O,) = (ww> PMM w w.w.1 .6))
2 w T ΣM AO) Em A(0,) w"
(38)
where the bound is tight for the optimal weights w = (A(0))Es: ((0))) A(0))Emw** The irreducible eтror determined by projection matrix which preserves the subspace where the fea- tures (bx 0o) have variance: - (A(0,) ( A(6o) A EM In general, this will preserve some fraction of the variance in the target function, but some variance in the target function will not be expressible by linear combinations of the features (b(æ, 0). We expect that random finite width N neural networks will have unexplained variance the target function on the order 1/N.
D.3 GAUSSIAN COVARIATE MODEL
Following prior works on learning curves for kernel regression (Bordelon et al. 2020 Canatar et al., 2021 Loureiro et al. 2021 we will approximate the learning problem with a Gaussian covariates model with matching second moments.
The features u (z) will treated as Gaussian over random draws of datapoints. We will assume centered features. We decompose the features in the orthonormal basis b(æ)), which we approximate as Gaussian vector (0, I).
4b w. y= u/ w
MM = Σ / (b(0)) = A(0)) abss + E!/
b (O, I), NNO, I)
(39)
This is special case of the Gaussian covariate model introduced by Loureiro et al. (2021)
) and sub-
sumes the popular two-layer random feature models (Mei Montanari, [222]] [ Adlam & Pennington 2020b) special case. In a subsequent section, go Loureiro et al. (2021) by comput- ing typical case learning curves over Gaussian A(0)) matrices. In particular, we have for the two layer random feature model in the proportional asymptotic limit P, N. D - with P/D D O(1) and P/N O(1) with (xx) ((Fx,) for fixed feature matrix F E xDD nonlinearity o and
Loureiro
x = b (
Σ 1, == - A F
= (~ 2. ()(2))) =~N(0011) ) of.
(40)
We refer readers to Hu & Lu 2020) for a discussion of this equivalence between random feature regression and this Gaussian covariate model.
D.4 REPLICA CALCULATION OF THE LEARNING CURVE
To analyze the typical case performance of kernel regression, we define the following partition function:
3 P
3 /2/2
JBM
Z(D. 001 = dw dw exp
w 2b, w" 3b
E, (w)
21
2
[1
(w(
11/2 "²
1/22 A(Oo) + M
1
E.w.
(41)
M
19

---

Published as a conference paper at ICLR 2023
Ea and (ee² M. E.. We note that in the + oe limit, the partition function is dominated by the unique minimizer of the regularized least squares objective (Canatar et al. 2021 Loureiro et al.) (02I). Further, for fixed realization of 0o the average generalization error over datasets D can be computed by differentiation of the source term J
For proper normalization, assume that (b if
2 BM DJ J= (In ZID, 20D,
/ Z
(-1 22
1
P
B x,,,
(42)
dw exp
Σ 1=1
w -
2/
2 u012 Eg(w)
Thus the B + oc limit of the above quantity will give the expected generalization error of the risk minimizer. We see the need to average the quantity In Z over realizations of datasets For this, we resort to the replica trick (In Z} = lim A In (? We will compute the integer moments (?*) for integer n and then analytically continue the resulting expressions to + 0 under a symmetry ansatz, The replicated partition function thus has the form
3
(2 = II a=1
dw E (b exp
W 3/ -w"
1o012
2X
2
#=1
a=1
(43)
JBM
Eg Eg(w))
exp
We now need to perform the necessary average over the random realizations of data points D (bu 6n) We note that the scalar quantities hy w 1/. w * are Gaussian with mean zero and covariance
(nGh)) Qab
1 Qab M (A(On)w) -w") EM (A(0))w" -w") + w E,w" M
1
(44)
We further see that the generalization error in replica a is Eg (ww) = Qaa Performing the Gaussian integral over (hr) gives
(Z") x / I dw
II
duudd exp (-2 det (AI + BQ)
J3M
TrQ
8 Σ 200222
ab
Σ ab
Qao (MQas [A(O)) -w"] "0'T"np EM ΣM [A(00)(W w - w01 + w" E, 200)
exp 2
Q which enforce the definition of order parameters Q. We now integrate over W Vec(w=)) 1==1- We let Σ, A Σ A
We introduced the Lagrange multipliers
dW exp
W
3I + Q 3 Σ. Σ.] W
2
W
Q 3 1 (1 @ A A -E=w"))
exp
(1 8 A E.w") T [ 2 1]
BI + Q (S, + [
(1 A ".E''))
-epp
exp
(-2 det
31 + Q ② E. + Σ.I
(45)
To take the + 0 limit, make the replica symmetry ansatz
3Q ql + 11
qI 11
(46)
20

---

Published as a conference paper at ICLR 2023
which is well motivated since this is convex optimization problem. Letting P/N, we find that under the RS ansatz the replicated partition function has the form
n.M S(q. qo. &. Qul
(Zm)
dqdqpdiido exp
(47)
2
In(A + q) + qu
S q9 + 909 + 990 - a
x+ q
3
w" [QEM k/ww"
w" [qE GA[@E.
M
1
+ In det G
90
TrG E.} J(q + 90)
M
M
G = I E + -.)
(48)
In limit where P/M is O(1), then this S is intensive S O1)( We can thus appeal to saddle point integration (method of steepest descent) to compute the set of order parameters which have dominant contribution to the free energy.
(Z") / dadqadâdgo exp
n.M S1g. 90: &. 8ol
n.M SIg -o. q- -6.6
exp
2
= (In Z) =
2 S(qa 90. .6).
(49)
The order parameters q², 46, 4 40 are defined via the saddle point equations as O. For purposes, it suffices analyze two of these equations Oúj
as
3q
3gg
04
às
a
J =0,
Oyo
X At
0s
q M
1
TrG
SS + Σ =0
(50)
Oio
We can now take the zero temperature (B - oo) limit to solve for the generalization error
Eg = DJ
1==
lim R B
1
O LiEn E A AEAM,G w".
(51)
M
We that need to compute the J derivatives on q. We let K À + q and note
0,ğ -ak -0,x
= - M
1 20,n +0 +1) ---xx)]---(xxx00
T [Σ, + E. 12 = =-(axx
TrG²
[Σ, + E.
12
(52)
M
We solve the equation for θyk which gives θ5k = x²
where = 2 M
TTG
Σ, + E.
With this definition we have θ.q = + 1- - 1-g.
Eg
1
11/
2//1/2 12 AT GAE!/2 // 2 A Σ. E. GAE!/2
1// w
M
M w
1/2 M
I !ー GA -/ 1/ A G2 AΣ!/2 1/2 11/2 w
(53)
This reproduces the derived expression from Loureiro al. 2021
The matching covariance Σ Σas and zero feature-noise limit Σ, = recovers the prior results of Bordelon et al. (220 Canatar al. (2021) Simon et al. (2021). In general, this error will asymptote the the irreducible error
lim F. M w² ΣM E.AT [I, + Σ.)
AΣ w"
(54)
21

---

Published as a conference paper at ICLR 2023
We see that this recovers the minimal possible enror in the P - limit. The derived learning curves depend on the instance of random initial condition 0g. To get the average case performance, take additional average of this expression over 0,
Eo, (00) Eso
M w 11/2
I GE!
A A G G AE!!
E1/2 w".
(55)
This average is complicated since y, q. G all depend on 0g. In the next section we go beyond this analysis to try average analysis for random Gaussian A.
D.5 QUENCHED AVERAGE OVER GAUSSIAN A
In this section we will define a distribution of features which allows an exact asymptotic predic- tion over random realizations of disorder 0g and datasets D. This is nontrivial extension of the result of Loureiro et al. (2021) since the number of necessary saddle point equations to solved doubles from two four. However, this
complicated theory allows exactly compute the expectation in equation 55 under an ansatz for the random matrix A. We construct our features with
1
/ A
whs E!/26, Aij NO)
We will now perform an approximate average over both datasets and realizations of
(Z / II 72
a
3 n
dw a E (b,
(b exp
W 2/
w"
Σ
(w
==
JMB 3
Eg (200))
x exp
2
a=1
(56)
As before, we first average over b,s E A and define order parameters Qab before.
(2") = /Il dwo
II
P dQaad exp (-2 In (
JBM
TrQ
Σ
/20
ab
E gg
exp
2
Σ
Qub (MQut [gg - w01 -ww Enrlg - -w"1 + wD Yo"E.w. E,w²)
ab
where defined the fields
g" Aw which are mean zero Gaussian with covariance 8² N ww w Performing the Gaussian integral over G Vec(g*).
VN
(g "T) VaL where Vab
we find
/ II dg exp
En +En 2 2 G @ (Exw* @ Q1)G
= dn (1 1VV)
1
2
1 ( 2 Q1) - EM 8 Q EErw" @ Q1)
1
EM @ Qv)).
exp
+
In
II
2
(57)
Next, we need to integrate over W Vee[w"} which gives
{ dW exp (-2 o w exp (-1 Indct
E. =q] dl) (58)
BI
2 I
22

---

Published as a conference paper at ICLR 2023
Now the replicated partition function has the form
M TrQQQ + VVV]
JBM
P
(2t) - dQdQdVdV exp
TrQ
In sX] + sQ)
2
2
[ @ Q)(w" 8 81))
x exp
(w" & 1)
(59)
EM & =Q1) I & &&v-i + EM @ Q]
(Ear @ Q1)
x exp
2
det
I+ E EM @ Qv) det 3I + E. 1 Q))
x exp
Now we make replica symmetry ansatz on the order parameters Q,Q.V,
- - 8V = vl + +
=- 4011", V = + 1611
(60)
We introduce the shorthand for normalized trace of matrix G as tr G 1. TrG. Under the replica symmetry ansatz, we find the following free energy
2 (In Z) qû + 9od 9a0
n + u vûo) [[g go)
In(A 2)
40
M
3 w" M 2= +
B
" 1
M
M
trlog I + gv -(00 + (u6) ur( qu 1-1]~>1
-'E
trlog [ + o 21 + Σ. tr [ + 01 + Σ.
[600 9o
(61)
Letting F = 2M-1 (In Z), the saddle point equations read
0F
==
0qo
At9
OF
v tr| + g -IEMI-EEM - tr| - - = 0,
-.
0qo
OF
qi 0 I + j - ΣE = 0,
0up
OF
- o² tr + 0² 61 + Σ.q = O.
(62)
06n
Now the generalization eπror can be determined from
8 lim
2 (In Z) i M w
F.g
[GEM II [= w². (63)
0J 8000 3M
We see that necessary to compute 0,g and Uyv in order to obtain the final result. For simplicity, we set x² = 1. The equations for the source derivatives are
a Øy9 (A + q)2
0,q + 1,
05g = tr[ of =
tr/ 61 + + O,4
n0 0 = tr(I + ug -
10 = -tr + 01 Σ. "100
(64)
Once the value of the order parameters (g, q, u, have been determined, these source derivatives can be obtained by solving a 4 x 4 linear system. Examples of these solutions are provided in Figure
16
23

---

Published as a conference paper at ICLR 2023
D.5.1 ASYMPTOTICS IN UNDERPARAMETERIZZR REGIME
We can compute the asymptotic (a - oc) generalization error due to the random projection A the limit of Σ. = O. First, note that if 6 + (1), then the asymptotic error would be zero. Therefore, we will assume that ~ axe for some a, C > 0. The saddle point equations give the following asymptotic conditions
X 1 tr iI + {EM 'EM tr Aaa
I Σas] EnM
(65)
For < 17 < 1, this equation can only be satisfied as a - 30 c = so that has the same scaling
with If < 1 then could get the equation 1) = 1. If 1, then the equation would give 0. The constant a solves the equation
r = tr|Aal + I Em.
(66)
Using this fact, our order parameters satisfy the following large scalings
9 0 é ac 0.
(67)
The source derivative equations simplify to و.6 1, و.g- ~ 0 and
nd (60.ς + (8,6) tr( +
Σ - Götr + -000=0 05qE31
~ q + -- + - 0, t 3 G
ut| + a Ma
EEM
0,0
(68)
uraal ΣM]
" M
We note that θyê only depends the product aA which is an implicit function of and M.. The generalization error is Eg
1 0 + 0) [[1 + i) + 92)
"Eww"
Eg
M w" I + I+a -
Enw" w
(69)
" 2 w x tr 1ー Eir
M " [daI
Ens -
" tr trlaXI + ΣM -2
We see that in the generic case, the asymptotic error has a nontrivial dependence on the task un* and the correlation structure ΣM- To gain more intuition, we will now consider the special case of isotropic features Σn I. In this case, we have 1+ so that λa = I-n This results in the following generalization error
1 M
n)2 172 (1 17) 2
Eg
|w"
w 12(1 17).
(70)
1 172
We that as 7 Nu 1, the asymptotic error converges to zero since all information in the original features is preserved.
D.5.2 SIMPLIFIED ISOTROPIC FRATURE NOISE
We can simplify the above expressions somewhat in the case where 0² and E. O²I. In this case, the order parameters become
1
T = n1 + + oZg)-1
+ 6 +026
n q tr I
Σ
ΣM = q t 02 tr( + 6 + +
'EM
+ + 026
?o!
q= tr (1
2 2
ΣM 1 +
(71)
24

---

Published as a conference paper at ICLR 2023
Letting G [[1 + 0 + + -1 the source derivatives have the form
0 = ()+ q)2
(72)
n0 ((1 (1+00-2200000000))0000)0 (1+ o/4)² (0,6 0 100000000000000000 -2?00)
+1 (x+ + q)²
1 +
(73)
(74)
This is a 2 linear system
ato 52 [ 0 0 2 EM Ir
[tr +
n
O
1
(1+6+o!9)
(h+a)
11+0+0 + q)
1 q 1
For each we can solve for O,4 and θ,ū get the final generalization error with the formula
1 E, 0. M w
[1+ é + o. w"
(0 + qo o. (1
01 à
(0 + o, + dqEM)] (75)
An example of these solutions can be found in Figure 16 theory and experiment.
where we show good agreement between
E RESNET ON CIFAR EXPERIMENTS
25

---

Published as a conference paper at ICLR 2023
Eg for ResNet CIFAR
Accuracy for ResNet CIFAR
0.96
0.94
0.5
0.92
0.3
0::
d=002
aw2.22
05005
0w5.6
0.2
0=0:0
0m300
0.522
2.46
10a
10
10a
10
(a) Generalization MSE
(b) Accuracy
variE, for ResNet on OFAR
var/E, for ResNet on CIFAR
0.40
0.50
0.000
P=8192
0.45
0.0002
16384
0.40
P-32768
0.35
variEz
0.30
0.464
0.255
0.25
0.20
0.10
0.05
10-3
10
10
10a
10a
10
(c) E.
(d) var/
Figure 8: A Wide ResNet Zagoruyko & Komodakis (2017) trained on a superclassed CIFAR task comparing animate inanimate objects. Each learning curve is averaged over different samples of the train set, yielding the means and eтror bars shown in the figures. a) Generalization error Eg- The dashed lines are the error 20-fold ensemble over different values of a. Across all P. lazy networks attain worse generalization error. As with the MLP task, the best performing networks are ensembles of rich networks. b) The accuracy also has the same trend: richer networks perform better and ensembling lazy networks helps them more. c) Once P is large enough, lazier networks tend to benefit more from ensembling. d) Very lazy networks transition to variance limited behavior earlier. For ResNets on this task, we see that rich, feature learning networks eventually begin reducing their variance on this task. Further details the experiment are given in section A.I
F ADDITIONAL EXPERIMENTS
26

---

Published as a conference paper at ICLR 2023
10
-15
-10
(a) Eg kN
(b) Eg for cNTKo 3
(c) Eg for cNTKo
and Soites
Le3,
-0
-0
(d) Eg for NN 2
(c) Eg for NN k =3 3
( Eg for NN k
Figure 9: Phase plots of log10 E, for initial eNTKs (top) and neural networks (bottom). The large behavior of the neural network generalization matches the generalization the corresponding eNTK,, As sanity check, the eNTK, generalization error is independent of re-scaling the network initialization because of the homogeneity of the ReLU network output.
27

---

Published as a conference paper at ICLR 2023
M-2000,
cnd N-2800
Me2000, aw22
10/9
10P4
(a)
(b)
Lm.. Nw00000 2-1.
Lm2. N.2000
N00.00000000.00 2w1
10-4
10
30-
10-4
30-4
10
10-0
10.4
1.2
(d)
(e)
(f)
30
1011
1074
10
157
1.7
107 30'
107 107
(g)
(h)
Figure 10: A fine-grained view of the generalization error across different datasets and ensembles. Solid curves are depth neural networks, dashed curves are the infinite width NTK (which only has variance over datasets). Each color is a set of networks trained on the same dataset but different initializations. Different colors correspond to different datasets indexed by d E {O.....9} 9}.
28

---

Published as a conference paper at ICLR 2023
Eg NN, Lm3, P-100
P-600
E. NN, Lm3, km1, 0000000
05
05
-05
-1
log
log
Eg for &
1, 600
Eg for
10000
Ey P=600
k-3,
20-
10-
0.5
555
-1.
loge
-1.
-20
-20
-4.-
-23
150
(c) Eg for l 3,
600
(d) Eg for 10000
Figure 11: Phase plots of log10 Eg for neural networks in the N-a plane. We plot these different train set sizes P and different tasks k. The colors are fixed to match across networks trained on the same task.
100
20'
N
Figure 12: Empirical plot of of the scaling of the variance the eNTK, with N with variance taken over 10 initializations and averaged 10 different datasets.
29

---

Published as a conference paper at ICLR 2023
L=2, 1=2, N-1000, D-10
L=3, k=2, N=1000. D=10
Led, km2, N=1090. D=10
10
100
12-4
NTK.
12-4
NTK.
12-1
NTK.
12-3
10-1
13
0-20.3
20.2
12"
12"
0-20
10
16-
(a) Eg for L 2
(b) Eg for L 3
(c) Eg for L
Erdembled Error Lm2. km2, No1000, D=10
Ersembled Eroor L-3.
N=1200, D-10
Ersembied Enor No1200. D-10
100
300
12-1
12'4
42
20
12-1
12-1
10-
13-4
1219
20 12-0
12a4
(d) Ensembled Eg for L = 2
(e) Ensembled Eg for L = 3
(1) Ensembled Eg for L 4
varis L-2, k-2, N- 1000, D-10
variz Lm3. k=2, 1000, D=10
3.2
veriz Lod, bm2 No00 D-10
-.1
0.1
0.5
10.0
10.0 044
0.4
20.0
20.0
6.4
0.2
(g) var Eg for L =2
(h) var Eg for L
(1) var / Eg for 4
Figure 13: Sweep over depth L (2. 3, 4). Deeper networks in the rich regime can more easily outperform the infinite width network for larger range of P. Also, for larger L is easier to deviate from the lazy regime at given a. By contrast, on this task the shallower NTK outperforms deeper As before, ensembled lazy networks approach NTK and the variance rises with P.
NTK S. S.
30

---

Published as a conference paper at ICLR 2023
D-30
KTK.
100
0
12-
12-
NTV.
10-
KTK.
13-
12-4
12-4
122
13-4
15-
2.0
1003
13.
12
12
1.2
109
201
18-
124
10
201
10-
122
11'
201
(a) Eg for L = 2
(b) Eg for L = 3
(c) Eg for L = 4
Ensemnledd Emert Emeri-3, k-2, N-1000, D-5
Ereambled Emor L-3, kn2, N-00000 2-25
Ensemblee raied Emar L-3, 2=2 N-1000, D-50
209
200
200
10-1
NTK.
13
15"
15
10-4
10.
20202
13-1
30
16-
10'
(d) Ensembled Eg for L = 2
(e) Ensembled Eg for L
(l) Ensembled Eg for 4
vedff c-2
vnnt,
3.2
vant,
k-2
D-150
0.1
0.2
0z
0.5
0.8 0.
0.4
0.5
d=10
4-10
0.6
6.6
0.6
0.4
20
102 10' 12a
10
112
20a
12a
10
122
207
12a
(g) var / Eg for L =2
(h) var Eg for L =
(i) var Eg for = 4
Figure 14: Sweep over input dimension D = {5, 25, 50). At larger input dimensions rich net- works can more easily outperform NTK This is consequence of the task depending on the low-dimensional projection B x.
uncemend Comernd MS C 2-2.1-1 M1289)
Unovrand and 1090
(a) Fg for centered vs uncentered
(b) Effect of ensembling
(c) F.g uncentered color plot
Figure 15: a) Ey for the centered predictor fa(x)) - Fo, (æ) (solid) compared to the generalization of the uncentered predictor (x) (dashed). At small the difference is negligible, while at large the uncentered predictor does worse and does not approach eNTK, The worse generalization can be understood as fe, (z) effectively adding an initialization -eepeedde noise to the target y. b) The effect of ensembling becomes less beneficial for uncentered lazy networks. e) Color plot of Eg. The lazy regime is different from the eNTK, generalization (c.f. Figure D.
31

---

Published as a conference paper at ICLR 2023
10a
10
10a
5h
4
Nv 200
Nu 100
10-3
10
0
1000
0 10
10
Ny 2000
of 10a
10a
10a
10a
10*
10
10a
10a
10
P
(a) Gaussian vary NR.
(b) No 750, vary of
10a
R-1
10
A 10
k= 20
10
= 600
10-1
10a
10a
10?
10a
10a
107
10a
10
P
(c) N2 1000, Vary
(d) Nx 1000, Vary A
Figure 16: Verification Gaussian A model. Solid lines are theory and dots are experiments. (a)
The effect of changing the student's RKHS dimension Nu. Double descent overfitting peaks occur Nz (b) The effect of additive noise the student features Σ, = -. (c) Learning curves for fitting the k-th eigenfunction. All mode errors exhibit double descent peak at P Nu regardless of the task. (d) Regularization can prevent the overfitting peak.
Multimodal k- (1, -1.333_-33.3 D 25,N~ 500
Multimodal k= (1,3), == 3,0 3, 25 550
verys
NTK.
10-2
0= 0.1
a=0.1
10
0 10.0
= 10.0
(a) E, for mixed mode task
(b) Var 9 for mixed mode task
Figure 17: Width 500 depth MLP learning D = 25 mixture of linear and cubic polynomials. a) Generalization error of NTK (solid black) and MLP (solid colored lines) mixed mode task. The dashed lines are convex combinations of the generalization curves for the pure mode k = 1, = tasks. For the NTK x0 the generalization curves sum to give the mixed mode curve, as observed in Bordeloo et al. (2020). We see that this also holds for the eNTKK for sufficiently lazy networks, as predicted by the simple renadom feature model considred in section 4 of this paper. b) The variance curves for the same task. Again, for sufficiently lazy networks the variance is sum of the variances of the individual pure mode tasks, as predicted by our random feature model.
32

---

Published as a conference paper at ICLR 2023
3, 177.
3, 3, 177. 20
P- 100
P- 180
- 400
P 100
P- 180
400
-1
<kTKc
cNTK
ehTTe cNTS
eNTX prodictors
ONTR predictors
0
P 4600
P- 6000
P 10000
P- 4000
P- 6000
P- 10000
preditto
NN predictor
(a) h: = 3, N = 177, a = 0.1
(b) 1: = 3, N 177, 20
L-3,k 1000,
L.3,
P 100
P- 180
P 400
100
P. 180
P 400
1000
18000
P 600
P 1000
ONTX predictors
ONTI predictors
-2
6000
10000
4000000
6000
P-
NN predictor
NN predictor
(c) & 3, N = 1000, = 0.1
(d) & 3, N 1000, 20
Figure 18: Comparison of the neural network predictor (I-axis) to eNTK, eNTK (blue, orange respectively, y-axis) across several training dataset sizes. N 177,a = 0.1, N 177, a 20, c) N 1000, = 0.1 d) N 1000, a = 20. All networks are depth 3. Note how in the rich regime there much stronger distinction between the eNTK, and the neural network. In all regimes, eNTK, matches the NN output.
33

---

Published as a conference paper at ICLR 2023
400
100
-2
P 600
1000
P 1800
P 600
P 11000
P: 1800
ONTK predictors
ONTX predictors
-2
-2
-2
NN predictor
NW predictor
N 177, a 0.1
(b) 1, N 177, a 20
L.3k 1. 1000, am 0.1
L=3,k= = 1. 1000, am
-2
P 600
P- 10000
p 1.800
P 600
P. 1000
P- 1800
ONTE predictors
ONTE
= 4000
= 6000
P 100000
4..000
P. 60000000
P-u 10000
-2
-2
NN predictor
NN predictor
(c)
1000, a
(d)
1000, a
Figure 19: The same figure 18 but for fitting linear mode. Because the task simpler, doesn't 't require as large of an to enter the lazy regime.
34


Reviews:
[{"review_id": "xs9u7s_GfI", "reviewer": "Reviewer_6LDR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**POST-REBUTTAL UPDATE:**\nThe authors did an excellent job responding to my concerns, in particular, with the take-home message. It appears as though the take-home message was not lost on the other reviewers. In light of the author response and the other reviews, I have increased my score and decreased my confidence. I recommend this paper be accepted.\n\n========\n\nOften analysing wide neural networks in a kernel regime is done by choosing a finite dataset size $P$ and taking the \"width\" (which may actually be many \"widths\" in different layers) $N$ to infinity. This approach leads to a tractable analysis but can often result in a performance gap in performance between the neural network model and the kernel method model. This is usually attributed to smaller width networks being able to perform feature learning. \n\nWhen the dataset size $P$ is asymptotically less than the square root of the number of parameters $\\sqrt{N}$, the authors find empirically that finite-width effects are particularly strong for polynomial interpolation tasks. The transition at $P \\sim \\sqrt{N}$ can be delayed to higher values of $P$ through ensembling or a type of feature learning. A particularly important parameter for feature learning is the output scale $\\alpha$. Feature learning may alternatively be performed by learning higher order polynomials. The authors find that ensembling according to the kernel is more effective than averaging the network predictions or features.", "summary_of_the_review": "This paper studies a special case of the important problem of reconciling infinite width theory with the behaviour of finite width networks. The paper is informed by a rigorous empirical analysis based on mathematical truth. Unfortunately, the story it told was not memorable and the theoretical and/or practical takeaways were not clear. I think this paper could be improved by trying to synthesise the isolated conclusions into a more holistic message.", "strength_and_weaknesses": "**Strengths Weaknesses:**\n- The data considered are all very simple. As far as I can tell, they are polynomials of order $k=1,2,3,4$. In this setting we would expect kernel methods to perform very well compared with (finite width) neural networks, even if the kernel is not a polynomials. Is there any evidence to suggest that the insights in this paper generalise to more complicated settings?\n- The paper has some very nice insights concerning some very important and interesting phenomena in non-kernel regime networks. The empirical analysis seems thorough and is based on what appears to be correct mathematically motivated calculations\n- However, I found it quite difficult to read. The paper feels like a set of isolated empirically established facts, rather than a cohesive and memorable story. In writing my summary, I was not able to find any single standout takeaway message that was theoretically or practically useful, beyond its immediate implication. To make matters worse, the studied situations (polynomial interpolation, MLPs, ...) are never considered in practice. Perhaps this kind of work is more suitable for a journal publication, where there is more room to establish some ground work and set the scene for the story. I note that there is lots of extra text in the appendix that is not all proof and figures.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SJ-Yw9pArX", "reviewer": "Reviewer_JK78", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper compares the generalization of finite width networks to their infinite width (NTK) counterpart. More precisely the paper analyse empirically the effect of increasing the number of datapoints $P$ for some fixed networks. These finite width effects depend on the regime (lazy or rich) the network are in:\n - In the lazy regime, the difference between finite width and infinite width network is due to an increase in variance for low widths. As a result the performence of finite width networks is strictly worse than the NTK limit.\n - In the rich regime, the presence of feature-learning (which is absent in the infinite-width NTK limit) helps generalization, thus outperforming the NTK limit for a range of dataset sizes, but for large $P$ the variance dominates, thus leading to worse performence.\n - The authors observe that the start of this variance dominated regime scales roughly as $P=\\sqrt N$ in both regimes, though it starts later in the rich regime.\n\nThe authors present a toy model to mathematically recover some of the properties observed empirically.", "summary_of_the_review": "The paper studies finite width effects in DNNs, in particular the opposing effects of feature learning (which improves generalization) and of the variance (which worsens generalization). These two effects remain ill understood in deep neural networks (though many paper have studied them) and their interaction is even less understoood. The paper study both of these effects empirically and with a few small theoretical results.", "strength_and_weaknesses": "With the recent focus on understanding infinite width networks, it is crucial to understand their discrepancy with finite width networks. The paper makes a number of interesting observation, supported by detailed experiments and a small toy model.\n\nWhile the authors have quite thoroughly studied the impact the of the number of datapoints $P$, width $N$ and initialization, the effect of the depth and of the task itself (input dimension, degree of the polynomial or more general function) are not really studied. Of course, I understand that it is not possible to study all of these effect simultaneously in a 9 page paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J43m3SpsdTl", "reviewer": "Reviewer_4Na8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the generalization error of deep ReLU feedforward networks trained with different initialization scales, widths, and training set sizes in comparison to kernel regression with Neural Tangent Kernels (the kernel in the infinite-width limit, the kernel at initialization, and the kernel at the end of training).  They demonstrate that the kernel corresponding to the infinite-width limit outperforms lazy networks (large initialization scales) and the kernel at initialization.  Furthermore finite width networks even in the rich regime (small initialization scales) are outperformed by the infinite-width NTK at large dataset sizes, however rich networks can outperform at intermediate dataset sizes.  Furthermore they study the dataset size after which most of the generalization error comes from the variance arising from the parameter initialization, and conclude that the transition occurs at dataset sizes on the order of the square root of the network width.  Consistent with other works, they show that the NTK at the end of training exhibits similar generalization error to the trained network (for a variety of widths and initialization scales).", "summary_of_the_review": "**Initial Recommendation:**\n\nAccept.  While the data model is a bit limited, the paper makes an interesting study of comparing networks in the lazy and rich regimes to NTK methods at different dataset sizes.  \n\n**Comments/Additional feedback:**\n\nFigure 2 (a) On the top of the figure worth replacing $E_g$ with $\\log(E_g)$ since it is the log generalization error\n\nFigure 3 (b) What width were the various networks trained with for the different $\\alpha$ values.  Was the width $N$ fixed and alpha varied to plot the stars?\n\nPage 7 Perhaps use \\eqref to reference equation 3\n", "strength_and_weaknesses": "**Strengths:**\n\nThis work considers the learning curves for finite widths and finite sample sizes, in contrast to previous works that require that one of the quantities tends to infinity.\nThere are plenty of experimental results with clear figures and illustrations.\nThe exposition is clear\n\n**Weaknesses:**\n\nThe structure of the data (degree $k$ polynomials on the sphere), uniformly distributed inputs is a bit limited.  In Figure 1 we already see that increasing the degree of the polynomial can already change the results for the ensembled networks (for $k=2$ the ensembles performs like the infinite-width kernel, for $k = 4$ the ensembles outperform).  It would be reassuring if we had an idea of which results are robust to the data distribution and target function.\nThe results may be specific to feedforward networks.  As was shown in [1] gap between NTK performance and finite width networks is greater for CNNs in comparison with feedforward networks.  While ensembled feedforward networks perform like the NTK, ensembled convolutional nets significantly outperform the NTK. \nI do have one reservation about the variance of the NTK over initialization.  Please see my question about the NTK variance in the questions section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BzLMT7fSKy", "reviewer": "Reviewer_PWNi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "*Please note that this is an emergency review, so I was not able to go as much into details of the paper as I would have liked to*\n\nA lot of recent work on deep learning theory has focused on understanding the\ndynamics and generalization of neural networks with infinitely wide hidden\nlayer. This is of course an idealisation, and recent work has sought to\nunderstand when finite-width effects become relevant as the size of the training\nset increases.\n\nThis work performs a careful study of two-layer ReLU networks trained on a\nsimple polynomial regression task to investigate precisely when finite-size\neffects become important for the performance of the network. The authors vary\nthree key parameters: the width of the network, $N$, the size of the training\nset $P$, and the *scale* of the network $\\alpha$, which was introduced by Chizat\net al. (2019) as a mean to interpolate between the \"lazy regime\" (large\n$\\alpha$) and the \"feature learning regime\" (small $\\alpha$).\n\nThe authors find through experiments that for large data sets, the\ninfinite-width limit always performs best. Feature learning does improve over\nthe infinite-width limit at intermediate data set size, while lazy learning at\nintermediate sizes does not. They relate these behaviours to the fluctuations due to \nvarious sources of noise in the system (for example in the initialisation of the weights),\nand show how ensembling can mitigate these fluctuations. The authors propose various scaling laws to explain\ntheir findings, which they obtain via a toy model which qualitatively reproduces\nthe effects (cf. Fig 6).\n", "summary_of_the_review": "To summarise, this paper provides a careful investigation of a problem which is\nof interest to the theory of neural networks community, and should therefore be\naccepted at ICLR.\n", "strength_and_weaknesses": "The paper tackles an important topic: given the ubiquity of papers studying\nneural nets in the infinite-width limit, it is important to understand the\nlimitations of this regime in practice. While previous work (which the authors discuss)\nhad already identified the variance-limited regime, or discussed the role of ensembling,\nhere the authors put everything together in a single case-study.\n\nThe paper is also well-written: it clearly states the setup and the results, and\nit connects nicely with the previous literature.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster