Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials 
Authors: Eshaan Nichani
Yu Bai
Jason D. Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning theory
neural tangent kernel
beyond NTK
optimization landscape
learning polynomials

Abstract: A recent goal in the theory of deep learning is to identify how neural networks can escape the “lazy training,” or Neural Tangent Kernel (NTK) regime, where the network is coupled with its first order Taylor expansion at initialization. While the NTK is minimax optimal for learning dense polynomials (Ghorbani et al, 2021), it cannot learn features, and hence has poor sample complexity for learning many classes of functions including sparse polynomials. Recent works have thus aimed to identify settings where gradient based algorithms provably generalize better than the NTK. One such example is the “QuadNTK” approach of Bai & Lee (2020), which analyzes the second-order term in the Taylor expansion. Bai & Lee (2020) show that the second-order term can learn sparse polynomials efficiently; however, it sacrifices the ability to learn general dense polynomials.

In this paper, we analyze how gradient descent on a two-layer neural network can escape the NTK regime by utilizing a spectral characterization of the NTK (Montanari & Zhong, 2020) and building on the QuadNTK approach. We first expand upon the spectral analysis to identify “good” directions in parameter space in which we can move without harming generalization. Next, we show that a wide two-layer neural network can jointly use the NTK and QuadNTK to fit target functions consisting of a dense low-degree term and a sparse high-degree term -- something neither the NTK nor the QuadNTK can do on their own. Finally, we construct a regularizer which encourages the parameter vector to move in the “good" directions, and show that gradient descent on the regularized loss will converge to a global minimizer, which also has low test error. This yields an end to end convergence and generalization guarantee with provable sample complexity improvement over both the NTK and QuadNTK on their own.

Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials
Eshaan Nichani Princeton University eshnich@princetonn.co edu
Yu Bai
Salesforce Research yu. www.bbsallsoooree.oom com
Jason D. Lee
Princeton University
jasonnle@@princeton..dd
Abstract
A recent goal in the theory of deep learning is identify how neural networks can escape the "lazy training," or Neural Tangent Kernel (NTK) regime, where the network is coupled with its first order Taylor expansion initialization. While the NTK is minimax optimal for learning dense polynomials 22 it cannot learn features, and hence has poor sample complexity for learning many classes of functions including sparse polynomials. Recent works have thus aimed to identify settings where gradient based algorithms provably generalize better than the NTK. One such example is the "QuadNTK" approach of Bai and Lee ? which analyzes the second-order term in the Taylor expansion. Bai and second-order term can learn sparse polynomials efficiently; however, it sacrifices the ability to learn general dense polynomials.
7 show that the
In this paper, we analyze how gradient descent on two-layer neural network can escape the NTK regime by utilizing spectral characterization of the NTK B and building on the QuadNTK approach. We first expand upon the spectral analysis identify "good" directions in parameter space which can move without harming generalization. Next, we show that wide two-layer neural network can jointly use the NTK and QuadNTK fit target functions consisting of dense low-degree term and sparse high- digheeegree term something neither the NTK nor the QuadNTK can do on their own. Finally, we construct regularizer which encourages the parameter vector to move in the "good" directions, and show that gradient descent on the regularized loss will converge to global minimizer, which also has low test error. This yields an end to end convergence and generalization guarantee with provable sample complexity improvement over both the NTK and QuadNTK on their own.
Introduction
In recent years, deep learning has acheived number of practical successes, in domains spanning computer vision, natural language processing, reinforcement learning, and the sciences. Despite these impressive empirical results, the theory underlying deep learning is far from complete. In fact, the dual questions of optimization - the mechanism by which neural networks trained with gradient descent are able to interpolate training data despite the nonconvexity of the loss landscape and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

generalization - why these solutions found by gradient descent require relatively few samples to generalize are still not well understood.
One successful approach for understanding optimization has been the Neural Tangent Kerel (NTK) theory [41 20
20 The NTK approach couples the gradient descent dynamics of wide neural network under specific initialization to the gradient descent dynamics of a particular kernel regression problem, with random, initialization dependent kernel. In the limit of infinite width, this kernel converges almost surely to a deterministic kernl,, also referred to as the NTK, and properties of this kernel and its corresponding Reproducing Kernel Hilbert Space can be studied.
However, recent work has shown that the NTK theory fails to explain the generalization capabilities of neural networks. While the equivalence between neural networks in the NTK regime and kernel methods implies that such models perform no better than kemels, in practice, neural networks have been shown outperform kernel methods
number of tasks [5] 33 Theoretically, recent has provided a precise statistical analysis of the generalization properties of rotationally invariant kernels on the unit sphere, which includes the NTK.
line of work 25 39, 33
[4 proves sample
complexity lower bound for such kernels, showing that dh samples are needed to learn any degree k polynomial in d dimensions. As result, the NTK is no better than polynomial kernel, and cannot adapt to low-dimensional structure.
The limitations of the NTK can be further understood from the linearization perspective. Consider two-layer neural network fx; W) with input x, width m, first layer weights initialized as Wo e Rdxm, second layer weights a E R activation function a, and displacement from initialization W E idxm;
1 Σ a. x.).
f(: W)
(1)
For simplicity we assume the second layer weights are held fixed, and so W are the trainable parameters. The NTK theory states that when W has small norm, the gradient descent dynamics can be well approximated by replacing the model with its first-order Taylor expansion about the initialization:
f(x: W) ~ f(x; +
Σ Ca,o(wwo XWx w. = fxx: 0) + vee W) (x().
(2)
Here, (wrrremm) and (wor)re(ml are the columns of W and Wo respectively, cx(x) = (xxx W) w=o) is a random feature vector with norm independent of m, and
fz(x: W) vec W) ((x))
hereafter referred to as the linear term. Ignoring dependence, there exists a global minimizer with [[]]e ~ 1 and ||w, 2 m 112 and thus due to local convexity gradient descent will stay in this small norm ball around the initialization while interpolating the training data. This small movement of each individual neuron gives rise the name lazy training for networks in the NTK regime [IS]. The equivalence to kernel methods and the poor generalization of neural networks in NTK regime, along with their failure to describe the dynamics of neural networks in practice, motivate our goal to understand how neural networks can escape the NTK regime. We concretely ask the following question:
Q: How encourage each neuron to move >> m m12 thus escaping the NTK regime? And does this allow us to break the NTK sample complexity lower bounds?
1.1 Motivation
Escaping the NTK Regime. To answer this, we invoke the statistical characterization of the NTK
developed in (25 39 38
understand the mechansism by which it overfits the training data, For d << n & ++1, consider a dataset of n training samples (x,, Mi). where the x; are sampled i.i.d
from Sat (Vdd) (the d-dimensional sphere of radius vd) and y; fx) for an unknown function f". Montanari and Zhong
3 decompose the SVD of the empirical feature matrix & E I@axmd into
the block matrix form
[x(x))"
Ä
0
+-
U1
U2
(4)
(xxx)T
0
V

---

where UI. A1. 11 are the top singular values/vectors and U2, A2. V2 are the bottom n Here, r O(d") chosen specifically so that V1 is the "high-variance" subspace which can express polynomials of degree S k and generalizes well ( while V2 is used to interpolate the training data while not affecting generalization.
39 show that the NTK will learn the projection of f onto degree à polynomials. Furthermore, show that gradient descent will first move in the subspace spanned by V, learn this projection, then move in the V2 directions to interpolate the training data, while not affecting the test predictions. This first stage is still desirable for our goal - if the network can fit and generalize from part of the signal using the NTK, then it should, and previous work [27] has shown that for general networks gradient descent will learn the optimal degree-1 polynomial in the early stages of training. The V2 directions, however, are "bad" directions the parameters should avoid moving in, as they are only used for the NTK to overfit. Instead, W should move in the null space of O,, where || |[]] r can be (2(1) while keeping the evaluation of (z; W) on the training data bounded by O(1). This heuristic argument yields our first criterion for escaping the NTK regime.
25
0
Goal #1: Move minimally in the V2 directions.
Generalizing to Test Data. While the previous criterion prevents the network from overfitting with the NTK, we must also prevent a movement of || > 1 from causing the test predictions to
explode. Since [|xxx))|22 en (1), is a priori possible for fz W) vec vec(W)"""pte (xx) to be 1
on the population, which would necessarily cause a large test loss.
To identify a set of good directions, define the feature covariance matrix Σ E mdddd by
Σ E Ex ((Va) [x(x))p(x)"]
(5)
Our first technical contribution a characterization of the cigendeccomposiiinn of E. Let the top r cigenvectors of Σ be Q1, the next s eigenvectors be Q2 and the bottom md-r eigenvectors be Q3. where A1, , A2, A3 are the corresponding diagonal matrices of eigenvalues, and (dd s ((k) We use the results of 39 38) to show that Σ has an eigenvalue gap in that ( >> Amaz (A2) and Amin(A2) >> Amaz (Aa). and furthermore that Qi can fit arbitrary degree < A polynomials. This partitioning of the eigenvectors tells us that Q₂ are informative, large eigenvalue directions which help the NTK to learn low degree signal, Q2 are the medium directions which will cause the test predictions to grow too large W moves too far from initialization, and Q3 are the "good" directions which W is free move distance of > 1 in. This yields the following criterion for generalizing well:
Goal #2: Move in the Q3 directions, but minimally in the Q2 directions.
One challenge is that we cannot distinguish the Q2 directions from the Qa directions with d samples. Nevertheless, the existence of an eigenvalue gap will allow us to constrain movement in Q2-
The Quadratic NTK. Once we have moved > 1 from the initialization, we can no longer couple to the network's ' lincarization. The network is still, however, in a local regime, and we instead can couple the training dynamics to the second-order Taylor expansion of our model, where the second-order term is denoted the Quadratic-NTK 77 fo(x; W):
f(x; W) RM ((x:0) + vec x
+
1
Σ o" (ww x()" w,)
(6)
2
2/m
= fx;; + fI. (x; W) + (x(:; W).
Bai and Lee 77] showed that fx( W) can effectively learn low-rank polynomials with better sample complexity than the NTK. In particular, they show that d samples are needed to fit target function I
of the form fxx = (33 x)+++ an improvement over the d samples needed by the NTK. In doing so, however, the QuadNTK sacrifices its ability to learn general dense polynomials; furthermore, Bai and Lee 7 require randomization trick to artificially delete the fr term. A later followup work [ showed that a number of standard experimental settings, the second-order Taylor expansion of the network better tracks the true gradient descent dynamics and acheives lower test loss than the network's linearization (i.e NTK) does. However, there is no existing result which shows that both the linear term and the quadratic term can provably leam a component of the signal.
3

---

Based on the preceeding motivation, we thus aim to show that we can jointly utilize both the NTK and the QuadNTK to learn a larger class of functions than either the NTK QuadNTK can learn on their own.
1.2 Our Contributions
With the previous intuition in hand, we outline the main contributions of our work. We first prove a technical result on the eigendecomposition of Σ, and show that the eigenvectors can indeed be partitioned into 3 categories corresponding large (degree S k), medium ("bad") and small ("good") eigenvalues. We then construct a regularizer, depending only on the covariate distribution and initialization, that enforces goals 1 and 2 by preventing the parameters from moving in either of the bad sets of directions (V2 and Q2). Furthermore, we show how to jointly use the NTK and QuadNTK to fit target signal f consisting of an arbitrary degree component. The key technical challenge is to construct a solution with large enough movement so that the QuadNTK can fit the high degree term and hence improve generalization, while simultaneously preventing this large movement from interfering with the NTK training predictions (Goal I) or greatly increasing test loss (Goal 2). Our main result, Theorem | is that gradient descent on a polynomially wide two-layer neural network converges to an approximate global minimizer of the regularized loss function, which generalizes well to the test distribution. As result, to learn f up to vanishingly small test loss. This ultimately gives us the "best of both worlds", as we leverage both the linear and quadratic term to learn the target f with sample complexity better than either the NTK or QuadNTK alone. Overall, our work identifies which directions weights can move further from initialization in and provably generalize better than the NTK.
component and sparse degree k + 1
show d samples are needed
The outline of our paper is as follows. In Section 2 we formally define the problem setup. In Section define regularizers, and present Theorem Section 4 is an outline of the proof of Theorem which we split into four components expressing f* with the linear-plus-quad model, showing the optimization landscape has favorable geometry, gradient descent convergence result, and generalization bound. We conclude with experiments supporting our main theorem and demonstrating the relevance of the low-degree plus sparse task to standard neural networks.
1.3 Related Work
The NTK approach
which couples a neural network to its linearization at initialization, has been utilized to show global convergence of gradient descent on neural networks [19
41
15
32
35 47 4 The equivalence to kernel methods has also been used to prove generalization bounds, [ However, neural networks have been shown to
based on generalization bounds for kernels 6
perform far better than their NTK in practice S 33 Further, (25] shows that kernels cannot adapt to
low-dimensional structure, and proves sample complexity lower bound of dh samples needed to learn a degree k polynomial in d dimensions. A number of recent works
4 18 46
24 16
45
14 37 have thus aimed to provide examples of learning problems where neural network trained with a gradient-bar based algorithm has a provable sample complexity improvement over any kernel method.
36
One such approach has been to understand higher-order approximations of the training dynamics 7 8, 28 Here, the network is no longer coupled to its linearization, but rather higher order terms in the Taylor expansion. On the empirical side, [ shows that these higher order Taylor expansions better track the optimization dynamics and can obtain lower test loss. Theoretically, 7 12) prove that the second-order term, the QuadNTK, can be used to obtain sample complexity improvements. However, the QuadNTK has poor sample complexity for learning dense polynomials, and 7 12 do not consider training on the original network, but rather only the sccond-order term after the linear term has been deleted. This paper, the other hand, provides end-to-end convergence and generalization result for training on the full two-layer neural network. We leverage the NTK to efficiently learn polynomials with both dense and sparse component, and are thus the first work showing that both the linear and quadratic term can learn part of the signal.
The technical results in our paper rely on the statistical characterization of the NTK developed Furthermore, our optimization results rely on a line of work showing that quadratically parameterized models have nice landscape properties such as all second- order saddle points are global minima [2] 22 41 18 ; the fact that gradient descent avoids saddle
in the series of works 25
39 3S
points 21 34
30
31 can then be used to show convergence.

---

2 Preliminaries
2.1 Problem Setup
Our problem setup is the standard supervised learning setting. Our dataset D, = {(x,, yi) Ne(x). has n samples, where (x,, Mi) E X ) are sampled i..d from a distribution μ on x V. μ is defined so that (x, 3) ~ μ satisfies x ~ Unif(S (vī)), ) the uniform distribution on the d-dimensional sphere of radius vd, and = f (x) for some deterministic, unknown function f -sd11( + R.
We assume that d < n << d for some integer k, and that the target I* has the following low-degree plus sparse structure:
Assumption (Low-degree plus sparse signal). Let (x) fa(x) + fxp(x), where
fx( is an arbitrary degree S k polynomial with Ex [ (x)² = 1. (Low Degree)
(x) = Ei (8T"x)) x where layl S 1, B.1l2 = 1. (Sparse)
We aim fit f with f(x; W), two-layer neural network as defined in I.. Here, W [W.,..., wr] E Rdxm is the first layer weight's distance from initialization and is the train- able parameter. Wo = [Wo.l..... Wo.r] denotes the first layer weight at initialization, and and a [a1,...,,m am E Rmm the second layer weight, which held fixed throughout training.
We consider the following symmetric initialization of a, Wo, which ensures that f(:; 0) = 0 identi- cally.
01 = 0m//2 = 1 [wo.rrsmm/2 Mi.i S(d- (1)
0m/2+1
am
(8)
Wo,m/2+r Wo,r
(9)
a E (3((T) is our nonlinear activation function. We make the following assumption on at
Assumption 2.
The activation a satisfies ||o|]oo; ||a' ||a"|lo < 1.
We also require a', o" to satisfy Assumption 3 a particular technical condition their harmonic expansions. These two assumptions are satisfied by commonly used activations, such as the sigmoid
with generic shift b: (=) [+exp(b-2)
assume the loss function 8 x R
20 satisfies E(v, S 1, E(y. v) = 0, Ely, =) convex in dgs Eloo I 1. The empirical loss f. and population loss L are defined
and 3 flloo ll 11 Elooo as
0a ll
Ł(W) = E. (e (x; WI)]
L(W)= = E,, [e(,, (x; W))l:
(10)
where for a function g(x, y), R. [g v) = 1 Σl (x,, yt) denotes the empirical expectation, while (x y)] denotes the population expectation over (x, v) p.
Notation. For f E (V(s) 11). define || = II II (s = [ffx)((((x){]] 1/2
We use big O notation to ignore absolute constants that do not depend on n. d, m. as well as polyno- mial dependencies on the rank R. We write as S ba if as Oba).. as < ba if lim aa ba
We also use Ô notation to ignore terms that depend logarithmically on d. We also treat k O()). Finally, all our results hold for d > C, where C' is a universal constant. For a matrix A, we let ||A
be its Frobenius norm, ||| = | |l op be the operator norm, and ||A||, = (E ||. // be the
2, p norm.
2.2 Linear and Quadratic Expansion
For W small, ((s W) can be approximated by its second order Taylor expansion about Wo:
1 Σ 0 (oo''w,,))x w, + 1 ja, x(", (x w.)
f(x: W)
(11)
5

---

We define fL(x, fx(x; W) to be the linear and quadratic terms of the network:
fx(x; W) = m P=
Σ
a,o a, x)x,,
fo(x: W) = m r= 2 a,
(x w,)2
(12)
3
Main Theorem
Define the NTK featurization map vp - (Vd) + Rnd as
6 vec W W)lw=o).
(13)
and the feature covariance mairix Σ Rmdxmd
Σ = Ex>p [x)))
(14)
Note that Σ depends only the network initialization and the imput distribution, and not on the target function f". In practice, can be approximated to arbitrary precision by using a large dataset of unlabeled data, by computing the harmonic expansion of o'. detailed in Appendix A
Let Σ admit the eigendecomposition E En (EEvvvy? where the X (E) nonnegative and nonincreasing. For r E [md], we let Hlr the projection operator onto span(Vi..... VI,..., Vy), and let Hy Ind Ir. Furthermore, define
Esr = Σ AEEE)NI
Σ>r = E Ezrr. =
(15)
i=1
We define our regularizers as follows
R1(W; r) = vec () vec(W)
'E, vec
(16)
(2(W;r) = vec Ee ve
(17)
Ra(W;r) = E, [(fr(x; (I: W))
(18)
(W( 1/
(19)
Intuitively, R3 constrains movement in the V2 directions to enforce Goal #1, R, constrains movement the Q2 directions to enforce Goal #2, and R2, R4 are weight-decay like terms necessary for generalization. Although R, does not know the directions Q2, the eigenvalue gap between Q2 and Qa ensures that whenever R, is small, movement in Q2 must small as well.
Given regularization parameters À = (A1, A2, A3. A1), define the regularized loss Lx (W)
LA(W) i(W) R1 W;r) R2 (Wr) + Ag (W; Ra (W).
(20)
Finally, we train our model trained via perturbed gradient descent 31 with learning rate and noise level 2 That if W denotes the weights at time step the update given by
W W 7 WWL"(W") =.).
(21)
where E, dx are i.idd random matrices with each entry i.i.d i.i.d a²
Given these definitions, now present our main theorem:
Theorem 1. Let E be target test accuracy, the number of samples be n 2 dh poly(R)
max(e-2 2, log d). and the width be m poly(n, d, R, = Let the sequence of iterates {W") follow the update in (21) with initialization W (A1,)2, X3, A4. o², 11) such that with high probability over Wa, Dn, and (Er)tzo. there exists T = poly(m) such that the predictor W = 9 satisfies L(W) S.
Then, there exists choice of parameters
Remark 1. Theorem tells us ((d+) samples are needed to learn the low-degree plus sparse function f This is an improvement over the sample complexity needed to learn f* via the NTK, which is 22d samples as f" is degree (A: 1o-polynomial [25 39 This also improves the upper bound for the sample complexity of the quadratic NTK given in 77. in which Ω d samples are needed learn rank K polynomial of degree bound since the dense, low-degree term fg have rank > d.
This polynomially worse than

---

Proof Sketch
The proof of Theorem follows similar high-level steps 77 which fits f and small on the regularizers. Next, we show that the optimization landscape of the regularized loss has favorable geometry, and result that gradient descent converges to global minimum. We conclude with generalization bound to show that global minima have low test loss. Throughout the proof sketch, we emphasize how the regularizers encourage regime, and discuss the challenges posed by the existence of the fr. term in the dynamics.
We first construct W R
escape the NTK
4.1 Expressivity
We begin by showing that the fr. and SQ terms can fit the low degree and sparse components, the derivations in this section rely on spherical harmonics;
respectively, the signal. As [25 39
an overview of the technical results used are presented in Appendix A
The following lemma shows that (x; W) can fit the sparse, high degree term in f"
Lemma (QuadNTK can fit high degree component). Let m d2h. With high probability over the initialization, there exists Wo such that
max [fo(x; We) (x)(( S d KE(M]
and &
(22)
Vm
This generalizes the corresponding result in to more activations. presented in Appendix B.1.2
The proof of this lemma is
Next, we show that fr (x: W) can fit the low degree term. Here, we choose r ny (dd)), where ny is defined in Appendix to be the dimension of the subspace which can express degree s k
polynomials. We define P Is be the projection on the top n, eigenvectors E.
Lemma (NTK can fit low degree component). Let m 2 d0k With high probability, there exists W, with vec(W (.) E span(P such that
d and W d
(ff WL) fx(())
(23)
n
The proof of this Lemma is presented in Appendix B.2.4. and relies on key lemmas from 39 relating the spherical harmonics of degree & k to the eigenstructure of the kernel. A key intermediate result is Lemma [5] which characterizes the spectrum of the population covariance matrix E. A similar result for random features was shown in (38]. Unlike (38. we do not characterize all the eigenvalues of Σ; however, simply partitioning them into the three categories is sufficient for our purposes.
Finally, use the WL. WQ to construct W which has small regularized loss. Recall the definition of the regularizers, after setting = ng:
R, (W) = R, (W;nk) (; P W) 16, Psx W)) R2( W) = R2 (: mx) = 1.f(; W) E,
(24)
(25)
R3 W) = Ra W; m²) = E,, PP
(26)
(W( = ||II 1
(27)
Also, define the empirical loss of the quadratic model as:
i (W) == E, [y fx; W) + fo(x; W))l
(28)
The following theorem is the central expressivity result:
Theorem 2. For Emin > O, let Z max(d$it-11g_ min d 2 max mmn dl log d). With
probability 1/poly(d), there exists W such that
L Emiin
(29)
and:
R(")") S mm +
R2(W)) $ Ra(W*) fm
Ri(W") S 1)(
(30)
7

---

The proof of this Theorem is presented in Appendix B.3 and again relies on the eigendecomppoition of Σ. As outlined in the introduction, we show Σ Q + Q24,Q Q3A, Q where A₂ are the medium eigenvalues and A3 are the small eigenvalues. Formally, A2 contain (d with multiplicity (d²), for integers E [k + 1, 2k), and the entires of A3 are equal to 1/m on average. This tells us that the medium directions Q2 are undesirable,
any 9 (1) movement in these directions will case (x; W) to grow large. On the other hand, movement in a "sufficiently random" Q3 direction will minimally affect the population value of (x W).
To prove Theorem 2 we will construct W to be of the form Wt. + We, where W,, fits the low- degree term and W Q fits the sparse term. The issue with this direct construction is that || > 1, so a priori Wa can have a large effect on the linear term. We thus require fL (x: We) to be small, both on the sample and over the population. The key insight is the following: since dim (1) < dim( V2). any sufficiently random direction lies almost entirely in V2. If u is this random direction, then E,, [(fr. (x; = is small. Similarly, the random direction lies almost entirely in Q3- Since the Q2 directions have eigenvalues 1/ 1/m on average, By, [(fr u))
= m ||ul <.
We thus consider "sufficiently random" version of W so that Wq minimally affects fr. Specifi- cally, we consider the weight SWa. where S E m is a diagonal matrix of random signs. By definition fg (x; SW Q) fq (x; We), furthermore, we show SWq is now sufficiently random in that E,, [If u}) Ep [(fu (x; ux; ) are both small. This allows us prove that in expectation over S, a solution of the form W i. + SWo acheives small regularized loss, which implies the existence of such desirable solution via the probabilistic method.
4.2 Landscape
Define first and second-order stationary points
31
follows:
Definition 1. W is u-first-order stationary point of f if (f(VS(()) S
Definition 2. W is a (V, ssscooddood stationary point (SOSP) of I if (f 2 -7I
The following lemma is our central landscape result:
Lemma 3. Let r dr A2 = Eminn A3- = m2dd" Emiin X1 = dd-((k- 10 miin- Assume m 2 ndd 201k=1) 3 Emn -22/3 and < -m- 3, Let W be a v-first order stationary point, and let W be the
solution constructed in 2 Then,
Es [V
- (VLA W 22, + WL) 2L2 2L (W") S Emin-
(31)
As corollary, we show that any (V,7)-SOSP of LA (W) has small loss.
Corollary 1. Set A as in Lemma Let W be -ocood-eoo staation point of Lx (W), with S m m-122, 7 < m 334 Then Lx (W) S Emin-
The proofs of these results are deferred to Appendix
4.3 Optimization
We next invoke the main theorem of 30 31, which is that perturbed gradient descent will find a (v, -)SSSP in poly(1/v, 1/// time. The challenge with applying these results directly is that Lx is no longer smooth or Hessian-Lipschitz due to the R2 regularizer. To circumvent this, we first prove that the iterates in perturbed gradient descent are bounded in a Frobenius norm ball. Then, it suffices in this ball to prove convergence. Our
to use the bound on smoothness and Hessian-Lipschite main optimization result, with proof in Appendix D is the following:
Theorem 3. There exists a choice of learning rate n and perturbation radius such that with probability 1 - 1/poly(d), perturbed gradient descent (c.f [37 Algorithm 1]) reaches a (v. (.-S)SSSSPP within T = poly(m) timesteps.
8

---

A: 0.0
Ag = 1e.05
300
2; 0,0001
0.8
A: -00000
R:(W)
=0.0.
Train Loss
0.6
Test Loss
0.6
200
As 0.1
0.4
0.4
0.2
0.2
0
0.0
0
2
0
2
logot
logut
logot
Figure 1: We train fz + fo with varying A3. When A is small, the NTK overfits the high degree signal and test error is large. When A3 is large, the QuadNTK can learn the high degree signal, and test error is smaller. Results are averaged over trials, with one standard deviation shown.
4.4 Generalization
Finally, we conclude by showing that any W with small (x(W) also has small test loss:
Theorem 4 (Main generalization theorem). Let r = di, a1 = m d k-1 Emino de = Emin. As
m 3 Emin. A d (--((kl)) E Assume m 2 Emin ((k--)) With probability 1 - 1/poly(d)
over the draw of D, any data dependent W with (x( S CEmin has population loss
d
L(W) S Emin + +V
(32)
The proof of this theorem is presented in Appendix The key is use the small values of the regularizers at W to bound the test loss. R2 and R, are used to bound the Rademacher complexities of the linear and quadratic terms respectively, while R, and R3 control the influence of the high degree component of the linear term on the population loss and empirical loss respectively.
Experiments
conclude with experiments to support our main theorem. In Figure I we train the joint linear and quadratic model (x; W) + fx W) via gradient descent the square loss, with signal (x) + (8T x)² where fi(x) = œ 1. Our covariates are dimension d = 100, we have (x) a 5 1000 samples, and our network has width mn 10000.
We train our model with the regularizer Ra (W) for varying values of Ag. Rather than computing E, we use the top næ right singular vectors of to estimate R3. We observe that zero training error. However, when A3 is zero or very small, the model struggles to learn the entire signal, and test error plateaus near 0.6. the leftmost pane, we plot the value of Rg (W) over course of training. We observe that R3 (W) grows large for small Ag. which implics that the model using the "bad" directions in V2 to overfit to the training data. For large values of λg. however, R3 prevents W from moving in the "bad" directions overfit the data. This is seen in the leftmost pane, where the value of R3 (W) is small. Instead, the parameter moves the "good" Q₂ directions, and the Quad-NTK term kicks in to fit the remaining component of the signal while generalizing. This leads to the consistently lower test loss (around 0.2) as shown in the rightmost pane.
Xa we reach near
On the regularizer R₁, Our proof required the existence of R1 to prevent movement in the bad Q2 directions. While R₁ being small is necessary for generalization, in Figure | we observe that we generalize well without explicitly regularizing R,. We hypothesize that the noise in the perturbed gradient descent update may be implicitly regularizing RR, A heuristic justification is as follows. Recall that any sufficiently random direction lies mostly in Q.s. Due the rotational symmetry of the perturbations, one might expect that the solutions P& W P WS, over choices of random signs S, can be reached with roughly equal probability. Since this set of solutions is "sufficiently

---

random," in expectation over S it generalizes well, and thus gradient descent is likely to converge to solution which generalizes well. In Appendix G we conduct experiments with both R₁ and R3 to support the claim that explicit regularization of Ri is not needed to acheive low test error; however, proving this claim left for future work.
On the Low-Degree Plus Sparse Task. Theorem shows that two-layer neural networks trained via noisy gradient de- scent with a specific regularizer can more efficiently learn low-degree plus sparse target functions. In Figure 2 empirically that neural networks with standard initialization and trained via vanilla gradient descent efficiently learn a "dense quadratic plus sparse cubic." For varying values of d, we train a two-layer neural network to convergence and plot the smallest such that the test loss < 0.1. We ob- serve this minimal n roughly scales with d², the optimal (in d) sample complexity. This provides evidence that standard networks trained with vanilla GD can effectively learn the low-degree plus sparse task, and hence that this is sensible task to study and that our work presents a step towards un- derstanding why standard neural networks perform better in practice more generally. See Appendix for more details.
10a
n 6d
we show
n samples
103
10
50
Figure 2: Neural networks optimally learn the "dense quadratic plus sparse cubic" task.
On Second-Order Taylor Expansions of Standard Networks. Throughout this paper we studied the quadratic Taylor expansion of a two-layer neural network. In Appendix G we complement the results of X and show that for a standard architecture and data distribution, the quadratic Taylor expansion better approximates the optimization dynamics and acheives lower test loss than the linearization (NTK) does.
Discussion
The goal of this work is to better understand how neural networks can escape the NTK regime. By analyzing the eigendecomposition of the feature covariance matrix, we identified sets of directions ones that can fit low degree signal, "bad" directions which either cause the NTK to overfit or the test predictions to explode, and "good" directions in which the parameters can move large distance. We then constructed regularizer which encourages movement in these good directions, and showed how a network can jointly use the linear and quadratic terms in its Taylor expansion to a low- low-degree plus sparse signal. Altogether, provided an end-to-ent tnd convergence and generalization guarantee with provable sample complexity improvement over the NTK and QuadNTK.
As discussed above, one interesting direction of future work is to understand the role of R1 Other directions of future work include understanding whether our analysis can be used to leverage higher- order terms in the Taylor expansion, understanding the connection between the QuadNTK and feature learning, and investigating whether increasing the depth of the network can allow the NTK and QuadNTK to jointly learn hierarchical representation [12]
Acknowledgements
EN acknowledges support from a National Defense Science & Engineering Graduate Fellowship. JDL and EN acknowledge support of the ARO under MURI Award 1911NF-11-100304 the Sloan Research Fellowship, NSF CCF 2002272, NSF IIS 2107304, NSF CIF 2212262, ONR Young Investigator Award, and NSF-CAREER under award #21 14494.. The authors would like to thank Alex Damian for helpful discussions throughout the project.
10

---

References
[1] Zeyuan Allen-Zhu and Yuanzhi Li. What can resnet learn efficiently, going beyond kernels? In Advances in Neural Information Processing Systems, 2019.
[2] Zeyuan Allen- Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep learning. 2020. URL https://aaxiiiii wttg/bbs//001.00 BBLL:tpp:://arr..ooo///////2000. 04413
[3] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overpa- rameterized neural networks, going beyond two layers. In Advances in Neural Information Processing Systems, 2019.
[4] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-paramettrization.. International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 242-252, 2019.
[5] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[6] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analy- sis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning (ICML), 2019.
[7] Yu Bai and Jason D. Lee. Beyond linearization: On quadratic and higher-order approximation of wide neural networks. In International Conference on Learning Representations (ICLR), 2020.
[8] Yu Bai, Ben Krause, Huan Wang, Caiming Xiong, and Richard Socher. Taylorized training: Towards better approximation of neural network training at finite width, 2020. URL https //arxiv.org/abaet
04010.
[9] Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: A statistical viewpoint. Acta Numerica, 30:87-201, 2021.
[10] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miin and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL http:/ http://github. cww/ wtt.a/u....ooooooo///eeee/// google/jax
[11] Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems, 2019.
[12] Minshuo Chen, Yu Bai, Jason D. Lee, Tuo Zhao, Huan Wang. Caiming Xiong, and Richard Socher. Towards understanding hierarchical learning: Benefits of neural representations. In Advances in Neural Information Processing Systems, 2020.
[13] Yuxin Chen, Yuejie Chi, Jianqing Fan, and Cong Ma. Spectral methods for data science: A statistical perspective. Foundations and Trends Machine Learning, 14(5):566-80) 2021. ISSN 1935-8237. doi: 10.1561/2200000079
[14] Zixiang Chen, Yuan Cao. Quanquan Gu, and Tong Zhang. A generalized neural tangent kernel analysis for two-layer neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[15] Lénaïc Chizat, Edouard Oyallon, and Francis Bach, On lazy training in differentiable program- ming. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[16] Amit Danicly and Eran Malach. Learning parities with neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[17] Chandler Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal Numerical Analysis, 7:1-46, 1970.

---

[18] Simon S. Du and Jason D. Lee. On the power of over over-vvrameen parametrization in neural networks with quadratic activation. In Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, pages 1329-1338, 2018.
[19] Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, 2019.
[20] Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations (ICLR), 2019.
[21] Rong Ge, Jason D. Lee, and Tengyu Ma. Matrix completion has no spurious local minimum. In Advances in Neural Information Processing Systems, 2016.
[22] Rong Ge, Jason D. Lee, and Tengyu Ma. Learning sae-hideen-layer neural networks with landscape design. In International Conference Learning Representations. arXiv, 2017.
[23] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limitations of lazy training of two-layers neural network. In Advances in Neural Information Processing Systems, 2019.
[24] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? In Advances in Neural Information Processing Systems, 2020.
[25] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 49:1029-1054, 2021.
[26] Nikhil Ghosh, Song Mei, and Bin Yu. The three stages learning dynamics in high-dimensional kernel methods. In International Conference Learning Representations (ICLR), 2022.
[27] Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[28] Jiaoyang Huang and Horng-Tzer Yau. Dynamics of deep neural networks and neural tangent hi- erarchy. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 4542-4551, 2020.
[29] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 8571 - 8580, 2018
[30] Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. How to escape saddle points efficiently. In International Conference on Machine Learning (ICML), 2017.
[31] Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M. Kakade, and Michael I. Jordan. On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points. Journal of the ACM, 68, 2021.
[32] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, 2019.
[33] Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. In Advances in Neural Information Processing Systems, 2020.
[34] Jason D. Lee, Max Simchowitz, Michael I. Jordan, and Benjamin Recht. In 29th Annual Conference on Learning Theory, Proceedings of Machine Learning Research, pages 1246-1257, 2016.
12

---

[35] Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, 2018.
[36] Yuanzhi Li, Tengyu Ma, and Hongyang R. Zhang. Learning over-parametrized two-layer neural networks beyond ntk. In Proceedings of Thirty Third Conference on Learning Theory, Proceedings of Machine Learning Research, pages
2020.
[37] Eran Malach, Pritish Kamath, Emmanuel Abbe, and Nathan Srebro. Quantifying the benefit of using differentiable learning over tangent kernels. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 7379-
7389, 2021.
[38] Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis, Special Issue on Harmonic Analysis and Machine Learning. 59:3-84, 2022.
[39] Andrea Montanari and Yiqiao Zhong. The interpolation phase transition neural networks: Memorization and generalization under lazy training. 2020. htttp:///a/a wrg. /ba/
2007. 12826
[40] Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl- Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020. URL https wtwp:hhub..ooooogggeeeeeeeeeee
tangents
[41] Mahdi Soltanolkotabi, Adel Javanmard, and Jason Lee. Theoretical insights into the op- timization landscape of over- parameterized shallow neural networks. IEEE Transactions on Information Theory, 65:742- 769, 2018
[42] Roman Vershynin. High-dimensional probability: An introduction with applications data science, volume 47. Cambridge University Press, 2018
[43] Martin J. Wainwright. High-dimenssonal statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
[44] Colin Wei, Jason D. Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets their induced kernel, 2018. URL https://arxxii...o abb//880. 0i5//99
[45] Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, Proceedings of Machine Learning Research, pages 3635-3673, 2020.
[46] Gilad Yehudai and Ohad Shamir, On the power and limitations of random features for under- standing neural networks. In Advances in Neural Information Processing Systems, 2019.
[47] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks, 2018. URL hattps://art //tr:i/./.ivvvv// orggabsg////eoooooooe...ooeaaert 181111111111 08888
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss the assumptions results rely on in Section 2
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This work a theoretical work, and does not have any direct societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Assumptions on our model are given in Section 2
(b) Did you include complete proofs of all theoretical results? [Yes] We give proof sketch of our main result, Theorem in Section and present the full proof in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We specify instructions in Appendix G and will submit code with the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify training details in Appendix G
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report error bars in all plots.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] Computational resources detailed in Appendix G
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite JAX [T]] and Neural Tangents 400 frameworks used for our experiments. Our experiments use synthetic data.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "exTXLk1PEB", "reviewer": "Reviewer_g4sF", "summary": "This paper builds on a solid theory for the optimization and generalization of two-layer neural networks fitting polynomials. In particular, they jointly utilize the power of NTK and QuadNTK to fit target functions consisting of a dense low-degree term and a sparse high-degree term, demonstrating its sample complexity is superior to either NTK and QuadNTK.", "questions": "Please see the Weaknesses part.", "limitations": "Please see the Weaknesses part.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* This paper improves upon the previous QuadNTK, showing that a special form of NN can avoid the kernel regime and achieve better sample complexity than any rotationally invariant kernels while fitting target functions consisting of a dense low-degree term and a sparse high-degree term. Compared to QuadNTK, it benefits from the linear term, providing its power to fit the dense polynomials.\n* The paper is well-structured and provides a clear proof sketch. \n**Weaknesses:**\n* As mentioned by the author self, some assumptions, i.e., smoothness and nonzero Hermite/Gegenbauer coefficients of the activation function, boundedness of the loss function, sphere distribution of the inputs. These assumptions are mainly technical, but some of them seem to be crucial for the proof techniques, like nonzero Hermite/Gegenbauer coefficients, which might prevent the general impact and application of the proposed methods. \n* The analysis of this paper heavily relies on the regularizers. Hence, my question is about the role of these regularizers. Are they just for the proof simplicity or necessary since we consider the quadratic approximation of the NN. If the answer is the latter, then it seems that there is something different from the model considered in this work might not reflect the practical success of general NNs.\n* This paper provides great intuition about how to avoid the blowup of the training loss and population and the usage of the four regularizers. However, it would be great if the authors can provide the intuition why the sample complexity can be reduced from $\\Omega(d^{k+1})$ to $O(d^k)$.\n* This paper fixed the second layer parameters, which also limits the general impact of this work.\n* (typo) The sign in line 273 is wrong. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YwBBjUE10Oj", "reviewer": "Reviewer_t53d", "summary": "The paper focuses on escaping the NTK regime for gradient descent and provided a comprehensive theoretic analysis toward understanding how a two-layer neural network could escape the NTK regime. Via eigendecomposition of the covariance matrix, the parameter space is partitioned into large, medium (bad) and small (good) directions. With the theoretical guide toward “good” directions, the authors develop a regularizer to encourage parameters moving in these directions. They also demonstrate that the linear and quadratic terms of the Taylor expansion can be used together to fit target functions and leverages from both NTK and QuadNTK to achieve provable improved sample complexity.", "questions": "See weakness and minor issues above.", "limitations": "The authors discussed the limitations of their work with regard to the assumptions made for their main theorem and the discussion is intact and reasonable to me. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well motivated and presented, with detailed explanations of the current limitations of NTK and QuadNTK and clear illustrations of why the focus of this paper raises interest. The structure and logic of the paper are easy to follow and the proof sketch is coherent and helpful for understanding. Generally, the writing of the paper is excellent.\n2. The contributions of this paper are well supported by solid evidence both theoretically and empirically. The theoretical proofs are solid as far as I read into, and the codes for the empirical results are available and runnable.\n\nWeakness:\n1. The introduced regularizer also introduces many hyperparameters $(\\lambda_1,\\lambda_2,\\lambda_3,\\lambda_4)$, which are not fully discussed in the paper. Does this bring limitations to the regularizer as there are many hyperparameters to attend to? In the experiments, there are ablation studies done on varying values of $\\lambda_3$ and in the Appendix there are ablation studies on $\\lambda_1$. What about the other hyperparameters $(\\lambda_2,\\lambda_4)$? \n\nMinor Issues:\n1. The legend for figure 1 is a bit unclear since the paper earlier defined $\\lambda = (\\lambda_1,\\lambda_2,\\lambda_3,\\lambda_4)$, but the experiment is on varying values of $\\lambda_3$. It could be clearer by changing $\\lambda$ to $\\lambda_3$ in the figure. Similarly in Appendix G figure 2, $\\lambda$ -> $\\lambda_1$.\n2. The following work is closely related to the interest of this paper on NTK analysis. The authors may find it valuable to be included in the discussion of related works: Chen et al. \"A generalized neural network tangent kernel analysis for two-layer neural networks.\" Advances in Neural Information Processing Systems 33 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Kfd6QxTzW23", "reviewer": "Reviewer_nemz", "summary": "The authors investigate very wide 2 layer neural networks and their learning dynamics on a special class of target distributions given by sparse polynomials. It has been established that neural networks in the NTK regime exhibit suboptimal sample complexities on such distributions, while QuadNTK, which incorporates a second order term, can improve said complexity. On the other hand, QuadNTK suffers from worse complexities on dense polynomials, compared to the NTK and is hence also not completely satisfactory. The authors show that combining NTK and QuadNTK together with specially designed regularizers can unite the best aspects of the two worlds, enabling the model to efficiently learn a combination of sparse and dense polynomials. The same cannot be achieved with neither NTK nor QuadNTK in isolation. The regularizers are specifically crafted such that weights only move within certain subspaces associated with big and small eigenvalues but ignoring mid-sized ones. While the large eigenspace allows to preserve the benefits from the NTK regime, the small eigenvalues enable large movement of neurons and thus feature learning.", "questions": "See section above.", "limitations": "As mentioned above, the relevancy of these results beyond the setting of polynomial regression is not clear and not addressed in the main text. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is very well-written and nicely structured, making it easy to read. \n2. The differences between the lazy learning regime and the “actual” regime in which neural networks usually operate in practice are still not well-understood and making progress in this area would be very helpful to the community. The paper definitely takes steps towards this direction and identifies benefits of operating outside of the kernel regime.\n3. The idea of dividing the spectrum into three categories, the large part where the NTK reduces the test error, the middle part responsible for overfitting and the small part where the quadratic term reduces the test error, is very appealing and intuitively makes sense.\n\n**Weaknesses:**\n1. Only investigating a polynomial regression setting seems like a very particular restriction and I wonder how much of these theoretical findings translate to real-world problems such as image classification. Is it a good idea (empirically speaking) to ignore the mid-sized eigenvalues for practical tasks (does such a clear clustering of eigenvalues even emerge for non-uniform inputs over the sphere?) or do those regularizers only make sense in the context of polynomial regression? The authors only evaluate their method on polynomial regression tasks but no other benchmark such as CIFAR10 or MNIST. Again, for these distributions the formulation might not even make sense but this would hence pose a significant limitation. I would appreciate if the authors could comment on this.\n2. Finding particular distributions where the NTK provably performs worse than standard neural networks is a very sensible thing to do. To my understanding, the polynomial regression task has been already identified as being suboptimally learnt by NTK while the QuadNTK can improve over it in sparse settings. This makes the contribution of this work less clear to me. The fact that QuadNTK is suboptimal for dense polynomials is interesting but I am not sure what is the significance of developping another (somewhat artificially-designed) approach that fixes this particular issue, other than enabling better polynomial regression. Especially if point 1. holds and the approach doesn’t easily extend to more realistic setups with empirical improvements, I am not sure what we can learn from these results.\n3. This brings me to my next point/question. The goal (as stated in the introduction) is to understand why NTK might perform suboptimal compared to realistic, finite-width networks. In this work (and the QuadNTK work), it is shown that higher order approximations, together with special regularizers and specific initialization schemes can outperform the NTK in terms of sample complexity. This however somehow does not answer the original question yet, do these new approaches also mimic better what is actually happening in finite networks? This is not entirely obvious to me, given the somewhat unintuitive form of the regularizers, exotic initialization schemes, perturbed SGD instead of SGD etc. For instance, if you were to train finite-width networks on these polynomial tasks, would they be able to efficiently learn both sparse and dense polynomials? Are there similar, bad directions in parameter space where we should not move? Would your regularisers also make sense and lead to performance boosts in this setting? As far as I understood, experiments only concern the newly developed approaches yet no standard setup. Experiments with standard neural networks would really complete the picture in my opinion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept