Explicit loss asymptotics in the gradient descent training of neural networks
Authors: Maksim Velikanov
Dmitry Yarotsky
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep learning theory
neural networks
gradient descent
Neural Tangent Kernel
spectral theory

Abstract: Current theoretical results on optimization trajectories of neural networks trained by gradient descent typically have the form of rigorous but potentially loose bounds on the loss values. In the present work we take a different approach and show that the learning trajectory of a wide network in a lazy training regime can be characterized by an explicit asymptotic at large training times. Specifically, the leading term in the asymptotic expansion of the loss behaves as a power law $L(t) \sim C t^{-\xi}$ with exponent $\xi$ expressed only through the data dimension, the smoothness of the activation function, and the class of function being approximated. Our results are based on spectral analysis of the integral operator representing the linearized evolution of a large network trained on the expected loss. Importantly, the techniques we employ do not require a specific form of the data distribution, for example Gaussian, thus making our findings sufficiently universal.   

Explicit loss asymptotics in the gradient descent training of neural networks
Maksim Velikanov
Dmitry Yarotsky Skolkovo Institute of Science and Technology d. y.yarotsky@skeltech..rr
Skolkovo Institute of Science and Technology maksim. maksim.vel www..kaaoowssooooe.......
Abstract
Current theoretical results on optimization trajectories of neural networks trained by gradient descent typically have the form of rigorous but potentially loose bounds the loss values. In the present work take different approach and show that the learning trajectory of a wide network in a lazy training regime can be characterized by explicit asymptotic at large training times. Specifically, the leading term in the asymptotic expansion of the loss behaves as a power law L(I) Ct- with exponent & expressed only through the data dimension, the smoothness of the activation function, and the class of function being approximated. Our results are based on spectral analysis of the integral operator representing the linearized evolution of a large network trained on the expected loss. Importantly, the techniques we employ do not require a specific form of the data distribution, for example Gaussian, thus making our findings sufficiently universal.
1 Introduction
A major challenge in the research of neural networks the quantitative theoretical description of their optimization by gradient descent. At present, many aspects of network training to be understood rather well on qualitative level, or admit convincing heuristic explanations, but we seem to lack tools for making reasonably accurate quantitative predictions, even for relatively simple models and data. In this sense, the theory of neural networks compares unfavorably to physics, which is also an application-driven field but with an apparently much more successful penetration of theoretical methods. The main difficulty here is probably the complex structure of the data and models, which are hard to describe in terms of convenient and simple mathematical abstractions.
In recent years, a significant progress in the theoretical analysis of gradient descent of neural networks has been associated with the limit of large networks, which be studied using various methods from partial differential equations 22 31
kernel methods (20 21), spin glass theory [2 random
matrix theory 29) dynamical systems
30 and other mathematical fields.
In the present work, we consider setting of large networks and large, smoothly distributed data sets that allows us to obtain explicit leading terms in the long-term evolution of the loss under gradient descent. We are inspired by the spectral theory of singular integral operators [3] which we apply to the linearized evolution of the network. While this linearized evolution has been widely studied recently, most related research seems to focus on theoretical convergence guarantees and upper bounds for the loss values [3 27],
highly symmetric problems admitting explicit solution In contrast, we focus on explicit loss evolution formulas, which we find as power laws
35 38
L(t) C
We argue that the exponents here exhibit some form of universality, in that they are essentially determined by the input dimension d and by the smoothness classes of the activation function and the target function. In particular, we find that in the case of ReLU networks approximating an indicator
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

109
10a
Figenvalues
partial
d=2
10a1
dec
10
10
10"
10"1
10
GP. 6-2
GP. d-2
GP. 6-4
10"
GP. dec
10-3
d-2
d-4
10
10
10
10
10a
102
10a
10-
10
The loss trajectories and spectral properties of the neural tangent kernels of shallow networks in the NTK regime. The target function (i.e., the initial displacement between the network output and the approximated function) either generated by Gaussian process (GP) modeled by larger network of the same architecture, or is an indicator function of a d-dimensional ball (Ind). The data distributions ja modeled as mixtures of 8 Gaussian distributions with random centers, and the data dimension is either d = 2 or d = 4. The solid lines show the numerically obtained values, while the dashed lines show the respective theoretical power-law asymptotics. The dataset size is M = 10a (see Section A (SM) for further details of experiments).
Figure 1:
Left: Loss evolution for shallow network with width N 3000. The scaling exponent giving the slope of the theoretical asymptotic is Ş a d+I for GP and & 1 for Ind (see Section 5.2). Center: Distribution of the infinite network NTK eigenvalues An. The theoretical scaling exponent
is v = 1 + à (see 5.1)). Right: Distributions of the coefficient partial sums (see Eq. 7
The theoretical scaling exponent is k _ 2 for GP and á for Ind (see Section 5.2
function of some region in the d-dimensional space (a classification problem target), the natural
value of the exponent is { ... On the other hand, in the case of target functions generated by randomly initialized wide ReLU network, the exponent is 5 = d+1 Our approach also allows us to obtain explicit expressions for the coefficient
3
these cases.
The power law [I is established using similar power laws (but with different exponents and coef- ficients) for the eigenvalues of the evolution operator and for the coefficients in the expansion of the target function over corresponding cigenvectors. These power laws are indeed confirmed by our experiments (see Figure IT.
Our main scenario is approximation by shallow ReLU network in the NTK regime, but we also briefly consider several modifications of this scenario, namely the activation functions (xr.) with g > approximation by a deep network in the NTK regime, and approximation in the mean field regime.
Related work
The approximation of linearized network evolution and its applications were studied in many works, in particular 21 1O 23 22 [9]. The role of the eigenvalues and eigenvectors of the NTK in the linearized network was emphasized (3). where the GD dynamic of the finite network was linked to the dynamic of its infinite width counterpart, determined by spectral properties of the corresponding NTK. In subsequent works, the NTK spectrum was central for description of network training [27] and generalization 7 These papers the assumption of power law NTK spectrum, but justify empirically or for highly symmetric problems.
Because of the importance of the NTK spectrum, number of works focused on its description in different settings. We first mention directions and settings that are different from ours. The case of very deep networks was studied in (37 18 17 2. This analysis is relying on convergence of the NTK to simple fixed points in the limit of infinite depth. [17] also studied the choice of activation function, in particular its smoothness. Another line of research [I [3] uses techniques from Random Matrix Theory to analyze the setting where the dataset size M goes to infinity together with data dimension and layer widths vig
In this work we consider the setting of fixed data dimension and effectively infinite network width and dataset size. In this case (35] showed that the network evolution can be described by a deterministic

---

integral operator that is easier to analyze than a large but finite matrix. Also, in that paper and in the papers 38 27. the integral operator was explicitly diagonalized in the special case of uniform distribution a sphere.
A power law upper bound for the training loss was obtained in 277. but with exponent & smaller than ours, The paper [4] describes a different power law, relating the test loss at the end of training to the dataset size and the network width. This result also relies on power law assumption for the NTK spectrum.
Another related line of research is the case of univariate functions. The gradient descent evolution of univariate shallow networks has been analytically studied in the papers [39 36].
3 Asymptotic evolution of the loss function
We consider linearized training of neural network by gradient descent. Such linear approximations arise naturally in various "lazy training" scenarios [TO]. Consider the standard quadratic loss function L(W) _/2 JI [f(W, ( function, f is the network, W are the network weights, and /4 is the data distribution on which the network trained. Gradient descent can be written as the differential equation d W = - LWL(W)
where x is the d-dimensional input, f is the approximated
zt
We assume that the weight vector W is close to global minimum W. where L(W.) x) 0 and
f(W., N x) (x) p-a.e., and that the evolution equation can be linearized at W W., .. It is convenient to write this linearized equation in terms of the difference 8f(x) = j(W, x) s(x) between the current output and the target. The corresponding linear equation is d 55 = -A5f,
where A can be written as the integral operator A = =gga x(x, (x))) ddx' with the NTK (neural tangent kernel)
0 x²) = VV
(2)
The evolution operator A is a symmetric non-negative definite operator with respect to the sealar product (f1.f2), = J fa fx (x))xxxxx
By multiplying funetions f by p1/2, the operator A can be brought to the form A = μ 12 AA 11/2 with symmetric kernel,
1/2
í = / n (x))(x (xx
In this form À is symmetric w.r.t. the usual scalar product (21, (x( Observe that the loss at time t can be written as L(t) 2 ||gr where = c ctAgg the norm corresponds to the scalar product (., +), and the function g is given by
g(x) μ (x)(()) 0), f(x)).
(4)
We can now describe the evolution of the loss by diagonalizing the operator A. We will be interested in the scenario where is smooth function that is compactly supported or falls off at infinity sufficiently fast. (In particular, in the context of a finite training set this means that this set is large enough to be legitimately approximated by μ.) In this case, for typical kernels O the operator Á will have a discrete spectrum with eigenvalues converging to 0. Let An denote the eigenvalues of A sorted in decreasing order, and let e, be the respective coefficients in the expansion of the initial error g (given by Eq. 4 over the normalized orthogonal eigenvectors. Then, the loss evolves by
I(t)= = Σ
22n |nn 2
(5)
To compute the asymptotic of L(t) at large times I, we need to know the distribution of the eigenvalues An and the coefficients en at large n. The key assumption of our work (verified later for certain scenarios) is that these distributions have power law forms. Specifically, regarding the eigenvalues An we assume that
An ~ An
(6)
3

---

with some coefficient A and exponent Regarding the coefficients we assume that they also have a power law distribution on a large length scale in n, but possibly deviate from this law locally (e.g,, due to oscillations). For this reason, we describe their large behavior by the partial sums
|ca|2
8n
We then assume that, for some coefficient K and exponent K,
Kn
(8)
Under assumption of the power laws (6) and (). it is easy to check (see SM, Section B that the loss also has a power law asymptotic constants A, K and exponents V, K.
(I) with the constant and exponent & expressible through the
L(t) K 2 K + 1) (2At)
where r(z) is the Gamma function.
In Figure D we illustrate this approach to the long-term loss evolution with several examples of target functions having different smoothness and dimension and. as a result, exhibiting different exponents.
In the remainder of the paper we show that the power laws (6) and (8), and hence the large-l asymptotic (9) of the loss, are indeed valid for some natural network training scenarios. The asymptotic (6)
of the eigenvalues is primarily determined by the singularities of the kernel e. These singularities
can be explicitly described for shallow neural network with piecewise smooth activations such as ReLU. Then, the power law (6) can be derived from general results on integral operators with singular kernels. In particular, in the case of ReLU we find that
The asymptotic (8) of the coefficients is more subtle, as it depends significantly on the class of the initial error function g (which in turn depends, by Eq. on the target function f and the initial approximation f). We derive this power law for one natural class of discontinuous functions g. and for functions that are realizations of Gaussian process of a particular "roughness".
Moreover, for these two classes, also find explicit form of the coefficient C appearing in loss asymptotic (I). This requires us, however, to modify the above derivation of the loss asymptotic (9) by what can be called "integrated localization". Roughly speaking, in the large-/ limit we can think of the eigenvectors of A infinitesimally localized Fd We then apply the above derivation of Eq. (9) not to the full set of eigenvectors, but separately to each infinitesimal sub-domain, and then integrate the results. See details in Sections 5.2 and (SM).
The subsequent exposition is structured as follows. In Section 4 we provide the background on the NTK kernel and spectral properties of singular integral operators. Then, in Section 5 we derive the loss asymptotic (1 for our main setting - the NTK training with a shallow ReL.U network. After that, in Section 6 we consider various modifications of this setting: other activation functions (Section deep networks (Section 6.2). and training in the mean field regime (Section 6.3
6.1
4 Background
4.1 Infinitely wide networks
Lazy training scenarios discussed in section 3 naturally arise for the networks in the limit of infinite width. However, there are several ways to scale parameters of the network with width, which lead to different operating regimes of infinitely wide networks [6].
NTK regime. The first option we consider is the NTK regime 200,, for which the NTK & defined in is deterministic and constant during training. This simplification immediately leads to linear dynamic in the space of network outputs. We consider feed-forward networks parametrized as
z Di= amuttaa + obbY
2 Li=1 vmi-T wl - orbly 1>1
ou
(10)
x o(2))

---

Here n' is the width of layer z; is network input and 2j is the network output. We also consider the last layer without bias term by and having width nz 1 (scalar output). Trainable parameters whys V/ are initialized as i.i.d. normal Gaussians.
The output of each layer at initialisation is a Gaussian process with covariance (z)(x) (x)) = E53 E (x, x'). By introducing the NTK's 6 (x(xx')) (x,x') of intermediate layers, one can recursively compute 21 the both NTK and covariance
{
-(++) = (0==)(()())))))))))))))))))))( to 0(++))= = E(+ o 0 (
(11)
Here 2² = (x) and zd (x') are draws from the Gaussian process with covariance E [x, and (...) is the averaging. It is also convenient to parametrize the covariance at input points x, x' in some layer 1 as
(x,x').
> (x,')
rî riri cos ypt
rerj cos 9I r?
(12)
To analyze the spectrum of the evolution operator À given by (3) we will intensively use the explicit
form of the NTK and covariance. For the ReLU activation ((z)= = (2)+ the averages in (TT) can be
computed analytically [[]] In the case of shallow network (L 2) the result is:
Σ(x, x')
o2 2 (sin yp + cos (p(r -p)
(13)
0(x, x') = -x(x x') + 2म
r'' cos pp .p)
(14)
Here r', are parameters from [22 with = 1 (we dropped index 1). They have a clear interpreta- tion in terms of extended input vectors x = (a.... x, σb) E RR++1 Specifically, r = ||||], r' = =||x|| and yp is the angle between x and x'.
Mean Field regime. This operating regime of infinitely wide networks naturally defined [26 for shallow networks of the form f(W, x) = N Σi= co((w; x b,) ト E (o(w, x). Here w; (c,, w., b.) denotes the collection of parameters, associated with single neuron. In the infinite width limit N - oo the evolution is described by a PDE on parameter density distribution p(w):
- 0.p = V w V / K(W, w/ (p(w)))) - Poo (w") Poooooo
(w")) K(w, w/) - - ā( x))((x) x)dx
Under mild assumptions [9], solution of the MF equation converges to the global optimum (w"). At large times t the MF equation can be linearized (39] around Poo. thus bringing the network to the lazy training regime discussed in section 3 The network NTK in this case is equal to
O x)) = (wo(w,x)pes V Vww(ww x'ddw.
(15)
4.2 Singular integral operators
Consider the evolution operator À given by Eq. Under our assumptions, the kernel xx/²(x)((xxxx)) x22xx of this operator quickly falls off infinity and is smooth outside the diagonal x = x'. but, as we will see later, has homogeneous singularity on this diagonal. In this setting, general theory developed in (5) allows us to obtain the asymptotic distribution (6 of the eigenvalues with explicit constants A and V.
Specifically, suppose that in a neighborhood of the diagonal the kernel θ has representation
O(x, (x x') + ....
(16)
where (-) is a (possibly x-dependent) even (0x(z) (-z)) homogeneous function of degree
(x(z)) = Ox)),
(17)
and the dots denote terms of higher smoothness.
5

---

Let NA denote the number of eigenvalues of À greater than A. Then, it is shown in [ that for small X, the leading term of N₂ is given by
Nx ( muudta (xx)dx)
d d+a.
(18)
Here, 7x = (2T)-4(( E IRd (k) > 1)/, where |- denotes the Lebesgue measure and 0x is suitably defined Fourier transform of the homogeneous function 0x (see SM, Section (
Formula (18) can be derived as follows. Divide the domain R into multiple small subsets 9.m and think of A as an operator matrix corresponding to the decomposition L (R()) = 3m LLmLL0 (92m). Using the fall off and smoothness of the kernel outside the diagonal, one can show that the leading term of NA is determined only by the diagonal elements of this operator matrix. Then, the leading term be found by considering each restriction A Al..( separately and summing the respective contributions to Nx, i.e., (A) ~ Em (A(A) If we decrease the size of each Qm by factor M, then the number of terms in this sum increases -old, but at the same time each (A (=..). decreases also roughly Mo-fold, due to rescaling of eigenvalues. In the limit of infinitely small subsets 9m: the operator (l.) can be approximated by the convolution with the homogeneous function xx. The asymptotic form of its eigenvalues can then be written in terms of the d in Eq. (18) can be deduced by observing that Fourier transform 8x given above. The power d+a
the volume of the k-space corresponding to fx À scales as ata.
The formula 188 can be translated into the power law (6) by inverting the relation between À and n (note that N - for any n)). Specifically, we find that the law (6) holds with
a d' A= "=(/xut""
V = ==++
(x)
(19)
In Section 5.2 we will show that this approach can be extended to yield the loss asymptotic (T).
Asymptotic analysis of wide networks
5.1 NTK operators and their singularities
In this section we demonstrate the asymptotic law (6 of the NTK spectrum well as outputs covariance spectrum, and find constants v, A. We consider a shallow ReLU network in the NTK regime and with the data distribution x(x) as described in Section 3 The NTK ( x') and outputs
covariance Σ(x,x² x') of such network is given by (14). (13) with r' and & explicitly depending on input points x, x':
rxx) = 10, of (xxx, x') = arccos (Xxx'-03
((x)r(xx))
(20)
To use the spectral theory described in section 4.2 we analyze the smoothness of the kernels and Σ. Firstly, they are smooth (infinitely differentiable) functions of r, r,r', ip on the whole domain. Now suppose that the bias term is not absent: σ, > O. Then r(x) is a smooth function for all x E Rd The argument of arccos in (20) is also smooth everywhere, but arccos(2) 22) itself is smooth on (-1, and has divergent derivative at the end points 2 = 1, -1 corresponding to o = 0, .. We see that condition is never realized, while sX(x, x') = 0 at all coinciding inputs. Thus we established that ((x, x') and E(x, x') are smooth everywhere except the diagonal x =xx where they might have singularity.
σb > 0 implies that the case yp
To analyze the behavior at the diagonal, we first expand y in Eq. (00 for small ox x x':
po(x x') = (x)) -2 x|2 cos? of
5x + ((xx²)
(21)
(xx)
Here v is the angle between òx and x. If ob O, then the expression under the square root is always
positive, therefore the angle (x(x, x') has a homogeneous singularity of degree I on the diagonal.
Now we expand expressions (3) and (14) for small yp and find that both NTK and covariance indeed have a singularity on the diagonal with leading singular terms.
Exing x')
o² -²x² 3 v0t (x, x') x |
(x)
(22)
2nt
Osing (x, (xxx)= -xxx(x))xxxxx x [5x]
(23)

---

dm3
10a
10a
10-3
2000
10
1050
1000
10
10--
10
10"7-
10a
10'1
10
10
10-10
10' 10-7 10 10 10 30 10a
10
30 10a
Figure 2: NTK cigenvalues An for networks with different input dimension and for different data set sizes M. Asym shows the theoretical power law (6) with parameters A, derived in Section 5.1 in particular, = + d: Singular corresponds to eigenvalues of the singular part of NTK 23 We that for moderately big the singular part has the same eigenvalues the full NTK. Observe that the number n at which the spectrum converges to its asymptotic form increases with dimension d.
Now we see that both NTK θ and covariance Σ have the form needed for application of spectral theory discussed section 4.2 For NTK we have singularity degree = 1, which by Eq. (9) leads to the already announced exponent the covariance higher (namely, 3), resulting in a faster fall off the corresponding cigenvalues. In the sequel (see Section 5.2). this latter degree will appear in the analysis of loss asymptotic for target functions generated by the neural network Gaussian process, and will be denoted by (3.
1+ 1 (see Figure I}. Interestingly, the singularity degree for
In the case of NTK ( x') the singularity is essentially given by 21 The corresponding Fourier
transform 0x and the function " be calculated analytically (see Section C). This leads to explicit expression for the constant A:
2 I + +1)++++0000 -1+()))) ())x)))(x(x(x)--))))))))))))>))) कोx (r(x)) 5नं 1+
(24)
where (u(x)), denotes the integral f [xx))xxx)dx. In the experiment, the value (u(x)), can be computed by averaging u over a y--distributed data set. In Figure we compare theoretical and numerical NTK eigenvalue distributions for several dimensions d and data set sizes M.
5.2 The loss function
We extend now the arguments of Section 4.2 derive the loss asymptotic TT We consider two classes of functions g representing the initial error
Scenario 1:
discontinuous function g- We assume that g is supported a bounded subset C dRdd with smooth boundary an so that has a discontinuity on this boundary but is smooth inside Q. An obvious example of g the indicator function of Q2, In this scenario we obtain
/ 80
L(t)
ziads + r + +1) (2t) aंa.
(25)
Here, x is the point on the surface 82, n is the unit normal to An, and Ag(x) is the size of the jump of g at x, given by the limit of g(y) as y approaches x from inside Q. The eigenvalue, coefficient, and loss exponents in this scenario are, respectively,
K = v=1+3 a. f d+a
(26)
Suppose that g is a realization of Gaussian process with a covariance matrix Σ(x, x') (x(x)g(x' }} and that Σ has homogeneous singularity Ex of degree 3 at the diagonal (in the sense of Eqs. (66.. 17)) this scenario we find
Scenario 2: generated by a Gaussian process.
(x')
[ / |n/=1
I(t)
((I)
P 0 dxdS 2g (ata + 1) (ff)-
(27)
Here, &x is the Fourier transform Sx (defined similarly to the Fourier transform 0x). The eigenvalue, coefficient, and loss exponents in this scenario are, respectively,
8 v 1
3 3. { = a+a
(28)
d
7

---

In our experiments we model GP by large network in the NTK regime. The corresponding covariance is analyzed Section 5.1 and has the singularity degree 3 =
We provide the full derivations of Eqs. 25 and (27) in Section of SM, and sketch now the main of ideas. Our general strategy is to complement the localized eigenvalue analysis of Section 4.2 by the
analysis of expansion coefficients. Note, however, that the simple approximation of Ā by direct sum of localized operators A A (as performed in [5] and sketched in Section 4.2 is "too rough" for the study of expansion coefficients (due to a stronger effect of boundary conditions in each Qm)) Accordingly, we replace this approximation by the short-time Fourier transform of g:
Fly, k) = (2x)- / g(x)w(( y)e ik- ik-xxx,
d/2
where w is window function such that f w² 1. The coefficient Fy, k) describes the component of having the wave number k and localized at Then at large i, using the stationary phase method,
gs (x) = e e- ~ (2x)- # 1 / Fly, k)e (10
dydkkddydk.
The leading contribution to this integral comes from large k. For such k, we can write the coefficients Fly, k) in an asymptotic form, primarily determined by the singularities of g. By integrating out y and k, we then arrive at the desired Eqs. (25) and (27).
The above argument establishes the loss asymptotic (1) while bypassing the computation of the asymptotic (8) of the expansion coefficients aligned with the sorted "global" eigenvalues. This latter asymptotic (including the coefficient K) can be found by similar computation, Section D.4
Extensions
The power law asymptotic of eigenvalues obtained in section 5.1 was based on the analysis of diagonal singularity of the NTK in the setting of ReI.U activation, shallow depth 2, and the NTK regime. We argue now that our general approach is not restricted to this narrow setting. To show this, separately consider several modifications of the network from Section [. and application to MNIST (see Figure 3). In this section we mostly limit ourselves to describing final results, with the derivations provided in SM, Section D
6.1 Activations different smoothness
Let's consider a shallow network in NTK regime with activation function (z(2) = (z) q 0 ("a ReLU with the altered smoothness g²). Similarly to the ReLU case, one can show that NTK in current setting has singularity on the diagonal for all values of y except half-integers The leading singular term is
3
o (x x') a " 20-1
(29)
2x (t) 9)
0g V /
(30)
Here r', sp are the same as in section
The singularity of NTK with degree
2q implies the eigenvalue power law asymptotic (6) d Thus, the singularity degree in the NTK is determined by the
with the exponent singularity degree of the activation function. The coefficient A, can be explicitly computed as
29-1
I( D(d+a) (
Ag = o 20, (2x)) 2
dim (u(x)==²2 a r(x)
28 2d-ad-a d+a
(31)
(T + 1))
In Figure 3a we compare the theoretical and numerical eigenvalue distributions for several values of d and
8

---

d-1
Loss
10
109
10
Pg
65pth
10"
e4p Sowerdh, 0..0
10
10"7
10"
Qw0.5
10
Figenvalues A,
10"1
10P
exp
10.6
10
10"
power ft,
10:
10a
d
10
Elgenvalues
Eigenvvaluss
10
10
g:0.5
10
gm0.77
10'
107
10:2
Q-1.5
100
10
sums Se
10-2
10-4
10
10
10:3
10
10
10-7
10"4
10-4
10
102
(a) Activations (=)
(b) Deep networks
(c) Mcan field
(d) MNIST
Figure 3: Extensions other activations, deep networks, mean field regime, and MNIST.
(a): NTK eigenvalues for activation functions <(z) = (=) with varying smoothness The
theoretical distributions (dashed lines) have the exponents v = 1 2q-1 (see Section 6.
(b): Loss evolution and NTK eigenvalue distribution for networks with varying number of hidden layers L trained 3-dimensional data and target generated by network GP (3 3). In agreement with theory, the numerical results (solid lines) show the scaling exponents Ç = d+1 and 1
for all depths L (see Section 6.2). The theoretical predictions (dashed lines) are only shown for L = (for other L the theoretical lines would have the same slopes but different positions determined by the coefficients C' and A; due to computation complexity, we have found C and only for = 1). (c): Loss dynamic and eigenvalue distribution at different moments of time for network in the MF regime (Section 6.3 ). The network approximates GP on inputs with dimension d 4. Top: comparison of the theoretical and experimental loss evolution. The theoretical exponent = 2+1 = 3
Bottom: the theoretical eigenvalue distribution and its experimental counterparts at different times. The theoretical exponent v = d+1 5
(d): Loss evolution, NTK eigenvalues X, and coefficients for MNIST. All three curves are reasonably well approximated by power laws with exponents Ş = 0.26, 1.35, x R 0.33 so that & Rs in agreement with our theory. At the same time, our current theory does not predict specific values for the exponents and x because the intrinsic dimension the MNIST manifold is very different from the large dimension of the ambient space. See SM, Section B for a detailed discussion.
6.2 Deep networks
We consider now a network of arbitrary depth the NTK regime and with the ReLU activation function. Similarly to the shallow ReLU case, the angles ip: (see Eq. (12) are singular on the diagonal: P x xx x/). Using relations
(TT) one can obtain recursive relations for yPr, Ti and finally for the
singular part of NTK O
@ (1+1) diag
2
+
(I) dlag
2
(32)
(I+1)
1
(1)
02
sing
2TT
diag
2
sing
Here ((++))
diag is the value of NTK on the diagonal. Since (1) O. we can see from 332 that +
(L)
sing
sing
a weighted sum -sp for 1....
Thus, the singularity degree of the NTK a
and the eigenvalue power law
( holds with 1 + 7. However, obtaining explicit formula for A
harder for this case and we leave for future work. in Figure 3h we compare the theoretical and
numerical cigenvalue distributions for NTK's 's of deep networks.

---

6.3 MF regime
In this section we consider a shallow network in the MF regime and with the ReLU activation function, with the NTK given by (15). The neural tangent kernel (14) of the shallow NTK network can be obtained from (5) with the distribution Pao (c, w, b) taken as a product of Gaussians for each variable. Therefore, neural tangent kernels in MF regime represent a broader class of kernels, containing our basic example (14). turns out that the diagonal singularity present for all sufficiently smooth and quickly decaying distributions Poo of this broader class:
X
/ w-X=-b
sing (x, x')
[ 5x 22
2
Here ōx X x' and 2 the second moment of the distribution Poo w.r.t. the variable c, i.e. 2 w.b) b) c' (c. w, b)dc. Thus, the eigenvalue power law exponent is the same in the NTK regime: = 1 2.
In Figure 3e we show the loss dynamic and the eigenvalue distribution at different moments of time for a network in the MF regime. Since the neural tangent kernel significantly changes during the training of the MF network, picking NTK's at different moments of training provides sufficiently distinct and general kernels of the form (15
7
Discussion
The main results of the paper are theoretical derivations of power laws for the NTK spectrum and the loss asymptotic. Spectral properties of neural networks is an active area of current research
It has been observed (e.g.. [32 33 that, close to a global minimum, the Hessian of the loss of neural networks in practical tasks typically has eigenvalues accumulating near O. This implies, particular, that the Hessian is ill-conditioned, which has profound implications for network training: convergence of gradient descent in this case is much slower than for well-conditioned optimization problems. power law spectral distribution assumed in some recent works (e.g.. for the study of generalization in (6), but we are not aware of any previous general theoretical derivations such a law. Our work essentially provides such a derivation (since at the global minimum the NTK spectrum coincides with the Hessian spectrum up to the eigenvalue 0). In sense, we prove theoretically (under some reasonable assumptions) that slow (power-law) convergence of the gradient descent is inevitable. Moreover, precisely quantify this convergence.
24 34 25 28
Our work provides simple explicit formulas for the asymptotic eigenvalue distribution and loss evolution. This allows, in principle, to make some quantitative predictions about optimization not feasible with theoretical methods only providing upper bounds. For example, the loss exponent & allows to directly estimate the number of GD steps needed to decrease the loss by one order of magnitude. principle, the full loss asymptotic (25) can be used to analyze the effect of different data distributions μ(z) on convergence speed of GD.
The key idea of our asymptotic analysis is the extraction of the diagonal singularity from the NTK kernel. This is new and promising way to relatively easily obtain information about the model spectrum: it only requires simple Taylor expansion and, as we show, applicable in wide range of scenarios, extending to deep networks, the mean field regime, and different activation functions. We expect this operation to be important for future studies of network training.
Our loss exponents { a+d and = and tend to 0 as the dimension d increases. However, this does not contradict the practical trainability of networks for nigh-dimensional tasks such image recognition, since real images occupy only tiny (i.c., effectively oow-dieensional) subsets in the ambient high-dimensional space. While our current theory does not predict specific exponents in this setting, we do empirically observe power laws for eigenvalues, coefficients and the loss, and the fitted exponents satisfy our theoretical relation a (see Figure 3d). This suggests that method can be extended to this setting, which will be the topic of a future work.
3
Funding transparency statement
Research supported by Russian Science Foundation, grant 21-11 1003733
10

---

References
[1] Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. In Hal Daumé III and Aarti Singh, edi- tors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 74 84. PMLR, 13 18 Jul 2020.
[2] Naman Agarwal, Pranjal Awasthi, and Satyen Kale. A deep conditioning treatment of neural networks. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato, editors, Proceedings of the 32nd International Conference Algorithmic Learning Theory, volume 132 of Proceedings of Machine Learning Research, pages 249 249-305. PMLR, 16-19 Mar 2021.
[3] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- sine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 322-332. PMLR, 09-15 Jun 2019.
[4] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021.
[5] M Š Birman and M Z Solomjak. Asymptotic behavior of the spectrum of weakly polar integral operators. Mathematics of the USSR-IZvestiya, 4(5): 10::1111-11 151-1168, oct 1970.
[6] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1034. PMLR, 2020.
[7] Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. arXiv preprint arXiv:2006. 13198, 2021.
[8] Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint (606.05660.05333 2020.
[9] Lénaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 31. Curran Associates, Inc., 2018.
[10] Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. Advances in Neural Information Processing Systems, 32:2937-2947, 2019.
[11] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Y. Bengio, D. Schu- urmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 22, pages 342-350. Curran Associates, Inc., 2009.
[12] Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. Artificcal intelligence and statistics, pages 192-204. PMLR, 2015.
[13] Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7710-77211 Curran Associates, Inc., 2020.
[14) Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232-2241. PMLR, 2019.
[15] Eugene Golikov. Towards a general theory of infinite-width limits of neural classifiers. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3617-3626. PMLR, 13-18 Jul 2020.

---

[16] Eugene A. Golikov. Dynamically stable infinitewwidth limits of neural classifiers. arXiv preprint arXiv:2006.06574, 2020.
[17] Soufiane Hayou, Arnad Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 2672-2680. PMLR, 09-15 Jun 2019.
[18] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau, Mean-field behaviour of neural tangent kernel for deep neural networks. arXiv preprint arXiv: 1654,, 2021.
[19] Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 171 17116-17128. Curran Associates, Inc., 2020.
[20] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[21] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[22] Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218. 2020.
[23] Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15954-15964. Curran Associates, Inc., 2020.
[24] Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning, pages 4284-4293, PMLR, 2019.
[25] Charles H Martin, Tongsu Serena Peng. and Michael W Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1):1-13, 2021.
[26] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665- E7671,, 2018.
[27] Atsushi Nitanda and Taiji Suzuki. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime. International Conference on Learning Representations, 2021.
[28] Courtney Paquette, Kiwon Lee, Fabian Pedregosa, and Elliot Paquette. Sgd in the large: Average-case analysis, asymptotics, and stepsize criticality, arXiv preprint 1xXiv:2102.04390 2021.
[29] Jeffrey Pennington and Yasaman Bahri. Geometry of neural network loss surfaces via random matrix theory. In International Conference on Machine Learning, pages 2798-2806. PMLR, 2017.
[30] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. arXiv preprint 0rX1v:1606.05340.1 2016.
12

---

[31] Grant M Rotskoff and Eric Vanden-Eijnden Trainability and accuracy of neural networks: An interacting particle system approach. arXiv preprint ftth:::8000009/55 2018.
[32] Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian deep learning: Singularity and beyond. arXiv preprini urXiv: 2016.
[33] Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv: 04454, 2017.
[34] Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning. pages 5827-5837. PMLR, 2019.
[35] Lili Su and Pengkun Yang. On learning wer-ppaameeterized neural networks: A functional ap- proximation perspective. In H. Wallach, Larochelle, Beygelzimer, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[36] Francis Williams, Matthew Trager, Claudio Silva, Daniele Panozzo, Denis Zorin, and Joan Bruna. Gradient dynamics of shallow univariate relu networks. Advances in Neural Information Processing Systems, 32, 2019.
[37] Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and generalization in deep neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10462-10472. PMLR, 13-18 Jul 2020.
[38] Greg Yang and Hadi Salman. fine-grained spectral perspective on neural networks. In Eighth International Conference on Learning Representutions (ICLR 2020), September 2019. Submitted to ICLR 2020.
[39] Dmitry Yarotsky. Collective evolution of weights in wide neural networks. arXiv preprint arXhv.1810003974.0 2018.
13


Reviews:
[{"review_id": "t0c1Xp2UJK", "reviewer": "Reviewer_xC4H", "summary": "The paper identifies a polynomial decay rate in the loss function during gradient descent under the condition that both the eigenvalues of the semigroup operator, and the coefficients in the eigenfunction expansion of an initial loss (g), exhibit power law decay. These assumptions are verified for the semigroup operator in the infinitely wide (NTK) regime, a mean field regime, and also, interestingly, under a homogeneity assumption in the transition kernel. Two scenarios for g are also considered. Throughout, the authors have taken care to be precise with coefficients and compute quantities exactly. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm fond of what this paper manages to achieve. There is a great attention to detail from a mathematical point of view. The exposition is clear, assumptions well-specified, and a number of scenarios are covered. The recent push to consider spectral distributions in theoretical machine learning is an exciting one, and I think this paper is a prime example of what can be accomplished. An enjoyable read for sure. \n\nHowever, much of the presentation seems too matter of fact for this venue, with little discussion surrounding consequences and interpretations. A concluding paragraph to interpret the findings more generally would be ideal. I recognize that this is hampered by space constraints. To cut down on space, there are a number of paragraphs that should probably belong in supplementary material instead. For this particular venue, lines 123-127 and 176-187 (for example) bog down the exposition. Otherwise, making more equations inline and cutting down on some unnecessary paragraph breaks could help with this.\n\nAs a side comment, there is a particular line of research establishing an empirical connection between tail exponents in power laws and generalization performance [1-3]. There has been a subsequent push to explain these connections theoretically. I believe the presented work could be an excellent contribution to this literature in the following sense: the belief to date is that decreasing the exponent nu in (6) should improve performance. This coincides well with the conclusion in (9). Furthermore, the relationship between nu, xi, alpha, and d, all seem to coincide well with empirical results. I recognize that the eigenspectrum of the operator A is not quite the same as the heavy-tailed distributions considered in the present literature, but I would expect there to be a relationship, due to the presence of the Jacobian in the NTK (2) and therefore in (3). Besides, much of the \"heavy-tailed\" literature is ad hoc about where the tail exponent comes from, since it is believed that many of the power laws encountered are related. As far as rate of convergence results are concerned, I also recall a similar strategy of using spectral distributions in [4] --- not sure if this is also worth mentioning. \n\nWith some of these changes, I believe this could be an outstanding paper --- at least a 7 or 8. \n\n[1] Mahoney, M., & Martin, C. (2019). Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning (pp. 4284-4293). PMLR.\n\n[2] Simsekli, U., Sagun, L., & Gurbuzbalaban, M. (2019). A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning (pp. 5827-5837). PMLR.\n\n[3] Martin, C. H., Peng, T. S., & Mahoney, M. W. (2021). Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1), 1-13.\n\n[4] Paquette, C., Lee, K., Pedregosa, F., & Paquette, E. (2021). SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality. arXiv preprint arXiv:2102.04396.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work presented is highly theoretical in nature. The limitations are predominantly to do with its assumptions, which are all clearly stated. Nevertheless, a concluding paragraph describing potential future improvements would be good to see.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "gGIPjL97Kc", "reviewer": "Reviewer_qRxk", "summary": "The paper studies the scaling law in the training time of neural networks in the NTK regime.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem is novel and the scaling relations are very important both theoretically and practically. While the results in this work are interesting, I feel that the work could have spent some paragraphs discussing the practical uses of knowing the scaling law in training time.\n\nAlso, I am not too sure if the title is appropriate. The title reads: \"Universal scaling laws in the gradient descent training of neural networks.\" However, the paper only studies the special form of scaling of the training loss with respect to the training time, but not all possible scaling laws. Therefore, I feel that the title needs to be changed to be more specific and avoid overclaiming. \n\nI will consider raising the score if the authors discuss the practical side of the work in the rebuttal and propose a more appropriate title (or convince me why the present title is indeed precise and accurate). \n\nSome other problems:\n1. Figure 2 lacks axes labels.\n2. line 158 \"asisciate\"\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Sufficiently addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hour"}, "extra_text": {}}, {"review_id": "N_jemh-gHXp", "reviewer": "Reviewer_ZGjL", "summary": "The paper studies the scaling law relation between $t$ (time) and the population loss in the NTK regime. The authors show that the scaling law is determined by two indices, (1) the index of the power law decay of the spectrum of the NTK (2) the index of the power decay of the residual of the loss on the eigenfunctions of the NTK.  The main technical contribution is the computation of the two indices. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Scaling law behaviors of neural networks have attracted a lot of recent attention recently. The goal of this paper is study the scaling law under the NTK regime where the where the distribution of the learning task is given, in another words, the distribution of the input and the target are given and scaling law is mainly about the dependence of the population loss on time $t$. In this setting, population loss could be decompose into independent eigen-mode in the sense the loss is \n$$\nloss(t) = \\sum_{n} e^{-2 \\lambda_n t} g_n^2\n$$ where $\\lambda_n$ are the eigenvalues of the NTK and $g_n$ are the projections of the residual.  As such, the scaling law is determined by the index ($\\nu$ in the paper) of the power law decay (assumed in the paper) of $\\{\\lambda_n\\}$ and the index ($\\beta$ in the paper) of the power law decay of the residual $\\sum_{n\\geq m} \\|g_m|^2$. The paper then proceeds to compute these two indices under various of settings, which are the most technical part of the paper and which are also a very classical topic in mathematical analysis. \n\nOverall, I think this is an interesting paper that draws a connection between scaling laws of neural network and oscillatory integrals although the analysis is limited to the NTK regime with popular loss. \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work focuses on the NTK regime where the kernel is fixed, in which there is a nice and simple eigen-decomposition of the residual and does not cover the more realistic setting (mean field) where the kernel and the eigenvalues/eigenvectors are all evolving. Clearly, scaling law behavior under the latter setting is a very challenging open question in the field. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "LX9f4ruWNEx", "reviewer": "Reviewer_ZAzG", "summary": "This paper derives asymptotic scaling laws that characterize the rate of decay of the loss for neural networks trained by gradient descent in the NTK regime. Interestingly, the exponent of the power law describing this decay depends only on a few natural parameters (properties of the target function, dimensional of the data, smoothness of the nonlinearity).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Many of the current guarantees for training are difficult to operationalize because they are weak upper bounds on the evolution of the loss. The authors here take a distinct approach that focuses instead on networks in the NTK regime which are formally infinite width. In essence, the approach can be outlined as follows: initially one assumes that the parameters are close to optimal (always the case in the infinite width limit) and linearize the dynamics. If the data density is smooth and compactly supported then the spectrum of the integral operator $\\tilde{\\mathcal{A}}$ that acts on the residual will have a well-behaved spectrum. In some cases, the spectrum of this operator has a power law form, leading to an evolution equation for the loss that decays like $t^{-\\xi}$.\n\nI found the exposition throughout, especially in Sec. 3, to be quite clear and precise. Appendix D gives a very complete accounting of the different function classes considered throughout. \n\nThe numerical results seem consistent with but not conclusive for the picture the authors advocate. \n\nSec. 6.3 did not provide enough context to argue for the power law.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The most obvious limitation in this work is the NTK assumption. The authors are well aware of this fact and have some suggestive results in the mean-field regime near the end of the paper. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four reviewers recommend this paper for acceptance. One reviewer indicated that s/he is not as concerned with the NTK assumption and that this work could initiate some interesting follow-up studies on the benefit of certain power law scalings, and raised her/his initial score. Another reviewer raised her/his score as the authors sufficiently addressed his/her questions and proposed convincing improvements. Another reviewer concludes that the primary limitation of this work is the NTK assumption, but that the that the clarity of the submission is high compared with other submissions, and that the reviewers agree that the results are sufficiently interesting and could initiate a number of different inquiries both theoretical and numerical. In summary, the general consensus is for acceptance. I agree with this view and hence I am recommending this submission for acceptance. The reviewers made several suggestions and several improvements were proposed in the authors’ responses. I request the authors take these carefully into consideration when preparing the final manuscript. 
Decision: Accept (Poster)