The asymptotic spectrum of the Hessian of DNN throughout training
Authors: Arthur Jacot
Franck Gabriel
Clement Hongler
Venue:  (2020)
Keywords: theory of deep learning
loss surface
training
fisher information matrix

Abstract: The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs: we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. 

Published as a conference paper at ICLR 2020
THE ASYMPTOTIC SPECTRUM OF THE HESSIAN OF DNN THROUGHOUT TRAINING
Arthur Jacot, Franck Gabriel & Clément Hongler Chair of Statistical Field Theory
Ecole Polytechnique Fédérale de Lausanne (arthur. jacot, franck. grabriel, clement hongler @ @epfl.ccfc..
ABSTRACT
The dynamics of DNNs during gradient descent described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian the cost of DNNs. When the NTK is fixed during training, we obtain full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mcan-field limit, where the NTK is not fixed during training, we describe the first two moments the Hessian at initialization.
INTRODUCTION
The advent of deep learning has sparked a of interest in the loss surface of deep neural nelworks (DNN), and in parlicular its Hessian. However to our knowledge, there is still no theoretical description the spectrum the Hessian. Nevertheless a number phenomena have been observed numerically.
The loss surface of neural networks has been compared to the energy landscape different physical models [Choromanska et al. 2015 Geiger et al. 2018 Mei et al. 2018). It appears that the loss surface of DNNs may change significantly depending on the width of the network (the number of neurons in the hidden layer). motivaling the distinction between the under-
and over-parametrized regimes (Baity-Jesi et al. 2018 Geiger el al. 2018 2019).
The non-convexity of the loss function implies the existence of very large number of saddle points, which could slow down training. In particular, in (Pascanu et al. 2014 Dauphin
et al. 2014). relation between the rank of saddle points (the number of negative eigenvalues of the Hessian) and their loss has been observed.
For overparametrized DNNs, possibly more important phenomenon is the large number of flat directions JBaity-Jesi et al.) 2018 The existence of these flat minima is conjectured to be related to the generalization of DNNs and may depend on the training procedure (Hochreterr & Schmidhuber 1997 Chaudhari et al. 2016 Wu et al. 2017).
In (Jacot et al. 2018)) has boen shown, using functional approach, that in the infinite- width limit, DNNs behave like kernel methods with respect to the so-called Neural Tangent Kernel, which is determined by the architecture the network. This leads to convergence
guarantees for DNNs (Jacot et al. 2018 Du et al. 2019 Allen-Zhu et. al. 2018 Huang &
Yau
2019
and strengthens the connections between neural networks and kernel methods
Neal
1996
Cho & Saul 2009 Lee et al. 2018).
Our approach also allows one to probe the so-called mean-fied fieldddddaaa /ccive limit (studied in (Rotskoff & Vanden-Fijnden 2018 Chizat & Bach 2018a Mci et al. 2018 for shallow networks), where the NTK varies during training.
these new results to gain insight into the behavior of the Hessian of the loss DNNs, at least in the small region explored by the parameters during training?
This raises the question: can

---

Published as a conference paper at ICLR 2020
1.1 CONTRIBUTIONS
Following ideas rodueee in (Jacol el al. 2018 we consider the training of L + 1-layered DNNs a functional setting. For a lunctional cost C, the Hessian of the loss RRP 0 - C (FFL) (0) is the sum of two P x P matrices I and S. We show the following results for large P and for a fixed number of datapoints N:
The first matrix 1 is positive semi-definite and its eigenvalues are given by the (weighted) kernel PCA of the dataset with respect the NTK. The dominating eigenvalues are the principal components of the data followed by a high number of small eigenvalues. The "flat directions" are spanned by the small eigenvalues and the null-space (of dimension at least P P P- when there is single output). Because the NTK is asymptotically constant (Jacot al. 2018). these results apply at initialization, during training and at convergence.
The second matrix can be viewed as residual contribution to H, since it vanishes the network converges to global minimum. We compute the limit of the first moment Tr (S) and characterize its evolution during training, of the second moment Tr (S²) which stays constant during training, and show that the higher moments vanish.
Regarding the sum =I++3 we show that the matrices and S are asymptotically orthogonal to each other at initialization and during training. In particular, the moments of the matrices I and S add up: tr(H)) s xr(1+)+ + +r(()).
These results give, for any depth and fairly general non non-linearity. complete description of the spectrum of the Hessian in terms of the NTK at initialization and throughout training.
Our theoretical results are consistent with number of observations about the Hessian
(Hochreiter & Schmidhuber
1997: Pascanu et al. 2014
Dauphin al. 2014 Chaudhari
et a.l 2016 Wu et al. 2017 Pennington & Bahri
2017
Geiger et al. 2018 and sheds
new light on them.
1.2 RELATED WORKS
The Hessian of the loss has been studied through the decomposition + S in a number of previous works (Sagun et al. 2017: Pennington & Bahri 2017 Geiger et al. 2018).
For least-squares and cross-entropy costs, the first matrix I is equal to the Fisher matrix Waagnaar 1998 Pascanu & Bengio 2013). whose moments have been described for shallow networks in (Pennington
Worah 2018). For deep networks, the first two moments and the operator norm of the Fisher matrix for least squares loss were computed at initialization in (Karakida et al., (018)) conditionally on a certain independence assumption; our method does not require such assumptions. Note that their approach implicitly uses the NTK.
The second matrix has been studied in (Pennington Bahri 2017 Geiger et al. 2018 for shallow networks, conditionally on a number of assumptions. Note that in the setting of (Pennington & Bahri 2017), the matrices and S are assumed to be freely independent, which allows them to study the spectrum of the Hessian; in our setting, we show that the two matrices and are asymptotically orthogonal to each other.
SETUP
We consider deep fully connected artificial noural networks (DNNs) using the setup and NTK parametrization of (Jacot et al. 2018), taking arbitrary nonlincarity E (R) (i.e. a RR + R that is 4 times continuously differentiable function with all four derivatives bounded). The layers are numbered from 0 (input) to L (oulput), each containing n& neurons for O,..,, L. The P = -I- (ne + 1)ne+1 parameters consist of the weight matrices
W(O) E RT<r1xng Xng and bias vectors b(5) E R for = O....,L 1. We aggregate the parameters into the vector 0 E R.P.
2

---

Published as a conference paper at ICLR 2020
The activations and pre-activations of the layers are defined recursively for an input R setting (0 (x; 1z; 0) r:
a 1 a (x; 9
_ VTu
(et (x; o ((a (z; 0))
The parameter ,9 is added to tune the influence of the bias on training All parameters are initialized iid (N00 1) Gaussians.
We will in particular study the network function, which maps inputs x to the activation of the output layer (before the last non-linearily):
fo(x) ā (D) (z:
In this paper, we will study the limit of various objects as ....... n2 - 0o sequentially, i.c. we first lake ni - oo, then ni - oo, elc. This greatly simplifies the proofs, but they could in principle be extended to the simultaneous limit, i.e. when our numerical experiments are done with 'rectangula'' networks (with 01 = ... = = and match closely the predictions for the sequential limit.
n1--1 + co. All
In the limit we study in this paper, the NTK is asymptotically fixed, as in (Jacot et al. 2018 Allen-Zhu et al. 2018 Du et al. 2019 Arora et al. 2019 Huang & Yau 2019 By resca ling
the outputs DNNs as the width increases, one can reach another limit where the NTK
is not fixed (Chizat & Bach 2018a b Rotskoff & Handen-Eijjdee
2018 Mei al. 2019).
Some of our results can be exlended to this setting, but only at initialization (see Section The behavior during training becomes however much more complex.
3.3
2.1
FUNCTIONAL VIEWPOINT
The network function lives in a function space So F == IR R= and we call the function F(L) RR + F that maps the parameters to the network function fe the realization function. We study the differential behavior of F(L);
The derivative (F E RR.. F funetion-valued vector of dimension P. The p-th entry F(F 0,, fo E F represents how modifying the parameter 6, modifies the function fo the space F.
The Hessian HF(L) E RR & F as funetion-valued P x P matrix.
The network is trained with respect to the cost functional:
c()= N Iaa
for strictly convex G, summing over a finite dataset Ti. E R of size N. The parameters are then trained with gradient descent on the composition Co p(L), which defines the usual loss surface of neural networks.
In this setting, we define the finite realization function Y(L) RP - No mapping parameters 0 to be the restriction of the network function fe to the training set Vik (z;) The Jacobian (L) is hence Nnt x P matrix and its Hessian HY is P x P Nnt tensor. Defining the restricted cost C'yy) = N E. (y)), we have Co (L) C (L))
For analysis, require that the gradient norm ||PO'| does not explode during training, The following condition is sufficient:
Definition 1. A loss C NnL + R has bounded gradients over sublevel sets (BGOSS) if the norm of the gradient is bounded over all sets U, {y e Na... C(Y) < a}.
For example, the Mean Square Error (MSE) C(Y) NN || 112 for the labels Y Nuz has BGOSS because VC(Y)] 1 |||*- cross-entropy the gradient is uniformly bounded, see Proposition in Appendix A
2C(Y). For the binary and softmax
In our experiments, we take 8 0.1.
3

---

Published as a conference paper at ICLR 2020
2.2 NEURAL TANGENT KERNEL
The behavior during training of the network function fo in the function space F described by (mulli-dimennional) kernel, the Neural Tongent Kernel (NTK)
(L) R (2,2²) E (0)
During training, the function fo follows the so-called kernel gradient descent with respect to the NTK, which defined as
N 6
(z) -e Gracs (2) Σ
In the infinite-width limit (letting n1 .o...., RL-1 - sequentially) and for losses with
BGOSS, the NTK converges to a deterministic limit (LL) + 0 @ Idn,, which is constant during training, uniformly finite time intervals [0, T] (acoo al. 2018). For the MSE
loss, the uniform convergence of the NTK was proven for T = oo in Arora et al. 2019
(T) R Re R constructed follows: The limiting NTK O%
1. For S.g
R +
R and a kernel K R x Ro + R, define the kernel Like
no R
nD
R
by
LR (zo, x1) = a.) [f(ao)g(ai)),
for (4g, a1) centered Gaussian vector with covariance matrix (K(a,, V R)))s=o.. For
f = we denote by T.g the kernel T.k/
2. We define the kernels 2i for each layer of the network, starting with D!! (xo, x1)
(n + 8 and then recursively by ((++1) 22 + B for l = I,.... L
where a is the work non-linearity.
3. The limiting NTK ((4) is defined in terms of the kernels (e) and the kernels 50
L 200
();
L
((L) no
5(4+1) 5(2) 00
The NTK leads to convergence guarantees for DNNs in the infinitewidthh limit, and connect
their generalization to that of kernel methods (Jacot et al. 2018 Arora et. al. 2019)
2.3 GRAM MATRICES
For finite dataset Ti, Tie EN E Inte and fixed depth L > 1, we denote by Ô E I X.n, the Gram matrix of ,,,..., UN with respect to the limiting NTK, defined by
Özggm = ( (I,, xzz) 6xm.
It is block diagonal because different outputs & + are asymptotically uncorrelated.
Similarly, for any (scalar) kernel (LL (such the limiting kernels Σ(h A m( $2 =0 introduced later), we denote the Gram matrix of the datapoints by K.
3
MAIN THEOREMS
3.1
HESSIAN AS S
Using the above setup, the Hessian H of the loss Co F(L) is the of two terms, with the entry Hp given by
Hp.'' HC]5, (Oo, F, F,Oo, F) + 100,,000,000 F).
4

---

Published as a conference paper at ICLR 2020
For a finite dataset, the Hessian matrix H (C ((1) is equal to the sum of two matrices
T = (DY HCDY(4) and == VC HY
where (L) is a Nn,, x P matrix, HC is a Nus. x Nus, matrix and HY is Px Px Nu lensor to which apply a scalar product (denoted by .) in its last dimension with the Nnr. vector VC to obtain a x P matrix.
Our main contribution is the following theorem, which describes the limiting moments Tr (H*) in terms of the moments of I and S:
Theorem For any loss C with BGOSS and a E (R),, in the sequential limit nı RT-1 00, we have for all k 2
Tr (T(H(O))) = Tr (T()*) TT
The limits of Tr (I (1)+t and Tr (s())) can be expressed in terms of the NTK O the kernels Th =(6) and the non-symmeetric kernels (L) A defined in Appendix C
The moments Tr (T(1)*) converge to the following limits (with the convention that
d++1 inj)
N
k II
Tr (1(1))) + Tr ((CC
Σ
the (foc))(r.))))
() (Xi Tim4il
NOW
The first moment Tr (I)) converges to the limit:
Tr (1)) = (G())² VC((()).
At inilialization (G(O), Y(0)) form Gaussian pair of N independent for differing output indices = 1,... nn and with covariance E (O) 6 (Ii, and E (0) (0)] = dEN bbb) (x1, for the limiting kernel =(6e) (x, y) and non symmetric kernel @ (2) (I, v). During training, both vectors follow the differential equations
0,G(t) = ÁVC(V())
0,Y -VVC(Y
The second moment Tr (s()²) converges to the following limit defined in terms of the Gram matrix T:
Tr (52) (
The higher moments Tr (S(t)+) for k 2 vanish.
Proof. The moments of and S can be studied separately because the moments of their sum is asymptotically equal to the of their moments by Proposition 5 below. The limiting moments of and S are respectively described by Propositions and below.
In the case MSE loss C(Y) = + || 11 the first and second derivatives take simple forms VC(Y)= = & (Y - YY* and HC(Y) = + IdNaz and the differential equations can be solved to obtain more explicit formulae:

---

Published as a conference paper at ICLR 2020
121
10
1
5x
121
20 129 121
15
122
Figure 1: Comparison of the theoretical prediction of Corollary [ for the expectation of the first 4 moments (colored lines) to the empirical average over 250 trials (black crosses) for rectangular network with hidden layers finite widths n1 n2 5000 (L = 3) with the smooth ReLU (left) and the normalized smooth ReLU (right), for the MSE loss on scaled down 14x14 MNIST with N 256. Only the first two moments are affected by at the beginning of training.
RL-1 - 0c, we have Corollary 1. For the MSE loss C and a E (R), in the limit ni,..., uniformly over [0, T]
Tr (H((())) NE Tr (6t) + Tr (S(1))
where
Tr (S()))
N (Y" - Y(O)) [IdNn. + T 1 -Y())(())
6 ( -Y(0))
+ N
Tr (S(+)²) + N (Y* e -(()--(()) Y(())
Tr (S(1)*) +0 when k: > 2.
In expectation we have:
1 Tr ((IdNn +--16) 0- -1 ( +
1 Ir (e-15
[T (S(t))
E Tr (S(+))) + N2 Tr (e ((+xrr))))+
Proof. The moments are constant because HC = 1/2 IdNaz is constant. For the moments of S, we first solve the differential equation for Y(t):
Y(t) Y - (Y* c Y())).
Noting Y(t) ( - VC(s)ds, we have
G(t) G(O) Ä ( VC(s)ds
= G(0) + A0- (0)('(((t)-)))) -(())
= G(0) + Ao-1 (Idwne + t--16) (0--Y(0)
The expectation of the first moment of S then follows.

---

Published as a conference paper at ICLR 2020
3.2 MUTUAL ORTHOGONALITY OF AND S
first key ingredient to prove Theorem is the asymptotic mutual orthogonality of the matrices I and S
Proposition (Proposition in Appendix D For any loss C with BGOSS and a E (RR), we have uniformly over [0,
lim 16-1700
lim ||]]||= m10000
=0.
As a consequence lim lim,,
Tr (1* [Tr (1*) + Tr (5*)) = 0.
Remark 1. If two matrices A and B are mutualy ort orthogonal (i.e. AB = 0) the range of A is contained in the nullspace of B and vice versa. The non-zero eigenvalues of the sum A B are therefore given by the union of the non-zero eigenvalues of A and B. Furthermore the moments of and B add up: Tr (IA+ + BJ²) Tr (A*) + Tr (B+). Proposition 5 shows that this is what happens asymptotically for T and S.
Note that both matrices I and S have large nullspaces: indeed assuming constant width w 71 = AL-1, we have Rank(I) < NuL and Rank(S) < 2(L 1)wNnz (see Appendix while the number of parameters P scales as w² (when L > 2).
Figure 2 illustrates the mutual orthogonality of / and S. All numerical experiments are done for rectangular networks (when the width of the hidden layers are equal) and agroe well with our predictions obtained in the sequential limit.
3.3 MEAN-FIELD LIMIT
For a rectangular network with width w, if the output the network is divided by vw and the learning rate is multiplied by (to keep similar dynamics at initialization), the training dynamics changes and the NTK varies during training when w goes to infinity. The new parametrization of the output changes the scaling of the two matrices:
H ( (
(DY(C)) T HCDY(L)
L.v HY - V
--1 Ls. VW
The scaling of the learning rate essentially multiplies the whole Hessian by w. In this sctting, the matrix I is left unchanged while the matrix S' is multiplied by vw (the k-th moment of is hence multiplied by w particular, the two moments the Hessian are dominated by the moments of S, and the higher moments of S (and the operator norm of S) should not vanish. This suggests that the active regime may be characterised by the fact that ||S|| > || Under the conjecture that Theorem [ holds for the infinite-wide wiith limit of rectangular networks, the asymptotic of the two first moments of H is given by:
1/ (H) + NO, VC"EVC) 1/ (H²) VV-iVC,
where for the MSE loss we have VC=
3.4 THE MATRIX S
The matrix S = VC (L) is best understood as a perturbation to I, which vanishes as the network converges because VC - 0. To calculate its moments, note that
T
Tr (VC. (VC. (.C.H)()))
Σ
Ogg
TC=GvV
where the vector G Dm 0g E R Nas. is the cvaluation of the function go(x) Σp_ fzz)((z) on the training set.

---

Published as a conference paper at ICLR 2020
For the second moment have
T: (((vc-ryuu))) vo"
Eb V (0 (0) VC= VOTiVC
p.P'=1
for the Gram matrix of the kernel r(4) (x, (I, Eps'= b fo(z) (08,0,
The following proposition desribes the limit of the function go and the kernel T()) and the vanishing of the higher moments:
Proposition (Proposition in Appendix [ For any loss C with BGOSS and a E (RR),
the first two moments of S Take the form
Tr (S(t)) VC()
Tr (S(t)²) (C(O)
At initialization, go and fo converge to a (centered) Gaussian pair with covariances ((g. = E()) (2,22)
(o) (x')) = SUN 1 (x,x²))
fe, òN 2/66) (I,
and during training go evolves according to
N
A (z, (x. 0.90,1 (x) = Σ
i=1
Uniformly over any interval [0, the kernel PLL) has deterministic and flaed limiil limmg 1000 limp r(l) 1 (I, = du ( (x,x") with limiting kernel:
1-1
T( (z, x') = Σ
+ (I, 225000000 (2, (I,x²) + 20 (2, 2')00 (x, (x, (1,t1)(z, ..6))...))-))) 111-11 (2, (x,
The higher moment k >>> 2 vanish: limaz ... lim Tr (s*) 0.
This result has number of consequences for infinitely wide networks:
1. At initialization, the matrix S has finite Frobenius norm ||S|| Tr (S2)
VCTiVC, because converges to a fixed limit. As the network converges, the derivative of the cost goes to zero VC(t) + 0 and so does the Frobenius norm of S.
2. In contrast the operator norm of S vanishes already at initialization (because for all even k, we have || S (S** - 0). At initialization, the vanishing of S in operator norm but not in Frobenius norm can be explained by the matrix S having a growing number of eigenvalues of shrinking intensity as the width grows.
3. When it comes to the first moment of S, Proposition 4 shows that the spectrum of S is general not symmetric. For the MSE loss the expectation of the first moment at initialization is
E [Tr(S)] =
= -
Tr ④ (o)-0
which may be positive or negative depending on the choice of nonlincarity: with smooth ReLU, it is positive, while for the arc-tangent or the normalized smooth ReLU, it can be negative (see Figure [].
This is in contrast to the result obtained in (Pennington & Bahri 2017 Geiger et al. 2018) for the shallow ReLU networks, taking the second derivative of the ReLU to be zero. Under this assumption the spectrum S is symmetric: the eigenvalues ordered from lowest to highest, \; ->p-i and Tr(S) 0.

---

Published as a conference paper at ICLR 2020
larget df langrd
64-
64-
1000
02
60
-0.10 -3.855 -0.050
-1.32
2.800 6622
0.500
8.055
3.200
vSV
Figure 2: Illutration of the mutual orthogonality of I and S. For the 20 first eigenvectors of I (blue) and S (orange), we plot the Rayleigh quo- tients 0 Iv and v"Se St (with L = 3, n1 n2 1000 and the normal- ized ReLU on 14x14 MNIST with N 256). We see that the direc- tions where I is large are directions where is small and vice versa.
Figure 3: Plot of the loss surface around global mini- mum along the first (along the coordinate) and fourth (x coordinate) eigenvectors of L. The network has 4, width
n2 = n3 1000 for the smooth ReLU (left) and the normalized smooth ReLU (right). The data unifor the unit disk. Normalizing the non-linearity greatly reduces the narrow valley structure of the loss thus speeding up training.
These observations suggest that S has little influence on the shape of the surface, especially towards the end of training, the matrix I however has interesting structure.
3.5
THE MATRIX I
At global minimizer 8", the spectrum of describes how the loss behaves around 8*, Along the eigenvectors of the biggest eigenvalues of I, the loss increases rapidely, while small eigenvalues correspond to flat directions. Numerically, it has been observed that the matrix I features a few dominating eigenvalues and a bulk of small eigenvalues (Sagun et al. 2016 2017 Gur-Ar et al. 20188 Papyan 2019). This leads to a narrow valley structure of the loss around minimum: the biggest eigenvalues the 'cliffs' of the valley, i.c. the directions along which the loss grows fastest, while the small eigenvalues form the 'flat directions or the bottom of the valley.
Note that the rank of is bounded by Nn, and in the overparametrized regime, when Nnn P, the matrix I will have large nullspace, these are directions along which the value of the function on the training set does not change. Note that in the overparametrized regime, global minima are not isolated: they lie in a manifold of dimension least P Nn and the nullspace of I is tangent to this solution manifold.
The matrix I is closely related to the NTK Gram matrix:
ô = DD (DrlL)² and I= (DVUU)) T HCDY(L))
As result, the limiting spectrum of the matrix can be directly obtained from the NTK Proposition 1. For any loss C with BGOSS and CR)(R), uniformly over any interval
[0, 7), the moments Tr (IK) converge to the following limit (with the convention that ig+1 i ):
N
lim lim Tr (1+) =T -T(CHCX)))) N
Σ
II
Co (Foc))()..)))0 (Iim Dimes)
$1
=1 m=1
2Thi result already obtained in Karakida et al. explicitely and only initialization.
2018 but without identifying the NTK
9

---

Published as a conference paper at ICLR 2020
Proof. It follows from Tr (1+) = Tr ((DDLL)) HODDwW)) = =Tr ((HCØ)") and the
asymptotic of the NTK (Jacot al. 2018
3.5.1 MEAN-SQUARE ERROR
When the loss is the MSE, HC is equal to IIdNNz. As a result, θ and I have the same non-zero eigenvalues up to a scaling of I/N. Because the NTK is assymptotically fixed, the spectrum of T is also fixed in the limit.
The cigenvectors the NTK Gram matrix are the kernel principal components of the data. The biggest principal components are the directions in function space which most favorised by the NTK. This gives a functional interpretation of the narrow valley structure in DNNss: the cliffs of the valley are the biggest principal components, while the flat directions are the smallest components.
Remark 2. As the depth L of the network increases, ctt al. 2016; Jacot et al.] 2019): Order Freeze where the NTK converges to a constant and Chaos where the NTK converges to Kronecker delta. In the Order/Frest /Freeze the Nn, Nng, Gram matrix approaches a block diagonal matrix with n.y, constant blocks, and as a result n eigenvalues of / dominate the other ones, corresponding to constanl directions along each outputs (this is in line with the observations of (Papyan 2019)). This leads to narrow valley for the loss and slows down training. In contrast, in the Chaos regime, the NTK Gram matrix approaches a scaled identity matrix, and the spectrum of 7 should hence concentrate around a positive value, hence speeding up training- Figure illustrates this phenomenon: with the smooth ReLU we observe a narrow valley, while with the normalized smooth ReLU (which lies in the Chaos according to (Jacot et al., 2019)) the narrowness of the loss is reduced. A similar phenomenon may explain why normalization helps smoothing the loss surface and speed up training (Santurkar et al. 2018 Ghorbani et al. 2019).
observe two regimes Poolc
3.5.2 CROSS-ENTRON Loss
For a binary cross-entropy loss with labels y* E +11,+11
1 C(Y) N
log (1 11+-- (1 xxxe)))))
HCC is a diagonal matrix whose entrics depend on Y (but not on Y"):
1 1
Ha,(Y) = N Flleee 11+e
The eigenvectors of / then correspond to the weighted kernel principal component of the data. The positive weights Itc V.+.,, approach 1/3 as Y, goes to 0, i.e. when it close to the docision boundary from one class to the other, and as Y, + too the weight go to zero. The weights evolve in time through Y, the spectrum of is therefore not asymptotically fixed as in the MSE case, but the functional interpretation of the spectrum in terms of the kernel principal components remains.
CONCLUSION
We have given an explicit formula for the limiting moments of the Hessian of DNNs throughout training. We have used the common decomposition of the Hessian in two terms I and S and have shown that the two terms are asymptotically mutually orthogonal, such that they can be studied soparately.
The matrix S vanishes in Frobenius norm as the network converges and has vanishing operator norm throughout training. The matrix I is arguably the most important it describes the narrow valley structure the loss around a global minimum. The eigendecomposition is related to the (weighted) kernel principal components of the data w.r.f. the NTK.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
Clément Hongler acknowledges support from the ERC SG CONSTAMIS grant, the NCCR SwissMAP grant, the NSF DMS-1106588 grant, the Minerva Foundation, the Blavatnik Family Foundation, and the Latsis foundation.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A Convergence Theory for Deep Learning via Over- Parameterization. CoRR, abs/ abs/1811.03992, 2018. URL http://arxiv org/ / 1811.03962
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint arXiv: 11955, 2019.
Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, Gerard Ben Arous, Chiara Cammarota, Yann LeCun, Matthieu Wyart, and Giulio Biroli. Comparing Dynamics: Deep Neural Networks versus Glassy Systems. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 314-323. PMLR, 10-15 Jul 2018. URL http://prooceedings mlr press//v0/baaty Desi18a.html
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. arXiv preprint arXi:: 617 1.01838. 2016.
Lénaïc Chizat and Francis Bach. On the Global Convergence of Gradient Descent for Over pheepparamteriied Models using Optimal Transport. In Advances in Neural Information Processing Systems 31, pp. 3040-30000 3050. Curran Associates, Inc., 2018a. URL http://paperr: nips wwtpc//pppp//.70-oo.o.o. the teaenntthhggbballae gllball convereleee-ceeeeeeeeeeeeeoee oraaeererrrrggooooo---r-a-eee--e wwwnssootrrt.r....mm
descent- over wwwppaataaaieeeeeeaa parerroroooeeaaaaareeerraaaaaaaa models sonendell-ssing--ee
Lenaic Chizat and Francis Bach. A note on lazy training in supervised differentiable
programming. Xiv preprint crXiv:1812.07956 2018b.
Youngmin Cho and Lawrence K. Saul. Kernel Methods for Deep Learning. In Advances in Neural Information Processing Systems 22, pp. 342 350. Curran Associates, Inc., 2009. URL -oweeeee--eettoddd--oo-eeeeeee Learning.ee.rnnnn.n.
http://papers wiis. www.:iii...p.peeeeeeeeii
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The Loss Surfaces of Multilayer Networks. Journal of Machine Learning Research, 38: 192 204, nov 2015. URL httts:///axxiv www.:/r...ppff... 1412. 0233. .pdf
Yann N. Dauphin, Razvan Pascamu, Caglar Gulcchre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and Attacking the Saddle Point Problem in High-dimensional Non-convex Optimization. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS'14, pp. 2933 2941, Cambridge, MA, USA, 2014. MIT Press.
Simon S. Du, Xiyu Zhai, Barnabás Póczos, and Aarti Singh. Gradient Descent Provably Optimizes Over-paramelerizeed Neural Networks. 2019.
Mario Geiger, Stefano Spigler, Stéphane d' Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart. The jamming transition as a paradigm to understand the loss landscape of deep neural networks. arXiv preprinl or:: 180.009499, 2018.
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. abs/1901.01600 2019. URL http:://axiv.org/// 01608
11

---

Published as a conference paper at ICLR 2020
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Mochine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2232-2241, Long Beach, California, USA, 09-15 Jun 2019. PML.R. URL htttp://proceeeing... mlr www.r.press/vve wttp:/s//////ghbbbaaii/ html
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. CoRR, abs/1812.04722 2018. URL
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Newral Computation, 9(1):1-42, 1997.
Jiaoyang Huang and Horng-Tzet Yau. Dynamics of deep neural networks and neural tangent hierarchy. arw preprint 1X0::190000811 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Ker- nel: Convergence and Generalization in Neural Networks. In Advances in Newral Informalion Processing Systems 31, pp. 8580 8589. Curran Associates, Inc., 2018. URL http://papers nips oc/paper//07--neurrall convergence - peadd-geneaaaatior...e in nww.aooo---nnnaallll-a----all--a networks..
kernel
Arthur Jacot, Franck Gabriel, and Clément. Hongler. Freeze and chaos for dnns: an NTK view of batch normalization, checkerboard and boundary effects. CoRR, abs/ hbbs/190.05777 .05115, 2019. URL
Ryo Karakida, Shotaro Akaho, and Shun-Ichi Amari. Universal Statistics of Fisher In- formation in Deep Neural Networks: Mean Field Approach. jun 2018, URL http
//arxiv. org/ aas/a/as/..../////// 01316
Jae Hoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep Neural Networks as Gaussian Processos. ICLR, 2018.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33): E7665-E7671, 2018.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. arXiv preprint arvi:: 1902. 06015. 2019.
Radford M. Neal. Bayesian Learning for Neural Networks. Springer-Vette Verlag New York, Inc., Secaucus, NJ, USA, 1996. ISBN 0387947248.
Vardan Papyan. Measurements of thrce-level hicrarchical structure in the outliers in the spectrum of deepnet hessians. CoRR, Abs/1190.008244 2019. URL http:://axxiv..o org/ abs/1901. 0bs/1901.08244.
Razvan Pascanu and Yoshua. Bengio. Revisiting Natural Gradient for Deep Networks. jan 2013. URL http:///axxv.. org/ abs//301.31313
Razvan Pascanu, Yann N Dauphin, Surya Ganguli, and Yoshua Bengio. On the saddle point problem for non-convex optimization. arXiv preprint, 2014. URL https:://axiv.... org/ pdf//440 2Bf/1405.4604.11 4604.pdff
Jeffrey Pennington and Yasaman Bahri. Geometry of Neural Network Loss Surfaces via Random Malrix Theory. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 2798 2806. PMLR, 06-11 Aug 2017. URL http:://prrocoeeinngs
mlr. wwt.perss//7///pmmnnnooooo/
Jeffrey Pennington and Pratik Worah. The Spectrum of the Fisher Informa- Neural Network. In Advances in Neu-
tion Matrix of a ral Information Processing Systems 31, pp. 5415-5424. Curran Associates, Inc., nips.
2018. URL http://pppeer.iiip..c//pppr//t1p http:///rm-0---ee-esttttt------ ofpectrum-of-oeeee the-thhhees aaformaaaaaraaaaaieaaaaaaaaaeeae
network..pp
12

---

Published as a conference paper at ICLR 2020
Ben Poole, Subhancil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya. Gan- guli. Exponential expressivity in deep neural networks through transient chaos. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Ad- vances in Newral Information Processing Systems 29, pp. 3360 3368. Curran Associates, Inc., 2016. URL http::ttppp:peeee htttp::/papers.nnss.ccca///////e www.:.iiiscccccccc..ee wttp:///eeeeeeooaaaaaaaaaaaeeeee in-deep neeral-laaaee -hwteooo-------ro-----------
dn-deep-neural
chaos. pdf
Grant Rotskoff and Eric Vanden-Fijnden. Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks. In Advances in Curran Associates, Inc., as interacting" of neural-
Newral Information Processing Systems 31, pp.
2018. URL
nips http://apppp///---/aaaaa.. hwtpapers.nissccppppeeeeeeeeeeee
particles-lome tong-time--ae oaanerrgncee-m---aasypoooo------ networks.por pdf
and
Levent Sagun, Léon Botton, and Yann LeCun. Singularity of the hossian in deep learning. CoRR, abs/1611.07476, 2016. URL http://arxiv wwwaaawwwww..a///////aa.a.aaae wtt./oo/g////../.e/oe 07476
Levent Sagun, Utku Evei, V. Ugur Güiney, Yann Dauphin, and L6on Botton. Empirical Analysis of the Hessian of -ver-Parametrized Neural Networks. CoRR, abs/1706.04454, 2017.
Shibani Santurkar, Dimitris Tsipras, Andrew Tlyas, and Aleksander Madry. How does batch normalization help optimization? In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Gesa-BBanchhi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2483 2493, Curran Associates, Inc., 2018. URL lnttp://papers nips wwwp:r///11111oooo-oo- batch- help optimization. pdf
Daniel Wagenaar. Information geometry of neural networks. 1998. ISSN 0302-9743.
Lei Wu, Zhamxing Zhu, and Weinan E. Towards Understanding Generalization of Deep
Learning: Perspective of Loss Landscapes. CoRR, abs/ 0bs/1706.10233 2017. URL http:
//arxiv. meexiiv.org/aaa hwwg/bb/11706.12333
A
PROOFS
For the proofs of the theorems and propositions presented in the main text, reformulate the setup of (Jacot et al.) 2018). For a fixed training set #1 EI....IN. we consider a (possibly random) time-varying training direction D(t) E R which describes how cach of the outputs must be modified. In the case of gradient descent on a cost C(Y), the training direction is D(L) (I) The parameters are updated according to the differential equation
0,0(t) = (O,Y D(t).
Under the condition that S [D(t)]l2 dt is stochastically bounded as the width of the network goes to infinity, the NTK (t) converges its fixed limit uniformly over [0, 7).
The reason we consider a general training direction (and not only a gradient a loss) is that can split a network in two at layer & and the training of the smaller network will be according to the training direction D.00 (t) given by
DI (() diag (o (z)) (_
...diag (6 ...diag
(all-))
1 (--_-u--))
T Di(t)
because the derivatives â are bounded and by Lemma I of the Appendix of (Jacol et al. 2018h, this training direction satisfies the constraints even though it is not the gradient loss. As consequence, as n1 To + oo the NTK of the smaller network ((6) also Te-1-
converges to its limit uniformly over [0, Tj. As we let ne + oo the pre-activations (()) and weights wi(E) move at rate of 1/yme. We will use this rate of change prove that other types of kernels constant during training.
13

---

Published as a conference paper at ICLR 2020
When network is trained with gradient descent loss C' with BGOSS, the integral fo' [t)()] dt is stochastically bounded. Because the loss is decreasing during training, the outputs (t) lie in the sublevel set (C(Y(()) (o)) for all times t. The norm of the gradient is hence bounded for all times t. Because the distribution of ( converges to multivariate Gaussian, (C(((O))) is stochastically bounded as the width grows, where b(a) is a bound on the norm of the gradient U.. We then have the bound fo [D(t) dt Tb(C(Y which itself stochastically bounded.
For the binary and softmax cross-entoon cross-entropy losses the gradient is uniformly bounded: Proposition 2. For the binary cross- coss-entropy loss and any Y E R,, || - VN
For the softmar cross-entree cross-entropy loss C on c N classes and any Y E RN,, |VC(Y)l VN
22c
Proof. The binary cross-entropy loss with labels Y
[0, [0,1)* is
C(Y)
1
Σ
N
log
exiY
1
Σ
N
log (1 +1+))) Y,Y,
N
1 e
N
E=1
and the gradient at an input is
1 e -1,--
a,C(Y)
N
1 1+ eV:
which is bounded in absolute value by N for both Y, = 0, such that VCC()) S J.
The softmax cross-cntropy loss over classes with labels Y E fl,.....c* is defined by
N
N Σ
C(Y)
log
log
Σ Vin
-Y
Ek=ret
2=1
R=
The gradient is at input and output class
m is
1 Vom
Om (()
8y;m
N Ei evau
2 which bounded in absolute value by N such that |VC(Y)]2 S 2
B PRELIMINARIES
To study the moments of the matrix S, we first have to show that two tensors vanish as 71 1 + oc:
(1L)) ky (oo, 21, x2) = [:00=[[Vooon (o)) H.fo. (zi)
(1L) k2, (To. T1, x2, x2, =
H.fo.. (21) (x2) (
need recursive definition for the first derivatives 0g fa,((z) and second derivatives 8% faxx). The value of these derivatives depend on the layer the parameters Op and Opr belong to, and on whether they are connection weights wi(f) or biases (()) The derivatives with respect to the parameters of the last layer mk are
We study these tensors recursively, for this,
① wil fz) VnL-I am
(L-1) (x))
0e (z) = 326kx'
for parameters 6p which belong to the lower layers the derivatives can be defined recursively
by
Bh-
So,
Σ
0u, aml 00,am
(x))
(all (x)) (((-1)) mnk
/Tc. m=l
14

---

Published as a conference paper at ICLR 2020
For the second derivatives, first note that if cither of the parameters fg fyr are bias of the last layer, or if they are both connection weights of the last layer, then _0,
(1, 0. Two cases are left: when one parameter is a connection weight of the last layer and the others belong to the lower layers, and when both belong to the lower layers. Both cases can be defined recursively in terms of the first and second derivatives of a!--1)
8² Op (x)
00 am (z)6
(x)) OY
VnL-1
b 60,
fo.
1
BL
(z))) w(ll-1) Opl am (z)) a.m
VLL
m=1
nz
(L 0g 1 m
10, (1) ( (x)ō am (x)) will-1) mi
m=1
Using these recursive definitions, the tensors l(++1) and I(L+1) are given in terms of ((L) (2(4) and p(4) in the same manner that the NTK ((L+1) defined recursively in terms of (()) in (Jacot et al. 2018).
Lemma 1. For any loss C' with BGOSS and a E (R),, we have uniformly over [0,T]
lim 20 ko (2o, 21, x2) 710
lim RL
Proof. The proof is done by induction. When L 1 the second derivatives 0 fo, 0 and ( ky (oo, 21, x2) = O.
For the induction step, we write ( ka (oo, 21 x:2) recursively as
8/2
Σ
O momm (oo, x 0 mg (o) xy 12 (zo) (o)))(((l)) (z1)) (22)) w(( will) w( make miki make
ng
Mo, m2
1/2
Σ
2 mo m2 (wo, 21 22) (zo) (Z) (o))) (a!l (x2) moko miks make
(8) w w(e
n2g
momma ma
+ n2.
8/2
+ (E)
mg.my (1 (2)) (2) ) (1²)) W(()) mako 8k,k2
1/2
Σ
mi.mm (11, (zo (a (z) (a (x2)) ( Makk'
+ n2g
As 71,
n--1 + do and for any times T, the NTK ( converges to its limit while (2))
vanishes. The second summand hence vanishes and the others converge to
2/2
(e) (ro, vi 1 (x), (a (xo)) ( (a (a
(9 (.2) W mio w(O ((())
ng
mk, mkz
+ n
2/2
(Dn ) (zo) (z) )o (x2)) W W mko 6k
+ n2 3/2
(11,
)o (z) à (a (I2) ((() (z))
mka'
At initialization, all terms vanish as ne bocause all summands are independent with zero mean and finite variance: in the n + 00...., 1 - oo limit, the (+)) (x) are independent for different m, see (Jacot et al. 2018). During training, the weights W and preactivations (!) move at a rate of 1/vor (see the proof of convergence the NTK (Jacot et al. 2018 ). Since ā is Lipschitz, -3/2+1/2 my As a result, uniformly over times & E [0, all the sums vanish.
obtain that the motion during training of each of the sums of
order n,
Similarily, have
15

---

Published as a conference paper at ICLR 2020
Lemma 2. For any loss C' with BGOSS and (RR), have uniformly over [0,
lim P(L) No Na.ka (Co D2, I2, 23) =
lim 700
my 700
Proof. The proof is done by induction. When L = 1 the hessian HF((1) = 0, such that I(L) k2,ks, (Zo, 21, 22, Fz3) O.
For the induction step, ((++)) can be defined recursively:
r(L+1) ko.ki,
(Io, 2i, 22, v2, Ta)
= =
E
r(L)
mo. (Zo, 21, T2, (a) (zo)) (2) (
( (1) o (x2) (a (x:))
ma. MMg.
moko wilg,, w(l) W maks
miki
mgge
-2
Σ
((L)
mo (ro, 11))11)s11 (52 mi.mmmm. 42, (a (cr)) (a
a(a ((2 ( (xx)
mako miki make w
w
maka
+
2
Σ
(L) (zo)) (all) (a (x1)) Dib mo (fo, 22 (L)) mg (I2, x3) ( (ams
8 (22)) (IS)) With w(l)) w make makss
w(l)
mako miki
+ L
2
Σ
(L)) mo
(oo, 2 (L))
(2) 22 ( ms (22, 23) )
(a (1l (2o)) (z1))
makeo miki mag w(ll 23 23) (2o) (a (o (L)
8 ( (2g) w(( will) w(t)
+
2
Σ
Di ma (41, (1
(a
òkoks migh will magks
+n
L
2
Σ
@ mi (31, 222) (01, (1)) ma (x2, x3)a (fo))d ( (( (a (H)))
mg
Oknks w w(ll ( (2z))
maks
Σ
( my 22) (a)))) (x) (z1))
(a (1)
mo mo
w(l) w wilk) Skzks moka
miki
+n
Σ
( ma (Io, (Jo, ((())
mi (21, 22) (zo)) (22) (xs))
mo
w(ll w(l) degks mako
+ 1 2
Σ
(L))
my my ((o) (a))) (x2)) (xx) 0kg Skyts
my min
(
mg (ooz (zo, ( m mi my (1z 23)6 (b) (o) (a)) (o) (a) (xx)) (x2)
+n
mp
w(t) 8 kg w mykes
mako
As W1,.. 11-1 + oo and for any times < T, the NTK ! converges its limit while (() and (() vanishes. (f++1)
ka (To, 31, x2, v3) therefore converges to:
2 Σ +n
(21, (z2, za) (ao)) ( (2 (z1)) (2 (22)) (a (œz))
m
wil w(ll) wil) wil) mako mky mik miks
16

---

Published as a conference paper at ICLR 2020
+ 2
Σ
(2o)) (o)
(12,
(x2) ( (1z))
will) Skoki mis mks
θ 1) ( ( (1) (x2) (22)) (o(am (as)) Σ
w w dugks máko mks
+,,
Σ
(L))
(T), 12) (a (12))o (1) (11)) (oa)))kok> Ōkzks
(12, (mo)) (21)) (x2) (2a)))(( (xs)) 00
2 +
Σ
(sn, 11 (
w(l)) wil 8e kz mkss mkko
For the convergence during training, we proceed similarily to the proof of Lemma [ At initialization, all terms vanish ne - oo because all summands are independent (after taking the RI..... R + oo limit) with zero mean and finite variance. During training, the weights W(()) and preactivations ((*) move at a rate of 1/ymz which leads to change of order nn
2+2 ny 1.5 which vanishes for all times too.
C THE MATRIX S
We now have the theoretical tools to describe the moments of the matrix S. We first give bound for the rank of
Proposition 3. Rank(S)= < 2(n +... + NN.
Proof. We first observe that S is given by sum of Nn, matrices:
n. Ou fo
Spp
therefore sufficiant to show that the rank of each matrices H.fox(z) Fex(zz)) DiP
is bounded by 2(n1 +... nz).
The derivatives fo, (x) have different definition depending on whether the parameter Op connection weight w((e) or bias 560
_ (x; (x; (0)
fox(()) VTt
fo 30 a (x:9) fz)
These formulas only depend on f through the values (z. (z: 0)) and (o) t,5 for ( L,..,, 4-1 (note that both aj (0) (x) x, and 2 (2:0) fox dn do not depend on 0). Together there are 2(n: + + of them. As consequence, the map 0 (o. can be written as a composition
OERP + (a a (1)
(o 8,3
E 2m (o,,))))) ERP
and the matrix H.fa (x) is equal to the Jacobian of this map. By the chain rule, H.fo,()) the matrix multiplication of the Jacobians of the two submaps, whose rank are bounded by 2(n1 + + nL-1),, hence bounding the rank of Hff.(()). And because S is of Nn,, matrices rank smaller than 2(n + +... + n1,1)); the rank is bounded by 2(m +... 1
n
17

---

Published as a conference paper at ICLR 2020
C.1 MOMENTS
Let us now prove Proposition 4
Proposition 4. For any loss C' with BGOSS and a E C% the first two moments of S take the form
(S(()) (T(5())=(((VC())
Tr (S(t)²) VC(H)
AA initialization, ga and fe converge to a (centered) Gaussian pair with covariances
(lgo,x (x')l = 5 (2.
(e (x')l ôu 1 (1)
(x, (I,
(1) (r') 6x x(6L (I, x¹)
and during training go evolues according to
0.go,((z) Σ (2,))
Uniformly over any interval [0, I] where So [[C(t)] dt is stochastically bounded, the kernel
r(L) has a deterministic and fixed Biitt lim lim +00 T (x. x') dEk T( (x, with limäting kernel:
r (z, x))
(x,x') 20 2')x) (2,+) ()(11) (x,
The higher moment k >>> 2 vanish: limaz-11100 lim 000 Tr (St) 0.
Proof. The first moment of S takes the form
(S) = -(C) Hpp H (
where G is the restriction to the training set of the function -o(x)=>> process is random initialization and varies during training. Lemma below shows that, in the infinite width limit, Gaussian process at initialization which then evolves according to simple differential equation, hence describing the evolution of the first moment during training.
= -,, fo This
The second moment of S takes the form:
N
Tr((²)= Σ
Σ
De Pr fo.k (21)0 fer (22)c,, (#x, )e,, (212)
m.P==lin.iz
= (VC)" TvC
-f 2p 03 pr fa, (x (z1)0 fo.v (22) multidimensional ker- nel and f is its Gram matrix. Lemma below shows that the infinite-width limit, r(L)
where T(L))
ky =
ky (#1, converges to deterministic and time-independdnt limit r (x1, 12 0k,
To show that Tr(S*) 0 for all k 2, it sulfices to show that || 0 [T(S**] || |S and we know that ||S] - (8rC)" foyC is finite. We have that
Th
||
Σ
Σ
We ko (Figg Tig Figs Fig )0 (m C0 C0fo.
(L))
ka=1
18

---

Published as a conference paper at ICLR 2020
Dsoing (1r2) Cd Coongglon)
= = 03
for v the Nuz x Nu x Nu x Nug finite version of
w (L) Ao. (Fior Diyy Vigy Iig) =
Σ
Lh 0n Jo (zo) Ooz fo (x1)
Po.PLoPa-Pa
pag Pr fo (Is). R6 py f0. (2 (
which vanishes the infinite width limit by Lemma below.
Lemma 3. For any loss C with BGOSS and a E (R),, at initialization go and fo converge to (centered) Goussian pair with covariances
(l (x')1 = 6N = (z, x²) (L)
Elgon(())fo,x = SE T 00 ( x1) (z) fo (x')l SNN E (x, x²)
and during training ge evolves according to
(zgo(x) Σ
Proof. When I, go(x) for any x and
For the inductive step, the trace 90,k (E+1) (x) is defined recursively as
L Σ ( (z) (c)) wiLl) + Tr (. (x) (V (=)) ø (x) w
VLL
First note that Tr fo (V (Vlom(()))*) mnn (x. x). Now let. 11.11 - xo, by the induction hypothesis, the pairs (0 alh) converge to iid Gaussian pairs of processes with covariance 2 at initialization.
At initialization, conditioned on the values of ( a!l the pairs (9k (L+1) ff) follow a centered Gaussian distribution with (conditioned) covariance
(L+1) (L+1) (1 Elgo.k (x)90 (x') a(10) = 5Kk
Σ ( (z) + (x, (al2(()) m=1
n
( (2())( + 2 (s' x (xx))
Blgo,
(1+1) (c)oo
(1) alll) 5kk Σ
90,m (z))
(I) ((()) (x)) + ( 2) (x))
m=1
(all
(e (L) [(0)
88k Σ o
a (L) a (alh1 (x') + g2.
n
As ny + oo, by the law of large number, these (random) covariances converge to their expectations which deterministic, hence the pairs (g( (L+1) fox) have asymptotically the same Gaussian distribution independent of gm amm)
(L) 90.6 (2) (1)) + dn ( (I,
E
19

---

Published as a conference paper at ICLR 2020
90 (o)
(D)
+ 82 (x,
jo (o)
+ OK I (x)
with E = (z. 0 and
=(t+))( (x, ((gg'
+ O (o((a))(a
+ (
= (1000 (r',)) ((0) ( (x,,2")
+ 10
"z "
0
and
(x, x²) (((+1)((
E Igo(a)) +
$ (D (z. x ) ( + +)),)))++(())))))) (x.x)) E E(ā(a)a(a'))
where (g.g', aa)a) is Gaussian quadruple of covariance
=ll (a. x) E (z,
EXe (a, x²)
$ ( (I,
$ L (x,
(z',a')
1 (z',z)
4 (1) (x',x')
2x)
20 (x,
>le (x,
(x,2²)
(z',x')
x(L) (x',x!)
During training, the parameters follow the gradient 0,0(t) (0, D(t). By the induction hypothesis, the traces 98.m then evolve according to the differential equation
(2) oigo. (a)
A (z, (,) (1) (z)) (wil)) D:(t)
VnL
and in the limit as Mı,.... RL-1 + oo, the kernel A (1) (x, x4) converges to a deterministic and fixed limit AmmmAsa (x., ax)). Note that as nr. grows, the go.m (a)) move at rate of l/yaz
(L)
just like the pre-activations =(4)) Even though they move less and less, together they affect (L+1) 1) which follows the differential equation
the trace 90.k
0.go. (x) (1+11(()= ΣV
Agk (1+1))
(s.
where
(x, (x,22) = Σ (L) (I, I'àà (1) (2)) w w(l)
A
nT. m.m!
Σ
-8.m ( (z)
mmm (I, (x)) (all (a')) w'll mike
m.m
1 Σ
H (x)δ (z) (al)((²)) SNx
20

---

Published as a conference paper at ICLR 2020
2
Σ
(L)
m' m'm 2,, (all (a (z') withe walk mik"
n
m.m!
Σ
O (z, 0) (2, (w)) (ann (o') witt wilk mike
T.
m.m!
1
0 (z, 2)0 @am (2)) a (al) 6KK.
a
As 11 1I + 0c, the kernels Oul (2t, x') and (x, (x,x²) converge to their limit and D
'n mmm (x', xz x) vanishes:
R1 (I, -
(1)
Σa a (I, (am) (al)(())) will will
n
m
Σ
(1) (2,,)"))))))) (o)) à (ath w w mnk!
Do.m
n
90,m (1) (x) σ (x)) dEN
T
O (I, (I, x')0 (all(()) (x)) (ath w w(()
mike
o (L) (I,
(x) a (al)((z)) Su
nL
By the law of large numbers, nt oc, initialization A ( (x, - du d (x), where
A = x') ('t (z,zz)
+ Lgū (a))
+ [g0 (a)
+ 14
+ (04)))))
(
+ "4""0
+ 14
+ 1 x (1
During training ( and (L) are fixed in the limit R1,,,, n1-1 - 0o, and the values Io.m (x), am (1)(()) and w to Ald (L) (z, z'), which is therefore asymptotically fixed during training as nn + oo.
m& vary at a rate of 1/ymz which induce a change of the same rate
The next lemma describes the asymptotic limit of the kernel T(4;;
Lemma For any loss with BGOSS and CH), the second moment of the Hession of the realization function HFFL) converges uniformly over jo, T a fized limit as 11.11111-111
T( (x, x²) + 8KK Σ
(000 (2, 2/0 x') 20 (z, x') 1210 (x,))) (x,)' 3(1-1)(xx,0)
21

---

Published as a conference paper at ICLR 2020
Proof. The proof is by induction on the depth L. The case == 1 is trivially true because 00 fz, = 0 for all p. p', k r. For the induction step we observe that
r r R (a,2")
Σ
8y Op2 & (x)0 8n
pp (xy)
PL-P2=1
n
r (L) m (2, z')ō (x)) o willww
molk
nL
1
Dll m.m (4', (H', (2 á (am)(())) (z'))
whllw mi
11
m,w!
1
2 mm m (z, x0, (x) (a (a')) wilh with)
TL
m.m!
=1
1
m.me (I, 1
(L)
ml mm (o', mmmmmmm x)) z) (a)) ö (amn (x') mi
wlll
mil
RL
m,m! m1
2
mome (z, d')ō (z) (xm))())))
(L)
(x')) duz
TEL
we now let the width of the lower layers grow to infinity -1,-. - oo, the tensor Q(4) vanishes and r(b) m. and the NTK @ converge to limits which are non-zero only when m'. As result, the term above converges to
r( do (r, x')6 (x) (a') w will)
mk
@ (a, 2'), (a(h) (x)) 8 (a'))'))
(a')) willd w(L) mk
7
2 nz
(1) (,,'')ō (al) 2
(alh (a') ORA
n
m=1
AL initialization, can apply the law of large numbers as nr. oo such that it converges to (((+1) (2,,") for the kernel r((+1) (z(()) (x,x²) defined recursively by
((tt)(2,,)) (r x' x') (2, (x, + ( (', 25) (x, x²) + (0 (x, x' x')))'' S(L) (x, x!)
and T2 0.
For the convergence during training, proceed similarily to the proof of Lemma 1 the activations a (z) and weights ((()) mk move rate of 1/yoz and the change to r (( is therefore of order 1/vaz and vanishes as - O.
Finally, the next lemma shows the vanishing of the tensor (L) daiks to prove that the
higher moments of S vanish.
Lemma 5. For any loss O' with BGOSS and a (RR), uniformly over [0,
lim (L)
lim 11-1-
a 000 h2. (Iig Tig Dig Vig) ) = 0
Proof. When L 1 the Hessian zero and W daiks (2 Ihe Lig Itay 0.
For the induction step, we write v (((+t1) (L+1) ka, (Fior Tny Dig Dig)
recursively, because mi
contains many terms, we change the notation, writing
Zo for
mg
22

---

Published as a conference paper at ICLR 2020
+ (L) mo (xo; Ti);
3o
01
T2
for Did Am (1, 2x2) and
30
21
322
3/3
mg
mi
m2
mn
m1
M2
m3
for I ((L)
mn.. (Io, #1, 22, 23). The value v (++1)
ko. ( Tai Finy Fig) is then equal to
Σ
vll
22, (2o)) o (all) (z11)
(al (122)
n
mp, m.
Mo
(1)
(3x)
W(LL)
Will
W(L)
Wil) maka
mms
mako
midi
make
2
Σ
Eo
21
21
2/2
122
I3
23
2o
3 (ai)
me
m
m/
m2
m3
mg
mo
(all)
(x1)
(a
(3ra)
Ö (a(h) (xx)
W W
mokoo
W mida
Woll mikks
2
Σ
No
21
822
32
F3
J3
To
(alhd (mo)
(ali) (z1)
mo
m
m2
m2
m3
m3
mo
mo, m2g
ö
m2
(L)
(122)
ms
(I.)
(I:3)
Wi(L) W(l W(L) will moko
miki
maka
maks
+n L
2
Σ
Zo
21
222
33
J3
To
á aalid (200)
(al)
102
m3
mg
vito
mg, mamma.mm...g
(((L)
(122
ams (1r3)
(L)
w motoo Wiyhs
w
will) make
m2
m2kk
+
2
Σ
To
21
x1
T2
1/2
J33
(alte (30)
(a (z1)
mo
m1
M2
m2
m3
mo
mo. mmama
ö
amm (122)
mms
(T.) (a:s)
moki miny wilh), will)
mzkg
myka
+
2
Σ
22
23
I33
JD
à (ahli (10)) (all((a))
me
7722
1/12
mg
mg
m1
ö
(a(h)
(12)
ams (1r)
(L)
w mmoko Whit w w makks
+ng
2
Σ
To
21
822
.2
43
ö (cl)((z)) à (alli(a)) .0
mo
m1
17/
m2
ma
mg
mo, mm.ma
ö
0
m2
(L)
(x22)
à (1) (wa)
(13)
Will) W miki mighs w(L)
make
+0
2
Σ
I1
E3
F3
Fu
F1
④ (alti) (20) ö (())
mi
1112
m3
mg
mg
mi
mg. m2
(a())
(x2)
8 (a())) (1a)
will) w(l)) Whigh w(l) make
mako
+n
2
Σ
To
822
43
F3
To
(athi) (2o) à (al)((a1)
mo
12
m3
ma
mg
0
m2
(L)
(w22
ö (L)) (I3)
Wil will)
makio
makk might w(l)) makes
+nF
2
Σ
To
Z1
32
E3
ö ((( (((hi)))) (2o) o (al)i())))
mo
mi
mi
m2
mg
mo
mo. my.mimm..mg m2
(L)))
(x2)
à
(a())) (ma)
wilk) w(l)) kn Whish w(l)
w(l
+n
2
Σ
x2
x22
x3
Fo
x1
á (00((())) ö (all) (x1)
mc
m2
m3
ma
mi
0
m2
(L)
(x2)
à (L) (Ia)
will) w(l) w(ll) w(l)) magto myky migi makks
23

---

Published as a conference paper at ICLR 2020
Σ
D2
33
2/3
20
T1
1/2
(all) (zo) à (xi)
+
m2
mg
Mg
m0
m
M2
mo, m2222
ö
0
(L)
(22)
ö
mma
(LL)
(17s)
Whl w Wil)) w(l) make
moko mikk
maka
+
2
Σ
20
21
21
22
22
33
(30) 8 (all (x1)) á
m
m
m1
m2
722
m
mm2
(xa2)
C'm
(L) (wa)
Woll),
maghy draks
T1
222
1/2
3/3
J3
Fo
(11h (z0))
(())))))
+
m
m2
m2
mg
mg
mm
ö
(L)
(122)
(L) ma
(Iz)
Welt) whl
mako
makk
Skgks
+
2
21
22
23
á ano (1o0) (al)) à (m1) 20
m7
77A.
772
m3
mg
(1ra)
ö
am (wa)
will) wiigk òeikz "okoo
Σ
F0
21
T2
T3
To mo
aalhd (200) (al)
+.
mg
m1
m1
m
m9 12. m1
(a (1ra)
(all (1a)
will)
with) misk ōkgka
moko
+7
2
Z0
21
22
22
I3
(al (2a) à (al))()))
m
m
1/2
m2
m
mmmmi.mn
ö
a((L) (x22)
à à (2z)
will) w(l) 8koks miki make
F1
22
23
T3
To
a (2) (zo) à (a)()))) (alnd (x2)
+B
m
m2
m3
mg
m
(L) mm (Da)) why
makka makks 0kghs
+n
2
21
22
T33
ö (alh)) (20)) à (ath (a1))
mo
m
m
m3
mp
mm
(L)) (x22
à
(L) (2z)
wi(L) wil)
moko makks Sky
Σ
D2
Z3
To
(x) ā (a1) m1
+n
m1
m
mo
Y. 111
(L)) (122)
(L) (1s))) with will) mikk dagka
à
à
am
+nz
2
Σ
21
F1
T3
④ (alh (()) (2o) ö (all((a))
172
mi
m1
M2
m
m,m.mm mm
à
[(())
(x22)
à (G(L) (xs)) will) will)
maka Bkoks
Z2
T22
Z3
a (mo)) ā (at)
+0
m
m2
m2
m3
m
ö
[L))
(Ir2)
à amm (1s))
(L)
Whlli will) makk Skoks
+n
2
22
23
.7'
.uu
.1
δ (1)( à (alh)
m
my
m3
mg
m
mm
(a(L) (122) ö (alh) (as) wiL wil
mgkk Sky
24

---

Published as a conference paper at ICLR 2020
2 +
Σ
F0
21
0:2
3/3
Tip
ö aa (z0) ā (
mo
m
m
m00
m. 17, m_
(((L) (122)
ā (ath (1:)) ) W Whih
mokio mike òkgka
2
Σ
20
21
22
Z3
à aa (2ro) 8 (11h à (o2) ā (all)(
172
1
m2
m
willk mirks make Skoks
w
2
Σ
222
223
Zig
à (a/h (co)) 8
(zo))
à (a!) à (ath) (2s)
171
1/22
1/
Will) wilk) Skonks
myke makk
2
Σ
222
23
20
21
(E3)) à (a/h à (a u (12)))))) ā ((s))
171
M3
Mg
m
m. ma
W wilh) Skika mako makia
2
Σ
33
370
31
32
a (alli (zoo) à (all) à (alh) (1z) ā (ath) (xs))
m71
m710
1
m
mm
will)
makio with Skgka
Σ
Zo
Z2
I3
à (au)) (al)) à (a (a))(a1)) (all (mza) à (a()(rs)) (ma)
m'
m
mmm!
Shukk Skeka
tnp 2
Σ
21
22
F3
à
(al) à (x1)) à (a (12)) à (L) (Ia) mm
m
m'
m'
m
m.
8kgks O kg
Even though this very large formula one can notice that most terms are "rotation of each other". Moreover, as Ri,,.., nL-1 + 00, all terms containing either an (p(L, an (LL) or
a (L) vanish. For the remaining terms, we may replace the NTKs ( by their limit and as result v Aa, (Ris Tiy Fig. Vig) converges to n2 Σ 2e (zo. 21) 22) (!) 15) (23.20)0 (a!! (zo)) o (a!(()))
(a δ aall (23)) will) min will) w(l)
mka
Σ
5ow (zo, 11 )0 (m, x2) 2 2z)0 (2o)) (all
+n
(1t ā ( (1s)) ) wilh) w(l) deoks
mis mky
Σ
0 (2), 22 0 (22 2o0ō (amh (zo)) (all
tn
(all( ā (aa) mirs will) deaks mig
Σ
0 (Zo, 3 e no (x2, 2y 0 (ns, 0n)ō (a (zo)) (all
t
m
(2t)) ā (zs)) wilhd mnk Seikz (z) 22 2 41) (Σz, 2olō (at (2o)) (ar)
22 Σ tnz
(m0,
(mo, x )e (L)
00
m
à ( (1a) à (1x) ) mkko mky deyka
25

---

Published as a conference paper at ICLR 2020
Σ
(Zo, (20, (x2, 0z) (am (xo) (amh (x1))
+
m
(x2) (1 (Ix) skoRs Oezks
Σ
O [1
82 22) (zy, Fu (xo)) (alh
m
( (12)
(aLh)) ( (Ix)) Seoks Ōkrkz
And all these sums vanish as ny thanks to the prefactor nz² proving the vanishing of y (L+1) d2. (Tig Li T2z Tig) in the infinite width limit.
During training, the activations a!! (x) and weights w mk move at a rate of 1/mr which induces a change to (p(+1)) of order "1 1/2 which vanishes the infinite width limit. L
D ORTHOGONALITY OF AND S
From Lemma and the vanishing of the tensor ( proven in Lemma 2 can easily prove the orthogonality of I and S of Proposition 5
Proposition 5. For any loss C with BGOSS and a E (R)) we have uniformly over [o, T]
lim lim nL
||S|| =0.
100 n1
As a consequence limm 1-000" lim
Tr (1+) [Tr (1 + Irr (S*)] = O.
The Frobenius norm of IS is equal to Proof.
|| p = DY (DY)² (VC- (VC-HY)
P
P
N
nz
0o Jo. (1i,) (oi, forn (2i, 002 fo (122 )c (fit))
Pl Pl.Pa=1
p= $1
ki 12=
N
Σ
Chi (Ir, )ok (xxz yoh (ot Deg 1k., (2r, Fy J da, (Iir Dizs Iit, Zi.
and vanishes as nı,.... n--1 + 00 by Lemma
The k-th moment of the sum Tr (1 + 5 is equal to the sum over all Ir (A1 ...A)) for any word A of A, E (T,S}. The difference Tr ( + ST*) - [Tr (7*) + Tr (S*)] is hence equal to the sum over all mixed words, ie. words A ... A& which contain at least one and one S. Such words must contain two consecutive terms Am+1 one equal to I and the other equal to S. We can then bound the trace by
A
ITr (Ai A+) ||Al ... 1|Am- IIr Am+ II Am44ll ||Ae
which vanishes in the infinite width limit because ||]| and ||S]] are bounded and Am 1l = ||S]]p vanishes.
26


Reviews:
[{"review_id": "SkgmG-iQ9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses the Neural Tangent Kernel (NTK) to presents an asymptotic analysis of the evolution of Hessian of the loss (w.r.t. model parameters) throughout training. The authors leverage the Neural Tangent Kernel to analyze the evolution of the Hessian of the loss w.r.t the model parameters. Specifically the authors show that as the width of the neural networks tend to infinity, the Hessian can be decomposed into two components: (1) one that reflects the initialization of the model parameters and reduces as training progresses; and (2) one that captures the principal directions of the data.\n\nTechnical Soundness: The paper appears to be technically sound, though I did not go through the 14 pages of proofs. \n\nPotential Impact:\nI found the focus of the paper to be quite narrow which I think will negatively affect the impact that this paper could have. The authors take the reader on a notational taxing voyage through the decomposition of the Hessian to finally arrive at a result that is rather intuitive if not entirely obvious. While I learned something about the evolution of the Hessian (in the limit of the infinitely wide NN),  I can not say that this paper will have significant impact on how I think about NN training. \n\nTo address this issue of significance and impact: What novel conclusions about Neural Network learning dynamics can you draw from your analysis? What are the implications for generalization or for future training algorithms? \n\nClarity: Beyond the possibly unduly heavy notation, the paper is rather clear and well written. There is a minor typo in the first equation of Sec. 2.3 (i -> j). \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklmK1PpKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses NTK and techniques from deriving NTK to study asymptotic spectrum of Hessian both at initialization and during training. For understanding neural network’s optimization and generalization property, understanding Hessian spectrum is quite important. \nThis paper gives explicit formula for limiting moments of Hessian of wide neural networks throughout training. \n\nIn detail, Hessian of neural networks can be decomposed into two components, denoted by H = I + S. In the infinite width I is totally described by NTK, and authors show that I and S are asymptotically orthogonal (both at initialization and during training).  Residual contribution is described by S, which captures evolution of Hessian by its first moments Tr (S) since Tr (S^2) remains constant and Tr (S^k ) for k>=3 vanishes. \n\nCorollary 1 has analytic dynamics of moment of Hessian in the case of MSE loss demonstrating power of this paper’s main Theorem.  This is also supported by experiments in Figure 1. \n\nfew comments:\nAuthors should follow format given by ICLR style file. The paper is more dense than typical submission and may have violated page limit (10 pages max) if the style guide line was followed. \nSimilar to the prior comment, the reference section should be cleaned and formatted better. The reference doesn’t count towards page limit and I don’t understand the reason for them to be formatted badly and become eligible. \nIt would be useful if the Figure axes are more legible. \nThere have been many variations of NTK beyond vanilla FC networks(Arora et al. 2019, Yang 2019). Is there a major block for the analysis given in the paper to extend beyond FC networks? \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeJ57o3FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the behavior of the Hessian of the loss of NN during training. There is no question that the subject is relevant, and the paper gives an interesting contribution to the problem.  I do support publication.\n\nMy main criticism is in the presentation of the paper: The title and the first pages are, I believe, really misleading. Unless I am missing an important point, all derivation of the papers are being made in the so-called NTK regime. In fact, this is explicitly stated in page 3 \"In the limit we study in this paper, the NTK is fixed\". While this is an interesting assumption, this is by no mean a trivial one that should be stated casually in the middle of the paper. This is an extremely strong assumption (essentially, one is either in the lazy Regime of [Chizat&Bach] with infinity small learning rate, or in the kernel regime of [Jacot&al] with astronomically large layers. This is a nice setting for mathematics, but by no means a natural one and this limits drastically the conclusion of the paper: what is studied is the behavior of the DNN in the \"Kernel\" regime, or the \"Lazy\" regime. Such a limitation should be explicitly mentioned, at least in the abstract and the introduction, if not in the title itself! It should, also, be mentioned in the conclusion (\"We have given an explicit formula for the limiting moments of the Hessian of DNNs throughout training\"). \n\nIn this respect, with the connection to the Kernel, it make sense that the Hessian is dominated by, essentially, the equivalent of Kernel PCA! This is to be expected, since one is essentially doing a Kernel Ridge Regression with random features.\n\nOther source of confusion (for me):\n* \"This is contrast to the results obtained in [Pennington&Bahri] etc....\" Is the only difference the fact that the network is more than two layers? \n* \"This gives theoretical confirmation of a number of observations about the Hessian\"... I do not believe all these references are in the NTK regimes. (e.g. Chaudhari et al., for instance). \n\nThat being said, the paper is a nice and interesting contribution to the NTK regime, that is the subject of intense studies currently. I do support acceptance if the paper clarify the range of applicability of the its results.\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the spectrum of the Hessian through training, making connections with the NTK limit. While many of the results are perhaps unsurprising, and more empirically driven, together the paper represents a valuable contribution towards our understanding of generalization in deep learning. Please carefully account for the reviewer comments in the final version.
Decision: Accept (Poster)