Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper uses the Neural Tangent Kernel (NTK) to presents an asymptotic analysis of the evolution of Hessian of the loss (w.r.t. model parameters) throughout training. The authors leverage the Neural Tangent Kernel to analyze the evolution of the Hessian of the loss w.r.t the model parameters. Specifically the authors show that as the width of the neural networks tend to infinity, the Hessian can be decomposed into two components: (1) one that reflects the initialization of the model parameters and reduces as training progresses; and (2) one that captures the principal directions of the data.

Technical Soundness: The paper appears to be technically sound, though I did not go through the 14 pages of proofs. 

Potential Impact:
I found the focus of the paper to be quite narrow which I think will negatively affect the impact that this paper could have. The authors take the reader on a notational taxing voyage through the decomposition of the Hessian to finally arrive at a result that is rather intuitive if not entirely obvious. While I learned something about the evolution of the Hessian (in the limit of the infinitely wide NN),  I can not say that this paper will have significant impact on how I think about NN training. 

To address this issue of significance and impact: What novel conclusions about Neural Network learning dynamics can you draw from your analysis? What are the implications for generalization or for future training algorithms? 

Clarity: Beyond the possibly unduly heavy notation, the paper is rather clear and well written. There is a minor typo in the first equation of Sec. 2.3 (i -> j). 

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper uses NTK and techniques from deriving NTK to study asymptotic spectrum of Hessian both at initialization and during training. For understanding neural network’s optimization and generalization property, understanding Hessian spectrum is quite important. 
This paper gives explicit formula for limiting moments of Hessian of wide neural networks throughout training. 

In detail, Hessian of neural networks can be decomposed into two components, denoted by H = I + S. In the infinite width I is totally described by NTK, and authors show that I and S are asymptotically orthogonal (both at initialization and during training).  Residual contribution is described by S, which captures evolution of Hessian by its first moments Tr (S) since Tr (S^2) remains constant and Tr (S^k ) for k>=3 vanishes. 

Corollary 1 has analytic dynamics of moment of Hessian in the case of MSE loss demonstrating power of this paper’s main Theorem.  This is also supported by experiments in Figure 1. 

few comments:
Authors should follow format given by ICLR style file. The paper is more dense than typical submission and may have violated page limit (10 pages max) if the style guide line was followed. 
Similar to the prior comment, the reference section should be cleaned and formatted better. The reference doesn’t count towards page limit and I don’t understand the reason for them to be formatted badly and become eligible. 
It would be useful if the Figure axes are more legible. 
There have been many variations of NTK beyond vanilla FC networks(Arora et al. 2019, Yang 2019). Is there a major block for the analysis given in the paper to extend beyond FC networks? 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper discusses the behavior of the Hessian of the loss of NN during training. There is no question that the subject is relevant, and the paper gives an interesting contribution to the problem.  I do support publication.

My main criticism is in the presentation of the paper: The title and the first pages are, I believe, really misleading. Unless I am missing an important point, all derivation of the papers are being made in the so-called NTK regime. In fact, this is explicitly stated in page 3 "In the limit we study in this paper, the NTK is fixed". While this is an interesting assumption, this is by no mean a trivial one that should be stated casually in the middle of the paper. This is an extremely strong assumption (essentially, one is either in the lazy Regime of [Chizat&Bach] with infinity small learning rate, or in the kernel regime of [Jacot&al] with astronomically large layers. This is a nice setting for mathematics, but by no means a natural one and this limits drastically the conclusion of the paper: what is studied is the behavior of the DNN in the "Kernel" regime, or the "Lazy" regime. Such a limitation should be explicitly mentioned, at least in the abstract and the introduction, if not in the title itself! It should, also, be mentioned in the conclusion ("We have given an explicit formula for the limiting moments of the Hessian of DNNs throughout training"). 

In this respect, with the connection to the Kernel, it make sense that the Hessian is dominated by, essentially, the equivalent of Kernel PCA! This is to be expected, since one is essentially doing a Kernel Ridge Regression with random features.

Other source of confusion (for me):
* "This is contrast to the results obtained in [Pennington&Bahri] etc...." Is the only difference the fact that the network is more than two layers? 
* "This gives theoretical confirmation of a number of observations about the Hessian"... I do not believe all these references are in the NTK regimes. (e.g. Chaudhari et al., for instance). 

That being said, the paper is a nice and interesting contribution to the NTK regime, that is the subject of intense studies currently. I do support acceptance if the paper clarify the range of applicability of the its results.







review_title: Official Blind Review #2

Meta-review: This paper studies the spectrum of the Hessian through training, making connections with the NTK limit. While many of the results are perhaps unsurprising, and more empirically driven, together the paper represents a valuable contribution towards our understanding of generalization in deep learning. Please carefully account for the reviewer comments in the final version.
Decision: Accept (Poster)