Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper studies the limiting behavior of neural tangent kernels when the depth grows to infinity. They show that the obtained limit kernels are trivial (a constant) unless one uses 'edge of caos' initialization, in which case they are close to the identity. The authors compare the convergence for different activations, showing a slower convergence (hence better propagation) for some piecewise smooth activations compared to ReLU. For residual networks, the 'edge of caos' behavior is claimed to be in place regardless of the initialization.

The detailed characterization of these limiting kernels for different activations and initializations is interesting. Yet, the work is quite incremental and of limited significance, in that such EOC initialization was known to be important for controlling propagation of both activations and gradients with such networks, so it is no surprise that the NTK has similar properties, given that it basically consists of the sum of similar gradient covariances.

some comments:
- title: isn't the "mean-field behavior" already subsumed in the definition of NTK?
- contribution 4.: "more suitable" seems a bit strong if it's just a log(L) factor?
- after lemma 1: generalization may not provide a strong argument here unless further discussed: the constant kernel is obviously bad for learning anything, but the EOC limiting kernel is also pretty bad for predicting anything outside the training data
- section 3: typo "ourselves"
- prop 3: specify that phi is the relu
- the proof of prop 3 should be more detailed. Also, it is not obvious that you are giving the correct formula for the NTK of resnets
- Table 1: is it meaningful to compare performance with such low accuracies?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper studies the role of weight/bias variance of neural network’s trainability via analyzing large depth behaviour of Neural Tangent Kernels(NTK). 

Recently NTK has been a popular topic of study in theoretical deep learning as it describes exact gradient descent dynamics of infinitely wide networks. Original NTK paper (Jacot et al. (2018)) and other follow up papers often gloss over role of weight/bias scales whereas in the setting of signal propagation or NNGP covariance, Schoenholz et al. (2017), Lee et al. (2018), Novak et al. (2019), Hayou et al. (2019) have shown understanding initialization scale is quite important as networks becomes deeper. This paper brings those analysis for NTK and discovers few interesting results. 

The good weight/bias initialization scale that propagates signal for very deep network is denoted edge of chaos (EOC). The authors show that 1) for fully connected feed forward networks, outside initialization edge of chaos (EOC) the NTK converges exponentially to constant kernel. This indicates non-trainability. 2) With EOC initialization the convergence is polynomial and the NTK remains invertible for very large depth implying trainability. 3) Certain class of activation function (denoted class S, including ELU/Tanh/Swish) it has even slower convergence(O(log(L)/L)) compare to ReLU (O(1/L)). 4) Residual FC networks NTK has polynomial convergence for all weight and bias variance. 

In terms of novelty, the paper is combining existing techniques and objects to study large depth behaviour of NTK. However the contribution of the paper is interesting and worth the ICLR audience to know about, especially with the current surge of interest in NTK.

One main weakness is that the experiments are weak and have a very weak connection to the early part of the paper. Section 5.1 is direct convergence comparison, which provides fair evidence. In section 5.2, I’m not certain whether experiments displayed connects to asymptotic NTK analysis. First of all, in order to obtain NTK in the infinite width limit, one has to use `NTK parameterization’ where one scales out 1/sqrt(fan_in) in network definition and not in weight initialization. It seems (by mention of He/Glorot init, and choice of learning rate O(1e-3/1e-4)) the authors experimented with standard parameterization. In this case the connection to very deep behaviour of NTK is not straightforward to actual network training since the training dynamics will be different.  I suggest authors try experiments in NTK parameterization and see if results are similar.


Few comments:
It should be emphasized that the infinite width is taken first before taking infinite depth limit. There are subtle effects when depth/width are taken to infinity at the same time. The analysis on asymptotic behaviour of NTK at infinite depth is only valid after width is taken to infinity. 

First sentence of the abstract is too strong in the sense that NTK has limitations. NTK certainly does not work for any kind of network, so I suggest authors to down tone the sentence. 

P4 paragraph after Lemma1 `'K^L(x, x’) = cte’ is a typo?

In section 5.2, the authors’ claim full batch GD is practically impossible. One could accumulate gradients over minibatch to simulate full batch GD. 

P13 typo in last paragraph, `  NTKl'

EDITS POST AUTHOR RESPONSE:
Regarding point 5) I was referring to solving the memory problem. In practice, if one is sampling without replacement, there is no randomness accumulating gradient of the full epoch. 

I also thank the authors for the clarification. My score still remains the same. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper discussed the property of the NTK with the increasing depth L with the help of the Edge of Chaos initialization. The authors show that if deep neural networks are not properly initialized, the NTK can have a large condition number, which leads to the poor performance of training and generalization. Moreover, the authors also introduce the conditions that make the neural network trainable by decreasing the convergence rate to a nearly constant kernel w.r.t the depth L by using the specific Edge of Chaos initialization as well as different activations and use residual connections. Experiment results show that the theoretical results are well aligned with the practice.

Detailed Comments:
1. At the paragraph after Lemma 1, the authors claimed that f_t(x) is entirely given by f_0(x), which means a generalization error of order O(1). This is a little inaccurate I think, as NTK can only characterize the training process and does not directly indicate generalization. Also, as t to infinity, the coefficient of initialization f_0(x) can be arbitrary small and thus the f_t(x) is entirely decided by initialization is not accurate as well. To analyze generalization with NTK, we need a little more, see [1]. This is not some core issue, but I think the authors should make the description more accurate.
2. The authors should explain what richer limiting NTK means at the end of the Sec 3. It is unclear how the convergence rate of the NTK related to the richer limiting NTK.
3. A typo in the first paragraph in Sec 4, ouerselvs to ourselves.
4. In definition 1, the second-order derivative of \phi is defined via the indicator function 1_{A_i}? It’s better to use another notation like \mathbf{1} or \mathbb{1} to make it more clear.
5. I think for clarity the authors should give a formal definition of ANTK, or if it is unnecessary, better directly use the original NTK. Also, it is better to show the invertibility of the NTK is equivalent to ANTK more formally before Proposition 2. This conclusion is not so straightforward at the first view. Meanwhile, I think the invertibility of ANTK can just indicate a bad condition number of NTK? I don’t think it directly related to the invertibility of NTK.
6. There are several typos in the appendix. Please go through the appendix and fix them.
7. In the proof of Lemma 3 and Lemma 4, I don’t get why |a_l| < l + |a_0|. It may depends on the property of q? For sufficiently large q, if a_0=0, a_1 = q + O(1) which is not necessarily smaller than l=1? But to get a_l/l bounded, it is not necessary to use this. Also, I think it’s better not omit the constant in the dynamic system of Lemma 3. 
8. Better use the Stirling’s approximation rather than k to infinity in the calculation of k!/(k-\alpha-1)!
9. The lemma number of Appendix C is in a chaos. And I don’t quite get the idea behinds Appendix C. Is it relevant to the contents in the main text?
10. I also feel the content in Appendix D is unnecessary for this paper.

Overall, this paper is interesting and gives a unified perspective on the recently developed NTK and Edge of Chaos initialization. It also sheds light on the impact of different activation function on NTK by generalizing the results of [2]. I feel this paper can help the communities have more understanding on the properties of deep neural networks. However, this paper can be better if the authors can polish their paper and reorganize the appendix part.

[1] Arora, Sanjeev, et al. "Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks." International Conference on Machine Learning. 2019.
[2] Hayou, S., Doucet, A. & Rousseau, J.. (2019). On the Impact of the Activation function on Deep Neural Networks Training. Proceedings of the 36th International Conference on Machine Learning, in PMLR 97:2672-2680

review_title: Official Blind Review #2

Meta-review: This paper focuses on studying the impact of initialization and activation functions on the Neural Tangent Kernel (NTK) type analysis. The authors claim to make a connection between NTK and edge of chaos analysis. The reviewers had some concern about (1) impact of smooth activations "any NTK-based training method for DNNs should use a Smooth Activation Function from the class S and the network should be initialized on the EOC" (2) proofs of residual networks (3) and why mixing NTK with EOC is interesting. Some of these concerns were addressed in the response. I do share the reviewer concerns about (2). The authors need to give a clear proof. I think this combination of NTK and EOC could be interesting but needs to be better motivated. As a result I do not recommend publication.
Decision: Reject