Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies the spectra of neural tangent kernels (NTKs) at large depth -- first let width go to infinity, and then let depth go to infinity. At infinite depth the kernel has the form a*identity+b*(all-one matrix), and the paper studies how the large-depth NTK converges to the limit in three cases: chaotic, ordered, and critical line. The paper draws connection between these behaviors with the trainability and generalization of corresponding neural networks. Furthermore, the difference between CNNs with and without global average pooling is studied.

NTK has been a popular subject of research in deep learning theory, and it's an interesting direction to study the NTK in large depth. However, the exposition is confusing and I'm missing some key points of this paper. Therefore I cannot recommend acceptance at this time. See below for detailed comments.

1. I don't really get how the spectrum of large-depth NTK is connected to generalization. At infinite depth, the NTK is just a trivial kernel Theta^*, as noted in the paper. It is claimed that a finite-depth correction Eqn. (7) "captures the generalization." How exactly does it capture the generalization? Generalization appears to be highly dependent on the data distribution. I don't understand how the paper arrives at its conclusions regarding generalization.

2. The paper (esp. Section 3) is written in a way very unfriendly to someone who is not familiar with previous work, with notation, derivations and conclusions buried in paragraphs. I wish there were some theorems clearly and formally summarizing the conclusions.

3. It's unclear whether the studied regime (large depth, probably even larger with) is relevant in practice. Although there are experimental results provided, the CNN experiments are for the infinite-width NTK. It's unclear how they look like for practical networks.

4. There are numerous typos and grammar errors in the paper, even in abstract and introduction.


------
update:
Thanks to the authors for the response, especially the clarification about what they mean by generalization. Since the concern about the exposition is still present, I can only update my rating to "weak reject." I hope the authors could further improve the exposition of this paper.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies the evolution of Neural Tangent Kernel (NTK) at large-depth regimes. By analyzing the conditional number and eigenvalues, they identify three phases of hyper-parameters; 1) In the chaotic phase NTK converges to an identity matrix, which is easy to train but hardly to generalize. 2) In the ordered phase NTK converges to an all-one matrix, which is hard to train but generalizes well. 3) In the critical phase the conditional number converges to a constant. Furthermore, they also analyze the influence of pooling and flattening in CNNs and identify potential regimes where pooling hurts the generalization. They conduct empirical experiments to supporting their theoretical analyses.

However, I think this paper is worth of more revisions because many theoretical analyses are unjustified. And some potential typos makes the analyses even more difficult to understand.

1) It looks to me that Eq(2) and Eq(6) are contradictory, where T already contains sigma_w and simga_b in Eq(2) but re-multiplied in eq(6).
2) The paper analyzes the dynamics by assuming the variances of inputs are q*, which is debatable. The variance q^l also evolves with the depth increases. It is unclear whether the condition number will change if you takes the evolution of q^l into considered.
3) It is unclear how Eq(9) comes directly from Eq(6), and there aren't any rigorous proofs in the Appendix. Similarly for eq(14).
4) In the paragraph below Eq(11) the paper states that \Theta* becomes an all one-matrix. However, Eq(11) states the diagonal converges to q*/(1-xi_1), but the paragraph below Eq(9) states the off-diagonal converges to q*_{ab}/(1- xi_c). Because q*=q*_{ab} as you stated nearby, do you mean xi_1 = xi_c ? 
5) In the first paragraph of Section 3.3, p^l = q* and p^l=l q*. 
6) In the 2nd contribution, you mentioned "eigenvector correlation", while I cannot find anywhere else introducing this.
7) The plots of Figure 1(b) should behave like convex if the kappa really evolves like x_1^l / l. However it is concave. 
8) In the first experiment, you state "To confirm that the maximal feasible learning rate are ... 2/(lambda_max)". However, learning rates are never discussed in this paper. It is confusing why this experiment is useful. 

Generally speaking, I think the paper needs careful revisions to support its theoretical analyses. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies the relation between trainability and generalization ability in deep neural networks. In the theoretical analysis, the authors used the Neural network Gaussian process (NNGP) kernel and Neural Tangent kernel (NTK). The paper clarified that the spectrum of the NTK and NNGP has an important role in investigating the generalization and trainability, i.e., the condition number of the NTK. Some numerical experiments showed an agreement of theory with the practical behavior of learning algorithms. 

In this paper, some existing theoretical results on deep neural networks were combined to extract new insight. Thought the attempt of this paper is interesting, the readability of the paper is not necessarily high. 

- In equation (2), the operator T is defined as the kernel K(x,x'). However, the definition seems different from that in equation (8).  The authors need to make clear the definition of T.
- What is the "DC" mode in the sentence above the equation (15)? 
- Is the derivation of the left part in equation (9) straightforward? How was the second term, chi_1 q^* p^(ell), derived?  I'm not sure how the dot{T} was dealt with. The argument below equation (3) should be used? 
review_title: Official Blind Review #2

Meta-review: The paper investigates the trainability and generalization of deep networks as a function of hyperparameters/architecture, while focusing on wide nets of large depth; it aims to characterize regions of hyperparameter space where networks generalize well vs where they do not; empirical observations are demonstrated to support theoretical results. However, all reviewers agree that, while the topic of the paper is important and interesting, more work is required to improve the readability and clarify the exposition to support the proposed theoretical results.
 
Decision: Reject