Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies overparameterized fully-connected neural networks trained with squared loss. The authors show that the resulting network can be decomposed as a sum of the solution of a certain interpolating kernel regression and a term that only depends on initialization. Based on this, the authors also derive a generalization bound of deep neural networks by transferring it to a kernel method. My major concern about this paper is the novelty and significance of its results:

In terms of connection to NTK, It seems that the connection between neural networks trained with squared loss and the result of NTK-based kernel regression has already been well-studied by 

Arora, Sanjeev, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. "On exact computation with an infinitely wide neural net." arXiv preprint arXiv:1904.11955 (2019).

which is a missed citation. Without a clear explanation on the difference between the submission and this paper above, I don’t think this paper is ready for publication.

In terms of generalization, it is also very difficult to judge whether this paper's result is novel. In fact this paper misses almost all citations on generalization bounds for neural networks. Moreover, the generalization bound given in this paper does not seem to be very complete and significant, since the authors do not show when can L_{test}^{int} be small. To demonstrate the novelty and significance of the result, the authors should at least compare their generalization result with the following generalization bounds for over-parameterized neural networks in Section 4: 

Allen-Zhu, Zeyuan, Yuanzhi Li, and Yingyu Liang. "Learning and generalization in overparameterized neural networks, going beyond two layers." arXiv preprint arXiv:1811.04918 (2018).
Cao, Yuan, and Quanquan Gu. "A generalization theory of gradient descent for learning over-parameterized deep relu networks." arXiv preprint arXiv:1902.01384 (2019).
Arora, Sanjeev, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. "Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks." arXiv preprint arXiv:1901.08584 (2019).
Cao, Yuan, and Quanquan Gu. "Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks." arXiv preprint arXiv:1905.13210 (2019).

Overall, I suggest that the authors should make a clear discussion on the relation of this paper to many existing works mentioned above. As long as the authors can give a convincing demonstration of the novelty and significance of their results, I will be happy to increase my score.

A minor comment: how can the bound in Theorem 3 be derived based on Theorem 2? Should there be a constant factor in the bound?


review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper studies the solution of neural network training in the NTK regime. The trained network can be written as the sum of two terms --- the first is the minimum RKHS norm interpolating solution, and the second term depends on the initialization. When the initialization scale is small, the second term almost vanishes, but when the initialization scale is large, it's likely that the second term becomes very large, leading to worse generalization.

The technical contribution of this paper is pretty low. The most important formula is (14), which only appears in the second half of the paper (the first half of the paper is almost all known results). The bounds in later part of the paper are also straightforward. Moreover, another paper https://arxiv.org/abs/1905.07777 already studied the same question and showed that non-zero output can increase the generalization error.


-----------
update:
I have read the authors' response. My assessment stays the same since I still think that the technical contribution of this paper is quite limited.

Also there is a negative effect of using small init, which the authors might have overlooked: when the init is smaller, you'd need a larger width for the NN to be in the NTK regime. See e.g. "Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks. Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruosong Wang. ICML 2019".
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper considers the impact of initialization bias on test error in strongly overparameterized neural networks. The study uses tools from recent literature on the generalization of overparameterized neural networks, i.e. neural tangent kernels and interpolating kernel method, to provide useful insights on how the variance of weights initialization affects the test error. I have a few questions about theoretical results, but the paper has a convincing experiment that supports its theoretical claims. Addressing the following points will improve the exposition of the paper. 
1. Please provide a little hint on how Lemma 2 rewrites the equation (13) for linearized function for easier readability without referring to the Appendix.
2. In the case of cross-entropy error, would the effect be similar? Could this be verified with a similar experiment as for MSE?
3. To what extent this result is observed in not as strongly overparameterized settings? In other words, it would be interesting to see what happens if you fix the architectural choice while increasing the number of training parameters, how long does the test error effect persist?

Minor remark:
- a few typos are present on pages 4, 5, 7, 8
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: 
[Summary]
This paper studies the impact of initialization noise on the theories of wide neural networks in the Neural Tangent Kernels (NTK) regime. The paper proves that the difference between the trained neural net and the kernel interpolator (with the NTK) can be bounded by O(\sigma^L + 1/\sqrt{m}), where \sigma^2 is the initializing variance of each individual weight entry. Relationships between the generalization error of these two functions are derived from the above bound.

[Pros]
The general message that this paper conveys is interesting -- the initial network f_{\theta_0}(x), which is typically omitted (or made small by making \sigma small) in NTK analyses, can deviate the converged NN from the kernel interpolator in terms of generalization error.

[Cons]
There are fundamental mistakes in the statements/proofs of Theorem 2, 3, 4:
-- Theorem 2: the statement is “whp over W, the bound … holds uniformly for x”. The proof relies on Lemma 3, whose statement is also uniform over x, but the proof applies the Markov inequality *for a single x* and is thus valid only for a single x. (As it’s Markov, it seems not sensible to apply the union bound upon it.)

-- Theorem 3: the difference between L^NN_test and L^int_test should be on the order of (\sigma^L + 1/\sqrt{m}) rather than it squared. To bound the difference in squared loss we have a^2 - b^2 <= O(1) * |a-b| (if a, b are bounded by O(1)). We don’t have a^2 - b^2 <= |a - b|^2.

-- Theorem 4: J(X_test) as defined is a vector whose dimension grows with the number of test data points, where the theorem requires it to be a scalar. Indeed the treatment of test data as a fixed matrix (rather than samples from a distribution) is already a bit atypical.

***

I have read the authors' rebuttal and the other reviews, and I'm glad to see the issues with Theorem 3 and 4 pointed out above are fixed in the revision. However, I also agree with the other reviewers that the paper in the present stage has not yet demonstrated sufficient technical contributions, and thus I am keeping my original evaluation.
review_title: Official Blind Review #1

Meta-review: This paper proves that fully-connected wide ReLU-NNs trained with squared loss can be decomposed into two parts: (1) the minimum complexity solution of an interpolating kernel method, and (2) a term depends heavily on the initialization. The main concerns of the reviewers include (1) the contribution are not significant at all given prior work; (2) flawed proof,  and (3) lack the comparison with prior work. Even the authors addressed some of the concerns in the revision, it still does not gather sufficient support from the reviewers after author response. Thus I recommend reject.
Decision: Reject