Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary: The authors of the paper examine how different activation functions affect training of overparametrized neural networks. They do their analysis in a general way such that it includes most activation functions such as ReLU, swish, tanh, polynomial, etc. 

The main point of their analysis is that they examine a matrix called the G-matrix which is described in equation (2), and this (positive semi-definite) G-matrix can determine the rate of convergence to zero of the training error. Namely, the minimum eigenvalue of the G-matrix is inversely proportional to the time required to reach a desired amount of error (Theorem 3.1 (Theorem 4.1 from Du et al. (2019a)). 

The main results separate into two cases: (1) activation functions with a kink (i.e. if the activation function is NOT in C^{r+1}, the space of r+1 continuously differentiable functions, for some finite r) and (2) smooth activation functions.

In the first case, the authors show that the minimum eigenvalue of the G-matrix is large, i.e. bounded away from zero after a few assumptions.

In the second case, the authors show that polynomial activations have many zero eigenvalues, and sufficiently smooth activations such as tanh or swish have many small eigenvalues, if the dimension of the span of data is sufficiently small.

The author’s initial problem setup works on a one-hidden-layer neural network where only the input layer is trained, but provide some extensions in the appendix.

The authors also provide some empirical experiments: one synthetic data, and on CIFAR10. The synthetic data experiments agreed with theory, but the experiment on CIFAR10 did have some gap between theory and experiment, although the CIFAR10 with ReLU experiment agreed with theory.


Stengths: I appreciate the author’s effort in providing needed theoretical analysis on how activation affects training error for deep neural networks. The authors also provide an extensive appendix that provide seemingly full proofs of the theorems (although this reviewer did not go into detail for most of the appendix). The authors also provide experiments that confirm the theory and also provide examples highlighting the gap (which this reviewer sees as a strength).


Weaknesses: A clear weakness of this paper is that the appendix is too long. The authors do provide a proof sketch of the main results and refer to the appendix, but I would have liked to have seen a more focused paper. Having extensions of the main results is nice, but it’s sometimes unclear what is being extended. Perhaps a list of extension would make clear what’s in the appendix.


Other comments: (i) I’d like to an explanation to why it’s called the DZXP setting.
(ii) On page 4, when explaining the M matrix, I think it should be grad_W F (instead of L)

(iii) On page 4, I think there is more to assumption 1, after looking at Allen-Zhu et al. (2019) (https://arxiv.org/pdf/1811.03962.pdf page 4, footnote 5)

(iv) the wording from page 4, “the matrix G^(t) does not evolve from its initial value G^(0)” is a bit awkward. Do you mean G^(t) does not change much from G^(0), and as to goes to infinity then G^(t) goes to G^(0)?
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper aims to characterize for different activation functions the minimum eigenvalue of a certain gram matrix that is crucial to the convergence rate for training over-parameterized neural networks in the lazy regime (small learning rate). On this front, the paper shows that for non-smooth activations the minimum eigenvalue is large under separation assumptions on the data improving on prior work. However, for smooth activations the authors show that the minimum eigenvalue can be exponentially small (or even 0 in case of polynomials) if the data has low-dimensional structure or the network is sufficiently deep. The authors experimentally validate the observations on synthetic data.

Overall, I vote to accept the paper. The paper does a thorough theoretical study of the behavior of the eigenvalues of the matrix corresponding to NTK which is crucial to the NTK analysis. The paper successfully makes the case for non-smooth activations versus smooth activations in the lazy regime. The authors use polynomial approximations and low-dimensionality in an interesting way to show an upper bound on the min eigenvalue for activations approximable by sufficiently low-degree polynomials. The paper is well written, self-contained and well-referenced.

Suggestions/Comments:
1. Please avoid referencing theorems in the appendix that do not have informal statements in the main paper. For example "We sketch the proofs of Theorem J.3, Theorem J.4 and Corollary J.4.1 showing that our results about the limitations of smooth activations are essentially tight when the data is smoothed.": Theorems/Corollary are not mentioned in the main text.
2. In real data as the authors point out, the dimension of the data is much larger than log n. In the setting of dimension being greater than log n, could you discuss how far the lower-bound from Theorem J.4.2 is from the upper bound from Theorem F.9. It would be useful to write it in similar notation.
3. The paper, in its current form, is long and probably hard for a general audience to parse. It would be useful to organize the appendix to emphasize the main techniques and ideas.
review_title: Official Blind Review #2

Meta-review: The article studies the role of the activation function in learning of 2 layer overparaemtrized networks, presenting results on the minimum eigenvalues of the Gram matrix that appears in this type of analysis and which controls the rate of convergence. The article makes numerous observations contributing to the development of principles for the design of activation functions and a better understanding of an active area of investigation as is convergence in overparametrized nets. The reviewers were generally positive about this article. 
Decision: Accept (Poster)