Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Overview: This work is an interesting work to understand the generalization capabilities of a two layered neural network in a high dimensional setting (samples, features and neurons tend to infinity). It studies the conditions under which the "double descent phenomenon" may be observed.

Summary: The work shows that in two layered neural networks with non-linearity
1) the double descent phenomenon of the bias-variance decomposition may be observed when the second layer weights are optimized assuming that the first layer weights are constant.
2) the bias-variance decomposition does not exhibit double descent when optimizing only the first layer with both vanishing and non-vanishing initialization of weights.
3) For vanishing initalization of weights for the first layer with non-linear activation , the gradient flow solution is asymptotically close to a two layered linear network. It is independent of overparametrization. However, the condition for this is smooth activation and the result does not hold for ReLU activation.
4) For non-vanishing initilization of the weights for the first layer with non-linear activation, the gradient flow solution is well approximated by a kernel model. However, the risk is independent of overparametrization.

I believe this is an interesting work that needs to be accepted.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper provides exact bounds on the risk when training a two-layer neural network in an asymptotic regime. Namely, the paper considers training under the square-loss objective, a two-layer neural network with $h$ hidden units on inputs of dimension $d$ and training on $n$ samples. The asymptotic regime is considered by making all of $d$, $h$, $n$ go to $\infty$, in a way that the ratio $d/n$ approaches $\gamma_1$ and the ratio $h/n$ approaches $\gamma_2$.

This paper considers the following scenarios of training described below, where the data is generated from a linear model on Gaussian inputs and with a zero-mean noise. The emphasis of the results is on understanding when a "double descent" type phenomenon occurs ("Double descent" is a recently coined phenomenon in literature where the risk, as a function of the "complexity of the model", initially has a classical U-shape behavior, but eventually decreases again once the complexity of the model exceeds the number of training points.)

1. Training only the second layer: The risk is first decomposed into a bias and a variance term. An exact bound on the variance term of the risk is obtained. While the exact nature of the bound is rather complex to parse, the takeaway is that a double descent phenomenon is observed in terms of $\gamma_2$, namely, the risk blows up when $h \approx n$, but decreases as $h$ is increased beyond $n$.

2. Training only the first layer: Two different regimes are considered here, depending on the scale of initialization, called "vanishing" and "non-vanishing" initializations. In both regimes, the risk is independent of $\gamma_2$, that is, the risk does not depend on number of hidden units (although the risk bounds are different and there is an additional assumption in the case of non-vanishing initialization to ensure that the initialized network computes the zero function). In other words, a "double descent" phenomenon is not observed in this setting.

Recommendation:
I recommend "weak acceptance". The paper extends prior works that obtain asymptotic risk bounds on linear models to the setting of two-layer neural networks (where only one layer is trained).  However, I am unable to assess the technical novelty of this work as it seems to heavily rely on prior work which in turn use techniques from random matrix theory.

Technical Comments:
- I felt that while it is valuable to have exact bounds on the risk, the form of the bounds are quite complex and hard to parse (especially in Thm 4, case of training only the second layer). Moreover, these bounds are just in the case where the teacher model is linear and while it is claimed that this could be relaxed to a more general class of functions, the specific bounds might change drastically. So any insights on the nature of these bounds will be valuable, especially with some comments on how these bounds change if the teacher model is itself realized as a 2-layer neural network.
- The parameter count of a 2-layer network with $h$ hidden units and input dimension $d$ is $O(dh)$. So perhaps it makes sense to study an asymptotic regime where $dh/n$ approaches $\gamma$, instead of both d and h growing linearly in n. While this issue is hinted at in the discussion section, I don't understand the statement "the mechanism that provably gives rise to double descent from previous works Hastie et al. (2019); Belkin et al. (2019) might not translate to optimizing two-layer neural networks."
- Another future direction that could be included in discussions is the setting where both layers are trained simultaneously.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The authors study the generalization error of two-layer neural nets, where an asymptotic point of view is taken. Their main results can be summarized as follows.
1. If only the second layer is optimized, they observe the double-descent phenomenon.
2. However, if only the first layer is optimized, the double-descent is not observed.
This shows that recent results for certain linear models (e.g. Song, Montanari 2019) do not directly transfer to neural networks. As the authors point out, however, if a different scaling is used in the asymptotics, double descent might still be observed.

I see the following strengths of the paper. 
-This is a very well-written paper with a clear message.
-The result is important and gives new insights into the generalization properties of neural networks.

In my view, this is an interesting contribution, which should be accepted. 

---------

Thank you for your response. I will leave the rating unchanged.
review_title: Official Blind Review #2

Meta-review: This paper focuses on studying the double descent phenomenon in a one layer neural network training in an asymptotic regime where various dimensions go to infinity together with fixed ratios. The authors provide precise asymptotic characterization of the risk and use it to study various phenomena. In particular they characterize the role of various scales of the initialization and their effects. The reviewers all agree that this is an interesting paper with nice contributions. I concur with this assessment.  I think this is a solid paper with very precise and concise theory. I recommend acceptance.
Decision: Accept (Spotlight)