Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: In this paper, the authors seek to examine carefully some assumptions investigated in the theory of deep neural networks. The paper attempts to answer the following theoretical assumptions: the existence of local minima in loss landscapes, the relevance of weight decay with small L2-norm solutions, the connection between deep neural networks to kernel-based learning theory, and the generalization ability of networks with low-rank layers.

We think that this work is timely and of significant interest, since theoretical work on deep learning has made significant progress in recent years.

Since this paper seeks to provide an empirical study on the assumptions in deep learning theory, we think that the results are somehow weak as the paper is missing extensive analysis, using several well-known datasets and several deep architectures and settings. For example, only the CIFAR-10 dataset is considered in the paper, and it is not clear whether the obtained results will generalize to other datasets. This also goes to the neural network architecture, as only MLP is considered to answer the assumption about the existence of suboptimal minima, while only ResNet is considered to study the generalization abilities with low-rank layers. We think that this is not enough for a paper that tries to provide an empirical study.

------- 
Reply to rebuttal

We thank the authors for taking into consideration our previous comments and suggestions, including going beyond MLP and adding experiments on other datasets. For this reason, we have increased the rating from "Weak Accept" to "Accept".
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors look at empirical properties of deep neural networks and discuss their connection to past theoretical work on the following issues:

* Local minima: they give an example of setting where bad local minima (far from the global minimum) are obtained. More specifically, they show such minima can be obtained by initializing with large random biases for MLPs with ReLU activation. They also provide a theoretical result that can be used to find a small set of such minima. I believe this is a useful incremental step towards a better understanding of local minima in deep learning, although it is not clear how many practical implications this has. One question that would ideally be answered is: in practical settings, to what degree does bad initialization cause bad performance specifically due to bad minima? (as opposed to, say, slow convergence or bad generalization performance).

* Weight decay: the authors penalize the size of the norm of the weights as it diverges from a constant, as opposed to when it diverges from 0 as is normally done for weight decay. They show that this works as well or better than normal weight decay in a number of settings. This seem to put into question the belief sometimes held that solutions with smaller norms will generalize better.

* Kernel theory: the authors try to reproduce some of the empirical properties predicted in the Neural Tangent Kernel paper (Jacot et al., 2018) in particular by using more realistic architectures. The results, however, do not appear very conclusive. This might be the weakest part of the paper, as it is hard to draw anything conclusive from their empirical results.

* Rank: The authors challenge the common belief that low rank provides better generalization and more robustness towards adversarial attacks. When enforcing a low or high rank weight matrices during training on ResNet-18 trained on CIFAR-10, the two settings have similar performance and are similarly robust to adversarial attacks, showing at least one counter example.

I think overall this is a useful although somewhat incremental paper, that makes progress in the understanding of the behavior of neural networks in practice, and can help guide further theoretical work and the  development of new and improved training techniques and initialization regimes for deep learning.

Other comments/notes:
* minor: the order of the last 2 sub topics covered (rank and NTK) is flipped in the introduction, compared to the abstract and the order of the chapters
* in the table confidence intervals are given, it would be nice to have more details on how they are computed, (e.g. +- 1.96 * std error)
* how is the constant \mu in the norm-bias chosen?
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The authors seek to challenge some presumptions about training deep neural networks, such as the robustness of low rank linear layers and the existence of suboptimal local minima. They provide analytical insight as well as a few experiments.

I give this paper an accept. They analytically explore four relevant topics of deep learning, and provide experimental insight. In particular, they provide solid analytical reasoning behind their claims that suboptimal local minima exist and that their lack of prevalence is due to improvements in other aspects of deep networks, such as initialization and optimizers. In addition, they present a norm-bias regularizer generalization that consistently increases accuracy. I am especially pleased with this, as the results are averaged over several runs (a practice that seems to be not so widespread these days). 

If I were to have one thing on my wish list for this paper, it would be the small issue of having some multiple experiment version of the local minima experiments (I understand why it is not all that necessary for the rank and stability experiments).

Nevertheless, I think this paper gives useful insight as to the behavior of deep neural networks that can help advance the field on a foundational level.
review_title: Official Blind Review #3

Meta-review: The authors take a closer look at widely held beliefs about neural networks. Using a mix of analysis and experiment, they shed some light on the ways these assumptions break down. The paper contributes to our understanding of various phenomena and their connection to generalization, and should be a useful paper for theoreticians searching for predictive theories.
Decision: Accept (Spotlight)