Title: Truth or backpropaganda? An empirical investigation of deep learning theory
Authors: Micah Goldblum
Jonas Geiping
Avi Schwarzschild
Michael Moeller
Tom Goldstein
Venue: ICLR 2020
Keywords: Deep learning
generalization
loss landscape
robustness

Abstract: We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike.  In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that small-norm parameters are not optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays a role;  (4) find that rank does not correlate with generalization or robustness in a practical setting.

TL;DR: We call into question commonly held beliefs regarding the loss landscape, optimization, network width, and rank.

Review discussion:
The authors seek to challenge some presumptions about training deep neural networks, such as the robustness of low rank linear layers and the existence of suboptimal local minima. They provide analytical insight as well as a few experiments.
I give this paper an accept. They analytically explore four relevant topics of deep learning, and provide experimental insight.
In particular, they provide solid analytical reasoning behind their claims that suboptimal local minima exist and that their lack of prevalence is due to improvements in other aspects of deep networks, such as initialization and optimizers.
In addition, they present a norm-bias regularizer generalization that consistently increases accuracy. I am especially pleased with this, as the results are averaged over several runs (a practice that seems to be not so widespread these days).
If I were to have one thing on my wish list for this paper, it would be the small issue of having some multiple experiment version of the local minima experiments (I understand why it is not all that necessary for the rank and stability experiments).
Nevertheless, I think this paper gives useful insight as to the behavior of deep neural networks that can help advance the field on a foundational level.
We appreciate the positive feedback, and we thank the reviewer for the thoughtful comments. We have added results from further suboptimal minima experiments to the appendix.
The authors look at empirical properties of deep neural networks and discuss their connection to past theoretical work on the following issues:
* Local minima: they give an example of setting where bad local minima (far from the global minimum) are obtained. More specifically, they show such minima can be obtained by initializing with large random biases for MLPs with ReLU activation. They also provide a theoretical result that can be used to find a small set of such minima. I believe this is a useful incremental step towards a better understanding of local minima in deep learning, although it is not clear how many practical implications this has. One question that would ideally be answered is: in practical settings, to what degree does bad initialization cause bad performance specifically due to bad minima? (as opposed to, say, slow convergence or bad generalization performance).
* Weight decay: the authors penalize the size of the norm of the weights as it diverges from a constant, as opposed to when it diverges from 0 as is normally done for weight decay. They show that this works as well or better than normal weight decay in a number of settings. This seem to put into question the belief sometimes held that solutions with smaller norms will generalize better.
* Kernel theory: the authors try to reproduce some of the empirical properties predicted in the Neural Tangent Kernel paper (Jacot et al., 2018) in particular by using more realistic architectures. The results, however, do not appear very conclusive. This might be the weakest part of the paper, as it is hard to draw anything conclusive from their empirical results.
* Rank: The authors challenge the common belief that low rank provides better generalization and more robustness towards adversarial attacks.
When enforcing a low or high rank weight matrices during training on ResNet-18 trained on CIFAR-10, the two settings have similar performance and are similarly robust to adversarial attacks, showing at least one counter example.
I think overall this is a useful although somewhat incremental paper, that makes progress in the understanding of the behavior of neural networks in practice, and can help guide further theoretical work and the  development of new and improved training techniques and initialization regimes for deep learning.
Other comments/notes:
* minor: the order of the last 2 sub topics covered (rank and NTK) is flipped in the introduction, compared to the abstract and the order of the chapters
* in the table confidence intervals are given, it would be nice to have more details on how they are computed, (e.g. +- 1.96 * std error)
* how is the constant \mu in the norm-bias chosen?
Thank you for your thoughtful input on our work. We address your comments in order:
* Our work here is focused on finding suboptimal minima, and we show that certain poor initializations motivated by theory can lead to this.  We agree that suboptimal local minima which arise from bad initializations in standard practice would be interesting to study in future work.
* We have changed the conclusion of the NTK section to more clearly discuss and conceptualize our findings, and we have added additional plots.
* The order of the topics has been fixed, thank you for bringing this to our attention.
* We have added details regarding the confidence intervals.
* The constant \mu is chosen heuristically by studying the norm of parameter vectors that result from standard weight decay, and setting \mu to be higher to make sure that networks trained with norm-bias indeed have a higher norm than those trained with weight decay. This explanation is now included in the section on weight norms.
In this paper, the authors seek to examine carefully some assumptions investigated in the theory of deep neural networks. The paper attempts to answer the following theoretical assumptions: the existence of local minima in loss landscapes, the relevance of weight decay with small L2-norm solutions, the connection between deep neural networks to kernel-based learning theory, and the generalization ability of networks with low-rank layers.
We think that this work is timely and of significant interest, since theoretical work on deep learning has made significant progress in recent years.
Since this paper seeks to provide an empirical study on the assumptions in deep learning theory, we think that the results are somehow weak as the paper is missing extensive analysis, using several well-known datasets and several deep architectures and settings.
For example, only the CIFAR-10 dataset is considered in the paper, and it is not clear whether the obtained results will generalize to other datasets. This also goes to the neural network architecture, as only MLP is considered to answer the assumption about the existence of suboptimal minima, while only ResNet is considered to study the generalization abilities with low-rank layers. We think that this is not enough for a paper that tries to provide an empirical study.
Reply to rebuttal
We thank the authors for taking into consideration our previous comments and suggestions, including going beyond MLP and adding experiments on other datasets.
For this reason, we have increased the rating from "Weak Accept" to "Accept".
We thank the reviewer for the time and effort spent on our paper. We agree about the note concerning the breadth of our experiments and have made the following additions to the paper.
* Experiments have been run on CIFAR-100 data and the results, which agree with our previous findings, are in the appendix.
* Our study of suboptimal minima had included experiments with ResNet-18. Results are in the appendix. As mentioned above, we have since added these experiments on CIFAR-100 for diversity of data sets.
* The section on rank has been updated to reflect further experiments with new architectures.
Specifically, we tested ResNet-18 without skip connections and MLP. See the updated appendix for full details and results.

Meta-review:
role: AC
data: comment: The work Bayesian Neural Network Ensembles by Pearce et al (https://arxiv.org/abs/1811.12188) proposes to use an ensemble of neural networks that are each trained with L2 regularization from a normal distribution sample. They show this is form of approximate Bayesian inference.

That idea is somewhat similar to the "norm-bias" regularizer proposed in this paper, with the difference that the weights attracted to a normal distribution sample rather than a fixed value.

I just wanted to point out this connection, which may be relevant to explain why norm-bias works.
role: Author
data: value: comment: Hi Pedro,
Thank you for pointing out this paper.  We agree that the relationship between explicit regularizers, like weight decay and norm-bias, during training and Bayesian priors at inference may be an interesting direction for future work.
role: PC
data: comment: The authors take a closer look at widely held beliefs about neural networks. Using a mix of analysis and experiment, they shed some light on the ways these assumptions break down. The paper contributes to our understanding of various phenomena and their connection to generalization, and should be a useful paper for theoreticians searching for predictive theories.

Decision: Accept (Spotlight)