Truth or backpropaganda? An empirical investigation of deep learning theory
Authors: Micah Goldblum
Jonas Geiping
Avi Schwarzschild
Michael Moeller
Tom Goldstein
Venue:  (2020)
Keywords: Deep learning
generalization
loss landscape
robustness

Abstract: We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike.  In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that small-norm parameters are not optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays a role;  (4) find that rank does not correlate with generalization or robustness in a practical setting.

Published as a conference paper at ICLR 2020
TRUTH OR BACKPROPAGANDA? AN EMPIRICAL IN- VESTIGATION OF DEEP LEARNING THEORY
Micah Goldblum'
Jonas Geiping'
Department of Mathematics University of Maryland goldolumd umd. edu
Department Computer Science and Electrical Engineering University of Siegen
jonas. get..geeppp. ng@uni eegen.ge.......
Avi Schwarzschild
Michael Moeller
Department of Mathematics University of Maryland avi 1 Bund edu
Department Computer Science and Electrical Engineering University of Siegen mi chael moe llerunn eegen.gnnn. de
Tom Goldstein
Department of Computer Science University of Maryland tomg 8 Bnnd edu
ABSTRACT
We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. In this work, we: (I) prove the widespread existence of suboptimal local minima in the loss landscape of neu- ral networks, and we use our thcory to find examples; (2) show that small-norm parameters
optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays role; (4) find that rank does not correlate with generalization or robustness in a practical setting.
INTRODUCTION
Modern deep learning methods descendent from such long-studied fields statistical learning. optimization, and signal processing, all which were built on mathematically rigorous foundations. In statistical learning, principled kernel methods have vastly improved the performance of SVMs and
PCA (Suykens & Vandewalle| 1999 Schölkopf et al. 1997), and boosting theory has enabled weak learners to generate strong classifiers (Schapire 1990p. Optimizers deep learning are borrowed from the field of convex optimization, where momentum optimizers (Nesterov) 1983) and conjugate gradient methods provably solve ill-conditioned problems with high efficiency (Hestenes & Stiefel, 1952). Deep learning harnesses foundational tools from these mature parent fields.
Despite its rigorous roots, deep learning has driven a wedge between theory and practice. Recent theoretical work has certainly made impressive strides towards understanding optimization and gen- eralization neural networks. But doing so has required researchers to make strong assumptions and study restricted model classes.
In this paper, we seck to understand whether deep learning theories accurately capture the behaviors and network properties that make realistic deep networks work. Following line of previous work, such as [Swirszzz et al. (2016), Zhang et al. (2016), Balduzzi et al. (2017) and Santurkar et al. (2018), put the assumptions and conclusions of deep learning theory to the test using experiments with both toy networks and realistic ones. We focus on the following important theoretical issues:
Authorr contrihuted equally.

---

Published as a conference paper at ICLR 2020
Local minima: Numerous theoretical works argue that all local minima of neural loss fune- tions are globally optimal or that all local minima are nearly optimal. In practice, we find highly suboptimal local minima in realistic neural loss functions, and we discuss reasons why suboptimal local minima exist in the loss surfaces of deep neural networks in general.
Weight decay and parameter norms: Research inspired by Tikhonov regularization suggests that low-norm minima generalize better, and for many, this is an intuitive justification for simple regularizers like weight decay. Yet for neural networks, it is not at all clear which form of fz-regularization is optimal. We show this by constructing a simple alternative: biasing solutions toward a non-zero norm still works and can even measurably improve performance for modern architectures.
Neural tangent kernels and the wide-network limit: We investigate theoretical results cerning neural tangent kernels of realistic architectures. While stochastic sampling of the tangent kernels suggests that theoretical results on tangent kernels of multi-layer networks may apply to some multi-layer networks and basic convolutional architectures, the predic- tions from theory do not hold for practical networks, and the trend even reverses for ResNet architectures. We show that the combination of skip connections and batch normalization is critical for this trend in ResNets.
Rank: Generalization theory has provided guarantees for the performance of low-rank net- works. However, we find that regularization which encourages high-rank weight matrices often outperforms that which promotes low-rank matrices. This indicates that low-rank structure is not a significant force behind generalization practical networks. We further investigate the adversarial robustness of low-rank networks, which are thought to be more resilient to attack, and we find empirically that their robustness is often lower than the baseline or even purposefully constructed high-rank network.
LOCAL MINIMA IN LOSS LANDSCAPES: Do SUBOPTIMAL MINIMA EXIST?
It is generally accepted that "in practice, poor local minima are rarely a problem with large net- works." (LeCun al. 2015). However, exact theoretical guarantees for this statement are elusive. Various theoretical studies of local minima have anvestigated spin- ppnnglass models (Choromanska et al.) 2014). deep linear models (Laurent Brecht 2018: Kawaguchi| 2016). parallel subnetworks (Ha-) effele & Vidal 2017), and dense fully connected models (Nguyen et al. 2018) and have shown that either all local minima are global or all have a small optimality gap. The apparent searcity of poor local minima has lead practitioners to develop the intuition that bad local minima ("bad" meaning high loss value and suboptimal training performance) are practically non-existent.
To further muddy the waters, some theoretical works prove the existence of local minima. Such results exist for simple fully connected architectures (Swirszez et al. 2016), single-layer networks (Liang et al.] 2018 Yun et al.) (218),, and two-layer ReLU networks (Safran & Shamir 2017). For example, YYun et al. 2019) show that local minima exist in single-layer networks with univariate output and unique datapoints. The crucial idea here is that all neurons are activated for all datapoints at the suboptimal local minima. Unfortunately. these existing analyses of neural loss landscapes require strong assumptions (e.g. random training data, linear activation functions. fully connected layers, or extremely wide network widths) - so strong, in fact, that it is reasonable to question whether these results have any bearing on practical neural networks or describe the underlying cause of good optimization performance in real-world settings.
In this section, we investigate the existence of suboptimal local minima from theoretical perspec- tive and an empirical one. If suboptimal local minima exist, they are certainly hard to find by standard methods (otherwise training would not work). Thus, we present simple theoretical results that inform us how to construct non-trivial suboptimal local minima, concretely generalizing pre- vious constructions, such as those by (Yun et al.) 2019). Using experimental methods inspired by theory, we easily find suboptimal local minima in the loss landscapes of range of classifiers.
Trivial local minima are easy to find in ReLU networks - consider the case where bias values are sufficiently low that the ReLUs are "dead" (i.e. inputs ReLUs are strictly negative). Such a point is trivially a local minimum. Below, we make a more subtle observation that multilayer perceptrons (MLPs) must have non-trivial local minima, provided there exists linear classifer that
2

---

Published as a conference paper at ICLR 2020
performs worse than the neural network (an assumption that holds for virtually any standard bench- mark problem). Specifically, we show that MLP loss functions contain local minima where they behave identically to linear classifier on the same data.
We now define a family of low-rank linear functions which represent an MLP. Let "rank-s affine function" denote an operator of the form (xx) Ax + b with rank s.
Definition 2.1. Consider a family of functions, {Fa R + R") JOER" parameterized by Φ. We say this family has rank-s affine expression if for all rank-s affine functions G I + Rh" and finite subsets Q C IRm there exists & with (x) = G(x), Vx E Q. If = min(n, m) we say that this family has full affine expression.
We investigate family of L-layer MLPs with ReLU activation functions, {F& R Rn" Jaea. and parameter vectors 6, i.e., ø = (Ai, b1, A2, .ag...,,A, bT). Fx(x) = H (H.,. (x)))), where f denotes the ReLU activation function and Hz(z) A,z bi-
Let A, E Ruixn--, b, E Rm with no m and nz = n.
Lemma 1. Consider a family of L-layer multilayer perceptrons with ReLU activations { FFa : IR" R" Joes. and let min, n, be the minimum layer width. Such a family has rank-s affine expres- sion.
Proof. The idea of the proof is to use the singular value decomposition of any rank-s affine function to construct the MLP layers and pick a bias large enough for all activations to remain positive. See Appendix I
The ability of MLPs to represent linear networks allows us to derive theorem which implies that arbitrarily deep MLPs have local minima at which the performance of the underlying model on the training data is equal to that of a (potentially low-rank) linear model. In other words, neural networks inherit the local minima of elementary linear models.
Theorem 1. Consider a training set, {(x, y:) a family {F o of MLPs with s min; n; being the smallest width. Consider purameterized affine function GA, solving
min L(GA,i (xx,3:)) A.b
subject to rank(A) Ss,
for a continuous loss function L. Then, for each local minimum, (A', b'), of the above training problem, there exists a local minimum, &' of the MLP loss L(Fas ((xx, v)) with the property that (x.) ) = G. (x) 2.,,...,NN
Proof. See appendix A.2
The proof of the above theorem constructs network in which all activations of all training examples are positive, generalizing previous constructions of this type such as Yun et al.] (2019) to more realistic architectures and settings. Another paper has employed similar construction concurrently to our own work (He et al. 2020). We do expect that the general problem in expressivity occurs every time the support of the activations coincides for all training examples, as the latter reduces the deep network to an affine linear function (on the training set), which relates to the discussion in Balduzzi et al. (2017). We test this hypothesis below by initializing deep networks with biases of high variance.
Remark 2.1 (CNN and more expressive local minima). Note that the above constructions I Lemma aand Theorem I are not limited to MIPs and could be extended to convolutional neural networks with suitably restricted linear mappings Go by using the convolution filters to represent identities and using the bias to avoid any negative activations on the training examples. Moreover, shallower MLPs can similarly be embedded into deeper MLPs recursively by replicating the behavior of each linear layer of the shallow MLP with several layers of the deep MLP. Linear classifiers, or even shallow MLPs, often have higher training loss than more expressive networks. Thus, we can use the idea of Theorem I to find various suboptimal local minima in the loss landscapes of neural networks. We confirm this with subsequent experiments.
We find that initializing a network at point that approximately conforms to Theorem is enough to get trapped in bad local minimum. We verify this by training linear classifier on CIFAR-10 with
3

---

Published as a conference paper at ICLR 2020
Table I: Local minima for MLPs generated via various initializations. We show loss, euclidean
norm of the gradient vector, and minimum eigenvalue of the Hessian before and after training. We use 500 iterations of the power method on a shifted Hessian matrix computed on the full dataset to find the minimum eigenvalue. The experiment in the last row is trained with no momentum (NM).
At Initialization
After training
Init. Type
Loss
Grad.
Min. EV
Loss
Grad.
Min. EV
Default
4.5963
0.5752
-1.5549
0.0061
0.0074
0.0007
Lemma
1.5702
0.0992
0.03125
1.5699
0.0414
0.0156
Bias+20
31.204
343.99
-1.7421
2.3301
0.0090
0.0005
Bias E (-(550, 50)
51,445
378.36
-430.49
2.3153
0.0048
0,0000
Bias E {=00- 10) NM
12.209
42.454
-47.733
0.2198
0.0564
0.0013
weight decay, (which has a test accuracy of 40.53%, loss of 1.57, and gradient norm of 0.00375 w.r. to the logistic regression objective). We then initialize multilayer network as described in Lemma to approximate this linear classifier and recompute these statistics on the full network (see Table When training with this initialization, the gradient norm drops futher, moving parameters even closer to the linear minimizer. The final training result still yields positive activations for the entire training dataset.
1..
Moreover, any isolated local minimum of a linear network results in many local minima of an MLP For, the weights o' constructed in the proof of Theorem | can undergo transformations such as scaling, permutation, or even rotation without changing Før as a function during inference, i.e. (x) = (x) for all x for an infinite set of parameters ø. as soon as has at least one hidden layer.
While our first experiment initializes deep MLP local minimum it inherited from a linear one to empirically illustrate our findings of Theorem [] Table I also illustrates that similarly bad local minima are obtained when choosing large biases (third row) and choosing biases with large variance (fourth row) conjectured above. To significantly reduce the bias, however, and still obtain sub- par optimum, we need to rerun the experiment with SGD without momentum, as shown in the last row, reflecting common intuition that momentum is helpful to move away from bad local optima.
Remark 2.2 (Sharpness of sub-optimal local optima). An interesting additional property of minima found using the previously discussed initializations is that they are "sharp". Proponents of the sharp- flat hypothesis for generalization have found that minimizers with poor generalization live in sharp attracting basins with low volume and thus low probability in parameter space (Keskar et al. 2016 Huang et al.) 2019), although care has to be taken correctly measure sharpness (Dinh al. 2017). Accordingly, we find that the maximum eigenvalue of the Hessian at each suboptimal local minimum significantly higher than those at near-global minima. For example, the maximum eigenvalue of the initialization by Lemma 1 in Table [ is estimated as 113, 598.85 after training, whereas that of the default initialization is only around 24.01. While our analysis has focused on sub-par local optima in training instead of global minima with sub-par generalization, both the scarcity of local optima during normal training and the favorable generalization properties of neural networks seem to correlate with their sharpness.
In light of our finding that neural networks trained with unconventional initialization reach subopti- mal local minima, we conclude that poor local minima can readily be found with a poor choice of hyperparameters. Suboptimal minima are less scarce than previously believed, and neural networks avoid these because good initializations and stochastic optimizers have been fine-tuned over time. Fortunately, promising theoretical directions may explain good optimization performance while re- maining compatible with empirical observations. The approach followed by Du et al. (2019) an- alyzes the loss trajectory of SGD, showing that it avoids bad minima. While this work assumes (unrealistically) large network widths, this theoretical direction is compatible with empirical stud- ies, such as Goodfellow et al. (2014), showing that the training trajectory of realistic deep networks does not encounter significant local minima.
4

---

Published as a conference paper at ICLR 2020
3
WEIGHT DECAY: ARE SMALL f2-NORM SOLUTIONS BETTER?
Classical learning theory advocates regularization for linear models, such as SVM and linear regres- sion. For SVM, l2 regularization endows linear classifiers with wide-margin property (Cortes & Vapnik 1995). and recent work on neural networks has shown that minimum norm neural network interpolators benefit from wver-parametrization (Hastie et al. (011)) Following the long history of explicit parameter norm regularization for linear models, weight decay is used for training nearly
all high performance neural networks (He et al.) 2015a Chollet 2016 Huang al. 2017 Sandler et al. 2018).
In combination with weight decay, all of these cutting-edge architectures also employ batch nor- malization after convolutional layers (loffe & Szegedy 2015). With that in mind, van Laarhoven (017) shows that the regularizing effect of weight decay is counteracted by batch normalization, which removes the effect of shrinking weight matrices, Zhang et al. (0188) argue that the synergistic interaction between weight decay and batch norm arises because weight decay plays a large role in regulating the effective learning rate of networks, since scaling down the weights of convolutional layers amplifies the effect of each optimization step, effectively increasing the learning rate. Thus, weight decay increases the effective learning rate as the regularizer drags the parameters closer and closer towards the origin. The authors also suggest that data augmentation and carefully chosen learning rate schedules are more powerful than explicit regularizers like weight decay.
Other work echos this sentiment and claims that weight decay and dropout have little effect on performance, especially when using data augmentation [Hermández-Garde & König 2018 Hoffer jet al. (2018) further study the relationship between weight decay and batch normalization, and they develop normalization with respect to other norms. Shah et (2018) instead suggest that minimum norm solutions may not generalize well in the over-parametrizee setting.
We find that the difference between performance of standard network architectures with and without weight decay is often statistically significant, even with a high level of data augmentation, for exam- ple, horizontal flips and random crops on CIFAR-10 10 (see Tables and 3). But is weight decay the most effective form of l2 regularization? Furthermore, is the positive effect of weight decay because the regularizer promotes small norm solutions? We generalize weight decay by biasing the lg norm of the weight vector towards other values using the following regularizer, which we call norm-bias.
Rp( =
1
Ro is equivalent to weight decay, but we find that we can further improve performance by biasing the weights towards higher norms (see Tables 2 and 3 In our experiments on CIFAR-10 and CIFAR-100, networks are trained using weight decay coefficients from their respective original papers. ResNet-18 and DenseNet are trained with μr² = 2500 and norm-bias coefficient 0.005, and MobileNetV2 is trained with μ²² 5000 and norm-bias coefficient 0.001. μ is chosen heuristically by first training model with weight decay, recording the norm of the resulting parameter vector, lower
and setting μ to be slightly higher than that norm in order to avoid norm-bias leading parameter norm than weight decay. While we find that weight decay improves results over a non- regularized baseline for all three models, we also find that models trained with large norm bias (i.e., large μ1) outperform models trained with weight decay.
These results lend weight to the argument that explicit parameter norm regularization is in fact useful for training networks, even deep CNNs with batch normalization and data augmentation. However, the fact that norm-biased networks can outperform networks trained with weight decay suggests that any benefits of weight decay are unlikely to originate from the superiority of small-norm solutions.
To further investigate the effect of weight decay and parameter norm on generalization, we also consider models without batch norm. In this case, weight decay directly penalizes the norm of the linear operators inside a network, since there are no batch norm coefficients to compensate for the effect of shrinking weights. Our goal is to determine whether small-norm solutions superior in this setting where the norm of the parameter vector is more meaningful.
In our first experiment without batch norm, experience improved performance training MLP with norm-bia (see Table B). In a state-of-the-art setting, we consider ResNet-20 with Fixup ini- tialization, ResNet variant that removes batch norm and instead uses a sophisticated initialization
5

---

Published as a conference paper at ICLR 2020
that solves the exploding gradient problem (Zhang al. 2019). We observe that weight decay sub- stantially improves training over SGD with no explicit regularization - in fact, ResNets with this initialization scheme train quite poorly without explicit regularization and data normalization. Still, we find that norm-bias with 2r² = 1000 and norm-bias coefficient 0.0005 achieves better results again refutes the theory that small-norm parameters generalize better and brings into doubt any relationship between classical Tikhonov regularization and weight decay in neural networks. See Appendix [A.5 for discussion concerning the final pa- rameter norms of Fixup networks as well as additional experiments on CIFAR-100 100, a harder image classification dataset.
than weight decay (see Table 3p. This
Table 2: ResNet-18, DenseNet-40, and MobileNetV2 models trained on non-normalized CIFAR- 10 data with various regularizers. Numerical entries are given by m where mn the average accuracy over 10 runs, and represents standard error.
Model ResNet
No weight decay (%)
Weight decay (%)
Norm-bin bias (%)
93.46 (+00.5)
94.06 (=:0.77)
94.86 (++0.05)
DenseNet
89.26 (=0.08)
92.27 (=0.06)
92.49 (#0.06)
MobilcNetV2
92.88 (=0.06)
92.88 (=0.09)
93.50 (#0.09)
Table 3: ResNet-18, DenseNet-40, MobileNetV2, ResNet-20 with Fixup initialization, and a 4-layer multi-layer perceptron (MLP) trained on normalized CIFAR-10 10 data with various regularizers. Nu- merical entries are given by m (),, where yn is the average accuracy over 10 runs, and 8 represents standard error.
Model
No weight decay (%)
Weight decay (%)
Norm-bias (%)
ResNet
93.40 (=0.04)
94.76 (=0.03)
94.99 (+0.05)
DenseNet
90.78 (=0.08)
92.26 (=0.06)
92.46 (#0.04)
MobileNetV2
92.84 (=0.05)
93.64 (=0.05)
93.64 (+0.03)
ResNet Fixup
10.00 (=:0.00
91.42 (#0.04)
91.55 (#0.07)
MLP
58.88 (=0.10)
58.95 (+0007)
59.13 (+0.09)
4 KERNEL THEORY AND THE INFINITE- CHINITE-WIDTH LIMIT
In light of the recent surge of works discussing the properties of neural networks in the infinite- width limit, in particular, connections between infinite-width deep neural networks and Gaussian processes, see Lee et al. (2017), several interesting theoretical works have appeared. The wide net- work limit and Gaussian process interpretations have inspired work on the neural tangent kernel (Jacot et al. 2018). while Lee et al. (2019) and Bietti et al. (2018) have used wide network assump- tions to analyze the training dynamics of deep networks. The connection of deep neural networks to kernel-based learning theory seems promising, but how closely do current architectures match the predictions made for simple networks in the large-width limit?
We focus on the Neural Tangent Kernel (NTK), developed in Jacot et al. (2018). Theory dictates that, in the wide-network limit, the neural tangent kernel remains nearly constant as a network trains. Furthermore, neural network training dynamics can be described as gradient descent on convex functional, provided the NTK remains nearly constant during training (Lee et al. 2019). In this section, experimentally test the validity of these theoretical assumptions.
Fixing network architecture, we use F to denote the function space parametrized by o E RP. For the mapping F R F, the NTK is defined by
(Φ)) Σ day F(O) 60 0g F(Ø),
where the derivatives 0₂ F(o) are evaluated at a particular choice of e deseribing a neural network. The NTK can be thought of as similarity measure between images; given any two images input, the NTK returns an n x n matrix, where n is the dimensionality of the feature embedding of the neural network. We sample entries from the NTK by drawing of N images (æ..) from dataset,

---

Published as a conference paper at ICLR 2020
and computing the entries in the NTK corresponding to all pairs of images in our image set. We do this for a random neural network f IR" + I²" and computing the tensor (()) E NNxNxax xnxn of all pairwise realizations, restricted to the given data:
(0)igh Σ (s, f σ),
(4)
By evaluating Equation using automatic differentiation, we compute slices from the NTK before and after training for a large range architectures and network widths. We consider image classi- fication CIFAR-I0 and compare two-layer MLP, a four-layer MLP, a simple 5-layer ConvNet, and a ResNet. We draw 25 random images from CIFAR-IO to sample the NTK before and after training. We measure the change in the NTK by computing the correlation coefficient of the (vec- torized) NTK before and after training. We do this for many network widths, and see what happens in the wide network limit. For MLPs we increase the width of the hidden layers. for the ConvNet (6-Layer, Convolutions, ReLU, MaxPooling), we increase the number of convolutional filters, for the ResNet consider the WideResnet (Zagoruyko & Komodakis 2016) architecture, where increase its width parameter. We initialize all models with uniform He initialization as discussed in He et al. (2015b), departing from specific Gaussian initializations in theoretical works to analyze the effects for modern architectures and methodologies.
++L-ay MLP
Relative change
2-Layer MLP Net 16-x ConvNet
Correlation coefituent
Number Parameters
Number of Farametors
(b)
Average Norm paramneter change
Layer MLP
Norm of changeein
2-Layer
WideltesNet 16-x ConvNet
Number of Parameters
Number of Parameters
(d)
Figure 1: (a) The relative norm of the neural tangent kernel as function of the number of parameters is shown for several networks. This figure highlights the difference between the behavior of ResNets and other architectures. Figure Ic visualizes the same data in logarithmic scale. (b) The correlation of the neural tangent kernel before and after training. We expect this coefficient to converge toward 1 in the Sfnite-wwdtth limit for multi-layer networks in Jacot et al. (2018). We do not observe this trend for ResNets as is clear from the curve corresponding to the WideResNet. (d) The average norm of parameter change decreases for simple architectures but stays nearly constant for the NideResNet.
The results are visualized in Figure I where plot parameters of the NTK for these different architectures, showing how the number of parameters impacts the relative change in the NTK ( / where Φ denotes the sub-sampled NTK before/after training) and cor- relation coefficient (Cov(#; (1, (a Jacot et al. (2018) predicts that the NTK should change very little during training in the infinite-wiet limit.
At first glance, it might seem that these expectations are hardly met for our (non-infinite) experi- ments. Figure Ta and Figure show that the relative change in the NTK during training (and also
7

---

Published as a conference paper at ICLR 2020
the magnitude of the NTK) is rapidly increasing with width and remains large in magnitude for whole range of widths of convolutional architectures. The MLP architectures do show a trend to- ward small changes in the NTK, yet convergence to zero is slower in the 4-Layer case than in the 2-Layer case.
However, closer look shows that almost all of the relative change in the NTK seen in Figure Tc is explained by simple linear re-scaling of the NTK. It should be noted that the scaling of the NTK is strongly effected by the magnitude of parameters at initialization. Within the NTK theory Lee et al. (2017), a linear rescaling of the NTK during training corresponds simply to change in learning rate, and so makes more sense measure similarity using a acale-invariant metric.
Measuring similarity between sub-sampled NTKs using the scale- invarianaaiiant correlation coefficient, as in Figure [bb is more promising. Surprisingly, we find that, as predicted Jacot et al.) (2018), the NTK changes very little (beyond a linear rescaling) for the wide ConvNet architectures. For the dense networks, the predicted trend toward small changes in the NTK also holds for most of the evaluated widths, although there is dropoff at the end which may be artifact of the difficulty of training these wide networks on CIFAR-10. For the Wide Residual Neural Networks, however, the general trend toward higher correlation in the wide network limit is completely reversed. The corre- lation coefficient decreases as network width increases, suggesting that the neural tangent kernel at initialization and after training becomes qualitatively more different network width increases. The reversal of the correlation trend seems to be a property which emerges from the interaction of hatch normalization and skip connections. Removing either of these features from the architecture leads to networks which have an almost constant correlation coefficient for wide range of network widths, see Figure 6 in the appendix, calling for the consideration of both properties in new formulations of the NTK.
In conclusion, we see that although the NTK trends towards stability as the width of simple archi- tectures increases, the opposite holds for the highly performant Wide ResNet architecture. Even further, neither the removal of batch normalization or the removal of skip connections fully recover the positive NTK trend. While we have hope that kernel-based theories of neural networks may yield guarantees for realistic (albeit wide) models in the future, current results do not sufficiently describe state-of-the-art architectures. Moreover, the already good behavior of models with unstable NTKs an indicator that good optimization and generalization behaviors do not fundamentally hinge on the stability of the NTK.
5 RANK: DO NETWORKS WITH LOW-RANK LAYERS GENERALIZE BETTER?
State-of-the-ar neural networks are highly wer-parameterized, and their large number of parameters is problem both for learning theory and for practical use. In the theoretical setting, rank has been used to tighten bounds on the generalization gap of neural networks. Generalization bounds from Harvey et al. (2017) are improved under conditions of low rank and high sparsity (Neyshabur ct al. 2017) of parameter matrices, and the compressibility of low-rank matrices (and other low- dimensional structure) can be directly exploited to provide even stronger bounds (Arora al., 2018). Further studies show a tendency of stochastic gradient methods to find low-rank solutions (Ji & Telgarsky 2018). The tendency of SGD find low-rank operators, in conjunction with results showing generalization bounds for low-rank operators, might suggest that the low-rank nature of these operators is important for generalization.
Langenberg et al. (219) claim that low-rank networks, in addition to generalizing well to test data, are more robust to adversarial attacks. Theoretical and empirical results from the aforementioned paper lead the authors to make two major claims. First, the authors claim that networks which undergo adversarial training have low-rank and sparse matrices. Second, they claim that networks with low-rank and sparse parameter matrices more robust to adversarial attacks. We find in our experiments that neither claim holds up in practical settings, including ResNet- 18 models trained on CIFAR-10.
We test the generalization and robustness properties of neural networks with low-rank and high- rank operators by promoting low-rank high-rank parameter matrices in late epochs. We employ the regularizer introduced in [Sedghi et al. (2018) to create the protocols RankMin, to find low-rank parameters, and RankMax, to find high-rank parameters. RankMin involves fine-tuning a pre-trained
8

---

Published as a conference paper at ICLR 2020
Model
Training
Clean Test Accuracy (%)
Robust (%)
Robust (%)
method
8/255
1/255
ResNet-18
Natural
94.66
0.00
31.98
RankMax
93.66
0.00
22.01
RankMin
94.44
0.00
31.53
Adversarial
79.37
35.38
74.27
RankMaxAdy
80.00
35.55
74.92
RankMinAdv
78.34
33.68
73.19
ResNet-18
Natural
92.95
0.01
31.34
w/o skips
RankMax
91.71
0.00
18.81
RankMin
92.42
0.00
30.37
Adversarial
79.57
35.95
74.88
RankMaxAdy
79.43
36.45
74.87
RankMinAdv 78.52
33.97
73.64
Table 4: Result presented here are from experiments with CIFAR-10 data and of the architec- tures we studied. Robust accuracy is measured with 20-step PGD attacks with the values specified the top of the column.
Untreired bedde
Natural
Estective Rame
Effective
Filter index
Filter
(a) Effective rank of naturally trained models.
(b) Effective rank of adversarially trained models.
Figure 2: This plot shows the effective rank of each filter for the ResNet-18 models. The filters indexed on the ar-axis, so moving to the right is like moving through the layers of the network. Our routines designed to manipulate the rank have exactly the desired effect shown here.
model by replacing linear operators with their low-rank approximations. retraining, and repeating this process. Similarly. RankMax involves fine-tuning a pre-trained model by clipping singular values from the SVD of parameter matrices in order to find high-rank approximations, We are able to manipulate the rank of matrices without strongly affecting the performance of the network. We use both natural training and 7-step projected gradient descent (PGD) adversarial training routines (Madry et al., (2017). The goal of the experiment is to observe how the rank of weight matrices impacts generalization and robustness. We start by attacking naturally trained models with the standard PGD adversarial attack with = 8/ 8/255. Then, we move to the adversarial training setting and test the effect of manipulating rank on generalization and on robustness.
In order to compare our results with Langenberg et al. (2019), we borrow the notion of effective rank, denoted by r(W) for some matrix W. This continuous relaxation of rank defined as follows.
r(W)- | where || ||+, || II,, and || || are the nuclear norm, the 1-norm, and the Frobenius respectively. Note that the singular values of convolution operators can be found quickly with a method from Sedghi et al. (2018), and that method is used here.
||
In experiments we investigate two architectures, ResNet-18 and ResNet-18 without skip con- nections. We train on CIFAR-10 10 and CIFAR- 100, both naturally and adversarially. Table shows that RankMin and RankMax achieve similar generalization CIFAR-10, More importantly, when adversarially training, setting when robustness is undeniably the goal, we see the RankMax out- performs both MankMin and standard adversarial training in robust accuracy. Figure confirms that

---

Published as a conference paper at ICLR 2020
these two training routines do, in fact, control effective rank. Experiments with CIFAR-100 yield similar results and are presented in Appendix A.7 It is clear that increasing rank using an analogue of rank minimizing algorithms does not harm performance. Morcover, observe that adversarial robustness does not imply low-rank operators, nor do low-rank operators imply robustness. The findings in Ji & Telgarsky (20T8) are corroborated here the black dots in Figures that ini- tializations are higher in rank than the trained models. Our investigation into what useful intuition in practical cases can be gained from the theoretical work on the rank of CNNs and from the claims about adversarial robustness reveals that rank plays little to no role in the performance of CNNs in the practical setting of image classification.
6 CONCLUSION
This work highlights the gap between deep learning theory and observations in the real-world setting. We underscore the need to carefully examine the assumptions theory and to move past the study of toy models, such deep linear networks or single-layer MLPs, whose traits do not describe those of the practical realm. First, we show that realistic neural networks on realistic learning problems contain suboptimal local minima. Second, show that low-norm parameters may be optimal for neural networks, and in fact, biasing parameters to a non-zero norm during training improves performance on several popular datasets and a wide range of networks. Third, we show that the wide- network trends in the neural tangent kernel do not hold for ResNets and that the interaction between skip connections and batch normalization play a large role. Finally, we show that low-rank linear operators and robustness are not correlated, especially for adversarially trained models.
ACKNOWLEDGMENTS
This work was supported by the AFOSR MURI Program, the National Science Foundation DMS directorate, and also the DARPA YFA and L.2M programs. Additional funding was provided by the Sloan Foundation.
REFERENCES
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In International Conference Machine Learning, pp- 254 263, 2018.
David Balduzzi, Marcus Frean, Lennox Leary, P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The Shattered Gradients Problem: If resnets the answer, then what is the ques- tion? arXiv:/702.0857 [cs, stat), February 2017. URL 08591
Alberto Bietti, Grégoire Mialon, Dexiong Chen, and Julien Mairal. A Kernll Perspective for Regularizing Deep Neural Networks. 11X.v:1810.00363 [cs, stat), September 2018. URL
http:///are org/
worg/abs. abs//8lss1111/ 00363
François Chollet. Xception: Deep Learning with Depthwise Separable Convolutions.
1611:22610 [es], October 2016.
wwwp:/wwwwg/////////////oo// 160. 011.0////611110000777
Anna Choromanska, Mikacl Henaff, Michael Mathicu, Gérard Ben Arous, and Yann LeCun. The Loss Surfaces of Multilayer Networks. arXiv:1412.0233 [es], November 2014. URL http:
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-29) 1995.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can General- ize For Deep Nets. 13X003770304933 [cs], March 2017. URL http:://arat org/ abs/ 1703. 04933
10

---

Published as a conference paper at ICLR 2020
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient Descent Finds Global Minima of Deep Neural Networks. In International Conference on Machine Learning. pp. 1675- 1685, May 2019. URL http:
wwwpppresss.... /v97 http//////////// /wwllc.l..cc.o/cccco html
lan J. Goodfellow, Oriol Vinyals, and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems, arXiv:1412.6544 [cs, stat], December 2014. URL http://arxiv.
org/ orq/abs/141. 6544
B. D. Haeffele and R. Vidal. Global Optimality in Neural Network Training. In 20/7 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 4390-4398, July 2017. doi: 10. 10.1109/CVPRR.217..47..
Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight ve-dimension bounds for piece- wise linear neural networks. CoRR, abs/1 abss/1033.0230, 2017. URL http:://rrxiv.crg//aae
1703.02 02930
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv: arXivvI903... 2019.
Fengxiang He, Bohan Wang, and Dacheng Tao. Piecewise linear activations substantially shape the loss surfaces of neural networks. International Conference of Learning Representations, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image
Recognition. arXiv:1512.03335 [cs], December 2015a. URL http:://axxv.../ggg//.oo/// 1512. 03385
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv:/502.00882 [cs), February 2015b. URL htR.httt://arxivvoom aas///bb////22/ 01t.s//1122012.222222
Alex Herrnndez-Gareia and Peter König. Do deep nets really need weight decay and dropout?
arXiv:1802.07042 [cs], February 2018. URL http:/t/a/xxi...o////// wwtpxxiiv.rrg/....oo 1at...//180..000
Magnus Rudolph Hestenes and Eduard Stiefel. Methods of conjugate gradients for solving linear systems, volume 49. NBS Washington, DC, 1952.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: Efficient and accurate normalization schemes in deep networks. /rXiv:/803.01814 [cs, staf], March 2018. URLhttt
01814
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
W Ronny Huang. Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. arXiv preprint arXiv:1906.03291, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:/502.03 1502.03 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. arXiv:1806.07577 [cs, math, stat], June 2018. URL http: 7/arxiv. orq www.aaas/111.0003
Ziwei Ji and Matus Telgarsky, Gradient descent aligns the layers of deep linear networks. arXiv 2018.
preprint
Kenji Kawaguchi. Deep Learning without Poor Local Minima. arXiv: 1605.07110 [es, math, stat), May 2016. Uttt:/ttpp///arrii..orraaaaa//ba 07110
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On -atch Training for Deep Learning: Generalization Gap and Sharp Min- ima. 008660000433/100 [cs, math], September 2016. URthttp://aaxiv.. http://wwww...o.a////abaaaa
048.3
11

---

Published as a conference paper at ICLR 2020
Peter Langenberg, Emilio Rafael Balda, Arash Behboodi, and Rudolf Mathar. On the effect of low- rank weights on adversarial robustness of neural networks. CoRR, abs/1901.10371, 2019. URL
httt:://axiiv org. hwtp///b////a 1t1//9991.00 httpg//ab//991..10000
Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International Conference on Machine Learning, pp. 2908-2913, 2018.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553))436-444
May 2015. ISSN 1476-4687. doi: 10.1038/nature 14539.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep Neural Networks as Gaussian Processes. arXiv:1711.00165 [cs, stat), October 2017.
00165
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. arXiv:1902.06720 [cs, stat], February 2019. URL htttp://arxxv
erg/abs/ / 1902. 06720
Shiyu Liang, Ruoyu Sun, Yixuan Li, and R. Srikant. Understanding the Loss Surface of Neural Networks for Binary Classification. (xXiv:/803.00900 [cs, staf), February 2018. URL (tth:
//arxii..rr//bbs118000 00909
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. arXiv preprint arXiv:1706.0608.
2017.
Yurii Nesterov. A method for unconstrained convex minimization problem with the rate of conver- gence o (1/k* 2). In Doklady AN USSR, volume 269, pp. 543-547, 1983.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. A pac-bayesian approach to spectrally--normaiized margin bounds for neural networks. CoRR, abs/1707.09564, 2017. URL (Rt::: org. als 1707 0ww.4
Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. the loss landscape of class of deep neural networks with no bad local valleys. 1809 [cs, stat], September 2018. URL
http://///ga///////// 10749
Itay Safran and Ohad Shamir. Spurious Local Minima are Common in Two-Layer ReLU Neural Networks. 1712.08998 [cs, stat), December 2017. URL http: org.blss
1712.08963
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How Does Batch Normalization Help Optimization? arXiv::805.1 1604 [cs, stat], May 2018. URL http:
1160
Robert E Schapire. The strength of weak learnability. Machine learning, 5(2):197-227, 1990.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernll principal component anal- ysis. In International conference on artificial neural networks, pp. 583-588. Springer, 1997.
Hanie Sedghi, Vineet Gupta, and Philip M Long. The singular values of convolutional layers. arXiv preprint arXiv: 10400 2018
Vatsal Shah, Anastasios Kyrillidis, and Sujay Sanghavi. Minimum norm solutions do not always generalize well for par-paraaeteriied problems. arXiv preprint arXiv:1811.07055 2018.
Johan AK Suykens and Joos Vandewalle. Least squares support vector machine classifiers. Neural processing letters, 9(3):293-300 1999.
12

---

Published as a conference paper at ICLR 2020
Grzegorz Swirszez, Wojciech Marian Czarnecki, and Razvan Pascanu. Local minima in training of neural networks. arXiv:1611.0631 [cs, stat], November 2016. URL /tt/attp///axxiv orq
abs/1611. 06310
Twan van Laarhoven. L2 Regularization versus Batch and Weight Normalization, arXiv:1706.053350 [cs, stat], June 2017. URL URLhttp:: http::/aaxxiy:: http:/iwww...raaa///ibbaaa 0535
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. arXiv: (rXiv:/802.03481 [cs, math, stat], February 2018. URL
http:// /ttt:://arxi..oo org/ abs/1802 03487 03487
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. In International Conference on Learning Representations, 2019.
URL http::://ppenreeeeeset
net forum? (aarrummmddreeeee YiRct?
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. arXiv:1605.07146 [cs], May 2016. WRRLtttp//aaattt wwtpp//wwwppppaa.....ooooo
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv:1611.03530 [cs], November 2016. URL
org. abs 1611 03530
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three Mechanisms of Weight Decay Regularization. 1810v.1810.12281 [cs, stat], October 2018. URL/ 1810. 12281
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup Initialization: Residual Learning Without Normalization. urXiv:/901.099: [es, stat], January 2019. URL
1901. 09321
A APPENDIX
A.1 PROOF OF LEMMA 1
Lemma 1. Consider a family of L-layer multilayer perceptrons with ReLU activations {Fa: : Rmm R") and let = min; be the minimum layer width. Then this family has rank-s affine expression.
Proof. Let be rank-s affine function, and Ω C R a finite set. Let G(x) = Ax with A UEV being the singular value decomposition of with U Raxs and V Rexm.
We define
A = ΣV
where 0 (possibly void) (n) - 8) x m matrix of zeros, and bj el for Ei, (A1x;)) + and 1 E R being a vector of all ones. We further choose A [ to have an 5 x s identity matrix in the upper left, and fill all other entries with zeros. This choice is possible since 2 for all I. We define bj O c c1j" E R" where 0 xg is vector of all zeros and (m: is (possibly void) vector of all ones.
Finally, we choose A,, U 0]. where now 0 is a (possibly void) n x (ML-1 s) matrix of all zeros, and b, -c b for 1 E - being vector of all ones.
Then one readily checks that (x) G(x) holds for E Note that entries of all activations are greater equal to 0, such that no ReLU ever maps entry zero.
A.2 PROOF OF THBOREM 1
Theorem 1. Consider a training set, {(x,,y)) NN family (F&) } of MLPs with = min, n, being the smallest width. Consider the training of a rank-s linear classifier G. i.e.
min L(GA.b U.
subject to rank( A) S s,
(5)
A,b
13

---

Note that due to Published as a conference paper at ICLR 2020
for any continuous loss function L. Then for each local minimum, (A',b'), of the above training problem, there exists local minimum, Ø, of [(F5; ((x, ()) with the property that (x(x)) (x.) (x.) I.
Proof. Based on the definition of local minimium, there exists an open ball D around (A',b') b')
such that
L(GAL (CCA,bi((s, ())))) & C(GA,bi ((x, M)) V(A, D with rank(A) s. (6)
First, we use the same construction as in the proof of Lemma I to find function Far with (x.)
G .b (x,)) ) for all training example x, Because the mapping ø + Fx(x,) continuous (not only for the entire network F but also for all subnetworks), and because all activations of For greater or equal to > 0, there exists an open ball B(o', 81) around ø' such that the activations of Fg remain positive for all x; and all Φ E ((0,,1))
Consequently, the restriction of F& to the training set remains affine linear for Φ E ((),))). In other words, for any & B(d', 51) we can write
(x A(o)x; + b(Ø)
VXi,
by defining A(6) = A A and b(Ø) = Σ = min; ni the resulting A(d) satisfies rank (n())(()) S 8.
After restricting 6 to an open ball B(d', for 82 I 6, sufficiently small, the above (A()),
satisfy (A(ø), b(Ø)) E D for all B(Ø', 82). On this set, we, however, already know that the loss can only be greater or equal to C( P(Fr; {(x,, yi } due to equation Thus, o' is local minimum
of the underlying loss function.
A.3 ADDITIONAL COMMENTS REGARDING THEOREM
Note that theoretical and experimental results do not contradict theoretical guarantees for deep linear networks (Kawaguchi/ 2016 Laurent & Brecht 2018) which show that all local minima are global. A deep linear network with min(n, m equivalent case, the local minima constructed by Theorem T are global. However, this observation shows that Theorem characterizes the gap between deep linear and deep nonlinear networks; the global minima predicted by linear network theories are inherited as (usually suboptimal) local minima when ReLU's are added. Thus, linear networks do not accurately describe the distribution of minima in non-linear networks.
linear classifier, and in this
A.4 ADDITIONAL RESULTS FOR SUBOPTIMAL LOCAL OPTIMA
Table 5 shows more experiments. As above in the previous experiment, we use gradient descent to train full ResNet-18 architecture CIFAR-10 until convergence from different initializations. We find that essentially the same results appear for the deeper architecture, initializing with very high bias leads highly non-optimal solutions. In this even solutions that are equally bad zero-norm initialization.
Further results CIFAR-100 shown in Tables 6 and 7 These experiments with MIP and ResNet- 18 show the same trends as explained above, thus confirming that the results are not specific the CIFAR-I0 dataset.
A.5 DETAILS CONCERNING LOW-NORM REGULARIZATION EXPERIMENTS
Our experiments comparing regularizers all run for 300 epochs with initial learning rate of 0.1 and decreases by a factor 10 at epochs 100, 175, 225, and 275. We use the SGD optimizer with momentum 0.9.
We also tried negative weight decay coefficients, which leads to ResNet-18 18 CIFAR-10 performance above 90% while blowing up parameter norm, but this performance is still suboptimal and is not informative concerning the optimality minimum norm solutions. One might wonder high norm-bias coefficients lead lower parameter than low weight decay coefficients. This
14

---

Published as a conference paper at ICLR 2020
Table 5: Local minima for ResNet-18 and CIFAR-10 generated via initialization and trained by vanilla gradient descent, showing loss, euclidean norm of the gradient vector.
At Initialization
After training
Init. Type
Loss
Grad.
Loss
Grad.
Default
2.30312
0.05000
0.00014
0,01410
Zero
2.30258
0.00025
2.30259
0,00013
Bias+20
12.95754
590.12170
2.30658
0.00004
Bias E EM((10) 10)
12.96790
214.68600
2.30260
0.00123
Bias E (1(-5
84.67800
1190.23500
2.30260
0.00702
Table 6: Local minima for ResNet-18 and CIFAR-100 generated via initialization and trained by vanilla gradient descent, showing loss, euclidean norm of the gradient vector
At Initialization
After training
Init. Type
Loss
Grad.
Loss
Grad.
Default
4.60591
0.02346
0.00030
0.00466
Zero
4.60517
0.00019
4.60517
0.00003
Bias+20
34.37053
655.51569
4.60517
0.00015
Bias E 40-100, 100)
178.74391
2615.72534
4.60517
0.00003
Table 7: Local minima for MLP and CIFAR- 100 generated via initialization and trained by vanilla gradient descent, showing loss, euclidean norm of the gradient vector.
At Initialization
After training
Init. Type
Loss
Grad.
Loss
Grad.
Default Zero
4.60670
0.16154
0.02579
0.01482
4.60517
0.00019
4.60517
0.00011
Bias+10
15.77286
359.65710
4.60517
0.00079
Bias U1(55, 5)
8,69149
63.59983
2.15917
0,09718
Bias E 240-10 10)
13.02693
158.78347
2.58368
0.09233
15

---

Published as a conference paper at ICLR 2020
question may not be meaningful in the case of networks with batch normalization. In the case of ResNet-20 with Fixup, which does not contain running mean and standard deviation, the average parameter & norm after training with weight decay is 24.51 while that of models trained with norm- bias is 31.2. Below, we perform the same tests on CIFAR- 100, substantially more difficult dataset. Weight decay coefficients chitecture. Norm- borm- coefficient/yz² is chosen to be 8100/0. /0.005, 7500/0.001, and 2000 2000/0.0000 00.0005 for ResNet-18, DenseNet-40, and ResNet-20 with Fixup, respectively, using the same heuristic as described in the main body.
chosen to be ones used in the original paper for the corresponding
Table 8: ResNet-18, 18, DenseNet-40, and ResNet-20 with Fixup initialization trained on normalized CIFAR-1ON data with various regularizers. Numerical entries are given by ma(+s)) where m the average accuracy over 10 runs, and represents standard error.
Model
No weight decay (%)
Weight decay (%)
Norm-biar (%)
ResNet
71.73 (#00225)
74.66 (=0.17)
75.90 (#0.16)
DenseNet
65.61 (#:0.33
68.98 (#0025)
69.24 (#0.11)
ResNet Fixup
1.000 (=0.00)
65.08 (+0.30)
65.58 (+0.17)
A.6 DETAILS ON NEURAL TANGENT KERNEL EXPERIMENT
For further reference, we include details on the NTK sampling during training epochs in Figure 3 We see that the parameter norm (Right) behaves normally (all of these experiments are trained with a standard weight decay parameter of 0.0005), yet the NTK norm (Left) rapidly increases. Most of this increase, however is scaling of the kernel, as the correlation plot (Middle) is much less drastic. We do see that most change happens in the very first epochs of training, whereas the kernel only changes slowly later on.
miton With
extornles With 100
ee2kayn Vur Wiit 12 -=3 Weem
wich
HLFtwwthh 110)
Spoda
Spodhs
Figure 3: Plotting the evolution of NTK parameters during training epochs. Left: Norm of the NTK Tensor, Middle: Correlation of current NTK iterate versus initial NTK. Right: Reference plot of the network parameter norms.
4-y-ay
Rel. Change in morm of NTK
-8... ................................ ....
Similarity confiicient
2 MLP
WideResNet 16-x
ConvNet
Number of Parameters
Number of Parameters
Figure 4: The similarity coefficient of the neural tangent kernel after training with its initialization. We expect this coefficient to converge toward I in the infinite-widee width limit for multi-layer networks. Also shown is the direct relative difference of the NTK norms, which behaves similarly to the nor- malized direct difference from figure
16

---

Published as a conference paper at ICLR 2020
Average Norm paremmeter changee
4-yerr MLP
2ayer
Accuraty
Wet 16x
ConvNet
Figure 5: For reference we record the test accuracy of all models from in the left plot and the relative change in parameters in the right plot.
dedefault
Norm of thange
Coreeation
no skip
Numbee of Parameters
Number of Parameters
Figure 6: The correlation coefficient of the neural tangent kernel after training with its initializa- tion for different WideResNet variants namely WideResNet without batch normalizations and WideResNet without skip connections. We interestingly find that removing either of both prop- erties, which widely regarding as beneficial for neural network training, stabilizes the trend in the default WideResNet. However both variants hardly converge toward 1, even when sampling very wide ResNets.
A.7 DETAILS ON RANKMIN AND RANKMAX
We employ routines to promote both low-rank and high-rank parameter matrices. We do this by computing approximations to the linear operators at each layer. Since convolutional layers are linear operations, we know that there is matrix whose dimensions are the number of parameters in the input to the convolution and the number of parameters in the output of the convolution. In order to compute low-rank approximations of these operators, one could write down the matrix corre- sponding to the convolution, and then compute low-rank approximation using singular value decomposition (SVD). In order to make this problem computationally tractable we used the method for computing singular values of convolution operators derived in (edghi et al.) (018). We were then able to do low-rank approximation in the classical sense, by setting each singular value below some threshold zero. In order compute high-rank operators, we clipped the singular values so that when mulitplying the SVD factors, we set each singular value to be equal to the minimum of some chosen constant and the true singular value. It is important note here that these approximations to the convolutional layers, when done naively, can return convolutions with larger filters. To be precise, an n X n filter will map to a k filter through our rank modifications, where & > n. We follow the method in [Sedghi et al. (2088) 8). where these filters are pruned back down by only using n n entries the output.
When naturally training ResNet- 18 and Skipless ResNet-13 18 models, we train with a batch size of 128 for 200 epochs with the learning rate initiated to 0.01 and decreasing by a factor 10 at epochs 100, 150, 175, and 190 (for both CIFAR- 10 and CIFAR-100), When adversarially training these two models on CIFAR-10 10 data, we use the same hyperparameters. However, in order to adversarially train on CIFAR-100 100,
train ResNet-18 with batch size of 256 for 300 epochs with initial learning rate of 0.1 and a decrease by factor 10 at epochs 200 and 250. For adversarially training Skipless ResNet-18 on CIFAR-100, we batch size of 256 for 350 epochs with an
17

---

Published as a conference paper at ICLR 2020
initial learning rate of 0.1 and decrease by factor of 10 epochs 200, 250, and 300. Adversarial training is done with an & 7-step PGD attack with step size of 2/255, and = 8/255. For all of the training described above augment the data with random crops and horizontal flips.
During 15 additional epochs of training we manipulate the rank as follows. RankMin and RankMax protocols employed periodically in the last 15 epochs taking care to make that the loss remains small. For these last epochs, the learning rate starts at 0.001 and decreases by a factor of 10 after the third and fifth epochs of the final 15 epochs. As shown in Table I0 we test the accuracy of each model on clean test data from the corresponding dataset, as well as on adversarial examples generated with 20-step PGD with 8/255 (with step size equal to 2/ 2/255) and 1/255 (with step size equal to 25/ 255)..
When training multi-layer perceptrons on CIFAR-10, train for 100 epochs with learning rate ini- tialized to 0.01 and decreasing by factor of 10 at epochs 60, 80 and 90. Then, we train the network for additional epochs, during which RankMin and RankMax networks undergo rank manipulation.
Table Results from rank experiments with multi-layer perceptron and CIFAR-IO.
MLP and CIFAR-10
Training method
Training Accuracy (%)
Clean Accuracy (%)
Robust
Robust (%) 1/255
8/255
Naturally Trained
100.00
58.79
3.76
28.94
RankMax
99.97
58.19
3.72
26.63
RankMin
100.00
58,06
3.76
28.48
Table 10: Results from rank experiments on CIFAR-100. Robust accuracy is measured with 20-step PGD attacks with the values specified at the top of the column.
ResNet-18 18 and CIFAR-100
Training method
Training Accuracy (%)
Clean Accuracy (%)
Robust (%)
Robust (%)
8/255
1/255
Naturally Trained
99.97
73.08
0.00
17.5
RankMax
99.90
72.67
0.00
16.95
RankMin
99.92
72.57
0.00
17.63
Adversarially Trained
99.92
50.88
17.81
45.99
RankMaxAdy
99.73
51.04
16.80
45.74
RankMinAdy
99.91
50.22
16.64
45.03
ResNet-18 18 w/o skip connections and CIFAR-100
Training method
Training Accuracy (%)
Clean Accuracy (%)
Robust (%)
Robust (%)
8/255
1 1/255
Naturally Trained
99.96
72.13
0.01
13.7
MankMax
99.82
71.35
0.04
11.74
RankMin
99.90
71.28
0.00
13.53
Adversarially Trained
99.92
50.47
17.62
45.18
RankMaxAdy
99.90
50.93
17.72
45.78
Aidy
99.91
49.37
16.77
44.41
18


Reviews:
[{"review_id": "SJgN-L5mcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors seek to examine carefully some assumptions investigated in the theory of deep neural networks. The paper attempts to answer the following theoretical assumptions: the existence of local minima in loss landscapes, the relevance of weight decay with small L2-norm solutions, the connection between deep neural networks to kernel-based learning theory, and the generalization ability of networks with low-rank layers.\n\nWe think that this work is timely and of significant interest, since theoretical work on deep learning has made significant progress in recent years.\n\nSince this paper seeks to provide an empirical study on the assumptions in deep learning theory, we think that the results are somehow weak as the paper is missing extensive analysis, using several well-known datasets and several deep architectures and settings. For example, only the CIFAR-10 dataset is considered in the paper, and it is not clear whether the obtained results will generalize to other datasets. This also goes to the neural network architecture, as only MLP is considered to answer the assumption about the existence of suboptimal minima, while only ResNet is considered to study the generalization abilities with low-rank layers. We think that this is not enough for a paper that tries to provide an empirical study.\n\n------- \nReply to rebuttal\n\nWe thank the authors for taking into consideration our previous comments and suggestions, including going beyond MLP and adding experiments on other datasets. For this reason, we have increased the rating from \"Weak Accept\" to \"Accept\".", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeQuWDTtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors seek to challenge some presumptions about training deep neural networks, such as the robustness of low rank linear layers and the existence of suboptimal local minima. They provide analytical insight as well as a few experiments.\n\nI give this paper an accept. They analytically explore four relevant topics of deep learning, and provide experimental insight. In particular, they provide solid analytical reasoning behind their claims that suboptimal local minima exist and that their lack of prevalence is due to improvements in other aspects of deep networks, such as initialization and optimizers. In addition, they present a norm-bias regularizer generalization that consistently increases accuracy. I am especially pleased with this, as the results are averaged over several runs (a practice that seems to be not so widespread these days). \n\nIf I were to have one thing on my wish list for this paper, it would be the small issue of having some multiple experiment version of the local minima experiments (I understand why it is not all that necessary for the rank and stability experiments).\n\nNevertheless, I think this paper gives useful insight as to the behavior of deep neural networks that can help advance the field on a foundational level.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1x75kATFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors look at empirical properties of deep neural networks and discuss their connection to past theoretical work on the following issues:\n\n* Local minima: they give an example of setting where bad local minima (far from the global minimum) are obtained. More specifically, they show such minima can be obtained by initializing with large random biases for MLPs with ReLU activation. They also provide a theoretical result that can be used to find a small set of such minima. I believe this is a useful incremental step towards a better understanding of local minima in deep learning, although it is not clear how many practical implications this has. One question that would ideally be answered is: in practical settings, to what degree does bad initialization cause bad performance specifically due to bad minima? (as opposed to, say, slow convergence or bad generalization performance).\n\n* Weight decay: the authors penalize the size of the norm of the weights as it diverges from a constant, as opposed to when it diverges from 0 as is normally done for weight decay. They show that this works as well or better than normal weight decay in a number of settings. This seem to put into question the belief sometimes held that solutions with smaller norms will generalize better.\n\n* Kernel theory: the authors try to reproduce some of the empirical properties predicted in the Neural Tangent Kernel paper (Jacot et al., 2018) in particular by using more realistic architectures. The results, however, do not appear very conclusive. This might be the weakest part of the paper, as it is hard to draw anything conclusive from their empirical results.\n\n* Rank: The authors challenge the common belief that low rank provides better generalization and more robustness towards adversarial attacks. When enforcing a low or high rank weight matrices during training on ResNet-18 trained on CIFAR-10, the two settings have similar performance and are similarly robust to adversarial attacks, showing at least one counter example.\n\nI think overall this is a useful although somewhat incremental paper, that makes progress in the understanding of the behavior of neural networks in practice, and can help guide further theoretical work and the  development of new and improved training techniques and initialization regimes for deep learning.\n\nOther comments/notes:\n* minor: the order of the last 2 sub topics covered (rank and NTK) is flipped in the introduction, compared to the abstract and the order of the chapters\n* in the table confidence intervals are given, it would be nice to have more details on how they are computed, (e.g. +- 1.96 * std error)\n* how is the constant \\mu in the norm-bias chosen?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors take a closer look at widely held beliefs about neural networks. Using a mix of analysis and experiment, they shed some light on the ways these assumptions break down. The paper contributes to our understanding of various phenomena and their connection to generalization, and should be a useful paper for theoreticians searching for predictive theories.
Decision: Accept (Spotlight)