SGD on Neural Networks learns Robust Features before Non-Robust
Authors: Vikram Nitin
Venue:  (2021)
Keywords: neural networks
gradient descent
sgd
adversarial
robustness
features

Abstract: Neural networks are known to be vulnerable to adversarial attacks - small, imperceptible perturbations that cause the network to misclassify an input. A recent line of work attempts to explain this behavior by positing the existence of non-robust features - well-generalizing but brittle features present in the data distribution that are learned by the network and can be perturbed to cause misclassification.

In this paper, we look at the dynamics of neural network training through the perspective of robust and non-robust features. We find that there are two very distinct pathways that neural network training can follow, depending on the hyperparameters used. In the first pathway, the network initially learns only predictive, robust features and weakly predictive non-robust features, and subsequently learns predictive, non-robust features. On the other hand, a network trained via the second pathway eschews predictive non-robust features altogether, and rapidly overfits the training data. We provide strong empirical evidence to corroborate this hypothesis, as well as theoretical analysis in a simplified setting. Key to our analysis is a better understanding of the relationship between predictive non-robust features and adversarial transferability. We present our findings in light of other recent results on the evolution of inductive biases learned by neural networks over the course of training.

Finally, we digress to show that rather than being quirks of the data distribution, predictive non-robust features might actually occur across datasets with different distributions drawn from independent sources, indicating that they perhaps possess some meaning in terms of human semantics.

Under review as a conference paper at ICLR 2021
SGD ON NEURAL NETWORKS LEARNS ROBUST FEATURES BEFORE NON-ROBUST
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks are known to be vulnerable to adversarial attacks small, imper- ceptible perturbations that cause the network to misclassify an input. recent line of work attempts to explain this behavior by positing the existence of non-robust features well-generalizing but brittle features present the data distribution that are learned by the network and can be perturbed to cause misclassification.
In this paper, we look at the dynamics of neural network training through the perspective of robust and non-robust features. We find that there are two very distinct "pathways" that neural network training can follow, depending on the hy- perparameters used. In the first pathway, the network initially learns only predic- tive, robust features and weakly predictive non-robust features, and subsequently learns predictive, non-robust features. On the other hand, network trained via the second pathway eschews predictive non-robust features altogether, and rapidly overfits the training data. We provide strong empirical evidence to corroborate this hypothesis, as well as theoretical analysis in a simplified setting. Key to our anal- ysis is better understanding of the relationship between predictive non-robust features and adversarial transferability. We present our findings in light of other recent results the evolution of inductive biases learned by neural networks over the course of training.
Finally, digress to show that rather than being "quirks" of the data distribution, predictive non-robust features might actually occur across datasets with different distributions drawn from independent sources, indicating that they perhaps pos- sess some meaning in terms of human semantics.
INTRODUCTION
Neural networks have achieved state of the art performance on tasks spanning an array of domains like computer vision, translation, speech recognition, robotics, and playing board games (Krizhevsky et al. (2012): Vaswani et al. (2017); Graves et al. (2013); ; Silver et al. (2016)). However in recent years, their vulnerability to adversarial attacks small, targeted input perturbations, has come under sharp focus (Szegedy et al. (2013); Papernot et al. (0017: ) Carlini & Wagner (2017): Athalye et al. 20118 Schmidt al. (2018)).
Ilyas et al. (2199) propose that neural network vulnerability is at least partly due to neural networks learning well- weellgeenerliiing but brittle features that are properties of the data distribution. From this point of view, an adversarial example would be constructed by modifying an input of one class slightly such that it takes on the non-robust features of another class.
They provide empirical evidence for their theory by training model adversarially perturbed examples labeled as the target class, and showing that this model generalizes well to the original, unperturbed distribution.
Another unrelated line of work (Brutzkus et al. 2018 Ji & Telgarsky (2019) Li & Liang 2018
aims to study the properties of the functions learned by gradient descent over the course of training. Nakkiran et al. (019 and Mangalam & Prabhu| 2019) independently showed that Stochastic Gradi- ent Descent (SGD) learns simple, almost linear functions to start out, but then learns more complex functions as training progresses. [i et al. (201)) showed that models trained with a low learning rate learn easy-to-generall but hard-to-fit features first, and thus perform poorly easy-to-fit patterns.

---

Under review as a conference paper at ICLR 2021
100
150-
Mruney
Monvesbue Famuue
Accusay
Non-rtturt Freture
Robuet fenturm
Rebuet Fearur
Simpli Medl cors.
Simplee Houdd Con..
Yum Wym
20-
60
60
50
100 120
0
20
60
00
80
100
120
Coech
Cotch
Pathway (Normal Training)
(b) Pathway (Overfitting)
(Best viewed in color). Neural network training follows two very different pathways based on the choices of hyperparameters. These are training graphs of Resnet50 models trained on relabeled CIFAR- 10 adversarial examples. See Section 4 for more details.
Figure I:
In this paper, we study gradient descent neural networks from the perspective of robust and non- robust features. Our main thesis is that based on choices of hyperparameters, neural network training follows one of two pathways,
Pathway 1 (Informal) The neural network first learns predictive robust features and weakly predictive non-robust features. As training progresses, it learns predictive non- robust features, and having learned both robust and non non-robust predictive features, achieves good performance held-out data. This is the pathway that Ilyas et al. (2019) used to prove their theory.
Pathway (Informal):
The neural network learns predictive robust features and weakly predictive non- non-robus, robust features (as in Pathway 1). But thereafter, it begins to fit the noise in the training set, and quickly achieves zero training error. this scenario, the network learns only the robust predictive features and shows modest generalization on held-out data.
Through series of experiments, we validate our two-pathway hypothesis, investigate the specific circumstances under which Pathway and Pathway 2 occur, and analyze some properties of the two pathways. We will also develop a closer understanding of the relationship between adversarial transfer and predictive non-robust features, which will aid our analysis of the two pathways.
The rest of this paper is organized as follows. Section sets up the notation and definitions we use. In Section 3, we investigate the link between adversarial features and transferability. In Section 4 we provide empirical evidence for the two-pathway hypothesis and analyze some characteristics of each pathway. Section presents a theoretical analysis of gradient descent toy linear model. We show that for different choices of initial parameters, the linear model exhibits properties the first and second pathways. We digress to explore whether non-robust features occur across datasets in Section 6, and discuss future research directions in Section 7.
DEFINITIONS AND PRELIMINARIES
Consider the binary classification setting, where D a joint distribution over the input space and the labels 1, 1 In this setting, Ilyas et al. (2019) define feature any function f: X'' - I, scaled such that Eiz.yeDD|
0 and E E J (x) 2 = 1. feature said be p-useful if
Bayyeyl f(x)]
(1)
for some 0, and r-bousst if
inf y f(x + 65 >7 (EA(m)
(2)
This framework easily be adapted the multi-class setting
2

---

Under review as a conference paper at ICLR 2021
for some yy 0 and some family of perturbations A. For brevity, we sometimes refer to p-useful, y-robust feature (p, ? > 0) simply as a robust feature. Let pD(f) be the largest p for which f is p-useful. A feature f is said to be (highly) predictive or weakly predictive if pD(f) is high low respectively.
A useful, non-robust feature as defined by [lyas et al. 2019 is that is p-useful for some p O, but is not y-robust for any y > 0. They propose the following experiment to demonstrate the existence of these features.
Let C' be a classifier trained with Empirical Risk Minimization (ERM) on an empirical distribution D. We operate under the following assumption.
Assumption 1. If a distribution D contains a useful feature, then a classifier C' trained with ERM on an empirical distribution D drawn from 'D will learn this feature, provided that we avoid finite sample overfitting through appropriate measures such as regularization and cross-validation.
Let (z, t) denote the loss of C' on input x, for target label t. Construct adversarial examples by solving the following optimization problem
Fadv
arg min Lo(a', 11z-z'ISe
(3)
In particular, construct a distribution called Daet comprised of (Nadu: t) pairs by using Equation with chosen deterministically according y for each (xr, y) E D. In the binary classification setting, & must be y. so
E B [t- f(Fdv))
}] > 0, if f is non-robustly useful under D E (F [-| [Fadv)]
(4)
]] > 0, if robustly useful under DD
(5)
is observed that neural network trained on Ddes achieves non-trivial generalization to the original conclude that non-robust features exist and are useful for
test set, that is D. From this, classification in the normal setting.
Remark Goh 2019b) showed that the Drand dataset constructed by choosing / randomly in the above procedure, suffers from a sort of "robust feature leakage". PGD introduces faint robust cues in the generated adversarial example that can be learned by the model. But on the Ddes dataset, the robust features are correlated with a deterministic label which is different from Hence we use the Duct dataset in preference to the Drand for all our experiments.
Two kinds of non-robust features Goh (2019a) points out a subtle flaw with the above definition of a non-robust feature highly predictive non-robust features can arise from "contamination" of robust feature with a non-robust feature, instead of something meaningful. To see how this can hap- pen, consider a highly predictive robust feature In and weakly predictive non-robust feature JNR- Let fc "contaminated" feature that a simple sum of In and J'RR (appropriately normalized). Then it is possible to construct scenario in which
Ely fn(x)) >
inf y --ff( + 8) (EA(Z)
> 0
(6)
Ely SNN(2)) 2
inf y SNN( + (EA())
< 0
(7)
fc()) > 0
inf y. fo(x + 8) (EA(Z))
<0
(8)
Sc thus a highly predictive non-robust feature. Now when you train a model on (x + 6, -y) pairs, fc fR + fNR is still correlated with -y. But for -fR + fNR is more correlated, so the model will learn this combination in preference to fc and will not generalize on the original distribution. In fact, thanks to learning -fn. it will generalize to the distribution with flipped labels, i..., y + -y. In our analysis and experiments, when we refer to non-robust features, we will exclude such contaminated features.
Illustrative Example Consider a dataset of dog and cat images, where most dog images have snouts and most cats do not have snouts. Most cats have slightly lighter eyes than dogs, and making
3

---

Under review as a conference paper at ICLR 2021
Name
Feature
Type of Feature
f1
Snout
otherwise -1
Predictive Robust
f2
Dark Eyes = otherwise -1
Predictive Non-Robust
fa
First pixel is an odd number = 1, otherwise -1
Weakly Predictive Non-Robust
f4
f f3
Contaminated Robust
Table 1: An example illustrating the different kinds of features. Dogs and cats are labeled +1 and -1 respectively. Most dogs have dark eyes and snouts. A small majority of dog images start with an odd numbered pixel value.
the eyes slightly darker lighter is part of the set of valid adversarial perturbations. Suppose that very small majority of the dog images start with pixel that has an odd numbered value. Then the different types of features this dataset are enumerated Table
For f2. (x + -y) pairs would be dogs with lighter eyes, labeled as cats. The network trained on these examples will learn Snout = Cat, Light Eyes = Cat.
Since the eye-color is predictive of the true label, the second feature will ensure that the neural network has non- non-trivial performance on the original distribution. This is what Ilyas et al. (2019) observed in their experiments. /2 is thus a true non-robust feature.
For f4, (x+55, y) pairs would be dog images with the first pixel value converted to an even number, labeled as cats. The network trained on these examples will learn Snout = Cat, Dark Eyes Cat, and Even Pixel = Cat.
None of these will be particularly helpful on the true distribution, but the first two will be useful on the flipped distribution, i.e., where dogs are relabeled as cats. f4 thus contaminated robust feature, and not a non-robust feature.
Remark : network that learns only robust features but with contaminants can still be very vulner- able to adversarial attacks, as the above example shows. The weakly predictive non-robust feature fs can be manipulated to consistently cause misclassification on out-of-distribution inputs.
3 NON-ROBUST FEATURES AND TRANSFERABILITY
The phenomenon of adversarial transferability (Papernot et al., 2016), where a non-trivial fraction of the adversarial examples generated for neural network are still adversarial to other neural networks trained independently on the same data, can be readily explained in terms of non-robust features.
By Assumption [ different neural networks trained using ERM on a distribution would learn the predictive non-robust features (like Dark Eyes = Dog) present in the distribution. One would then construct an adversarial example by modifying an input such that the predictive non-robust features flip (modify dog images to have lighter eyes). Then this adversarial example would transfer to all the different networks that have learned to rely on the non-robust features.
natural question ask is, does all adversarial transferability arise from predictive non-robust features? Nakkiran (2019) showed that by explicitly penalizing transferability during PGD, one can construct adversarial examples that do not transfer, and from which it is not possible learn generalizing model. This establishes that adversarial examples that do not transfer, do not contain predictive non-robust features.
Here provide simpler experiment that constructs con-transferable adversarial examples without explicitly penalizing transferability. This experiment also establishes a stronger claim, that adver- sarial examples transfer if and only they exploit predictive non-robust features.
Let the CIFAR-10 10 dataset form the data distribution D. Train two Resnet50 models (He et 2016) on and ensure by Assumption that both networks have learned the predictive non-robust features of the distribution by using regularization and cross-validation across a grid of hyperparameters.
4

---

Under review as a conference paper at ICLR 2021
Construct Daet dataset for the first network using Equation where is chosen deterministically (y+1) 11%%.... We use Projected Gradient Descent (PGD) (Madry et al., 2018) to solve the optimization problem in Equation [ Split the adversarial examples into two categories those that transfer to the second network with their target labels, and those that do not. Relabel all adversarial examples Fade with their target label and train Resnet50 model on (Iade, t) pairs from each category.
according to y using the transformation
As Equations and 5 suggest, for (Fado, 1) DDett the non-robust features of are predictive of I, but the robust features of D are predictive of (t 1)%10. So if neural network trained on a subset Deet learns predictive non- robust features, it will generalize to D. and if it learns predictive robust features, will generalize the shifted distribution Dshift"
100
Accuracy Accuracy on Pan
Accuracy (%)
60
Danise = {(z, (y + 1)%10) (I, D}
(9)
40
Figure 2 shows the performance of these two networks on D and Denift- We can see that the network trained on the examples that transfer generalizes well to D, but the network trained on the examples that do not transfer generalizes to Setsfe. The configuration in the figure is as result of a thorough grid search over hyperparameters with the metric for success being performance on
Non transferabbl (34%)
All examples (100%)
Figure 2: Clean accuracy of Resnet50 model trained on subsets of relabeled adversarial examples
Along with Assumption 1 our experiment establishes that the examples that transfer contain predictive non-robust features, and the examples that don't transfer don't 't contain predictive non-robust features. In particular, we claim the following: :
Claim 1. Train two networks N1 and N2 on a common dataset such that both networks learn the predictive non-robust features present in the dataset. Then adversarial example generated for N₂ transfers to the second network if and only if this example contains predictive non-robust features.
Further, if neural network C. has learned predictive non-robust features, then PGD will construct some adversarial examples with predictive non-robust features (see Equation D. and vice-versa. This allows to infer the following property, which will use in our analysis the next section:
Claim 2. If neural network N2 has learned the predictive non-robust features in a dataset, then adversarial examples generated for another network N1 using PGD will transfer N2 if and only if N₁ has also learned predictive non- con-robust features.
4 THE TWO PATHWAY HYPOTHESIS
4.1 EXPERIMENTAL SETUP
We use the CIFAR-I0 training our empirical distribution D, and train a neural network Ni using ERM on D with voss-valiiotion and regularization such that it learns non-robust features by Assumption [ Construct the Dáet dataset according to the procedure described in Section 2 where the reference model is N₁ and the adversarial target is chosen deterministically as neural network N2 on the
(y 10%10. Split Daet into training and validation sets and train training set.
As we discussed in Section 3 if Nr is able to generalize to D, then N2 must have learned the predic- rive non-robust features of D. and if N2 is able to generalize to Denift. then N2 must have learned the predictive robust features of D. This is depicted in Figure in the context of our illustrative example from Section 2
We the accuracy on D (respectively, Dshift) a proxy for how much of the model's perfor- mance can be attributed to its learning predictive non-robust (respectively, robust) features. We refer to these "non-robust feature accuracy" and "robust feature accuracy".
Finally, the accuracies on the training and validation splits of Dact tell us how well the model has fit the training data, and whether the model is overfitting. We train the network N₂ using SGD for
5

---

Under review as a conference paper at ICLR 2021
x + 8
Cal
Predict
Train
Model
Dog
Using
Using Dark Eyes (Non-robust)
Snout (Robust)
Cat
Dog
Dogs with Lighter Eyes, Cats with Darker Eyes
Train a model on dogs with light eyes labeled as cats, and cats with dark eyes labeled as dogs. the model classifies a clean dog image as a cat, then it must have learned the predictive robust feature (snout), and if it classifies dog dog, it must have learned the predictive non- robust feature (dark eyes).
Figure 3:
120 epochs using different combinations of learning rate and regularization, and plot the evolution of these four metrics over the course of training.
4.2 RESULTS AND DISCUSSION
We observe that training follows two very distinct regimes or pathways depending on the choice of hyperparameters.
The first pathway, illustrated in Figure la occurs when the model trained with regularization of some sort either in the form of a high initial learning rate (LR), L2 weight decay, or data augmenta- tion. The model starts out by learning only predictive robust features (possibly with contaminants), but at some point switches learning combination of robust and non-robust predictive features. Training and validation accuracy steadily increase, and the model ends with both training and vali- dation accuracy close to 100%.
The second pathway, illustrated in Figure 1b occurs when the model is trained with low starting learning rate, little no L2 weight decay and data augmentation. The model starts out similar to the first pathway, but then starts overfitting the training data before it can learn non-robust predictive features. At this point, validation accuracy stagnates. The model finishes with training accuracy of 100% but validation accuracy of 81 %. Nearly all the performance of the model can be attributed its learning predictive robust features.
Hyperparameters: In Section C of the Appendix, we present a study of the effect of different hyperparameters for a Resnet-18 18 model trained on Dest We observe that the model makes sharp transition from Pathway to in the space of hyperparameters, with a narrow "middle ground".
On clean data: : Training on the Daet dataset allows us to decompose the accuracy into robust and non-robust, but similar decomposition doesn't 't exist for a model trained on D. Instead utilize Claim 2 and use adversarial transferability as a proxy for whether or not the model has learned non-robust features.
Train two models (1 and M with different random initializations on the unaltered CIFAR- 10 dataset, with both data augmentation and some weight decay. Train two more models M and MM (2) with neither weight augmentation nor weight decay. We plot the training and validation accuracies over the course of training for M(( and M in Figure and Figure 4b
Simultaneously, we also plot the targeted adversarial attack success, as well as the transfer accuracy to M and ! We observe that targeted adversarial attack success is high for both models. However, while adversarial examples generated for M( transfer to MM with reasonable success, adversarial examples generated for M(22) fail transfer either M M

---

Under review as a conference paper at ICLR 2021
20-
80
Training
Accrrccy
60-
Accuracy
60
Validietion
Targetoo Ady Sucooss
Transter to xe"
40-
Traning Acc
40
Transfer % wue
Validetion Art
Targeted Adv Succoss
20-
20
Transler % At
120
(a) M (Normal Training)
(b) M (Overfitting)
Figure 4: (Best viewed in color). Training and validation accuracy of M and M along with accuracy of targeted adversarial attacks and adversarial transfer accuracy to M22 and M!
We conclude that 1 learns predictive non-robust features training progresses, and 2 does not. Thus, ( follows Pathway and _M(3) follows Pathway 2. We observe that learning pre- dictive non-robust features seems to be essential for good generalization on the validation set, and robust features alone not suffice.
Remark Although Figure la suggests that the model eventually generalizes better to the non- robust feature mapping than the robust, this is not a generally applicable rule over different datasets, architectures and combinations of hyperparameters. Table in the Appendix illustrates this point.
4.3 RELATION TO OTHER RESULTS ABOUT NEURAL. NETWORK TRAINING
Low and high learning rates The regularizing effect of high initial learning rate has been studied in detail by Li al. (2019). They construct dataset with two types of patterns those that are hard-to-fir but easy-to- -e--eralzeer (i.e., low in variation), and those that are easy-to-fit but hard-to-generalle (i.e., noisy).
They show that a neural network trained with small learning rate first focuses on the hard-to-fit patterns, and because of the low noise in them, quickly overfits the training As result, is not able to learn easy-to-fit pattern effectively later on. In contrast, model that starts out with a high learning rate learns easy-to-fit pattern first, and since these are noisy, doesn't 't overfit the training set. Later on, once the learning rate is annealed, the model is able to effectively learn the harder-to-fit patterns.
These two cases can be crudely mapped onto our two pathways. The model in Pathway 2, trained with low LR, learns only robust features start out, indicating that these features are hard-to-fit. overfits the training set and thereafter is unable to learn the non-robust features, which are easy-to-fit.
The model in Pathway trained with high LR, quickly begins learn the non-robust features which are casy-to-fit. However, it learns the robust features too alongside, indicating that this map- ping from the low and high LR scenarios to our two pathway theory is not perfect.
Complexity of learned functions: Another perspective the training of neural networks is given by Nakkiran et al. (2019). They define a performance correlation metric between two models that captures how much of the performance of one model can be explained by the other, and show that as training progresses, the peformance correlation between the current model and the best simple model decreases. This indicates that the functions learned by neural network become increasingly complex as training progresses.
Although their metric is defined for binary classification setting, adapt for the multi-class setting, and use multi-class logistic regression classifier as the "best simple classifier". We measure the performance correlation between the model trained Dact and the simple classifier training

---

Under review as a conference paper at ICLR 2021
progresses. The performance correlation, scaled by 100 and smoothed, is shown in Figure | allong with the training curves.
We observe in Figure la that the point at which the performance correlation plateaus corresponds with when the robust accuracy decreases sharply. Similarly in Figure [bb] the correlation levels off along with the robust accuracy. We conjecture that the initial robust features learned by the model are simple, linear functions, and the non-robust features are more complex and non-linear, This is in line with the findings of Tramèr al. (2017) that transferable adversarial examples occur in space of high dimensionality.
THEORETICAL ANALYSIS
In this section, we present some results for gradient descent on toy linear model on a distribution with robust and non-robust predictive features. We get results that mirror our two pathways, for dif- ferent choices of initial conditions. The setting et al. (2019). For proofs of these theorems, refer to Section A of the Appendix.
is adaptation of the used by Nakkiran
Define data distribution P over Rd {-1, as follows:
y u.0.7 {-1,1}, q v-a.r {3....,d d}
\ ~ Bernouli(p)) over {+]}, x ~ Bernoulli(p/)) over {+1}, p (0,1/2), k>
= ydier + eyy2e22 + teg
where e is some small positive constant, and e; denotes the 7th natural basis vector. Now sample a "training set" (X,, Y1),... (.....,) (Xa, Y,) from this distribution. Let A = [X;:... xT E Raxd and
B [Yi,... Yn "r E R Consider training linear classifier using gradient descent with learning rate of to find w E Rd that minimizes the squared loss:
1 _ B
((w)
Awl
U E Rd
2n
We operate in the overparameterized setting, where n < So with high probability, the coordinates {3, ..., d} of the training data orthogonal for all the training points. B
The idea is that the data consists of a "robust" feature given by the first coordinate, a "non-robust" feature (one that can become anti-correlated with a perturbation of 2e) given by the second coor- dinate, and noisy component that comprises the rest of the coordinates, making it possible for a model to fit the data exactly. The robust component is predictive of the true label with probability 1- p. and the non-robust component predictive of the true label with probability 1 - (p/ (p/k). For simplicity, assume that the initial weight vector We O, and that is sufficiently large.
Theorem 1 (Robust before Non-robust). If e S /(1 2p)/(1 2p/k), then at the end of the first step, with high probability, the model will rely on the robust feature for classification, i.e., wi 2 ew! (2) and will have a population accuracy of Theorem (Two Pathways). Define
2p(1 + ne²) 2p
k 2p(1 + ne=) (1 e)
Then in s 2/(1 2 (1/n)), the number of gradient steps goes infinity,
k 2 kú, sample accuracy approaches and population accuracy approaches 1 (p/k) with high probability.
if < ka, sample accuracy approaches and population accuracy approaches p with high probability.
Discussion The two cases of Theorem very roughly correspond to Pathways I and 2. Since this is strongly convex problem, gradient descent with small enough learning rate will converge to a fixed solution, so we cannot mimic the setting where different training hyperparameters lead to Pathway 1 or 2. But we see that if the non-robust feature is predictive enough, the model lears the non-robust feature, otherwise it learns the robust feature.
CCaveat here, d is both the input dimensionality and the number of parameters. Although deep learning models overparameterized, is uncommon for datasets have dimensions than data points.
8

---

Under review as a conference paper at ICLR 2021
Trained On FEAAR-minu--Flictt
On CIFAR-minus--liicr
On CINIC-10
93.4
65.7
CINIC-10
82.1
84.6
CIFAR-minus- -lickr Deet CINIC-10 Daet
33.7
20.5
19.6
30.1
Table 2: Accuracy of Resnet50 models on the CIFAR-minus-Flickr and CINIC-10 test sets. The two numbers bold are the ones focus on.
DIGRESSION DATASSD TRANSFER
One view of non-robust features is that they are peculiarities or quirks in the data distribution. We provide evidence that allows us tentatively refute this assumption by showing that one struct two datasets from completely independent sources, and model that learns only the predictive non-robust features of one dataset can achieve non-trivial generalization on the other dataset.
The CINIC-10 dataset Darlow et al. (2018) is a distribution-shifted version of CIFAR-10 constructed by sampling from the ImageNet synsets corresponding each of the CIFAR-IO classes. Although may seem like CIFAR-10 10 and CINIC-10 could be candidates for two datasets drawn from indepen- dent sources, ImageNet constructed by querying Flickr, and Flickr is also one of the seven sources for the 80 million Tinylmages dataset (Torralba et al. (2008) that was used to construct CIFAR-10 (Kizhevsky et (2009). So roughly one in seven CIFAR- 10 images from Flickr.
To be even more certain that there are no spurious correlations creeping in because of common source, we construct the CIFAR-minus-Flick dataset that consists of those CIFAR-10 10 images that haven't been sourced from Flickr. This comprises 52, 172 out of the 60,000 CIFAR-10 images.
and CINIC-10, 10, and train Resnet50) models on them. These models can only learn non-robust features to help them generalize to the original unperturbed datasets, because the robust features are correlated with the shifted labels.
We construct Dett datasets as described in Section 4 for
The results are shown Table 2 the other dataset, which is long way from the expected 10% accuracy of random model.
Both Daet trained models achieve an accuracy of close to 20% on
7 CONCLUSION AND FUTURE DIRECTIONS
In this paper, we've shown that from the perspective of predictive robust and non-robust features, neural network training follows two very different pathways, corresponding to the training and overfitting regimes. In both regimes, the model starts out by learning predictive robust features first.
This decomposition into two distinct pathways has several interesting implications. For instance, adversarial transferability means that successful attack by constructing adversarial examples for proxy model. But a model trained via Pathway 2 learns no predictive non-robust features, and adversarial examples generated for another model will in general not transfer to this model. Thus an adversary cannot perform a successful attack on this model without least the ability to query the model and observe its outputs for large number of inputs.
adversary with access model can mount
line of enquiry that arises naturally from our work is understanding precisely why this behavior occurs in neural networks. What characterstics do predictive non-robust features have that ensure that they are learned only subsequent to predictive robust features? We pose finding a more pre- cise definition of non-robust features that will allow to theoretically analyze and explain these properties as an important direction for future work.
Finally, show in Section 6 predictive non-robust features can occur across datasets sampled from independent sources. Although this needs to be investigated more thoroughly, our results challenge the view that non-robust features are pecularities of the data distribution. We speculate that some of these features could have meaning in terms of human semantics, like our illustrative example where the eye color predictive non-robust feature.

---

http://i.....bb////////o/eehhase Under review as a conference paper at ICLR 2021
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense se- curity: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274-283, 2018.
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. Sgd learns over- parameterized networks that provably generalize on linearly separable data. In International Conference on Learning Representations, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017.
Luke N. Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey, Cinic- 10 is not imagenet or cifar-10, 10, 2018.
Gabriel Goh. A discussion of 'adversarial examples are not bugs, they are features': Two examples of useful, non-robust features. Distill, 2019a. doi: 2229111/istill..0001..3
Gabriel Goh. A discussion of 'adversarial examples are not bugs, they are features': Robust fea- ture leakage. Distill, 2019b. doi: 00123915/iistill0000122 https://distill.pubbb 11120009/dvex-bbiee discussion/response-..
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal IEEE, 2013.
processing, pp.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-But Buc, E. Fox, and R. Garett (eds.), Ad- vances in Neural Information Processing Systems 32, pp. 125-136. Curran Associates, Inc., 2019. URL http: http:::///ppp..appeeeee wwwpsscccpaaeee are not--oot---- bugs - they- are features. pdf
8307 adversaries -eaamplexaplssss
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory, pp. 1772-1798, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, pp. 8157- 8166, 2018
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 32, pp. 11674-11685. Curran Associates, Inc., nips wttp://.c...ppp/////-aaaaaa--ee. the- sraiining-neurrll-meewwrt..
Wallach, H. Larochelle,
2019. URL
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
10

---

Under review as a conference paper at ICLR 2021
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? 2019.
Preetum Nakkiran. A discussion of 'adversarial examples are not bugs, they are features':
Adversarial examples are just bugs, too. Distill, 2019. doi: 10.23915/distill.00011.. https://dsstill.pbbbt http:al......//10///////bbbugee wwwpawww-bbbsssssuussooeereeeeee
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity, arXiv preprint arXiv: 11604, 2019.
Nicolas Papernot, Patrick McDaniel, and lan Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference computer and communications security, pp. 506-519, 2017.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems, pp. 5014-5026, 2018.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6799 2013.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 1):1958-11199
2008.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv:1704.03455 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
A PROOFS OF THEOREMS IN SECTION 5
Consider using Gradient Descent with a learning rate of to minimize the squared loss as described in Section [ Then,
Wi+1 == w. + 1 - Aw,)
Letting = n/n, it can be proved by induction that
w; ww AAT
Σ Lk=
- AAATYA
(B (B-Aus) Awa)
Let the largest eigenvalue of AA be Amar- If [1 admaal 5 1, then
+ AT (AA")
(T a AAT) (B Awo)
ur = A"A Awo) w00
(10)
for some very large number of steps T. This achieves zero empirical training error, as we can verify that Awr B.
11

---

Under review as a conference paper at ICLR 2021
Let the first and second column of A be and respectively. Since (with high probability) coordi- nates to are orthogonal for all training points,
AA" + 88" + trp"
Other than 1, the eigenvalues of this matrix are
1 (5 rT + (SS' r r) 4()
2
It's easy to see that s² = n., n = nE? Let s² r = r" 8 = neB. Then since e is small,
(s 4 = / 22) 4(E3)²
= n(1 + 22 1))
Then the eigenvalues
A 1=1++n(1 =1=++11+ 2 1+ nc² 2 + 33)
Theorem 1: I S. 211-2p)(112pp) then at the end of the first step, with high probability,
the model will rely on the robust feature for classificatin,, i.e, ur 2 ew] (2) and will have population accuracy of
Proof.
w i W + as TBB.BB. B- Awo)
(1) (1)
w (2) w + or (B - Awwp)
2 aw (2) = as TT 2 Ear T B
w
It is easy to see that
E c B = 2(1
2p k
E
T B = (1 2p),
Since sufficiently large, these random variables are close to their mean with high probability.
So,
(1 (1-2p) 2 2 (1 (1-
2p
This is true by the assumed bound on
Theorem 2: Define
2p(1 + no2) 2p(1 - t) k;
2p(1 + nc?) c)
Then nn < + as the number of gradient steps goes to infinity,
k 2 kú, sample accuracy approaches and population accuracy approaches 1 (p/k) with high probability.
if k < kis sample accuracy approaches and population accuracy approaches with high probability.
Proof. Gradient descent will converge if
2
<
17
+ 2 (1/n)
Using the fact that 3 < 1 gives us the bound on learning rate in the theorem statement.
Next, using Equation 10
WT AT S8 + Awo) + up
12

---

Under review as a conference paper at ICLR 2021
(1 + +rr"t (1- (1+ 8" ()(r"" + (1 + r" r)8"") - r) s)(s²)
(1 + 8 s) + + (s?
8" + 55 + + - Awo) + w (1)
WT
(1 ( + n (1 + ne )²² 8J (n² (2²
8
B
(1 + n1 + ne²) - n 23
(1 nc²) n (ne3)r
B
(1 + [1 + :n22) - 2,22232
w
rT + 88 +rr ) Awo) + w (2)
nT (1+n)(nee)) )2² + (1 + ne= (ncB) (n22282)r" (n²
B
(1+ + nE2) n22232
(1+ - -( (ne (1+ + nE2) - 2222
where have used the fact that wo O. Now suppose we sample new point (X,Y) from the data distribution. Let denote the index of the noise coordinate of X; and let g denote the index of the noise coordinate of X. With high probability, q # q: vi. So,
X X = x xf (1) + x (1 + X x( (4)
= = (1) + x() w/ (2) + w (a)
= =x (1) + () (2)
We want to analyze the case when the first and second coordinate disagree. Let X = -1 and (²) = In this scenario if the model always predicts X WT 2 0, it will match the prediction of the second coordinate and achieve a population accuracy of p/k. On the other hand it always predicts X X"ur "T < 0, it will match the prediction of the first coordinate and achieve a population accuracy of 1 - p.
X WT > 0
(2) = Ew, > WT
(1)
(1+ + n) n)r T (nE
(1 ne²)s² ((nE3))
' + (1+n)(1 + nc2) 2,32
B
B
(1 + n1 + nc?) - 2222
With high probability, sTB n(1 11 2p), rT B ne(1 -1 -2p/k), and 3 (1 p(: 1)/k+ A/2/kk)
- = (1(1 =c((+n)ne n)ne (1-2) 2p' - (ne 220] > (1
(1+ nEn
- 2p) (nEß)ne (1-
2p
k
E2) 2 n(e2 - -1) 2pm (1-
+ 26? (
(1- k
(1-1) 1
(1
2 0
k:
x 2p(1 + nc2) 2p(11 - c)
2p(1 1 + nE2) - (1 - E)
13

---

Under review as a conference paper at ICLR 2021
B OTHER DATASETS AND ARCHITECTURES
In this section, we provide training graphs illustrating Pathway 1 and 2 for Resnet18 and Resnet50 models trained the Datt versions of the CIFAR-10 and CINIC-10 (Darlow al. (2018) datasets. Along with each graph, we mention the hyperparameters used.
60
Traningg
Traring
60-
Valiiationn
Non-rozunt Ace
Acurrayy
60-
Mon robutt Acc Robbuct Acc
klation
Robbst ACc
(a) Pathway (No aug. 0.1. L2 Se-4)
(b) Pathway 2 (No aug, LR 0.01, L2 0)
Figure Resnet50, CIFAR-10 D
30-
80-
Vall Validicion Acc Mon- ror
Robbsi
Training Ass
60
Vilcce
Man-rabum
23-
60
60 Spoce
60
100
330
40
60
10
100
125
Tpach
(a) Pathway (No aug, LR 0.1, L2 Se-4)
(b) Pathway 2 (No aug, LR 0.01, L2
Figure Resnet18, CIFAR-IO D
100
162
30-
80
Traiiing Vall Ialidisionn.o Acc Man-robuuct Rebbuct
Train biinng
Acc
/ wm
23-
120
180
125
bpoch
Spoch
(a) Pathway (Aug, LR 0.1, L2 Sc-4)
(b) Pathway (Aug, LR 0.1, L, Sc-4)
Figure 7: Resnet50, CINIC-10 D
14

---

Under review as a conference paper at ICLR 2021
100-
150-
Tarinn
Non-mouutt Aer Robuutt
Accusay
40
60
50
100
20
40
00
100
120
Pathway
Pathway 2
Figure 8: Resnet CINIC-10 Det
15

---

Under review as a conference paper at ICLR 2021
C HYPERPARAMETERS
In this section, we list the hyperparameters used in our experiments. We also take the case of Resnet18 trained on Dett CIFAR-10 and look which hyperparameters lead Pathway and which lead to Pathway 2. As we note in Section there is a sharp transition between the two pathways the space of hyperparameters.
Common Parameters:
Parameter
Value
Batch Size
128
LR Annealing
x0.1 at Epoch 100
Optimizer
SGD
Momentum
0.9
PGD norm
L2
PGD epsilon
0.5
PGD steps
100
Step size
0.1
Table These parameters were used throughout all our experiments
Training on Non-transferable examples (Section 3
We trained the model using 120 epochs of SGD, and did a grid search over the following combina- tions of hyperparameters.
Parameter
Values {0.01, 0.1) (Tree, False]
Learning Rate
Data Augmentation
L2 Weight Decay
{0, le-5, 5e-4)
Table 4: Hyperparameter grid for training on con-transferable adversarial examples
CIFAR Daet (Figure 1
Pathway LR 0.1, No data augmentation, L2 5e-4
Pathway 2: LR 0.01, No data augmentation, L2
CIFAR (Figure
Pathway 1: LR 0.01, Data augmentation, L2 5c-4
Pathway 2 LR 0.01, No data augmentation, L2
Which hyperparameters lead to Pathway and Pathway 2?
We pick one model (Resnet18) and one dataset (CIFAR-10) 10), and explore which hyperparameters lead to Pathway and which lead Pathway 2. To re-iterate, Pathway 2 is characterized by the model learning almost exclusively only the robust features. All models were trained with SGD for 120 epochs, and the reported accuracies after the final epoch.
16

---

Under review as a conference paper at ICLR 2021
Learning Rate
Data Aug
L2 decay
0.1
0.01
0
30.9/ 43.0
24.4/ 50.66
Yes
5e-4
38.3 26.2
30.9 43.0
0
2
2.6 80.7
le-5
4.2 79.1
No
5e-5
2.7 / 80.1
le-4
2.6/
2e-4
4.0/
2.25e-4
18.6 43.44
2.5e-4
13.5 49.3
2.75e-4
30.4 24.7
3e-4
27.1 30.6
5e-4
35.9/ 18.6
20.9 /
Table 5: (Test/ Shifted Test) accuracy for Resnet1 8 model trained on CIFAR-IO Daet- Corresponds to (Non-robust Accuracy Robust Accuracy). The higher the two is bold, and the configurations which follow Pathway 2 have been highlighted in green
in the case with LR 0.1 and data augmentation, the network exhibits sharp transition from Pathway 2 to Pathway 1 in the space of hyperparameters. There is narrow "middle ground" around L2 2.5e-4.
Remark: As
Cross-Datase Transfer (Table
TIFAR-minus-Flicke (clean): Adam optimizer, LR 1e-3, L2 le-5, with data augmentation. CINIC-10 (clean): : LR 0.01, data augmentation, L2 5e-4.
CIFAR-minus-Fiice Daet LR 0.1, No data augmentation, L2 5e-4.
CINIC-10 Deet: LR 0.1, No data augmentation, L2 5e-4.
17


Reviews:
[{"review_id": "gmLVOKwhl_i", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work studies the learning dynamics of neural networks in terms of robust and non-robust features. In particular, the authors argue that depending on various factors (e.g. learning rate, data augmentation), neural networks will have learning dynamics according to 1 of 2 pathways. Neural networks will either (1) first learn predictive robust features and weakly predictive non-robust features, followed by predictive non-robust features; or (2) only learn robust features, then overfit the training set, thereby not learning non-robust features. The paper has a good discussion expanding upon the robust/non-robust features model of Ilyas et. al. 2019, interesting experiments measuring what features the models is learning, and a digression that presents further results on non-robust features in different datasets.\n\nRecommendation and Explanation:\n\nI would currently recommend a rejection, but I believe the paper is on the borderline and may be improved after clarifications.\n\nThe strengths of this paper to me are the discussion in section 2 regarding the robust and non-robust features model of Ilyas et. al, and the experiments measuring “robust accuracy” (accuracy derived from robust features only) and “Non-robust Accuracy” (accuracy derived from non-robust features only). It was interesting to see the discussion of contaminated robust features and the distinction between weakly predictive and highly predictive non-robust features. The experiment relating adversarial transferability and non-robust features is clear and shows that transferable adversarial examples likely have more non-robust features than non-transferable adversarial examples. Overall, the paper has good ideas and studies a topic that merits further investigation.\n\nThe weaknesses of this paper to me are unclear explanations of Li et. al 2019, some confusion I have about Pathway 1 vs. Pathway 2, and problems with the organization of the paper that could be improved (e.g. a definition that was abandoned after its introduction). After reading the paper, I have many more questions regarding what the authors did. In rough order of their appearance in the paper:\n\nSection 2: In Ilyas et. al, don’t their experiments show that models generalize to the original distribution (and not to the distribution with flipped labels)? Does this mean that there are not many “contaminants” being learned? How can you evaluate if a robust feature contains “contaminants”?\n\nSection 2: You propose a new definition to exclude contaminated robust features (which are non-robust features), but I did not see you use it again. Is there a way to leverage this definition to further our understanding?\n\nSection 4.1: A diagram explaining what each type of accuracy means could improve clarity.\n\nSection 4.2: How many of the changes described (LR, weight decay, data augmentation) are necessary for transitioning from Pathway 1 to Pathway 2? Can you perform an ablation study to see if any one factor is the most important?\n\nSection 4.2: Is Pathway 2 ever preferable to Pathway 1? Or does robust accuracy matter less than overall accuracy? Explaining when one might be better than the other would be insightful.\n\nSection 4.3: I feel that using the terminology “easy-to-generalize” (as is written in the Li et. al 2019 paper) as opposed to “well-generalizing” is more clear. “Well-generalizing”/”not so well-generalizing” seems to imply that those features will lead to good/bad generalization accuracy, but the Li et. al 2019 paper describes “easy-to-generalize”/”hard-to-generalize” features as those that are not noisy/noisy, and thus easy/hard to learn from and generalize from. In Li et. al 2019, my understanding is that you can get good generalization accuracy from both easy-to-generalize and hard-to-generalize features; one is simply easier to learn from. Furthermore, Li et. al does not claim that all hard-to-fit features are easy-to-generalize or vice versa; they simply analyze how learning rate matters when the dataset does satisfy those properties. Thus, the authors of this paper need to significantly improve their discussion of Li et. al 2019 and how their work relates to it.\n\nSection 5: Isn’t it unlikely that n<<d in deep learning? If so, this should be mentioned as a caveat.\n\nSection 6: Explaining why the D_rand experiment was not done for earlier sections should be placed earlier, rather than in Section 6.\n\nIf these questions can be clarified, I believe that will make the paper better.\n\n\nPost Rebuttal Update:\n\nAfter reading the author's rebuttal as well as discussion with other reviewers, I will maintain my score. I do appreciate the changes the authors made, and believe that they improve the paper (especially the new example with cat and dog features and the associated figure). I would encourage the authors to incorporate the remaining feedback, as it will be helpful as well.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YTfQOdeDyTY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper posits some phenomena on neural network training: 1. With some proper regularizing effect, NN training tends to learn predictive robust features (and weakly predictive non-robust features) first and non-robust features next. 2. Without regularization, NN training does a similar thing as case 1 first but does not learn predictive non-robust features and overfits the training examples. \n\n\nI find the results vague and hard to interpret. The paper is written in a sloppy way. The intuition is clear but many experimental settings are quite hard to follow. Specifically, the paper constantly omits names of the model or dataset mentioned in the context and refer to them with \"it\" or very long descriptive language later, which is quite ambiguous or informal. The discussions and results are not very well-organized. It is also not clear what is the main result and contribution. \n\nThe empirical studies seem to be very far away from all the terms \"gamma-robust features\" and \"rho-useful features\" defined before. It is tested on some contrived test sets that vaguely correlated with the so-called robust/non-robust accuracy. I also don't understand how the empirical studies verify that with both procedures, they first learn \"weakly predictive non-robust features\". What even is \"weakly predictive non-robust features\" and how is it reflected in plots?\n\n\nThe biggest problem is the results do not give us constructive ways to improve the training. Even if the paper could give a strong empirical verification that a network overfits due to the fact that it does not learn predictive robust features without proper regularization, it is unclear to me how we could take action to improve the training procedure. The contributions of the paper are not very obvious to me. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jq9pTa2dNIu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper reveals interesting phenomenons of the training dynamics of SGD on convolutional neural networks for image classification. From carefully designed experiments and simplified theories, it argues that SGD in this setting follows two different pathways, depending on the regularizations and hyperparameters. In the first pathway where regularizations are applied and initial learning rate is large, the network first learns predictive robust features and weakly-predictive non-robust features, and then learns to incorporate predictive non-robust features. In the second pathway where little or no regularization is applied and initial learning rate is small, the network does the same thing as the first pathway in the beginning, but then starts overfitting to noise. Overall, I think this paper may have discovered something interesting, but I also feel uncertain about the correctness of the messages due to the lack of clearness on the experimental results. I strongly recommend the authors to improve the clearness in the next version.\n\nStrengths:\n1. The phenomenon demonstrated in this paper seems to be interesting. The authors found that CNNs with SGD behave differently in using robust and non-robust features under different hyperparameters.\n\n2. Simplified theoretical analysis is provided to justify the findings. But I have not carefully checked the correctness.\n\nWeaknesses:\n1.  My major concern is about the clearness of the experimental results, from which I cannot tell whether the conclusions are correct. For all the plots, there is not any written definition for the metrics such as \"Shifted Test Acc\" and \"Transfer % to Path 1\", and it is quite difficult to guess the definitions given the complicated setups, which makes me confused and feel quite uncertain about my understanding of the paper. For example, in Figure 2, I can guess what \"shift\" means, but I am not sure whether these accuracies are evaluated on the clean samples or the adversarially perturbed samples from a certain network, and on which subset are the networks tested. With Figure 2, the authors claim that \"the network trained on the examples that transfer generalizes well to the true distribution\". If the accuracies are evaluated on clean samples, then I cannot understand why \"Test Acc\" is higher than \"Shifted Test Acc\" when the model is trained with \"All Examples (66%)\". Intuitively, even the training examples are perturbed, the model should have higher \"Shifted Test Acc\" than \"Test Acc\", since the adversarial examples should have more robust features for the original class than the shifted class. If it is evaluated on perturbed samples, then the network might have just learned to depend on the adversarial perturbations, since we already know the perturbations for \"Non-Transferable (33%)\" do not transfer well but those for \"All Examples (66%)\" transfer well. If Figure 2 cannot justify \"a neural network being unable to learn a generalizing model as evidence that there are no highly predictive non-robust features\", then all the following experimental results will be questionable. \n\n2. The definition of robust and non-robust features in Section 2 are confusing, not related to the experimental and theoretical results, and might have some minor issues. First, from $\\mathbb{E}[y\\cdot f_{NR}(x)]>0$ and $\\mathbb{E}[\\inf_{\\delta\\in \\Delta(x)} y\\cdot f_{NR}(x+\\delta)]\\ll 0$, we cannot conclude that $\\mathbb{E}[\\inf_{\\delta\\in \\Delta(x)} y\\cdot f_{NR}(x+\\delta)]\\gg 0$, since we know for $\\delta=0$,  $\\mathbb{E}[-y\\cdot f_{NR}(x+0)]<0$. Second, the sum $f_C$ of the robust $f_R$ and weakly-predictive non-robust $f_{NR}$ does not necessarily satisfy $\\mathbb{E}[\\inf_{\\delta\\in \\Delta(x)} y\\cdot f_{C}(x+\\delta)] < 0$. Let $\\delta_1=\\arg\\min_{\\delta\\in \\Delta(x)} y\\cdot f_{R}(x+\\delta)$, $\\delta_2=\\arg\\min_{\\delta\\in \\Delta(x)} y\\cdot f_{NR}(x+\\delta)$. Given the conditions, it is possible that $f_{NR}(x+\\delta_1)>0$, and $f_R (x+\\delta_2)\\gg 0$, and therefore it is possible that $\\mathbb{E}[\\inf_{\\delta\\in \\Delta(x)} y\\cdot f_{C}(x+\\delta)] > 0$, i.e., the \"contamination\" does not always happen when robust and weakly-predictive non-robust features are both present, which might also be consistent with Remark 1. Section 2 is too confusing that I do not quite follow the motivation for Definition 1, and cannot find where Definition 1 is used later in the paper. \n\n3. Figure 1 (b) seems to conflict with Figure 3 (b). Figure 1 (b) shows the robust accuracy is around 80% at convergence, but Figure 3 (b) says the targeted adv success rate is around 80%, meaning the robust accuracy should be around 20%. \n\n4. The number of hyperparameters tried are not dense enough to show \"there is no middle ground\". I would suggest testing at least 20 points within the current interval for each hyperparameter including lr and weight decay.\n\n5. The title might be a bit misleading. It only describes the process of pathway 1, not pathway 2. \n\n---\n### Updates after rebuttal\nI appreciate the authors' efforts in revising the paper into a more rigorous and illustrative way. However, I still feel the current version difficult to follow, and I am not sure whether the conclusions are really correct or not. The conclusions are drawn from assumptions for empirical observations, but the scale of the experiments is not large enough, and the conditions for the assumptions are not specified rigorously. Therefore, I feel quite reluctant in raising my scores.\n\nMy final suggestions for improvements in the future version:\n\n1. Instead of using accuracies on $\\mathcal{D}_{shift}$ as a proxy for how much of the model’s performance can be attributed to its learning predictive robust features, showing the accuracy of the model on the shifted dataset under adversarial attacks would make it more convincing to me. \n\n2. I would prefer the figures to be close to the text that elaborate the figures. For example, Figure 1 is in Page 2, but the details of this complicated figure is not discussed until Sec. 4.2.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GH1530iAXz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies a novel phenomenon of two different pathways in DNN training that determine whether it learns robust or non-robust features (or both). The paper uses experiments based on adversarial training to validate their hypotheses, and also devise a toy model under which their hypotheses hold.\n\nPros\n- Interesting contribution\n- Provides both empirical and theoretical evidence for their claims\n\nCons\n- Very poorly written\n\nOverall, I think the insight the paper is studying is very interesting. Better understanding of the training dynamics of DNNs is an important problem that merits further study, and the authors are taking an important step in this direction. I also appreciate that the authors provide both empirical and theoretical evidence for their claims.\n\nHowever, this paper is currently poorly written and was very hard to follow. First, their experimental methodology is poorly explained -- e.g., Section 3 and the first paragraph in Section 4.1. They do not use precise terminology, for instance saying they “perturb each example to the next class” (I assume this means they compute an adversarial example for a “7” that is adversarially labelled “8”), and “relabel each adversarial example with its target label (I assume this means label “8”, but I am not 100% sure). Furthermore, they say accuracy on the “shifted labels” corresponds to “robust accuracy”, yet they never explain why this should be the case (again, I think I understand after several re-reads, but the lack of explanation makes it hard to be sure).\n\nAlong the same lines, they never explain *why* they are using their methodology. For instance, I’m not sure exactly how Section 3 is connected to Section 4. They say the point is to establish that “a neural network being unable to learn a generalizing model as evidence that there are no highly predictive non-robust features”, but I don’t understand why they need to replicate the results of Nakkiran (2019) to make this claim. Maybe they are replicating to be careful (which is good!), but they never say so. In addition, I also don’t understand this claim very well; e.g., what is the purpose of this connection? Is it to establish their pathways? Section 4 similarly lacks explanation of their methodology.\n\nFinally, it would also be helpful to have an explanation of why these two pathways are interesting. I do think there is some intrinsic value in decomposing different learning pathways in DNNs, and I can imagine several possible implications, but concretely pointing out some implications of these hypotheses would be very helpful.\n\n-------------------------------------------------------------------------------------------------------------------------------\n\nPost rebuttal: I appreciate the improvement in clarity of the paper; however, I think some work remains to improve the clarity of the paper, especially regarding why the two pathways are interesting. I think expanding on potential implications of this mechanism would be very helpful, in addition to the addressing the remaining issues raised by the other reviewers.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses the dynamics of training neural nets and how they are related to features that are robust and predictive (following Ilyas et al). The reviewers had many comments regarding the presentation of the claim and the validity of the empirical results, as well as their unclear practical implications. The authors have improved the writing somewhat but reviewers still thought the manuscript should be substantially improved so that the claims are clearer and empirical validation is more convincing. 
The authors are also encouraged to discuss their results in the context of results on  inductive bias of deep-learning (e.g., results on NTK, rick-regimes, margin maximization etc). 
  
Decision: Reject