SGD on Neural Networks learns Robust Features before Non-Robust

1 INTRODUCTION . Neural networks have achieved state of the art performance on tasks spanning an array of domains like computer vision , translation , speech recognition , robotics , and playing board games ( Krizhevsky et al . ( 2012 ) ; Vaswani et al . ( 2017 ) ; Graves et al . ( 2013 ) ; Silver et al . ( 2016 ) ) . However in recent years , their vulnerability to adversarial attacks - small , targeted input perturbations , has come under sharp focus ( Szegedy et al . ( 2013 ) ; Papernot et al . ( 2017 ) ; Carlini & Wagner ( 2017 ) ; Athalye et al . ( 2018 ) ; Schmidt et al . ( 2018 ) ) . Ilyas et al . ( 2019 ) propose that neural network vulnerability is at least partly due to neural networks learning well-generalizing but brittle features that are properties of the data distribution . From this point of view , an adversarial example would be constructed by modifying an input of one class slightly such that it takes on the non-robust features of another class . They provide empirical evidence for their theory by training a model on adversarially perturbed examples labeled as the target class , and showing that this model generalizes well to the original , unperturbed distribution . Another unrelated line of work ( Brutzkus et al . ( 2018 ) ; Ji & Telgarsky ( 2019 ) ; Li & Liang ( 2018 ) ) aims to study the properties of the functions learned by gradient descent over the course of training . Nakkiran et al . ( 2019 ) and Mangalam & Prabhu ( 2019 ) independently showed that Stochastic Gradient Descent ( SGD ) learns simple , almost linear functions to start out , but then learns more complex functions as training progresses . Li et al . ( 2019 ) showed that models trained with a low learning rate learn easy-to-generalize but hard-to-fit features first , and thus perform poorly on easy-to-fit patterns . In this paper , we study gradient descent on neural networks from the perspective of robust and nonrobust features . Our main thesis is that based on choices of hyperparameters , neural network training follows one of two pathways , : • Pathway 1 ( Informal ) : The neural network first learns predictive robust features and weakly predictive non-robust features . As training progresses , it learns predictive nonrobust features , and having learned both robust and non-robust predictive features , achieves good performance on held-out data . This is the pathway that Ilyas et al . ( 2019 ) used to prove their theory . • Pathway 2 ( Informal ) : The neural network learns predictive robust features and weakly predictive non-robust features ( as in Pathway 1 ) . But thereafter , it begins to fit the noise in the training set , and quickly achieves zero training error . In this scenario , the network learns only the robust predictive features and shows modest generalization on held-out data . Through a series of experiments , we validate our two-pathway hypothesis , investigate the specific circumstances under which Pathway 1 and Pathway 2 occur , and analyze some properties of the two pathways . We will also develop a closer understanding of the relationship between adversarial transfer and predictive non-robust features , which will aid our analysis of the two pathways . The rest of this paper is organized as follows . Section 2 sets up the notation and definitions we use . In Section 3 , we investigate the link between adversarial features and transferability . In Section 4 we provide empirical evidence for the two-pathway hypothesis and analyze some characteristics of each pathway . Section 5 presents a theoretical analysis of gradient descent on a toy linear model . We show that for different choices of initial parameters , the linear model exhibits properties of the first and second pathways . We digress to explore whether non-robust features can occur across datasets in Section 6 , and discuss future research directions in Section 7 . 2 DEFINITIONS AND PRELIMINARIES . Consider the binary classification setting , where D is a joint distribution over the input space X and the labels { −1 , 1 } 1 . In this setting , Ilyas et al . ( 2019 ) define a feature as any function f : X → R , scaled such that E ( x , y ) ∈D [ f ( x ) ] = 0 and E ( x , y ) ∈D [ f ( x ) 2 ] = 1 . A feature is said to be ρ-useful if E ( x , y ) ∈D [ y · f ( x ) ] > ρ ( 1 ) for some ρ > 0 , and γ-robust if E ( x , y ) ∈D [ inf δ∈∆ ( x ) y · f ( x+ δ ) ] > γ ( 2 ) 1This framework can easily be adapted to the multi-class setting for some γ > 0 and some family of perturbations ∆ . For brevity , we sometimes refer to a ρ-useful , γ-robust feature ( ρ , γ > 0 ) simply as a robust feature . Let ρD ( f ) be the largest ρ for which f is ρ-useful . A feature f is said to be ( highly ) predictive or weakly predictive if ρD ( f ) is high or low respectively . A useful , non-robust feature as defined by Ilyas et al . ( 2019 ) is one that is ρ-useful for some ρ > 0 , but is not γ-robust for any γ > 0 . They propose the following experiment to demonstrate the existence of these features . Let C be a classifier trained with Empirical Risk Minimization ( ERM ) on an empirical distribution D̂ . We operate under the following assumption . Assumption 1 . If a distribution D contains a useful feature , then a classifier C trained with ERM on an empirical distribution D̂ drawn from D will learn this feature , provided that we avoid finite sample overfitting through appropriate measures such as regularization and cross-validation . Let LC ( x , t ) denote the loss of C on input x , for a target label t. Construct adversarial examples by solving the following optimization problem : xadv = arg min ‖x−x′‖≤ LC ( x ′ , t ) ( 3 ) In particular , construct a distribution called D̂det comprised of ( xadv , t ) pairs by using Equation 3 with t chosen deterministically according to y for each ( x , y ) ∈ D̂ . In the binary classification setting , t must be −y , so E ( xadv , t ) ∈D̂det [ t · f ( xadv ) ] > 0 , if f is non-robustly useful under D ( 4 ) E ( xadv , t ) ∈D̂det [ −t · f ( xadv ) ] > 0 , if f is robustly useful under D ( 5 ) It is observed that a neural network trained on D̂det achieves non-trivial generalization to the original test set , that is D. From this , we can conclude that non-robust features exist and are useful for classification in the normal setting . Remark : Goh ( 2019b ) showed that the D̂rand dataset constructed by choosing t randomly in the above procedure , suffers from a sort of “ robust feature leakage ” . PGD introduces faint robust cues in the generated adversarial example that can be learned by the model . But on the Ddet dataset , the robust features are correlated with a deterministic label which is different from t. Hence we use the D̂det dataset in preference to the D̂rand for all our experiments . Two kinds of non-robust features : Goh ( 2019a ) points out a subtle flaw with the above definition of a non-robust feature - highly predictive non-robust features can arise from “ contamination ” of a robust feature with a non-robust feature , instead of something meaningful . To see how this can happen , consider a highly predictive robust feature fR and a weakly predictive non-robust feature fNR . Let fC be a “ contaminated ” feature that is a simple sum of fR and fNR ( appropriately normalized ) . Then it is possible to construct a scenario in which E [ y · fR ( x ) ] > 0 E [ inf δ∈∆ ( x ) y · fR ( x+ δ ) ] > 0 ( 6 ) E [ y · fNR ( x ) ] & 0 E [ inf δ∈∆ ( x ) y · fNR ( x+ δ ) ] 0 ( 7 ) E [ y · fC ( x ) ] > 0 E [ inf δ∈∆ ( x ) y · fC ( x+ δ ) ] < 0 ( 8 ) fC is thus a highly predictive non-robust feature . Now when you train a model on ( x + δ , −y ) pairs , fC = fR + fNR is still correlated with −y . But fC′ = −fR + fNR is more correlated , so the model will learn this combination in preference to fC and will not generalize on the original distribution . In fact , thanks to learning−fR , it will generalize to the distribution with flipped labels , i.e. , y → −y . In our analysis and experiments , when we refer to non-robust features , we will exclude such contaminated features . Illustrative Example : Consider a dataset of dog and cat images , where most dog images have snouts and most cats do not have snouts . Most cats have slightly lighter eyes than dogs , and making the eyes slightly darker or lighter is part of the set of valid adversarial perturbations . Suppose that a very small majority of the dog images start with a pixel that has an odd numbered value . Then the different types of features in this dataset are enumerated in Table 1 . For f2 , ( x + δ , −y ) pairs would be dogs with lighter eyes , labeled as cats . The network trained on these examples will learn Snout =⇒ Cat , Light Eyes =⇒ Cat . Since the eye-color is predictive of the true label , the second feature will ensure that the neural network has non-trivial performance on the original distribution . This is what Ilyas et al . ( 2019 ) observed in their experiments . f2 is thus a true non-robust feature . For f4 , ( x+δ , −y ) pairs would be dog images with the first pixel value converted to an even number , labeled as cats . The network trained on these examples will learn Snout =⇒ Cat , Dark Eyes =⇒ Cat , and Even Pixel =⇒ Cat . None of these will be particularly helpful on the true distribution , but the first two will be useful on the flipped distribution , i.e. , where dogs are relabeled as cats . f4 is thus a contaminated robust feature , and not a non-robust feature . Remark : A network that learns only robust features but with contaminants can still be very vulnerable to adversarial attacks , as the above example shows . The weakly predictive non-robust feature f3 can be manipulated to consistently cause misclassification on out-of-distribution inputs . 3 NON-ROBUST FEATURES AND TRANSFERABILITY . The phenomenon of adversarial transferability ( Papernot et al. , 2016 ) , where a non-trivial fraction of the adversarial examples generated for one neural network are still adversarial to other neural networks trained independently on the same data , can be readily explained in terms of non-robust features . By Assumption 1 , different neural networks trained using ERM on a distribution would learn the predictive non-robust features ( like Dark Eyes =⇒ Dog ) present in the distribution . One would then construct an adversarial example by modifying an input such that the predictive non-robust features flip ( modify all dog images to have lighter eyes ) . Then this adversarial example would transfer to all the different networks that have learned to rely on the non-robust features . A natural question to ask is , does all adversarial transferability arise from predictive non-robust features ? Nakkiran ( 2019 ) showed that by explicitly penalizing transferability during PGD , one can construct adversarial examples that do not transfer , and from which it is not possible to learn a generalizing model . This establishes that adversarial examples that do not transfer , do not contain predictive non-robust features . Here we provide a simpler experiment that constructs non-transferable adversarial examples without explicitly penalizing transferability . This experiment also establishes a stronger claim , that adversarial examples transfer if and only if they exploit predictive non-robust features . Let the CIFAR-10 dataset form the data distributionD . Train two Resnet50 models ( He et al. , 2016 ) on D̂ and ensure by Assumption 1 that both networks have learned the predictive non-robust features of the distribution by using regularization and cross-validation across a grid of hyperparameters . Construct a D̂det dataset for the first network using Equation 3 where t is chosen deterministically according to y using the transformation t = ( y+1 ) % 10 . We use Projected Gradient Descent ( PGD ) ( Madry et al. , 2018 ) to solve the optimization problem in Equation 3 . Split the adversarial examples into two categories - those that transfer to the second network with their target labels , and those that do not . Relabel all adversarial examples xadv with their target label t , and train a Resnet50 model on ( xadv , t ) pairs from each category . As Equations 4 and 5 suggest , for ( xadv , t ) ∼ D̂det , the non-robust features ofD are predictive of t , but the robust features of D are predictive of ( t− 1 ) % 10 . So if a neural network trained on a subset of D̂det learns predictive nonrobust features , it will generalize to D , and if it learns predictive robust features , it will generalize to the shifted distribution Dshift : Dshift = { ( x , ( y + 1 ) % 10 ) : ( x , y ) ∼ D } ( 9 ) Figure 2 shows the performance of these two networks on D and Dshift . We can see that the network trained on the examples that transfer generalizes well to D , but the network trained on the examples that do not transfer generalizes to Dshift . The configuration in the figure is as a result of a thorough grid search over hyperparameters with the metric for success being performance on D. Along with Assumption 1 , our experiment establishes that the examples that transfer contain predictive non-robust features , and the examples that don ’ t transfer don ’ t contain predictive non-robust features . In particular , we claim the following : Claim 1 . Train two networks N1 and N2 on a common dataset such that both networks learn the predictive non-robust features present in the dataset . Then an adversarial example generated forN1 transfers to the second network if and only if this example contains predictive non-robust features . Further , if a neural network C has learned predictive non-robust features , then PGD will construct some adversarial examples with predictive non-robust features ( see Equation 4 ) , and vice-versa . This allows us to infer the following property , which we will use in our analysis in the next section : Claim 2 . If a neural network N2 has learned the predictive non-robust features in a dataset , then adversarial examples generated for another network N1 using PGD will transfer to N2 if and only if N1 has also learned predictive non-robust features .

Summary: The paper posits some phenomena on neural network training: 1. With some proper regularizing effect, NN training tends to learn predictive robust features (and weakly predictive non-robust features) first and non-robust features next. 2. Without regularization, NN training does a similar thing as case 1 first but does not learn predictive non-robust features and overfits the training examples.