How Benign is Benign Overfitting ?

1 INTRODUCTION . Modern machine learning methods achieve a very high accuracy on wide range of tasks , e.g . in computer vision , natural language processing etc . However , especially in vision tasks , they have been shown to be highly vulnerable to small adversarial perturbations that are imperceptible to the human eye ( Dalvi et al. , 2004 ; Biggio & Roli , 2018 ; Goodfellow et al. , 2014 ) . This vulnerability poses serious security concerns when these models are deployed in real-world tasks ( cf . ( Papernot et al. , 2017 ; Schönherr et al. , 2018 ; Hendrycks et al. , 2019b ; Li et al. , 2019a ) ) . A large body of research has been devoted to crafting defences to protect neural networks from adversarial attacks ( e.g . ( Goodfellow et al. , 2014 ; Papernot et al. , 2015 ; Tramèr et al. , 2018 ; Madry et al. , 2018 ; Zhang et al. , 2019 ) ) . However , such defences have usually been broken by future attacks ( Athalye et al. , 2018 ; Tramer et al. , 2020 ) . This arms race between attacks and defenses suggests that to create a truly robust model would require a deeper understanding of the source of this vulnerability . Our goal in this paper is not to propose new defenses , but to provide better answers to the question : what causes adversarial vulnerability ? In doing so , we also seek to understand how existing methods designed to achieve adversarial robustness overcome some of the hurdles pointed out by our work . We identify two sources of adversarial vulnerability that , to the best of our knowledge , have not been properly studied before : a ) memorization of label noise , and b ) improper representation learning . Overfitting Label Noise : Starting with the celebrated work of Zhang et al . ( 2016 ) it has been observed that neural networks trained with SGD are capable of memorizing large amounts of label noise . Recent theoretical work ( e.g . ( Liang & Rakhlin , 2018 ; Belkin et al. , 2018b ; a ; Hastie et al. , 2019 ; Belkin et al. , 2019a ; b ; Bartlett et al. , 2020 ; Muthukumar et al. , 2020 ; Chatterji & Long , 2020 ) ) has also sought to explain why fitting training data perfectly does not lead to a large drop in test accuracy , as the classical notion of overfitting might suggest . This is commonly referred to as memorization or interpolation . We show through simple theoretical models , as well as experiments on standard datasets , that there are scenarios where label noise causes significant adversarial vulnerability , even when high natural ( test ) accuracy can be achieved . Surprisingly , we find that label noise is not at all uncommon in datasets such as MNIST and CIFAR-10 ( see Figure 1 ) . Our experiments show that robust training methods like Adversarial training ( AT ) ( Madry et al. , 2018 ) and TRADES ( Zhang et al. , 2019 ) produce models that incur training error on at least some of the noisy examples , but also on atypical examples from the classes ( Zhang & Feldman , 2020 ) . Viewed differently , robust training methods are unable to differentiate between atypical correctly labelled examples ( rare dog ) and a mislabelled example ( cat labelled as dog ) and end up not memorizing either ; interestingly , the lack of memorizing these atypical examples has been pointed out as an explanation for slight drops in test accuracy , as the test set often contains similarly atypical ( or even identical ) examples in some cases Feldman ( 2019 ) ; Zhang & Feldman ( 2020 ) . We point out this phenomenon for robust models through visual examples on MNIST , CIFAR10 , and Imagenet ( c.f . Figure 4 ) . Representation Learning and Robustness : Recent works ( Tsipras et al. , 2019 ) and Zhang et al . ( 2019 ) have argued that the trade-off between robustness and accuracy might be unavoidable . However , their setting involves a distribution that is not robustly separable by any classifier . In such a situation there is indeed a trade-off between robustness and accuracy . In this paper , we focus on settings where robust classifiers exist , which is a more realistic scenario for real-world data . At least for vision , one may well argue that “ humans ” are robust classifiers , and as a result we would expect that classes are well-separated at least in some representation space . In fact , Yang et al . ( 2020 ) show that classes are already well-separated in the input space . In such situations , there is no need for robustness to be at odds with accuracy . A more plausible scenario which we posit , and provide theoretical evidence in support of in Theorem 2 , is that depending on the choice of representations , the trade-off may exist or can be avoided . Recent empirical work ( Sanyal et al. , 2020a ; Mao et al. , 2020 ) has also established that modifying the training objective to favour certain properties in the learned representations can automatically lead to improved robustness . However , we show in Section 3.2 that some training algorithms can create an apparent trade-off even though the trade-off might not necessarily be fundamental to the problem . On a related note , it has been suggested in recent works that adversarially robust learning may require more “ complex ” decision boundaries , and as a result may require more data ( Shah et al . ; Schmidt et al. , 2018 ; Yin et al. , 2019 ; Nakkiran , 2019 ; Madry et al. , 2018 ) . However , the question of decision boundaries in neural networks is subtle as the network learns a feature representation as well as a decision boundary on top of it . We develop concrete theoretical examples in Theorem 2 and 3 to establish that choosing one feature representation over another may lead to visually more complex decision boundaries on the input space , though these are not necessarily more complex in terms of statistical learning theoretic concepts such as VC dimension . Summary of Theoretical Contributions . 1 . We provide simple sufficient conditions on the data distribution under which any classifier that fits the training data with label noise perfectly is adversarially vulnerable . 2 . There exists data distributions and training algorithms , which when trained with ( some fraction of ) random label noise have the following property : ( i ) using one representation , it is possible to have high natural and robust test accuracies but at the cost of having training error ; ( ii ) using another representation , it is possible to have no training error ( including fitting noise ) and high test accuracy , but low robust accuracy . ( See Theorem 2 ) . The second example shows that the choice of representation matters significantly when it comes to adversarial accuracy , and that memorizing label noise directly leads to loss of robust accuracy . Summary of Experimental Contributions . 1 . As predicted theoretically , neural nets trained to convergence with label noise have greater adversarial vulnerability . ( See Section 3.1 ) . 2 . Robust training methods , such as AT and TRADES that have higher robust accuracy , avoid overfitting ( some ) label noise . This behaviour is also partly responsible for their decrease in natural test accuracy . ( See Section 3.2 ) . 3 . To demonstrate the benefit of representation learning for adversarial robustness , we show that learning richer representation by training with more fine-grained labels , subclasses within each class , leads to higher robust accuracy . ( Due to lack of space we moved this to Appendix C.3 ) . While our primary contribution is showing the effect of overfitting label noise on adversarial robustness , we hope our theoretical and experimental evidences on the importance of representation learning for robustness will inspire further research in this direction . 2 THEORETICAL SETTING . We develop a simple theoretical framework to demonstrate how overfitting , even very minimal , label noise causes significant adversarial vulnerability . We also show in Theorem 2 and 3 how the choice of representation can significantly affect robust accuracy . Although we state the results for binary classification , they can easily be generalized to multi-class problems . We formally define the notions of natural ( test ) error and adversarial error . Definition 1 ( Natural and Adversarial Error ) . For any distribution D defined over ( x , y ) ∈ Rd × { 0 , 1 } and any binary classifier f : Rd → { 0 , 1 } , • the natural error is R ( f ; D ) = P ( x , y ) ∼D [ f ( x ) 6= y ] , ( 1 ) • if Bγ ( x ) is a ball of radius γ ≥ 0 around x under some norm1 , the γ-adversarial error is RAdv , γ ( f ; D ) = P ( x , y ) ∼D [ ∃z ∈ Bγ ( x ) ; f ( z ) 6= y ] , ( 2 ) In the rest of the section , we provide theoretical results to show the effect of overfitting label noise on the robustness of classifiers . 2.1 OVERFITTING LABEL NOISE . The following result provides a sufficient condition under which even a small amount of label noise causes any classifier that fits the training data perfectly to have significant adversarial error . Informally , Theorem 1 states that if the data distribution has significant probability mass in a union of ( a relatively small number of , and possibly overlapping ) balls , each of which has roughly the same probability mass ( cf . Eq . ( 3 ) ) , then even a small amount of label noise renders this entire region vulnerable to adversarial attacks to classifiers that fit the training data perfectly . Theorem 1 . Let c be the target classifier , and let D be a distribution over ( x , y ) , such that y = c ( x ) in its support . Using the notation PD [ A ] to denote P ( x , y ) ∼D [ x ∈ A ] for any measurable subset A ⊆ Rd , suppose that there exist c1 ≥ c2 > 0 , ρ > 0 , and a finite set ζ ⊂ Rd satisfying PD ⋃ s∈ζ Bpρ ( s ) ≥ c1 and ∀s ∈ ζ , PD [ Bpρ ( s ) ] ≥ c2|ζ| ( 3 ) 1Throughout , we will mostly use the ( most commonly used ) ` ∞ norm , but the results hold for other norms . ( a ) Toy-MNIST , = 64 255 ( b ) Full-MNIST where Bpρ ( s ) represents a ` p-ball of radius ρ around s. Further , suppose that each of these balls contain points from a single class i.e . for all s ∈ ζ , for all x , z ∈ Bpρ ( s ) : c ( x ) = c ( z ) . Let Sm be a dataset of m i.i.d . samples drawn from D , which subsequently has each label flipped independently with probability η . For any classifier f that perfectly fits the training data Sm i.e . ∀ x , y ∈ Sm , f ( x ) = y , ∀δ > 0 and m ≥ |ζ|ηc2 log ( |ζ| δ ) , with probability at least 1 − δ , RAdv,2ρ ( f ; D ) ≥ c1 . The goal is to find a relatively small set ζ that satisfies the condition as this will mean that even for modest sample sizes , the trained models have significant adversarial error . We remark that it is easy to construct concrete instantiations of problems that satisfy the conditions of the theorem , e.g . each class represented by a spherical ( truncated ) Gaussian with radius ρ , with the classes being well-separated satisfies Eq . ( 3 ) . The main idea of the proof is that there is sufficient probability mass for points which are within distance 2ρ of a training datum that was mislabelled . We note that the generality of the result , namely that any classifier ( including neural networks ) that fits the training data must be vulnerable irrespective of its structure , requires a result like Theorem 1 . For instance , one could construct the classifier h , where h ( x ) = c ( x ) , if ( x , b ) 6∈ Sm for b = 0 , 1 , and h ( x ) = y if ( x , y ) ∈ Sm . Note that the classifier h agrees with the target c on every point of Rd except the mislabelled training examples , and as a result these examples are the only source of vulnerability . The complete proof is presented in Appendix B.1 . There are a few things to note about Theorem 1 . First , the lower bound on adversarial error applies to any classifier f that fits the training data Sm perfectly and is agnostic to the type of model f is . Second , for a given c1 , there maybe multiple ζs that satisfy the bounds in ( 3 ) and the adversarial risk holds for all of them . Thus , smaller the value of |ζ| the smaller the size of the training data it needs to fit and it can be done by simpler classifiers . Third , if the distribution of the data is such that it is concentrated around some points then for a fixed c1 , c2 , a smaller value of ρ would be required to satisfy ( 3 ) and thus a weaker adversary ( smaller perturbation budget 2ρ ) can cause a much larger adversarial error . In practice , classifiers exhibit much greater vulnerability than purely arising from the presence of memorized noisy data . Experiments in Section 3.1 show how label noise causes vulnerability in a toy MNIST model , the full MNIST and CIFAR10 for a variety of architectures .

Summary: The main contribution of the paper is to study the connection between adversarial robustness, on the one hand, and label noise & data representation on the other hand. Here, an algorithm is said to be robust if for every training example xi with label yi, one cannot find an instance x within a small distance of xi that is assigned a different label from yi by the model. This is a standard definition of robustness in the literature.