How Benign is Benign Overfitting ?

1 INTRODUCTION . Modern machine learning methods achieve a very high accuracy on wide range of tasks , e.g . in computer vision , natural language processing etc . However , especially in vision tasks , they have been shown to be highly vulnerable to small adversarial perturbations that are imperceptible to the human eye ( Dalvi et al. , 2004 ; Biggio & Roli , 2018 ; Goodfellow et al. , 2014 ) . This vulnerability poses serious security concerns when these models are deployed in real-world tasks ( cf . ( Papernot et al. , 2017 ; Schönherr et al. , 2018 ; Hendrycks et al. , 2019b ; Li et al. , 2019a ) ) . A large body of research has been devoted to crafting defences to protect neural networks from adversarial attacks ( e.g . ( Goodfellow et al. , 2014 ; Papernot et al. , 2015 ; Tramèr et al. , 2018 ; Madry et al. , 2018 ; Zhang et al. , 2019 ) ) . However , such defences have usually been broken by future attacks ( Athalye et al. , 2018 ; Tramer et al. , 2020 ) . This arms race between attacks and defenses suggests that to create a truly robust model would require a deeper understanding of the source of this vulnerability . Our goal in this paper is not to propose new defenses , but to provide better answers to the question : what causes adversarial vulnerability ? In doing so , we also seek to understand how existing methods designed to achieve adversarial robustness overcome some of the hurdles pointed out by our work . We identify two sources of adversarial vulnerability that , to the best of our knowledge , have not been properly studied before : a ) memorization of label noise , and b ) improper representation learning . Overfitting Label Noise : Starting with the celebrated work of Zhang et al . ( 2016 ) it has been observed that neural networks trained with SGD are capable of memorizing large amounts of label noise . Recent theoretical work ( e.g . ( Liang & Rakhlin , 2018 ; Belkin et al. , 2018b ; a ; Hastie et al. , 2019 ; Belkin et al. , 2019a ; b ; Bartlett et al. , 2020 ; Muthukumar et al. , 2020 ; Chatterji & Long , 2020 ) ) has also sought to explain why fitting training data perfectly does not lead to a large drop in test accuracy , as the classical notion of overfitting might suggest . This is commonly referred to as memorization or interpolation . We show through simple theoretical models , as well as experiments on standard datasets , that there are scenarios where label noise causes significant adversarial vulnerability , even when high natural ( test ) accuracy can be achieved . Surprisingly , we find that label noise is not at all uncommon in datasets such as MNIST and CIFAR-10 ( see Figure 1 ) . Our experiments show that robust training methods like Adversarial training ( AT ) ( Madry et al. , 2018 ) and TRADES ( Zhang et al. , 2019 ) produce models that incur training error on at least some of the noisy examples , but also on atypical examples from the classes ( Zhang & Feldman , 2020 ) . Viewed differently , robust training methods are unable to differentiate between atypical correctly labelled examples ( rare dog ) and a mislabelled example ( cat labelled as dog ) and end up not memorizing either ; interestingly , the lack of memorizing these atypical examples has been pointed out as an explanation for slight drops in test accuracy , as the test set often contains similarly atypical ( or even identical ) examples in some cases Feldman ( 2019 ) ; Zhang & Feldman ( 2020 ) . We point out this phenomenon for robust models through visual examples on MNIST , CIFAR10 , and Imagenet ( c.f . Figure 4 ) . Representation Learning and Robustness : Recent works ( Tsipras et al. , 2019 ) and Zhang et al . ( 2019 ) have argued that the trade-off between robustness and accuracy might be unavoidable . However , their setting involves a distribution that is not robustly separable by any classifier . In such a situation there is indeed a trade-off between robustness and accuracy . In this paper , we focus on settings where robust classifiers exist , which is a more realistic scenario for real-world data . At least for vision , one may well argue that “ humans ” are robust classifiers , and as a result we would expect that classes are well-separated at least in some representation space . In fact , Yang et al . ( 2020 ) show that classes are already well-separated in the input space . In such situations , there is no need for robustness to be at odds with accuracy . A more plausible scenario which we posit , and provide theoretical evidence in support of in Theorem 2 , is that depending on the choice of representations , the trade-off may exist or can be avoided . Recent empirical work ( Sanyal et al. , 2020a ; Mao et al. , 2020 ) has also established that modifying the training objective to favour certain properties in the learned representations can automatically lead to improved robustness . However , we show in Section 3.2 that some training algorithms can create an apparent trade-off even though the trade-off might not necessarily be fundamental to the problem . On a related note , it has been suggested in recent works that adversarially robust learning may require more “ complex ” decision boundaries , and as a result may require more data ( Shah et al . ; Schmidt et al. , 2018 ; Yin et al. , 2019 ; Nakkiran , 2019 ; Madry et al. , 2018 ) . However , the question of decision boundaries in neural networks is subtle as the network learns a feature representation as well as a decision boundary on top of it . We develop concrete theoretical examples in Theorem 2 and 3 to establish that choosing one feature representation over another may lead to visually more complex decision boundaries on the input space , though these are not necessarily more complex in terms of statistical learning theoretic concepts such as VC dimension . Summary of Theoretical Contributions . 1 . We provide simple sufficient conditions on the data distribution under which any classifier that fits the training data with label noise perfectly is adversarially vulnerable . 2 . There exists data distributions and training algorithms , which when trained with ( some fraction of ) random label noise have the following property : ( i ) using one representation , it is possible to have high natural and robust test accuracies but at the cost of having training error ; ( ii ) using another representation , it is possible to have no training error ( including fitting noise ) and high test accuracy , but low robust accuracy . ( See Theorem 2 ) . The second example shows that the choice of representation matters significantly when it comes to adversarial accuracy , and that memorizing label noise directly leads to loss of robust accuracy . Summary of Experimental Contributions . 1 . As predicted theoretically , neural nets trained to convergence with label noise have greater adversarial vulnerability . ( See Section 3.1 ) . 2 . Robust training methods , such as AT and TRADES that have higher robust accuracy , avoid overfitting ( some ) label noise . This behaviour is also partly responsible for their decrease in natural test accuracy . ( See Section 3.2 ) . 3 . To demonstrate the benefit of representation learning for adversarial robustness , we show that learning richer representation by training with more fine-grained labels , subclasses within each class , leads to higher robust accuracy . ( Due to lack of space we moved this to Appendix C.3 ) . While our primary contribution is showing the effect of overfitting label noise on adversarial robustness , we hope our theoretical and experimental evidences on the importance of representation learning for robustness will inspire further research in this direction . 2 THEORETICAL SETTING . We develop a simple theoretical framework to demonstrate how overfitting , even very minimal , label noise causes significant adversarial vulnerability . We also show in Theorem 2 and 3 how the choice of representation can significantly affect robust accuracy . Although we state the results for binary classification , they can easily be generalized to multi-class problems . We formally define the notions of natural ( test ) error and adversarial error . Definition 1 ( Natural and Adversarial Error ) . For any distribution D defined over ( x , y ) ∈ Rd × { 0 , 1 } and any binary classifier f : Rd → { 0 , 1 } , • the natural error is R ( f ; D ) = P ( x , y ) ∼D [ f ( x ) 6= y ] , ( 1 ) • if Bγ ( x ) is a ball of radius γ ≥ 0 around x under some norm1 , the γ-adversarial error is RAdv , γ ( f ; D ) = P ( x , y ) ∼D [ ∃z ∈ Bγ ( x ) ; f ( z ) 6= y ] , ( 2 ) In the rest of the section , we provide theoretical results to show the effect of overfitting label noise on the robustness of classifiers . 2.1 OVERFITTING LABEL NOISE . The following result provides a sufficient condition under which even a small amount of label noise causes any classifier that fits the training data perfectly to have significant adversarial error . Informally , Theorem 1 states that if the data distribution has significant probability mass in a union of ( a relatively small number of , and possibly overlapping ) balls , each of which has roughly the same probability mass ( cf . Eq . ( 3 ) ) , then even a small amount of label noise renders this entire region vulnerable to adversarial attacks to classifiers that fit the training data perfectly . Theorem 1 . Let c be the target classifier , and let D be a distribution over ( x , y ) , such that y = c ( x ) in its support . Using the notation PD [ A ] to denote P ( x , y ) ∼D [ x ∈ A ] for any measurable subset A ⊆ Rd , suppose that there exist c1 ≥ c2 > 0 , ρ > 0 , and a finite set ζ ⊂ Rd satisfying PD ⋃ s∈ζ Bpρ ( s ) ≥ c1 and ∀s ∈ ζ , PD [ Bpρ ( s ) ] ≥ c2|ζ| ( 3 ) 1Throughout , we will mostly use the ( most commonly used ) ` ∞ norm , but the results hold for other norms . ( a ) Toy-MNIST , = 64 255 ( b ) Full-MNIST where Bpρ ( s ) represents a ` p-ball of radius ρ around s. Further , suppose that each of these balls contain points from a single class i.e . for all s ∈ ζ , for all x , z ∈ Bpρ ( s ) : c ( x ) = c ( z ) . Let Sm be a dataset of m i.i.d . samples drawn from D , which subsequently has each label flipped independently with probability η . For any classifier f that perfectly fits the training data Sm i.e . ∀ x , y ∈ Sm , f ( x ) = y , ∀δ > 0 and m ≥ |ζ|ηc2 log ( |ζ| δ ) , with probability at least 1 − δ , RAdv,2ρ ( f ; D ) ≥ c1 . The goal is to find a relatively small set ζ that satisfies the condition as this will mean that even for modest sample sizes , the trained models have significant adversarial error . We remark that it is easy to construct concrete instantiations of problems that satisfy the conditions of the theorem , e.g . each class represented by a spherical ( truncated ) Gaussian with radius ρ , with the classes being well-separated satisfies Eq . ( 3 ) . The main idea of the proof is that there is sufficient probability mass for points which are within distance 2ρ of a training datum that was mislabelled . We note that the generality of the result , namely that any classifier ( including neural networks ) that fits the training data must be vulnerable irrespective of its structure , requires a result like Theorem 1 . For instance , one could construct the classifier h , where h ( x ) = c ( x ) , if ( x , b ) 6∈ Sm for b = 0 , 1 , and h ( x ) = y if ( x , y ) ∈ Sm . Note that the classifier h agrees with the target c on every point of Rd except the mislabelled training examples , and as a result these examples are the only source of vulnerability . The complete proof is presented in Appendix B.1 . There are a few things to note about Theorem 1 . First , the lower bound on adversarial error applies to any classifier f that fits the training data Sm perfectly and is agnostic to the type of model f is . Second , for a given c1 , there maybe multiple ζs that satisfy the bounds in ( 3 ) and the adversarial risk holds for all of them . Thus , smaller the value of |ζ| the smaller the size of the training data it needs to fit and it can be done by simpler classifiers . Third , if the distribution of the data is such that it is concentrated around some points then for a fixed c1 , c2 , a smaller value of ρ would be required to satisfy ( 3 ) and thus a weaker adversary ( smaller perturbation budget 2ρ ) can cause a much larger adversarial error . In practice , classifiers exhibit much greater vulnerability than purely arising from the presence of memorized noisy data . Experiments in Section 3.1 show how label noise causes vulnerability in a toy MNIST model , the full MNIST and CIFAR10 for a variety of architectures .

Summary: The generalization ability of networks with zero training error has been heavily studied.  This paper extends beyond generalization to test sets to study the network's robustness to adversarial examples.  The paper provides two theoretical contributions demonstrating that a very low training error can indicate poor robustness under reasonable conditions.  They illustrate this with experiments using label noise, demonstrating that adversarially robust networks spurn overfitting on incorrectly labelled data.  They additionally experimentally demonstrate that unusual training examples, even if correctly labelled, are unlikely to be correctly predicted by adversarially robust networks.