Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training

1 Introduction . Although machine learning ( ML ) models have achieved superior performance on many challenging tasks , their performance can be largely deteriorated when the training and test distributions are different . For instance , standard models are prone to make mistakes on the adversarial examples that are considered as worst-case data at test time [ 10 , 113 ] . Compared with that , a more threatening and easily overlooked threat is the malicious perturbations at training time , i.e. , the delusive attacks [ 82 ] that aim to maximize test error by slightly perturbing the correctly labeled training examples [ 6 , 7 ] . In the era of big data , many practitioners collect training data from untrusted sources where delusive adversaries may exist . In particular , many companies are scraping large datasets from unknown users or public websites for commercial use . For example , Kaspersky Lab , a leading antivirus company , has been accused of poisoning competing products [ 7 ] . Although they denied any wrongdoings and clarified the false rumors , one can still imagine the disastrous consequences if that really happens in the security-critical applications . Furthermore , a recent survey of 28 organizations found that these industry practitioners are obviously more afraid of data poisoning than other threats from adversarial ML [ 64 ] . In a nutshell , delusive attacks has become a realistic and horrible threat to practitioners . Recently , Feng et al . [ 32 ] showed for the first time that delusive attacks are feasible for deep networks , by proposing “ training-time adversarial data ” that can significantly deteriorate model performance on clean test data . However , how to design learning algorithms that are robust to delusive attacks still remains an open question due to several crucial challenges [ 82 , 32 ] . First of all , delusive attacks †Corresponding author : Songcan Chen < s.chen @ nuaa.edu.cn > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can not be avoided by standard data cleaning [ 59 ] , since they does not require mislabeling , and the perturbed examples will maintain their malice even when they are correctly labeled by experts . In addition , even if the perturbed examples could be distinguished by some detection techniques , it is wasteful to filter out these correctly labeled examples , considering that deep models are data-hungry . In an extreme case where all examples in the training set are perturbed by a delusive adversary , there will leave no training examples after the filtering stage , thus the learning process is still obstructed . Given these challenges , we aim to examine the following question in this study : Is it possible to defend against delusive attacks without abandoning the perturbed examples ? In this work , we provide an affirmative answer to this question . We first formulate the task of delusive attacks as finding the worst-case data at training time within a specific ∞-Wasserstein ball that prevents label changes ( Section 2 ) . By doing so , we find that minimizing the adversarial risk on the perturbed data is equivalent to optimizing an upper bound of natural risk on the original data ( Section 3.1 ) . This implies that adversarial training [ 44 , 74 ] on the perturbed training examples can maximize the natural accuracy on the clean examples . Further , we disclose that adversarial training can resist the delusive perturbations by preventing the learner from overly relying on the non-robust features ( that are predictive , yet brittle or incomprehensible to humans ) in a simple and natural setting . Specifically , two opposite perturbation directions are studied , and adversarial training helps in both cases with different mechanisms ( Section 3.2 ) . All these evidences suggest that adversarial training is a promising solution to defend against delusive attacks . Importantly , our findings widen the scope of application of adversarial training , which was only considered as a principled defense method against test-time adversarial examples [ 74 , 21 ] . Note that adversarial training usually leads to a drop in natural accuracy [ 119 ] . This makes it less practical in many real-world applications where test-time attacks are rare and a high accuracy on clean test data is required [ 65 ] . However , this study shows that adversarial training can also defend against a more threatening and invisible threat called delusive adversaries ( see Figure 1 for an illustration ) . We believe that adversarial training will be more widely used in practical applications in the future . Finally , we present five practical attacks to empirically evaluate the proposed defense ( Section 4 ) . Extensive experiments on various datasets ( CIFAR-10 , SVHN , and a subset of ImageNet ) and tasks ( supervised learning , self-supervised learning , and overcoming simplicity bias ) demonstrate the effectiveness and versatility of adversarial training , which significantly mitigates the destructiveness of various delusive attacks ( Section 5 ) . Our main contributions are summarized as follows : • Formulation of delusive attacks . We provide the first attempt to formulate delusive attacks using the∞-Wasserstein distance . This formulation is novel and general , and can cover the formulation of the attack proposed by Feng et al . [ 32 ] . • The principled defense . Equipped with the novel characterization of delusive attacks , we are able to show that , for the first time , adversarial training can serve as a principled defense against delusive attacks with theoretical guarantee ( Theorem 1 ) . • Internal Mechanisms . We further disclose the internal mechanisms of the defense in a popular mixture-Gaussian setting ( Theorem 2 and Theorem 3 ) . • Empirical evidences . We complement our theoretical findings with extensive experiments across a wide range of datasets and tasks . 2 Preliminaries . In this section , we introduce some notations and the main ideas we build upon : natural risk , adversarial risk , Wasserstein distance , and delusive attacks . Notation . Consider a classification task with data ( x , y ) ∈ X ×Y from an underlying distribution D. We seek to learn a model f : X → Y by minimizing a loss function ` ( f ( x ) , y ) . Let ∆ : X ×X → R be some distance metric . Let B ( x , , ∆ ) = { x′ ∈ X : ∆ ( x , x′ ) ≤ } be the ball around x with radius . When ∆ is free of context , we simply write B ( x , , ∆ ) = B ( x , ) . Throughout the paper , the adversary is allowed to perturb only the inputs , not the labels . Thus , similar to Sinha et al . [ 109 ] , we define the cost function c : Z ×Z → R ∪ { ∞ } by c ( z , z′ ) = ∆ ( x , x′ ) +∞ · 1 { y 6= y′ } , where z = ( x , y ) and Z is the set of possible values for ( x , y ) . Denote by P ( Z ) the set of all probability measures on Z . For any µ ∈ Rd and positive definite matrix Σ ∈ Rd×d , denote by N ( µ , Σ ) the d-dimensional Gaussian distribution with mean vector µ and covariance matrix Σ . Natural risk . Standard training ( ST ) aims to minimize the natural risk , which is defined as Rnat ( f , D ) = E ( x , y ) ∼D [ ` ( f ( x ) , y ) ] . ( 1 ) The term “ natural accuracy ” refers to the accuracy of a model evaluated on the unperturbed data . Adversarial risk . The goal of adversarial training ( AT ) is to minimize the adversarial risk defined as Radv ( f , D ) = E ( x , y ) ∼D [ maxx′∈B ( x , ) ` ( f ( x′ ) , y ) ] , ( 2 ) which is a robust optimization problem that considers the worst-case performance under pointwise perturbations within an -ball [ 74 ] . The main assumption here is that the inputs satisfying ∆ ( x , x′ ) ≤ preserve the label y of the original input x. Wasserstein distance . Wasserstein distance is a distance function defined between two probability distributions , which represents the cost of an optimal mass transportation plan . Given two data distributions D and D′ , the p-th Wasserstein distance , for any p ≥ 1 , is defined as : Wp ( D , D′ ) = ( infγ∈Π ( D , D′ ) ∫ Z×Z c ( z , z ′ ) pdγ ( z , z′ ) ) 1/p , ( 3 ) where Π ( D , D′ ) is the collection of all probability measures on Z × Z with D and D′ being the marginals of the first and second factor , respectively . The∞-Wasserstein distance is defined as the limit of p-th Wasserstein distance , i.e. , W∞ ( D , D′ ) = limp→∞Wp ( D , D′ ) . The p-th Wasserstein ball with respect to D and radius ≥ 0 is defined as : BWp ( D , ) = { D′ ∈ P ( Z ) : Wp ( D , D′ ) ≤ } . Delusive adversary . The attacker is capable of manipulating the training data , as long as the training data is correctly labeled , to prevent the defender from learning an accurate classifier [ 82 ] . Following Feng et al . [ 32 ] , the game between the attacker and the defender proceeds as follows : • n data points are drawn from D to produce a clean training dataset Dn . • The attacker perturbs some inputs x in Dn by adding small perturbations to produce x′ such that ∆ ( x , x′ ) ≤ , where is a small constant that represents the attacker ’ s budget . The ( partially ) perturbed inputs and their original labels constitute the perturbed dataset D̂n . • The defender trains on the perturbed dataset D̂n to produce a model , and incurs natural risk . The attacker ’ s goal is to maximize the natural risk while the defender ’ s task is to minimize it . We then formulate the attacker ’ s goal as the following bi-level optimization problem : max D̂∈BW∞ ( D , ) Rnat ( fD̂ , D ) , s.t . fD̂ = arg minf Rnat ( f , D̂ ) . ( 4 ) In other words , Eq . ( 4 ) is seeking the training data bounded by the∞-Wasserstein ball with radius , so that the model trained on it has the worst performance on the original distribution . Remark 1 . It is worth noting that using the∞-Wasserstein distance to constrain delusive attacks possesses several advantages . Firstly , the cost function c used in Eq . ( 3 ) prevents label changes after perturbations since we only consider clean-label attacks . Secondly , our formulation does not restrict the choice of the distance metric ∆ of the input space , thus our theoretical analysis works with any metric , including the ` ∞ threat model considered in Feng et al . [ 32 ] . Finally , the∞-Wasserstein ball is more aligned with adversarial risk than other uncertainty sets [ 110 , 147 ] . Remark 2 . Our formulation assumes an underlying distribution that represents the perturbed dataset . This assumption has been widely adopted by existing works [ 111 , 118 , 145 ] . The rationale behind the assumption is that generally , the defender treats the training dataset as an empirical distribution and trains the model on randomly shuffled examples ( e.g. , training deep networks via stochastic gradient descent ) . It is also easy to see that our formulation covers the formulation of Feng et al . [ 32 ] . On the other hand , this assumption has its limitations . For example , if the defender treats the training examples as sequential data [ 24 ] , the attacker may utilize the dependence in the sequence to construct perturbations . This situation is beyond the scope of this work , and we leave it as future work .

Summary: The paper proposes a defense method based on adversarial training to defend against delusive adversaries at the training time. The paper shows that adversarial training can largely improve the accuracy under delusive attacks if the defender applies pessimistic budget. Both experiments and theory verify the claims.