Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training

1 Introduction . Although machine learning ( ML ) models have achieved superior performance on many challenging tasks , their performance can be largely deteriorated when the training and test distributions are different . For instance , standard models are prone to make mistakes on the adversarial examples that are considered as worst-case data at test time [ 10 , 113 ] . Compared with that , a more threatening and easily overlooked threat is the malicious perturbations at training time , i.e. , the delusive attacks [ 82 ] that aim to maximize test error by slightly perturbing the correctly labeled training examples [ 6 , 7 ] . In the era of big data , many practitioners collect training data from untrusted sources where delusive adversaries may exist . In particular , many companies are scraping large datasets from unknown users or public websites for commercial use . For example , Kaspersky Lab , a leading antivirus company , has been accused of poisoning competing products [ 7 ] . Although they denied any wrongdoings and clarified the false rumors , one can still imagine the disastrous consequences if that really happens in the security-critical applications . Furthermore , a recent survey of 28 organizations found that these industry practitioners are obviously more afraid of data poisoning than other threats from adversarial ML [ 64 ] . In a nutshell , delusive attacks has become a realistic and horrible threat to practitioners . Recently , Feng et al . [ 32 ] showed for the first time that delusive attacks are feasible for deep networks , by proposing “ training-time adversarial data ” that can significantly deteriorate model performance on clean test data . However , how to design learning algorithms that are robust to delusive attacks still remains an open question due to several crucial challenges [ 82 , 32 ] . First of all , delusive attacks †Corresponding author : Songcan Chen < s.chen @ nuaa.edu.cn > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can not be avoided by standard data cleaning [ 59 ] , since they does not require mislabeling , and the perturbed examples will maintain their malice even when they are correctly labeled by experts . In addition , even if the perturbed examples could be distinguished by some detection techniques , it is wasteful to filter out these correctly labeled examples , considering that deep models are data-hungry . In an extreme case where all examples in the training set are perturbed by a delusive adversary , there will leave no training examples after the filtering stage , thus the learning process is still obstructed . Given these challenges , we aim to examine the following question in this study : Is it possible to defend against delusive attacks without abandoning the perturbed examples ? In this work , we provide an affirmative answer to this question . We first formulate the task of delusive attacks as finding the worst-case data at training time within a specific ∞-Wasserstein ball that prevents label changes ( Section 2 ) . By doing so , we find that minimizing the adversarial risk on the perturbed data is equivalent to optimizing an upper bound of natural risk on the original data ( Section 3.1 ) . This implies that adversarial training [ 44 , 74 ] on the perturbed training examples can maximize the natural accuracy on the clean examples . Further , we disclose that adversarial training can resist the delusive perturbations by preventing the learner from overly relying on the non-robust features ( that are predictive , yet brittle or incomprehensible to humans ) in a simple and natural setting . Specifically , two opposite perturbation directions are studied , and adversarial training helps in both cases with different mechanisms ( Section 3.2 ) . All these evidences suggest that adversarial training is a promising solution to defend against delusive attacks . Importantly , our findings widen the scope of application of adversarial training , which was only considered as a principled defense method against test-time adversarial examples [ 74 , 21 ] . Note that adversarial training usually leads to a drop in natural accuracy [ 119 ] . This makes it less practical in many real-world applications where test-time attacks are rare and a high accuracy on clean test data is required [ 65 ] . However , this study shows that adversarial training can also defend against a more threatening and invisible threat called delusive adversaries ( see Figure 1 for an illustration ) . We believe that adversarial training will be more widely used in practical applications in the future . Finally , we present five practical attacks to empirically evaluate the proposed defense ( Section 4 ) . Extensive experiments on various datasets ( CIFAR-10 , SVHN , and a subset of ImageNet ) and tasks ( supervised learning , self-supervised learning , and overcoming simplicity bias ) demonstrate the effectiveness and versatility of adversarial training , which significantly mitigates the destructiveness of various delusive attacks ( Section 5 ) . Our main contributions are summarized as follows : • Formulation of delusive attacks . We provide the first attempt to formulate delusive attacks using the∞-Wasserstein distance . This formulation is novel and general , and can cover the formulation of the attack proposed by Feng et al . [ 32 ] . • The principled defense . Equipped with the novel characterization of delusive attacks , we are able to show that , for the first time , adversarial training can serve as a principled defense against delusive attacks with theoretical guarantee ( Theorem 1 ) . • Internal Mechanisms . We further disclose the internal mechanisms of the defense in a popular mixture-Gaussian setting ( Theorem 2 and Theorem 3 ) . • Empirical evidences . We complement our theoretical findings with extensive experiments across a wide range of datasets and tasks . 2 Preliminaries . In this section , we introduce some notations and the main ideas we build upon : natural risk , adversarial risk , Wasserstein distance , and delusive attacks . Notation . Consider a classification task with data ( x , y ) ∈ X ×Y from an underlying distribution D. We seek to learn a model f : X → Y by minimizing a loss function ` ( f ( x ) , y ) . Let ∆ : X ×X → R be some distance metric . Let B ( x , , ∆ ) = { x′ ∈ X : ∆ ( x , x′ ) ≤ } be the ball around x with radius . When ∆ is free of context , we simply write B ( x , , ∆ ) = B ( x , ) . Throughout the paper , the adversary is allowed to perturb only the inputs , not the labels . Thus , similar to Sinha et al . [ 109 ] , we define the cost function c : Z ×Z → R ∪ { ∞ } by c ( z , z′ ) = ∆ ( x , x′ ) +∞ · 1 { y 6= y′ } , where z = ( x , y ) and Z is the set of possible values for ( x , y ) . Denote by P ( Z ) the set of all probability measures on Z . For any µ ∈ Rd and positive definite matrix Σ ∈ Rd×d , denote by N ( µ , Σ ) the d-dimensional Gaussian distribution with mean vector µ and covariance matrix Σ . Natural risk . Standard training ( ST ) aims to minimize the natural risk , which is defined as Rnat ( f , D ) = E ( x , y ) ∼D [ ` ( f ( x ) , y ) ] . ( 1 ) The term “ natural accuracy ” refers to the accuracy of a model evaluated on the unperturbed data . Adversarial risk . The goal of adversarial training ( AT ) is to minimize the adversarial risk defined as Radv ( f , D ) = E ( x , y ) ∼D [ maxx′∈B ( x , ) ` ( f ( x′ ) , y ) ] , ( 2 ) which is a robust optimization problem that considers the worst-case performance under pointwise perturbations within an -ball [ 74 ] . The main assumption here is that the inputs satisfying ∆ ( x , x′ ) ≤ preserve the label y of the original input x. Wasserstein distance . Wasserstein distance is a distance function defined between two probability distributions , which represents the cost of an optimal mass transportation plan . Given two data distributions D and D′ , the p-th Wasserstein distance , for any p ≥ 1 , is defined as : Wp ( D , D′ ) = ( infγ∈Π ( D , D′ ) ∫ Z×Z c ( z , z ′ ) pdγ ( z , z′ ) ) 1/p , ( 3 ) where Π ( D , D′ ) is the collection of all probability measures on Z × Z with D and D′ being the marginals of the first and second factor , respectively . The∞-Wasserstein distance is defined as the limit of p-th Wasserstein distance , i.e. , W∞ ( D , D′ ) = limp→∞Wp ( D , D′ ) . The p-th Wasserstein ball with respect to D and radius ≥ 0 is defined as : BWp ( D , ) = { D′ ∈ P ( Z ) : Wp ( D , D′ ) ≤ } . Delusive adversary . The attacker is capable of manipulating the training data , as long as the training data is correctly labeled , to prevent the defender from learning an accurate classifier [ 82 ] . Following Feng et al . [ 32 ] , the game between the attacker and the defender proceeds as follows : • n data points are drawn from D to produce a clean training dataset Dn . • The attacker perturbs some inputs x in Dn by adding small perturbations to produce x′ such that ∆ ( x , x′ ) ≤ , where is a small constant that represents the attacker ’ s budget . The ( partially ) perturbed inputs and their original labels constitute the perturbed dataset D̂n . • The defender trains on the perturbed dataset D̂n to produce a model , and incurs natural risk . The attacker ’ s goal is to maximize the natural risk while the defender ’ s task is to minimize it . We then formulate the attacker ’ s goal as the following bi-level optimization problem : max D̂∈BW∞ ( D , ) Rnat ( fD̂ , D ) , s.t . fD̂ = arg minf Rnat ( f , D̂ ) . ( 4 ) In other words , Eq . ( 4 ) is seeking the training data bounded by the∞-Wasserstein ball with radius , so that the model trained on it has the worst performance on the original distribution . Remark 1 . It is worth noting that using the∞-Wasserstein distance to constrain delusive attacks possesses several advantages . Firstly , the cost function c used in Eq . ( 3 ) prevents label changes after perturbations since we only consider clean-label attacks . Secondly , our formulation does not restrict the choice of the distance metric ∆ of the input space , thus our theoretical analysis works with any metric , including the ` ∞ threat model considered in Feng et al . [ 32 ] . Finally , the∞-Wasserstein ball is more aligned with adversarial risk than other uncertainty sets [ 110 , 147 ] . Remark 2 . Our formulation assumes an underlying distribution that represents the perturbed dataset . This assumption has been widely adopted by existing works [ 111 , 118 , 145 ] . The rationale behind the assumption is that generally , the defender treats the training dataset as an empirical distribution and trains the model on randomly shuffled examples ( e.g. , training deep networks via stochastic gradient descent ) . It is also easy to see that our formulation covers the formulation of Feng et al . [ 32 ] . On the other hand , this assumption has its limitations . For example , if the defender treats the training examples as sequential data [ 24 ] , the attacker may utilize the dependence in the sequence to construct perturbations . This situation is beyond the scope of this work , and we leave it as future work .

Summary: The paper tackles the problem of delusive attacks. It reveals that adversarial training, traditionally considered as one of the best defense methods against adversarial examples, can serve as a powerful and elegant defense against delusive attacks. This expands the defense ability of adversarial training from test-time evasion attacks to training-time availability attacks, making the technique significantly more applicable in practice.