Fooling Adversarial Training with Induction Noise

1 INTRODUCTION . In recent years , deep learning has achieved great success , while the existence of adversarial examples ( Szegedy et al. , 2014 ) alerts us that existing deep neural networks are very vulnerable to adversarial attack . Crafted by adding imperceptible perturbations to the input images , adversarial examples can dramatically degrade the performance of accurate deep models , raising huge concerns in both the academy and the industry ( Chakraborty et al. , 2018 ; Ma et al. , 2020 ) . Adversarial Training ( AT ) is currently the most effective approach against adversarial examples ( Madry et al. , 2017 ; Athalye et al. , 2018 ) . In practice , adversarially trained models have been shown good robustness under various attack , and the recent state-of-the-art defense algorithms ( Zhang et al. , 2019 ; Wang et al. , 2020 ) are all variants of adversarial training . Therefore , it is widely believed that we have already found the cure to adversarial attack , i.e. , adversarial training , based on which we can build trustworthy models to a certain degree . In this paper , we challenge this common belief by showing that AT could be ineffective when injecting some small and specific poisonings into the training data , which leads to a catastrophic drop for AT on CIFAR-10 dataset in the test accuracy ( from 85 % to 56 % on clean data ) and the test robustness ( from 51 % to 0.6 % on adversarial data ) . Previously , Huang et al . ( 2021 ) and Fowl et al . ( 2021b ) have shown that injecting some special noise into the training data can make Standard Training ( ST ) ineffective . However , these kinds of noise can be easily removed by AT , i.e. , AT is still effective . While in this work , we are the first to explore whether there exists a kind of special and irremovable poisoning of training data that could make AT ineffective . Specifically , we first dissect the failure of Huang et al . ( 2021 ) and Fowl et al . ( 2021b ) on fooling AT and find that they craft poisons on a standardly trained model . As pointed out by Ilyas et al . ( 2019 ) , ST can only extract non-robust features , which will be discarded in AT because it only extracts robust features . In view of this , we should craft poisons with robust features extracted from adversarially trained models , which may be more resistant to AT . However , only using robust features is not sufficient to break down AT because we find that AT itself still works well when taking robust-feature perturbations during training . The key point is that we need to utilize a consistent misclassified target label for each class , and only with this consistent bias can we induce AT to the desired misclassification . Based on this , we instantiate a kind of irremovable poisoning , ADVersarially Inducing Noise ( ADVIN ) , for the training-time data fooling . ADVIN can not only degrade standard training like previous methods but also successfully break down adversarial training for the first time . To summarize , our main contributions are : • We are the first to study how to make adversarial training ineffective by injecting irremovable poisoning . It is more challenging since all previous fooling methods designed for standard training fail to work under adversarial training . • We instantiate a kind of irremovable noise , called ADVersarially Inducing Noise ( ADVIN ) , to poison data . Extensive experiments show that ADVIN can successfully make adversarial training ineffective and outperform ST-oriented methods by a large margin . • We apply ADVIN to prevent unauthorized exploitation of personal data , where ADVIN is shown to be effective against both standard and adversarial training , making our privacypreserved data truly unlearnable . 2 RELATED WORK . Data poisoning . Data poisoning aims at fooling the model to have a poor performance on clean test data by manipulating the training data . For example , Biggio et al . ( 2012 ) aims at poisoning an SVM model . While previous works mainly focus on poisoning the most influential examples using adversarial noise ( Koh & Liang , 2017 ; Muñoz-González et al. , 2017 ) , these methods can only play a limited role in the destruction of the training process of DNNs . Recently , Huang et al . ( 2021 ) and Fowl et al . ( 2021b ) propose error-minimizing noise and adversarial example noise , respectively , which lead standardly trained DNNs on them to have a test accuracy close to or even lower than random prediction . Unfortunately , their poisons can be removed by adversarial training . Therefore , we focus on how to generate poisons that could not be removed by adversarial training and deconstruct the training process at the same time , i.e. , making adversarial training ineffective . Addition discussion about recentlt related work could be found in Appendix E Adversarial Attack . Szegedy et al . ( 2014 ) has demonstrated the vulnerability of deep neural networks , which could be easily distorted by imperceptible perturbations . Typically , adversarial attacks utilize the error-maximizing noise ( untargeted attack ) to fool the models at test time ( Goodfellow et al. , 2015 ) . Specifically , the adversarial examples can be divided into two categories , untargeted ( Goodfellow et al. , 2015 ; Madry et al. , 2017 ) and targeted attack . Compared to the untargeted manner , targeted attack generates adversarial examples such that they are misclassified to the target class ( different from the original label ) . While iterative untargeted attack ( Madry et al. , 2017 ) is more popular in solving the inner loop of adversarial training , some recent works find that targeted attack can achieve comparable , and sometimes better , performance ( Xie & Yuille , 2020 ; Kurakin et al. , 2017 ; Wang & Zhang , 2019 ) . 3 THE DIFFICULTY ON FOOLING ADVERSARIAL TRAINING . Considering a K-class image classification task , we denote the natural data as Dc = { ( xi , yi ) } , where xi ∈ Rd is a d-dimensional input , and yi ∈ { 1 , 2 , . . . , K } is the corresponding class label . To learn a classifier f with parameters θt , Standard Training ( ST ) minimizes the following objective on clean data , where ` CE ( · , · ) denotes the cross entropy loss : min θt LST ( Dc , θ ) = min θt E ( xi , yi ) ∼Dc ` CE ( fθt ( xi ) , yi ) . ( 1 ) Instead , Adversarial Training ( AT ) aims to improve robustness against adversarial attack by training on adversarially perturbed data , resulting in the following minimax objective , min θt LAT ( Dc , θ ) = min θt E ( xi , yi ) ∼Dc max‖δt‖p≤εt ` CE ( fθt ( xi + δ t ) , yi ) , ( 2 ) where the sample-wise perturbation δt is constrained in a ` p-norm ball with radius εt and the inner maximization is typically solved by PGD ( Madry et al. , 2017 ) . Fooling Standard Training ( FST ) . Intuitively , the goal of the poisoned data Dp is to induce standard training to learn a model on Dp with parameters θt that is ineffective for classifying natural images from Dc . However , their fooling can only work for standard training while being easily alleviated under adversarial training . In other words , their “ unlearnable examples ” are actually learnable . Specifically , Huang et al . ( 2021 ) adopt the error-minimizing noise generated with the following min-min optimization problem for fooling standard training : min θs E ( xi , yi ) ∼Dc min‖δp‖≤εp ` CE ( fθs ( xi + δ p ) , yi ) , ( 3 ) where θs is the source model that is used to generate poisons with perturbation radius εp . The inner loop seeks the Lp-norm bounded noise δ by minimizing the loss with PGD steps , and the outer loop further optimizes the parameters θ by minimizing the loss on the adversarial pair ( xi + δp , yi ) . To investigate how error-minimizing noise can fool standard training , we compare the training process of clean data and error-minimizing perturbed data in Figure 1 ( a ) ( b ) , where the training loss of error-minimizing data is significantly smaller . This indicates that error minimization is designed to minimize the loss of the perturbed pair ( xi + δ p i , yi ) to near zero such that the poisoned sample can not be used for model updating . While for adversarial training , as shown in Figure 1 ( c ) ( d ) , its inner maximization process can easily remove the error-minimizing noise by further lifting the loss of the perturbed pair ( xi + δ p i + δ t i , yi ) with the error-maximizing noise δ t i . In this way , the hidden information is uncovered and makes those unlearnable examples learnable again . Thus , to fool adversarial training , we need to go beyond the paradigm of unlearnable examples and design a stronger type of poisoning , for which we need it to be irremovable and resistant to error-maximizing perturbations . Below , we introduce our attempts to design this irremovable noise . 4 DESIGNING OF IRREMOVABLE NOISE . Based on the investigation in Section 3 , we can easily see that it is more challenging for fooling adversarial training than standard training . In the following , we will design effective irremovable noise from the aspects of features , labels , and training strategies . 4.1 THE NECESSITY OF ROBUST FEATURES . First , we notice that it is necessary to use robust features for fooling AT . Specifically , we compare poisons generated using Fowl et al . ( 2021b ) from two different pre-trained models , a standardly trained model and an adversarially trained model , both with εp = 32/255 . Note that although here we use a larger perturbation radius εp , this factor can only slightly fool AT by ∼ 10 % performance drop in Huang et al . ( 2021 ) and can not guarantee irremovability . We compare the poisons generated from robust and non-robust features . The results are shown in Figure 2a . We can see that even with a larger εp , poisons generated from the ST source model are almost useless ( orange lines ) . In contrast , the poisons generated from the robust source model can effectively bring down the final robustness from∼ 50 % to∼ 30 % ( blue lines ) . More details of experiments for poison generations and training process could be found in Appendix A.2 This observation indicates that robust features are necessary for fooling AT . According to Ilyas et al . ( 2019 ) , ST can only extract non-robust features , and thus the generated poisons only contain nonrobust features , which , however , will be discarded under AT since it only relies on robust features . Therefore , to fool AT effectively , the source model itself must contain robust features so that the generated poisons could contain robust features that are resistant to AT . To achieve this goal , we adopt the adversarially trained models to craft poisons . 4.2 THE NECESSITY OF CONSISTENT LABEL BIAS . As shown in Section 3 , the error-minimizing noise can be easily removed by the error-maximizing process of AT . Recalling that AT itself can learn good models with error-maximizing noise generated by itself using untargeted attack , we consider to use alternative target labels that are different from the error-maximizing objective . Formally , given a source model fθs and a natural pair ( xi , yi ) ∈ Dc , we pick a target class y′i and generate the poison δ p i by δpi = argmin ‖δpi ‖≤εp ` CE ( fθs ( xi + δ p i ) , y ′ i ) . ( 4 ) Specifically , we consider the following strategies for assigning fooling labels : • Random ( Xie & Yuille , 2020 ) : a randomly drawn label y′i u.a.r.∼ { 1 , 2 , . . . , K } ; • LL ( Kurakin et al. , 2017 ) : the Least Likely label y′i = argmaxy 6=yi ` CE ( fθs ( xi ) , y ) ; • MC ( Wang & Zhang , 2019 ) : the Most Confusing label y′i = argminy 6=yi ` CE ( fθs ( xi ) , y ) ; • NextCycle ( ours ) : the next label in a cyclic order y′i = ( yi + 1 mod K ) ; • NearSwap ( ours ) : label swapping with y′i = { yi + 1 mod K , if yi = 2k + 1 , yi − 1 mod K , if yi = 2k , k ∈ N. We list their performance against AT in Figure 2b . We can see that like error-minimizing and errormaximizing noise , both Random , LL , and MC methods also fail to poison AT ( blue , orange , and green lines ) . Instead , we can see that both NextCycle and NearSwap can effectively degrade robust accuracy to 30 % – 35 % ( red and purple lines ) . Comparing the five strategies , we can find a common and underlying rule for the effective ones , e.g. , NextCycle and NearSwap , that the label mapping g : yi → y′i is consistent among samples in the same class while being different for samples from different classes . As a result , they impose a consistent bias on the poisoned data such that all samples in the class A are induced to a specific class B . In this way , they can induce AT to learn a false mapping between features and labels , resulting in a low robust accuracy on test data . Details of noise generation for these five label mapping strategy can be seen in Appendix A.3 .

Summary: This work tackles the problem of data poisoning adversarial training in the (image classification) setting where the adversary can add $\ell_p$ constrained perturbations to each training example. Others have previously studied this problem in the standard ERM setting by adding $\ell_p$ perturbations to examples that minimize loss on pretrained models: when the downstream learner tries to minimize risk on these perturbed examples, they can do well on the train set but not on a held-out set from the original distribution (this work is all performed in CIFAR). It turns out that adversarially trained models can largely circumvent this class of attacks, as they can modify the adversarial $\ell_p$ perturbations in the adversarial training process to make learning require appropriately generalizable features.   The authors try a new attack that focusses on adding "robust" features to examples: in the most basic attack, for each (example, label) combination the authors perturb the example to maximize the probability of a different (label-consistent --- i.e. each image of a given label is perturbed towards the same class, one such mapping could be target_label = label + 1 % num_classes ) label. The authors specialize the attack further by performing an alternating optimization routine in which one repeatedly (starting from a pretrained adv-trained model $f$): (a) completes the perturbation above on the examples using $f$ and then (b) adversarially trains on $f$ using the new label scheme. Using this technique the authors can force the model to perform well on the train set, but not on a held-out set. Using only step (a) (i.e. no alternating optimization routine) the authors can fool the model in the same sense but to a lesser extent.