Fooling Adversarial Training with Induction Noise

1 INTRODUCTION . In recent years , deep learning has achieved great success , while the existence of adversarial examples ( Szegedy et al. , 2014 ) alerts us that existing deep neural networks are very vulnerable to adversarial attack . Crafted by adding imperceptible perturbations to the input images , adversarial examples can dramatically degrade the performance of accurate deep models , raising huge concerns in both the academy and the industry ( Chakraborty et al. , 2018 ; Ma et al. , 2020 ) . Adversarial Training ( AT ) is currently the most effective approach against adversarial examples ( Madry et al. , 2017 ; Athalye et al. , 2018 ) . In practice , adversarially trained models have been shown good robustness under various attack , and the recent state-of-the-art defense algorithms ( Zhang et al. , 2019 ; Wang et al. , 2020 ) are all variants of adversarial training . Therefore , it is widely believed that we have already found the cure to adversarial attack , i.e. , adversarial training , based on which we can build trustworthy models to a certain degree . In this paper , we challenge this common belief by showing that AT could be ineffective when injecting some small and specific poisonings into the training data , which leads to a catastrophic drop for AT on CIFAR-10 dataset in the test accuracy ( from 85 % to 56 % on clean data ) and the test robustness ( from 51 % to 0.6 % on adversarial data ) . Previously , Huang et al . ( 2021 ) and Fowl et al . ( 2021b ) have shown that injecting some special noise into the training data can make Standard Training ( ST ) ineffective . However , these kinds of noise can be easily removed by AT , i.e. , AT is still effective . While in this work , we are the first to explore whether there exists a kind of special and irremovable poisoning of training data that could make AT ineffective . Specifically , we first dissect the failure of Huang et al . ( 2021 ) and Fowl et al . ( 2021b ) on fooling AT and find that they craft poisons on a standardly trained model . As pointed out by Ilyas et al . ( 2019 ) , ST can only extract non-robust features , which will be discarded in AT because it only extracts robust features . In view of this , we should craft poisons with robust features extracted from adversarially trained models , which may be more resistant to AT . However , only using robust features is not sufficient to break down AT because we find that AT itself still works well when taking robust-feature perturbations during training . The key point is that we need to utilize a consistent misclassified target label for each class , and only with this consistent bias can we induce AT to the desired misclassification . Based on this , we instantiate a kind of irremovable poisoning , ADVersarially Inducing Noise ( ADVIN ) , for the training-time data fooling . ADVIN can not only degrade standard training like previous methods but also successfully break down adversarial training for the first time . To summarize , our main contributions are : • We are the first to study how to make adversarial training ineffective by injecting irremovable poisoning . It is more challenging since all previous fooling methods designed for standard training fail to work under adversarial training . • We instantiate a kind of irremovable noise , called ADVersarially Inducing Noise ( ADVIN ) , to poison data . Extensive experiments show that ADVIN can successfully make adversarial training ineffective and outperform ST-oriented methods by a large margin . • We apply ADVIN to prevent unauthorized exploitation of personal data , where ADVIN is shown to be effective against both standard and adversarial training , making our privacypreserved data truly unlearnable . 2 RELATED WORK . Data poisoning . Data poisoning aims at fooling the model to have a poor performance on clean test data by manipulating the training data . For example , Biggio et al . ( 2012 ) aims at poisoning an SVM model . While previous works mainly focus on poisoning the most influential examples using adversarial noise ( Koh & Liang , 2017 ; Muñoz-González et al. , 2017 ) , these methods can only play a limited role in the destruction of the training process of DNNs . Recently , Huang et al . ( 2021 ) and Fowl et al . ( 2021b ) propose error-minimizing noise and adversarial example noise , respectively , which lead standardly trained DNNs on them to have a test accuracy close to or even lower than random prediction . Unfortunately , their poisons can be removed by adversarial training . Therefore , we focus on how to generate poisons that could not be removed by adversarial training and deconstruct the training process at the same time , i.e. , making adversarial training ineffective . Addition discussion about recentlt related work could be found in Appendix E Adversarial Attack . Szegedy et al . ( 2014 ) has demonstrated the vulnerability of deep neural networks , which could be easily distorted by imperceptible perturbations . Typically , adversarial attacks utilize the error-maximizing noise ( untargeted attack ) to fool the models at test time ( Goodfellow et al. , 2015 ) . Specifically , the adversarial examples can be divided into two categories , untargeted ( Goodfellow et al. , 2015 ; Madry et al. , 2017 ) and targeted attack . Compared to the untargeted manner , targeted attack generates adversarial examples such that they are misclassified to the target class ( different from the original label ) . While iterative untargeted attack ( Madry et al. , 2017 ) is more popular in solving the inner loop of adversarial training , some recent works find that targeted attack can achieve comparable , and sometimes better , performance ( Xie & Yuille , 2020 ; Kurakin et al. , 2017 ; Wang & Zhang , 2019 ) . 3 THE DIFFICULTY ON FOOLING ADVERSARIAL TRAINING . Considering a K-class image classification task , we denote the natural data as Dc = { ( xi , yi ) } , where xi ∈ Rd is a d-dimensional input , and yi ∈ { 1 , 2 , . . . , K } is the corresponding class label . To learn a classifier f with parameters θt , Standard Training ( ST ) minimizes the following objective on clean data , where ` CE ( · , · ) denotes the cross entropy loss : min θt LST ( Dc , θ ) = min θt E ( xi , yi ) ∼Dc ` CE ( fθt ( xi ) , yi ) . ( 1 ) Instead , Adversarial Training ( AT ) aims to improve robustness against adversarial attack by training on adversarially perturbed data , resulting in the following minimax objective , min θt LAT ( Dc , θ ) = min θt E ( xi , yi ) ∼Dc max‖δt‖p≤εt ` CE ( fθt ( xi + δ t ) , yi ) , ( 2 ) where the sample-wise perturbation δt is constrained in a ` p-norm ball with radius εt and the inner maximization is typically solved by PGD ( Madry et al. , 2017 ) . Fooling Standard Training ( FST ) . Intuitively , the goal of the poisoned data Dp is to induce standard training to learn a model on Dp with parameters θt that is ineffective for classifying natural images from Dc . However , their fooling can only work for standard training while being easily alleviated under adversarial training . In other words , their “ unlearnable examples ” are actually learnable . Specifically , Huang et al . ( 2021 ) adopt the error-minimizing noise generated with the following min-min optimization problem for fooling standard training : min θs E ( xi , yi ) ∼Dc min‖δp‖≤εp ` CE ( fθs ( xi + δ p ) , yi ) , ( 3 ) where θs is the source model that is used to generate poisons with perturbation radius εp . The inner loop seeks the Lp-norm bounded noise δ by minimizing the loss with PGD steps , and the outer loop further optimizes the parameters θ by minimizing the loss on the adversarial pair ( xi + δp , yi ) . To investigate how error-minimizing noise can fool standard training , we compare the training process of clean data and error-minimizing perturbed data in Figure 1 ( a ) ( b ) , where the training loss of error-minimizing data is significantly smaller . This indicates that error minimization is designed to minimize the loss of the perturbed pair ( xi + δ p i , yi ) to near zero such that the poisoned sample can not be used for model updating . While for adversarial training , as shown in Figure 1 ( c ) ( d ) , its inner maximization process can easily remove the error-minimizing noise by further lifting the loss of the perturbed pair ( xi + δ p i + δ t i , yi ) with the error-maximizing noise δ t i . In this way , the hidden information is uncovered and makes those unlearnable examples learnable again . Thus , to fool adversarial training , we need to go beyond the paradigm of unlearnable examples and design a stronger type of poisoning , for which we need it to be irremovable and resistant to error-maximizing perturbations . Below , we introduce our attempts to design this irremovable noise . 4 DESIGNING OF IRREMOVABLE NOISE . Based on the investigation in Section 3 , we can easily see that it is more challenging for fooling adversarial training than standard training . In the following , we will design effective irremovable noise from the aspects of features , labels , and training strategies . 4.1 THE NECESSITY OF ROBUST FEATURES . First , we notice that it is necessary to use robust features for fooling AT . Specifically , we compare poisons generated using Fowl et al . ( 2021b ) from two different pre-trained models , a standardly trained model and an adversarially trained model , both with εp = 32/255 . Note that although here we use a larger perturbation radius εp , this factor can only slightly fool AT by ∼ 10 % performance drop in Huang et al . ( 2021 ) and can not guarantee irremovability . We compare the poisons generated from robust and non-robust features . The results are shown in Figure 2a . We can see that even with a larger εp , poisons generated from the ST source model are almost useless ( orange lines ) . In contrast , the poisons generated from the robust source model can effectively bring down the final robustness from∼ 50 % to∼ 30 % ( blue lines ) . More details of experiments for poison generations and training process could be found in Appendix A.2 This observation indicates that robust features are necessary for fooling AT . According to Ilyas et al . ( 2019 ) , ST can only extract non-robust features , and thus the generated poisons only contain nonrobust features , which , however , will be discarded under AT since it only relies on robust features . Therefore , to fool AT effectively , the source model itself must contain robust features so that the generated poisons could contain robust features that are resistant to AT . To achieve this goal , we adopt the adversarially trained models to craft poisons . 4.2 THE NECESSITY OF CONSISTENT LABEL BIAS . As shown in Section 3 , the error-minimizing noise can be easily removed by the error-maximizing process of AT . Recalling that AT itself can learn good models with error-maximizing noise generated by itself using untargeted attack , we consider to use alternative target labels that are different from the error-maximizing objective . Formally , given a source model fθs and a natural pair ( xi , yi ) ∈ Dc , we pick a target class y′i and generate the poison δ p i by δpi = argmin ‖δpi ‖≤εp ` CE ( fθs ( xi + δ p i ) , y ′ i ) . ( 4 ) Specifically , we consider the following strategies for assigning fooling labels : • Random ( Xie & Yuille , 2020 ) : a randomly drawn label y′i u.a.r.∼ { 1 , 2 , . . . , K } ; • LL ( Kurakin et al. , 2017 ) : the Least Likely label y′i = argmaxy 6=yi ` CE ( fθs ( xi ) , y ) ; • MC ( Wang & Zhang , 2019 ) : the Most Confusing label y′i = argminy 6=yi ` CE ( fθs ( xi ) , y ) ; • NextCycle ( ours ) : the next label in a cyclic order y′i = ( yi + 1 mod K ) ; • NearSwap ( ours ) : label swapping with y′i = { yi + 1 mod K , if yi = 2k + 1 , yi − 1 mod K , if yi = 2k , k ∈ N. We list their performance against AT in Figure 2b . We can see that like error-minimizing and errormaximizing noise , both Random , LL , and MC methods also fail to poison AT ( blue , orange , and green lines ) . Instead , we can see that both NextCycle and NearSwap can effectively degrade robust accuracy to 30 % – 35 % ( red and purple lines ) . Comparing the five strategies , we can find a common and underlying rule for the effective ones , e.g. , NextCycle and NearSwap , that the label mapping g : yi → y′i is consistent among samples in the same class while being different for samples from different classes . As a result , they impose a consistent bias on the poisoned data such that all samples in the class A are induced to a specific class B . In this way , they can induce AT to learn a false mapping between features and labels , resulting in a low robust accuracy on test data . Details of noise generation for these five label mapping strategy can be seen in Appendix A.3 .

Summary: This paper studies a data poisoning method for adversarial training. I like this paper in general: The derivations are clear and empirical results are clearly presented. However, this paper may suffer from unclear formulations of threat model, as I will discuss below in more detail.