Fooling Adversarial Training with Induction Noise
Authors: Zhirui Wang
Yifei Wang
Yisen Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Data poisoning
adversarial training
data privacy

Abstract: Adversarial training is widely believed to be a reliable approach to improve model robustness against adversarial attack. However, in this paper, we show that when trained on one type of poisoned data, adversarial training can also be fooled to have catastrophic behavior, e.g., $<1\%$ robust test accuracy with $>90\%$ robust training accuracy on CIFAR-10 dataset. Previously, there are other types of noise poisoned in the training data that have successfully fooled standard training ($15.8\%$ standard test accuracy with $99.9\%$ standard training accuracy on CIFAR-10 dataset), but their poisonings can be easily removed when adopting adversarial training. Therefore, we aim to design a new type of inducing noise, named ADVIN, which is an irremovable poisoning of training data. ADVIN can not only degrade the robustness of adversarial training by a large margin, for example, from $51.7\%$ to $0.57\%$ on CIFAR-10 dataset, but also be effective for fooling standard training ($13.1\%$ standard test accuracy with $100\%$ standard training accuracy). Additionally, ADVIN can be applied to preventing personal data (like selfies) from being exploited without authorization under whether standard or adversarial training. 

Under review as a conference paper at ICLR 2022
FOOLING ADVERSARIAL TRAINING WITH INDUCING NOISE
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial training is widely believed to be a reliable approach to improve model robustness against adversarial attack. However, in this paper, we show that when trained on one type of poisoned data, adversarial training can also be fooled to have catastrophic behavior, e.g., < 1% robust test accuracy with > 90% ro- bust training accuracy on CIFAR-10 dataset. Previously, there are other types of noise poisoned in the training data that have successfully fooled standard training (15.8% standard test accuracy with 99.9% standard training accuracy on CIFAR- 10 dataset), but their poisonings can be easily removed when adopting adversar- ial training. Therefore, we aim to design new type of inducing noise, named ADVIN, which an irremovable poisoning of training data. ADVIN can not only degrade the robustness of adversarial training by a large margin, for exam- ple, from 51.7% to 0.57% on CIFAR-10 10 dataset, but also be effective for fooling standard training (13.1% standard test accuracy with 100% standard training ac- curacy). Additionally, ADVIN can be applied to preventing personal data (like selfies) from being exploited without authorization under whether standard or ad- versarial training.
INTRODUCTION
In recent years, deep learning has achieved great success, while the existence of adversarial examples (Szegedy et al. 2014) alerts us that existing deep neural networks are very vulnerable to adversarial attack. Crafted by adding imperceptible perturbations to the input images, adversarial examples can dramatically degrade the performance of accurate deep models, raising huge concerns in both the academy and the industry (Chakraborty et al. 2018 Ma et al., 2020).
Adversarial Training (AT) is currently the most effective approach against adversarial examples (Madry et al. [201] Athalye et al. 2018). In practice, adversarially trained models have been shown good robustness under various attack, and the recent -f-te-of-the-artt defense algorithms (Zhang et al.) 2019 Wang et al. (2020) are all variants of adversarial training. Therefore, it widely believed that we have already found the cure to adversarial attack, i.c., adversarial training, based on which we can build trustworthy models to certain degree.
In this paper, we challenge this common belief by showing that AT could be ineffective when inject- ing some small and specific poisonings into the training data, which leads to a catastrophic drop for AT on CIFAR- 10 dataset in the test accuracy (from 85% to 56% on clean data) and the test robust- ness (from 51% % 0.6% adversarial data). Previously, Huang et al. (202)) and Fowl al. 202 have shown that injecting some special noise into the training data can make Standard Training (ST) ineffective. However, these kinds of noise can be easily removed by AT, i.c., AT is still effective. While in this work, we the first to explore whether there exists kind of special and irremovable poisoning of training data that could make AT ineffective.
Specifically, we first dissect the failure of Huang et al. (202I) and Fowl et al. (202lb) on fooling AT and find that they craft poisons on a standardly trained model. As pointed out by llyas et al. (2019). ST can only extract non-robust features, which will be discarded in AT because it only ex- tracts robust features. In view of this, we should craft poisons with robust features extracted from adversarially trained models, which may be more resistant to AT. However, only using robust fea- tures is not sufficient break down AT because robust-feature perturbations during training. The key point is that we need utilize a consistent misclassified target label for each class, and only with this consistent bias can we induce AT to the desired misclassification. Based on this, we instantiate a kind of irremovable poisoning, ADVer- sarially Inducing Noise (ADVIN), for the training-time data fooling. ADVIN not only degrade
find that AT itself still works well when taking

---

Under review as a conference paper at ICLR 2022
standard training like previous methods but also successfully break down adversarial training for the first time. To summarize, our main contributions are:
We are the first to study how to make adversarial training ineffective by injecting irremov- able poisoning. It is more challenging since all previous fooling methods designed for standard training fail to work under adversarial training.
We instantiate a kind of irremovable noise, called ADVersarially Inducing Noise (ADVIN), to poison data. Extensive experiments show that ADVIN can successfully make adversarial training ineffective and outperform ST-oriented methods by large margin.
We apply ADVIN to prevent unauthorized exploitation of personal data, where ADVIN is shown to be effective against both standard and adversarial training, making our privacy- preserved data truly unlearnable.
2 RELATED WORK
Data poisoning. Data poisoning aims at fooling the model to have poor performance on clean test data by manipulating the training data. For example, Biggio et al. (2012) aims at poisoning an SVM model. While previous works mainly focus on poisoming the most influential examples using adversarial noise (Koh & Liang 2017 Muñoz- Gonzzzz- et al. 2017). these methods can only play a limited role in the destruction of the training process of DNNs Recently, [Huang et al. (222)) Fowl et al. (2021b) propose error-minimizing noise and adversarial example noise, respectively. which lead standardly trained DNNs on them to have a test accuracy close to or even lower than random prediction. Unfortunately, their poisons can be removed by adversarial training. Therefore, we focus on how to generate poisons that could not be removed by adversarial training and decon-
struct the training process at the same time, i.e.. making adversarial training ineffective. Addition discussion about recentlt related work could be found in Appendix E
Adversarial Attack. [Szegedy et al. (2014) has demonstrated the vulnerability of deep neural net- works, which could be easily distorted by imperceptible perturbations. Typically, adversarial attacks utilize the grror-mxxmizing noise (untargeted attack) to fool the models at test time (Goodfellow eet al., (015).. Specifically, the adversarial examples can be divided into two categories, untargeted Goodfellow et al. 2013 |015|Madry et al. 2017) and targeted attack. Compared to the untargeted man- ner, targeted attack generates ad adversarial examples such that they are misclassified to the target class (different from the original label). While iterative untargeted attack (Madry et al. 2017) is more popular in solving the inner loop of adversarial training, some recent works find that targeted attack can achieve comparable, and sometimes better, performance (Xie & Yuille, 2020 Kurakin et al., 2017 Wang& & Zhang 2019).
3 THE DIFFICULTY ON FOOLING ADVERSARIAL TRAINING
Considering a K-class image classification task, we denote the natural data as D. = {(x, y:) }. where 2; E Rd is d-dimensional input, and yi E 21, K} ..k is the corresponding class label. To learn classifier f with parameters O,, Standard Training (ST) minimizes the following objective on clean data, where (CE(.,) denotes the cross entropy loss:
min LST (Dey 0) min ((E.,)) fCE (æ:), 0,
0,
Instead, Adversarial Training (AT) aims to improve robustness against adversarial attack by training on adversarially perturbed data, resulting in the following minimax objective.
min LAT(DA Pe,00 O) = min (a,, 0.
max ÊcE (fe, (x; + 8)),
(2)
|
where the sample-wise perturbation 5' is constrained in a norm ball with radius e and the inner maximization is typically solved by PGD (Madry et al. 2017].
Fooling Standard Training (FST). Intuitively, the goal of the poisoned data Dp is to induce stan- dard training to learn a model on D, with parameters θ that is ineffective for classifying natural images from D.. However, their fooling can only work for standard training while being easily alleviated under adversarial training. In other words, their "unlearnable examples" are actually learnable. Specifically, HHuag et al. (2021) adopt the error-minimizing noise generated with the following min-min optimization problem tor fooling standard training:
min E (I....))) 0,
(I...))
min fee (fo, (x, + ++8%) y:)
(3)
88 SSp
2

---

Under review as a conference paper at ICLR 2022
Standard Training
Adversarial Training
22
0.8
0.6
00 Acouray
1.5
8.6
Cean Enorentn
raining
Claan
0.6
Ero-str
12
0.5
0.8
180
(6)
(6)
The training loss and natural test accuracy of models with 1) standard training on clean data (a) and pror-miiimizzig poisoned data (b); 2) adversarial training on clean data (c) and error- minimizing poisoned data (d). All experiments are conducted with ResNet-18 18 on CIFAR- 10 dataset.
Figure 1:
where 02 is the source model that is used to generate poisons with perturbation radius Ep. The inner loop seeks the Ln-norm bounded noise ö by minimizing the loss with PGD steps, and the outer loop further optimizes the parameters 0 by minimizing the loss the adversarial pair (æ; + 8", 9/).
To investigate how error-mininn eroo-mmnimizzin noise can fool standard training, we compare the training pro- cess of clean data and error- mooom-iiiiizing perturbed data in Figure [a)(()). where the training loss of error-miiiizing data is significantly smaller. This indicates that error minimization is designed to minimize the loss of the perturbed pair (æ;; + 6f, ys)) to near zero such that the poisoned sample can not be used for model updating. While for adversarial training, as shown in Figure (d), its inner maximization process can easily remove the error pror-minimizing noise by further lifting the loss of the perturbed pair (x; + 6% + 6), y/) with the marrmiiaxiiizing noise 6 In this way, the hidden information uncovered and makes those unlearnable examples learnable again.
Thus, to fool adversarial training, we need to go beyond the paradigm of unlearnable examples and design stronger type of poisoning, for which we need it to be irremovable and resistant to error-maximizing perturbations. Below, we introduce our attempts to design this irremovable noise.
4 DESIGNING OF IRREMOVABLE NOISE
Based on the investigation Section we can easily see that it is more challenging for fooling adversarial training than standard training. In the following, we will design effective irremovable noise from the aspects of features, labels, and training strategies.
4.1 THE NECESSITY OF ROBUST FEATURES
First, we notice that is necessary to use robust features for fooling AT. Specifically, we compare poisons generated using Fowl et al.] (2021b) from two different pre-trained models, a standardly trained model and an adversarially trained model, both with E, = 32/ use larger perturbation radius ε.. this factor can only slightly fool AT by 10% performance drop in Huang et al. (202T) and cannot guarantee irremovability. We compare the poisons generated from robust and non non-robust features. The results are shown in Figure 2a We can see that even with a larger Ep. poisons generated from the ST source model almost useless (orange lines). In contrast, the poisons generated from the robust source model can effectively bring down the final robustness from ~ 50% to ~ 30% (blue lines). More details of experiments for poison generations and training process could be found in Appendix A.2
32/255. /55 Note that although here
This observation indicates that robust features are necessary for fooling AT. According to Ilyas al. (2019). ST can only extract non-robust features, and thus the generated poisons only contain non- robust features, which, however, will be discarded under AT since it only relies on robust features. Therefore, to fool AT effectively, the source model itself must contain robust features so that the generated poisons could contain robust features that are resistant to AT. To achieve this goal, we adopt the adversarially trained models to craft poisons.
4.2 THE NECESSITY OF CONSISTENT LABEL BIAS
As shown in Section 3 process of AT. Recalling that AT itself can learn good models with mror-maximizing noise generated
the error-minimizing noise can be easily removed by the error-maximizing
3

---

Under review as a conference paper at ICLR 2022
Tool Rob
620
022
650
102
20
60
42
60
43
130
20
62
Traniin spochs
Touning spodie
Traninng spocts
Traning spocts
(a) Robust features V.S. non-robust features
(b) Target assignments
The training loss and robust test accuracy of adversarial training with poisoned data gen- erated with (a) robust (AT) and non-robust (ST) pre-trained models; and (b) different target assign- ments. All experiments are conducted with ResNet-18 18 on CIFAR-IC 10 dataset.
Figure 2:
by itself using untargeted attack, we consider to use alternative target labels that are different from the error-maximizing objective. Formally, given source model fa, and natural pair (Ii yi) E Do. we pick a target class v/ and generate the poison de by
6 arg min ECE (fe, (z; + 8.).
(4)
|| Sep
Specifically, we consider the following strategies for assigning fooling labels:
Random
Xie & Yuille y020 a randomly drawn label 3/ u.g.r. f1.2...., K};
LL (Kurakin et al.
2017): y the Least Likely label y/ = arg max (CE (fo, (2:)), v);
MC (Wang Zhang 2019; : the Most Confusing label y/ arg min yy lce (fo. (x:). v):
NextCycle (ours): the next label in a cyclic order y/ = (y) + mod K);
NearSwap (ours): label swapping with y/ = y/ + 1 mod K, if y: 2k + 1, KEN.
y) mod K, if y: = 2k,
We list their performance against AT in Figure 2b We can see that like groo-minimizing and error- maximizing noise, both Random, LL, and MC methods also fail to poison AT (blue, orange, and green lines). Instead,
can that both NextCycle and NearSwap can effectively degrade robust
accuracy to 30% 35% (red and purple lines). Comparing the five strategies, we can find a common and underlying rule for the effective ones, e.g., NextCycle and NearSwap, that the label mapping g V/ + y/ is consistent among samples in the same class while being different for samples from different classes. As result, they impose a consistent bias on the poisoned data such that all samples in the class A are induced to a specific class B. In this way, they can induce AT to learn a false mapping between features and labels. resulting in low robust accuracy on test data. Details of noise generation for these five label mapping strategy can be in Appendix A.3
4.3 TRAINING STRATEGY: INDUCING ADVERSARIAL TRAINING (IAT)
From the above two sections, we have known that poisons generated from robust models with con- sistent label bias can successfully fool AT to some extent. Nevertheless, we still notice there are some discrepancies between the fooling process and the adversarial training process. Specifically, for fooling, utilize pre-trained source model fe. while for AT, we train a target model fe. from scratch. Even though the source model is robust enough, its loss landscape could be very differentt from that of a randomly initialized target model. Besides, the source model is learned with clean data, while the target model is learned with poisoned data instead. These discrepancies between the source model fo, and the target model fo, will make the poisons generated from the source model less effective for fooling the target model.
To close these source-target discrepancies, we believe that it is better to generate poisons also from randomly initialized model that is adversarially trained for predicting the target labels y/. This will lead to an alternating procedure between two steps:
a) generating poisons D, from the source model fo, by
8 -arg min ECE (Se, (x; +
E D..
(5)
118 SS
4

---

Under review as a conference paper at ICLR 2022
0.8
IAT
Prottrrddd Clean
Natural Accuracy
Test Test Robust Accuracy
0.5
0.7
0.4
Training
0.6
0.3
0.5
0.2
0.1
0.3
0
20
100
120
20
40
60
80
100
120
40
80
80
100
120
(4)
(b)
(c)
Figure 3: The training loss (a), the natural test accuracy (b), and the robust test accuracy (c) of adversarial training under clean datasets, AT-pre-trained poisons, and IAT poisons, respectively.
b) adversarial training of the source model fe. such that it could robustly predict the poisoned images (a; + 6,') to the inducing target labels y'. i.e.
min E (a:+8) a,) 1
max ECE (fe, (x, + 8% 58, 3/4)
(6)
0,
In practice, we will keep involving the loop until the following Poisoning Success Rate (PSR)
PSR(Dp) = Ee + M.) [v/ = arg max fe, (z: + 6?)]
reaches a certain threshold 1). In this way, the source model will robustly classify the poisoned data to the target classes and generate poisons with desired inducing features. Therefore, we name this iterative poisoning process as Inducing Adversarial Training (IAT) as it involves the induction into the adversarial training process. As shown in Figure 3 when compared to poisoning with pre-trained models (blue line), our IAT (orange line) can achieve an even smaller training loss (plot a), while having worse natural test accuracy (plot b) and much worse robust accuracy 31.4% 0.6% (plot c). This shows that our IAT is much better fooling AT by causing worse test robustness while inducing a smaller training loss.
At last, combining IAT with inducing label assignments, arrive at our instantiation of irre- movable noise, namely Adversarially Inducing Noise (ADVIN), that could induce AT to a catas- trophic behavior. In particular, ADVIN could combine the advantages of using robust features and consistent label bias for fooling adversarial training. As shown in Eq. 5 the noise is generated by the source model through targeted PDG attack towards the targeted label y' with consistent bias. Meanwhile, the source model is adversarially trained as shown in Eq. 6 this way, we can inject the robust features of the targeted classes s/ into y-class samples consistently. When the poisoning success rate (PSR) reaches the given threshold, the poisons are trained to have enough robust fea- tures about the target class y'. Therefore, when the target model is trained on the y-class poisoned data, will be induced to use the y -class robust features in the perturbation to predict the true la- bel However, it will have catastrophically poor performance on natural test data where y-class samples only contain y-class features. The overall procedure for generating poisoned data is shown in Algorithm]]
Algorithm Generating Poisoned Data with Adversarially Inducing Noise
Input: Source model fe.. clean training dataset D. = {(x,, y:)}, training steps M, poison steps per sample T, threshold of fooling success rate v)
Output: Poisons 8", poisoned training datasets D,
For all x; E De. randomly initialize perturbation 8 within the ep-ball
2: while the poison success rate PSR(D,) S 17 (Eq. 7 do
3: Update each perturbation 8 by PGD for T steps using Eq. 5 (with NextCycle by default)
4: Adversarially train fo. on ((x + 8", .//)} with inducing labels for M steps (Eq. 6
5: end while
6: return Poisoned training dataset D, = ((x: + 5, (.y))}
5

---

Under review as a conference paper at ICLR 2022
Table 1: The natural and robust test accuracy of the target model, which is trained poisoned data
(generated with different poisoning methods) on CIFAR-10, SVHN, and CIFAR-100.
CIFAR-10
SVHN
CIFAR- 100
Poisoning Methods
Natural
PGD
Natural
PGD20
Natural
PGD
Clean (baseline)
85.14%
51.71%
91.43%
56.88%
57.52%
27.18%
Huang et al. 20221
73.42%
13.34%
58.54%
2.84%
51.45%
21.84%
Fowl el al. 2021b
78.83%
47.89%
87.52%
43.49% 0.46% %
51.21%
24.81% %
ADVIN (ours)
56.52%
0.57%
63.88%
46.72%
11.52% %
0.8
0.5
Training robust socuracy
Test Accuracy
Tost Robust R Accuracy
0.4
0.3
0.5
//
0.2
Clean
Enror-min
0.4
0.1
Adversarial examples
ADVIN
0
20
40
80
100
120
20
40
60
80
100
120
20
60
80
100
120
(b)
(c)
Figure 4: Comparison among baselines (clean data, Huang et al. (2021). Fowl et al. (2021b) and ADVIN (ours) of robust training accuracy (a), natural test accuracy (b) and robust test accuracy (c) with adversarial training. All experiments are conducted with ResNet- 18 on the CIFAR- 10 dataset.
5 EXPERIMENTS
In this section, we first evaluate our poisoning methods against previous methods on the bench- mark datasets and then verify the transferability of our methods across different training algorithms, network architectures, and poisoning ratios. At last, we provide comprehensive analysis of our ADVIN w.r.f. inductive training, target assignments, noise shapes, well as poisoning threshold.
Poison Generation. For the souree model used to generate poisons, we adopt the ResNet-18 18 (He et al. 2016) and train it by AT, where the optimizer is an SGD with a momentum of 0.9. The initial learning rate is set to 0.1 and the weight decay is set to 5e-4, Following Huang et 2022). we generate poisons for adversarial training using relatively large perturbation range Ep 32/ 25 Specifically, we use 60 steps of PGD with step size 2/255 and generate poisons every 30 training steps until the terminal threshold for PSR is met at 1) 0.99. Besides, we select the NextCycle strategy mentioned in Section 4.22 our target label mapping function. Typically the poisoning process is very quick and ends within five steps (less than one training epoch). We adopt this setting as default across all our experiments unless specified.
5.1 EVALAUTION ON BENCHMARK DATASETS
We evaluate the effectiveness our poisoning method on fooling AT against previous poisoning methods: emror-minimizing (Huang et al., (202)) and adversarial example noise (Fowl al., 202lb). We conduct experiments on three benchmark datasets, CIFAR-10, SVHN, and CIFAR 100. For each dataset, we adversarially train a ResNet-18 18 (as the target model) with poisoned data generated with different poisoning methods for 120 epochs. Specifically, we set the initial learning rate 0.1, 0.1, and 0.01 for CIFAR-10, CIFAR-100, and SVHN, respectively. The learning rate decays by 0.1 at epoch 75, 90, and 100. We adopt an SGD optimizer with momentum of 0.9 and weight decay of 5e After training, evaluate the target model on natural (unpoisoned) and adversarial data 20 with E 8/255) and get the natural and robust test accuracy, respectively.
Evaluation Protocol.
As shown in Figure in Appendix it will affect the semantics of the raw images.

---

Under review as a conference paper at ICLR 2022
Table 2: The natural accuracy and robustness against different AT' defense algorithms for ADVIN. Here we conduct experiments on CIFAR-I0 with ResNet-18. 18.
Madry's
MART
TRADES
Poisoning Methods
Natural
PGD20
Natural
PGD
Natural
PGD20
Clean data
85.14%
51.71% 0.57 %
81.78%
55.56%
82.44%
55.12% 1.66 %
ADVIN (ours)
56.52%
56.40%
0.68%
56.51 %
Table The natural accuracy and robustness of various network architecture for adversarial training. The poisons are generated on CIFAR-10 10 with ResNet-18 as source model fe.
ResNet-18 18
ResNet-34
VGG-11
MobileNet-v2
Poisoning Methods
Natural
PGD
Natural
PGD
Natural
PGD
Natural
PGD
Clean data
85.14%
51.71%
86.21%
51.64%
79.05%
46.60%
80.42%
51.01% 0.85 %
ADVIN (ours)
56.52%
0.57%
56.13%
0.54%
67.34%
4.97%
57.53% %
Robustness Drop at Test Time. We report the performance among clean test datasets of the last epoch in Table [ For all kinds of datasets, we obtain the lowest robust test accuracy of models trained on ADVIN. Specifically, in terms of robustness, the adversarially trained ResNet-18 18 have catastrophic behavior on both CIFAR-10 and SVHN, where the robustness decreases from 51.71 % to 0.57% and from 56.88 to 0.46% respectively, showing that ADVIN has severely led the training process disrupted. Besides, on CIFAR-100 100, Huang et al. (2021) and Fowl et al. (2021b) poisons can only have a slight influence on both natural and robust test accuracy, while we obtain relative decrease of robust accuracy of 57.6% (Compared to ADVIN, prror-minmmizing noise and adversarial example noise only obtain a relative reduction of 19.6% and 8.7%. respectively). As for natural test accuracy, we obtain the lowest points on both CIFAR-10 10 and CIFAR-100. Although error- minimizing noise outperforms our ADVIN by little margin on SVHN we still achieve very low natural accuracy.
Fooling the Training Process. Intuitively, since we want to fool the models to learn something from poisoned datasets, should make sure that models perform well train datasets and behave badly on test datasets simultaneously. Figure shows the adversarial training process of ResNet- 18 on clean examples, error mirom-mmniiizing noise Huang et al.] 2021). adversarial example noise (Fowl et al.) 202 and our ADVIN for CIFAR- 10 datasets. Figure D draws the training robust accuracy curve along training epochs (0 120) of four datasets (clean poisoned). On clean datasets, the ResNet- 18 model gets about 70% robust training accuracy while error-minimiza minimiziii noise and adversarial example noise achieve about 80% and 60% respectively. Surprisingly, the robust training accuracy of ResNet- 18 trained on ADVIN reaches over 95%, which makes the model fully believe that it has been well fitted on the training set, while it will perform very poor on the test datasets as shown in Figure 4b and Figur
5.2 EMPIRICAL UNDERSTANDINGS
In the above, we have shown the effectiveness of our poisoning method on benchmark datasets when the poisoming stage and the training stage share the same architecture and training objective. However, in practice, if adopt our poisoning method to protect personal data, the users agnostic to know how their data will used for training. Therefore, we further apply our poisons to other training methods and model architectures to study their black-box effectiveness. We also explore whether poisons still work when they are partially applied. For integrity, performance of standard training and adversarial training (using different E2) on our ADVIN. The results for AT with different training E, and poisoning rate can be found in Appendix [B]
also test the
Against Different Defenses. To valid the generalization among other adversarial training methods, use TRADES (Zhang et al., 2019) and MART (Wang et al.] 2020) as defense algorithms. As shown in Table 2 when training the model with ADVIN, the robustness under PGD 20 attack will decrease about 1.7% (trained with TRADES) and 0.7% (trained with MART) respectively, while the natural accuracy remains about 56%. Results show that although the training algorithms of the source model fo, and the target model fo, are different, our poisons can still fool the adversarial training process and make it ineffective.
7

---

Under review as a conference paper at ICLR 2022
Table 4: The test accuracy of standard training. Here we conduct experiments on three typical datasets (CIFAR-10, SVHN and CIFAR-100) with ResNet-18. The poisons are generated as de- scribed in 5.1
Poisoning Methods
CIFAR-10
SVHN
CIFAR-100
Clean data
94.60%
95.61%
71.11%
ADVIN (ours)
13.12%
9.15%
2.39% %
Transferability Across Architectures. Intuitively, the poisoned datasets should also destruct the training process whatever the network architectures are used during training. To valid the trans- ferability, we adversarially train our ADVIN on different network architectures. Table 3 shows that poisoned data crafted by ResNet-18 could also transfer other models. The performance on ResNet-34 34 and MobileNet-v2 is almost equal to the performance on ResNet-18, with natural accu- racy decreasing to about 57% and robust accuracy decreasing to lower than 1% respectively.
Effectiveness on Standard Training, Since ADVIN has shown the effectiveness of fooling various adversarial training methods, it is reasonable that ADVIN can even disrupt the standard training process more thoroughly. Therefore, we replace the adversarial training for the target model with standard training. For all the three datasets, we train them for 60 epochs with initial learning rate of 0.1. An SGD and MultiStepLR are used for optimization. Table 4 reports the accuracy of ResNet- 18 trained on benchmark datasets, which demonstrates that our ADVIN can destroy the per- formance of models and lead the accuracy close to random guess (e.g.. 9.15% accuracy for SVHN).
5.3 PARAMETER ANALYSIS
Here, we provide a thorough analysis of the generation process of our proposed inducing noise in terms of the following aspects. First, show that adversarial training is necessary in our inductive training process by comparing with standard training. Then, we show our poisoning still effec- tive when being applied to a small region in the image. Besides, in Appendix C we study the effect of different adversarial training algorithms for source models and showing that they are all useful for various defense algorithms. We also analyze the effect of alternative label assignments and PSR threshold. Results show our method is relatively robust to these choices.
Adversarial Inducing Noise v.s. Standard Inducing Noise. As discussed in Algorithm I gener- ating poisons by ADVIN is an iterative process that could be divided into two stages, adversarial training and noise generation with the source model. Here, we replace the adversarial training with standard training instead and name the poisons as STanDard Inducing Noise or STDIN. Note that in contrast to Fowl et al. (b021b). we train the source model from scratch instead of using a pre-trained model. As shown Figure although STDIN achieves higher robust training accuracy than clean data, there is still a big gap between STDIN and our ADVIN, which indicates that ADVIN can fool the source model more thoroughly. As for the poisoning effect on the test data, STDIN is also ef- fective to some extent, with natural test accuracy dropping by about 16% and robust test accuracy dropping by about 46%. Nevertheless, our ADVIN still outperforms STDIN by a large margin (13% natural test accuracy and 5% robust test accuracy). This comparison illustrates the effectiveness of adopting adversarial training for generating poisons in our ADVIN.
Effectiveness of Small-size Poisons. In previous experiments, we apply the noise to full images, while here, we explore the effectiveness of small-size noise. We expect the smaller-patched noise to fool the training of target models like triggers and at the same time retain semantic information much as possible. We choose 8*8, 16*16, and 24 as patched sizes. As expected, they can all destroy AT as shown in Table Surprisingly, when the model trained on 16 16 patched noise, it gains the lowest test natural accuracy of 44.16%, which is about 12% lower than 32*32 patched noise, and second-lowes, test robust accuracy of 0.76%, which is also less than 1%.
6 REAL-WORLD APPLICATION FOR DATA PRIVACY PROTECTION
As introduced above, can utilize poisoning to protect personal data from being exploited by commercial companies. Here we consider a real-world scenario where personal profile photos online could be crawled down for training face recognition systems without permission. Below, we show that adding our poisoning to the data can successfully protect the data from being exploited by either standard or adversarial training and make them truly unlearnable examples.
8

---

Under review as a conference paper at ICLR 2022
0.9
0.5
Robust Training Acquracy
0.8
Natural Accurapy
0.7
Robust Test Accurany
0.4
0.7
0.6
0.3
0.8
0.5
0.2
0.1
STDIN
0.3
0.2
data
0.0
0
20
40
60
80
100
120
0
20
40
60
80
100
120
20
40
60
80
100
120
(b)
(c)
Figure Comparison between ADVersarial Inducing Noise (ADVIN) and STanDard Inducing Noise (STDIN) robust training accuracy (a), natural test accuracy (b) and robust test accuracy (c) with adversarial training. All experiments are conducted with ResNet-18 on the CIFAR-10 dataset.
Table 5: The natural accuracy and robustness of ResNet-18 for AT. The poisons are generated on CIFAR-10 with ResNet- 18 as source model fo.- We set the shape of noise 8*8, 16*16, 24*24 and 32*32 (the size of full images) respectively.
8*8
16*16
24*24
32*32
Natural
2GD
Natural
PGD
Natural
PGD
Natural
PGD>20
72.19%
19.69%
44.16%
0.76%
66.93%
6.49%
56.52%
0.57
Setup. We choose Webface our raw dataset, which includes about 490k images of over 10k identities. For simplicity, we select the ten most frequent classes of images as our sub-dataset and name it Webface-10, The Webface-10 10 dataset consists of 5338 images for training and 1340 images for testing. Specifically, we want to protect the selected sub-datasets and generate noise for them, which leads the models to get fooled by the noise. Therefore, both clean and poisoned Webface- 10, we train a ResNet-18 18 where we set the learning rate to 0.01 and weight decay to 5e for ST (60 epochs) and AT (120 epochs). Also, choose CosinnAnnealingLA. learning rate drops by 0.1 for ST. An SGD with a momentum of 0.9 used for optimization.
the scheduler of ST, and the
Table Both the accuracy under standard training and the natural/robust test accuracy under adver- sarial training for Webface-10. 10. We use ResNet-18 18 for both the source model and the target model.
Poisoning Methods
Natural (ST)
Natural acc (AT) Robust acc (AT)
Clean data ADVIN (ours)
89.18%
81.34%
43.81 %
27.46 %
40.82
22.69 %
Results. As shown in Table 6 the natural accuracy could reach 89% and 81% with standard training and adversarial training, respectively. In comparison, with poisoned data, standard training could only obtain 27.46% natural accuracy, and adversarial training can only achieve 10.82% natural accuracy and 22.69% robust aecuracy. This shows that our poisons can actually protect the users' data from being mined and utilized for training.
7 CONCLUSION
In this paper, we have designed new kind of poisoning method, Adversarial Inducing Noise (AD- VIN), for fooling adversarial training. Extensive experiments on a range of benchmark datasets show that the generated poisons make adversarial training ineffective training strategies and model architectures are adopted. Besides, we can conduct thorough anal- ysis of the poisoning generating process, showing that our poisoning is effective under different stopping criteria and (fixed) label assignment strategies. At last, we successfully apply our method to protecting personal data privacy against adversarial training on face recognition.
matter what different

---

Under review as a conference paper at ICLR 2022
REFERENCES
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector ma-
chines. In ICML, 2012.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey, arXiv preprint arXiv:1810. www.1110.0000 1.00000069, 2018.
Ivan Evtimov, Ian Covert, Aditya Kusupati, and Tadayoshi Kohno. Disrupting model training with adversarial shortcuts. arXiv preprint arXiv:2106. 0666644 2021.
Ji Feng. Qi-Zhi Cai, and Zhi-Hua Zhou. Learning to confuse: generating training time adversarial data with auto-encoder. 2019.
Liam Fowl, Ping-ych Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use of proprictary data: Poisoning for secure dataset release. arXiv preprint arXiv:2103.0266: 2021a.
Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojtek Czaja, and Tom Goldstein.
Adversarial examples make strong poisons. arXiv preprint arXiv:2106. 10807, 2021b.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015. URL /tattp://arxiv...t arxiv. org.o/gg///..... abs http//abs/411..... 6572
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp. 770-778, 2016.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. In ICLR, 2021.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. In NeurIPS, 2018
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, pp. 1885-1894. PMLR, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In ICLR, 2017.
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang. Yitian Zhao, James Bailey, and Feng Lu. Under- standing adversarial attacks on deep learning based medical image analysis systems. Pattern Recognition, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2017.
Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 27-38, 2017.
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. Fawkes: Protecting privacy against unauthorized deep learning models. In 29th (USENIX) Security Sym- posium ( (USENIX) Security 20), pp. 1589-1604, 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Pre- venting delusive adversaries with adversarial training. 2021.
10

---

Under review as a conference paper at ICLR 2022
Qi Tian, Kun Kuang, Kelu Jiang, Fei Wu, and Yisen Wang. Analysis and applications of class-wise robustness in adversarial training. 2021.
Jianyu Wang and Haichao Zhang. Bilateral adversarial training: Towards fast training of more robust models against adversarial attacks. In ICCV, pp. 6629-6638, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In ICLR, 2020.
Chia-Hung Yuan and Shan-Hung Wu. Neural tangent generalization attacks. In ICML, pp. 12230- 12240. PMLR, 2021.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent HI Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICMI., pp. 7472-7482. PMLR, 2019.
A EXPERIMENTAL DETAILS
A.1 DETAILS OF WHY ERROR- CECOORMMNIMMZING NOISE FAILS
We adversarially and standardly train ResNet- 18 with clean datasets and error-minimizing poisoning datasets for CIFAR-10 10, respectively. Specifically, we set Ep, to 8/255 for error-minimizing noise and run the command released http://gihub.ccm//aaauttt htwp:/aannmm/nnllaaaaaaeeeeE-- Examlless For standard training. we use an SGD with a momentum of 0.9 and set the learning rate to 0.1 with weight decay 5e The learning rate drops through a Ansealinnn we all set the batch size to 128. The setting adversarial training is very close, except we train the model for 120 epochs, and the learning rate drops by 0.1 at epoch 75, 90 and 100. In addition, we set the perturbation budget of adversarial training E 8/255.
scheduler. For training and testing,
A.2 DISCUSSION ABOUT ROBUST AND NON-ROBUST FEATURES
We first standardly and adversarially train a ResNet-18 model the source model, and then use PGD attack to generate adversarial examples as poisons. The standardly pre-trained ResNet-18 trained for 60 epochs with an initial learning rate of 0.1, weight decay 5e -4, and CosineAmealinggL.. as the scheduler, while we train the robust R model by AT (Madry et al. 2017) for 120 epochs with MultiStepLR. The learning rate drops by 0.1 at epoch 75. 90, and 100. For both adversarial training and standard training, set the training batch size to 128. An SGD with momentum of 0.9 is used for optimization. For noise generation, we use targeted PGD200 with step size 2/255 to generate adversarial examples as poisons. In terms of targeted attack, we choose NextCyele as mentioned in Section 4.2 for label mapping.
A.3 EXPERIMENTS OF LABEL CONSISTENT BIAS
As Section 4.1 has discussed, we use an adversarially pre-trained ResNet-18 model to generate adversarial examples poisons. The process of training ResNet-18 source model and poison generation is just the same as A.2 except that we use five different target label mapping functions (Random,LL, MC, NextCycle, and NearSwap).
B EFFECTIVENESS ACROSS DIFFERENT TRAINING SETTINGS
Results for AT of different training e.. Intuitively, the poisons generated should also fool the mod- els adversarially trained with different e, (e.g., 2/255, 4/255, 8/ 8/555, 16/25% and 32/ 225). There- fore, we train a ResNet-18 on poisoned CIFAR-10 with AT under different é,. From Table 7 our ADVIN can decrease the performance among all the models trained with different é,, especially when the E, less and equal 16/255. For example, whether the models are adversarially trained with E, 2/ 2/255 or 4 4/255, they entirely lose the robustness and could be easily attacked by 2GD Besides, the natural test accuracy (about 20%) is also much lower than the results on clean data (about 90%). While the models are trained with e, = 32 /55, our ADVIN could also slightly affect the robust and natural test accuracy (about a drop of 2%). The effectiveness of our ADVIN seems

---

Under review as a conference paper at ICLR 2022
Table 7: The natural accuracy and robustness of AT under different Et- The models are trained on poisoned CIFAR-10, 10, which are generated with ResNet-18 18 as source model fo.
2/ 225
4/255
8/255
16/255
32/255
Poisoning Methods
Natural
PGD22
Natural
PGD
Natural
PGD
Natural
PGD
Natural
PGD*
Clean data
92. 92.62%
29.26%
90.17%
41.05%
85.14% 14%
51.71%
67 94%
52.54%
34.50%
29.38%
ADVIN (ours)
18.05%
0%
22.16%
0%
56.52%
0.57% %
67.36%
44.25%
32.53%
27.83%
Table 8: The natural accuracy and robustness under partially poisoned datasets D2 + DD and part of clean datasets De for ad adversarial training. The poisons are generated on CIFAR-10 with ResNet-18.
Dataset
0%
20%
40%
60%
80%
100%
Natural
PGD*
Natural
PGD
Natural
PGD
Natural
PGD
Natural
PGD
Natural PGD
DD D. D,
85.14%
51.71%
84.05%
49.51%
82.61%
46.67%
79.03%
42.64%
73.98%
35.15% 37.92%
84.69%
51.64%
84.25%
48.68%
82.57%
44.47%
80.57%
56.52%
0.57%
be reduced with E, = 32/255. This may be due to the fact that the noise could be distorted by the training perturbation with the same order of magnitude as ADVIN.
Poisoning Rate. In the practice scenario, it is very likely that not all the training examples poisoned as mentioned in |Huang et al.) (202T). Therefore we randomly select a certain portion of data to add noise while keeping the remaining training examples clean. The partially poisoned datasets are denoted poison rates. In comparison, we also use the subset of clean training data for adversarial training. where the portion of clean examples is the same as partially poisoned datasets D3 + De. and we denote it as De. Results show that a small portion of clean examples could help DNNs get trained well and lead to a huge increase both test robust and natural accuracy shown in Table & For example, when we select 80% of poisoned examples and 20% clean data jointly as D, + D.. After adversarially train the mixed datasets, 37.92 37.92%. which are even little higher than the models trained on only D... A similar result can be observed in previous work (Huang et al. (2021) and (Shan et al. 2020).
D, + De- We adversarially train the poisoned datasets with different
gain natural and robust test accuracy of 80.57% and
C ANALYSIS ON POISON GENERATION
C.I INDUCING ADVERSARIAL TRAINING WITH OTHER AT VARIANTS
Here we utilize MART (Wang et al. 2020) and TRADES (Zhang al. 2019) for training the source model. For a fair comparison, we also evaluate the target model trained by MART and TRADES instead. As expected, from Table 9 we can see that the poisons crafted by other AT variants could also generate useful noise. Especially the noise generated by TRADES could lead the natural test accuracy even lower than 50% whatever the training algorithms for the target model are. In comparison, ADVIN could only lead the models to reach a natural test accuracy of about 56%.
C.2 INDUCING TARGET LABEL
From the discussion of Section 4.2 we know that consistently target label bias is critical for noise generation. We replace the label mapping function mentioned in Algorithm [] (We use NextCycle strategy as default for previous experiments.) and test the ability to disrupt the training process of the noise with other target label induction. Table 110 lists the test robust and natural accuracy of R8 on CIFAR-10. The models are all trained on ADVIN but with different targeted inducing label mapping. NearSwap denotes the swap between the nearest classes as mentioned in Section4.22 while SimilarSSay and DissimilarSwap indicate the swap between semantically similar classes and semantically dissimilar classes. Motivated by Tian et al. (02T). we use the confusion matrix to estimate the semantic similarity. Here we also want to explore the class-wise property of the datasets. Since we assign every class an induced label during noise generation, the poisoned training examples should include the features of induced classes to some extent, which may lead to the results that when we test the clean datasets, the predicted results should show some bias to the reversely induced label (e.g.. when training we use class 2 as the inducing label for class I under NexiCycle setting, then during the test time, the predicted results of class should have tendency towards class 2).
12

---

Under review as a conference paper at ICLR 2022
Table 9: Poisons are generated from source models trained with different methods. Here lists the performance of target models trained with Mardy's (Madry et al., 2017), MART (Wang et al.) 2020) and TRADES (Zhang et al. 2019). Rows correspond to the training methods for source models, and cols correspond the training methods for target models.
Defense
Madry's
MART
TRADES
Poisoning
Natural
PGD
Natural
PGD
Natural
PGD
Madry's MART
56.52%
0.57 %
56.40%
0.68%
56.51 1%
1.66 %
52.83%
7.05%
50.20%
7.54%
53.44%
11.73%
TRADES
44.87
1.49%
41.99%
1.61%
46.18%
3.90%
NextCycle
Clean
1Swap
59
14
31
56
12
50
23
0.6
24
40
22
16
.63 22
02 2.4
L2
25
22
64
10
2.6
02
61
2.5
05
25
74
25
22
18
-13 66
40
033
44
53
25
25
62
86
2.9
3.5
0.8
74
18
15
51
83
17
14
-2-2 63
64
211
2.4
03
61
05
05
05
4.4
12
77
16
1.8
-37 14
50
3.7
32
05
30
3.6
-20
6.7
10
2.5
-10
Figure 6: Confusion matrix for clean CIFAR-IO test datasets of models trained with different poi- sons(ADAIN with NexiCycle label assignment, Clean training data, and ADAIN with NearSwap label assignment, respectively.) where rows correspond to the ground truth labels and cols corre- spond to the predicted labels.
Figure [6] shows the confusion matrix under clean and poisoned data for CIFAR-10. From that, we see whether under NextCycle NearSwap setting, the misclassifidd labels represent certain degree of consistency compared with the confusion matrix under clean datasets. For example, under clean data setting, adversarial examples of class have a probability of 12% to be predicted to class 1. while under ADVIN of NexlCycle and NearSwap wap, the probability increase up to 53% and 54% respectively. Besides, we indeed observe that the induced labels have some effects on the predicted classes. The adversarial examples of class tend to predicted as class under the NextCycle setting, where they have a tendency towards class 9 under the NearSwap setting.
C.3 STOPPING CRITERION
Recall that the signal for terminating the loops of noise generation is PSR(D,) surpasses the thresh- old of fooling success rate 7). Here we discuss the influences of different threshold n) on the effec- tiveness of poisons. Specifically, we choose 90%, 99%, and 99.9% as rate threshold to control the termination for noise generation. As shown in Table 11 when we set the threshold to 90%, the natu- ral and robust test accuracy achieve about 64% and 8%, respectively. Compared the results when the threshold is set to 99% and 99.9%. only gets about 8% improvement. In general, although the performance of models trained on noise with different thresholds has some slight differences, the effectiveness of ADVIN is insensitive to the PSR(D,) threshold.
D VISUALIZATION OF NOISE GENERATED FROM PRE-TRAINED MODEL
We randomly select part of images of CIFAR-1O 10 and generate robust features noise from a adver- sarially pre-trained model as mentioned in 4.1 Here we compare the poisoned images with noise under different Ep- From Table we can see that although the Ep is increased to 32/25 255, the poisoned images could still remain semantic information well.
13

---

Under review as a conference paper at ICLR 2022
Table 10: The natural accuracy and robustness of ResNet-18 for adversarial training. The poisons are generated on CIFAR-10 10 with ResNet- 18 as the source model Ja,, where four kinds of inductive label mapping function are utilized as mentioned Algorithm
NexeCycle
NearSwap
SimilarSwap
DissimilarSwap
Natural
PGD
Natural
PGD
Natural PGD
Natural
PGD
56.52%
0.57%
54.92%
2.96%
63.69%
7.92%
55.91%
5.95%
Table 11: The natural accuracy and robustness under AT, with being trained on ADVIN generated from different fooling success rate The experiments are conducted on CIFAR-10 with ResNet-18.
90%
99%
99.9%
PSR(D,) threshold
Natural
PGD
Natural
PGD 20
Natural
PGD
ADVIN
64.21%
8.44%
56.52%
0.57%
56.52%
0.57%
E SUPPLEMENTARY DISCUSSION ABOUT RELATED WORK
Feng et al. 2019) first propose the data poisoning problem where training data are allowed to be modified with a bounded perturbation such that the DNNs trained on the poisoned data could have poor performance
clean test data. However, Feng et al. (2019) adopt auto-encoder to generate poisons 6, under white-box setting with full access the targeted model, making them not practi- cal in real-world scenario. Fowl et al. (2021a) instead achieve this by aligning the training gradient of the perturbed data with the gradient of the reverse cross-entropy loss on the clean data, and Yuan & Wu (222) propose Neural Tangent Generalization Attacks based on Neural Tangent Kernels (Ja-) cot et 20.8 Besides, Desides, et al. (202T) explore three class-wiss patterns of shortcuts which control the trade-off between disrupting training and preserving visual features in the data. While these work focus poisoning standard training. Tao et al. (2021) theoretically show that adversarial training is an effective method defend these attacks, Different from theirs, the goal this work to develop a poison algorithm that works on both adversarial training as well as standard training.
F MORE POISONED IMAGES
To show that poison budget Ep 32/ 25/255 will not modify the clean labels, randomly select ten poisoned images from each class in CIFAR-10 and compare them with clean ones. The poisons are generated through an adversarially pre- trained model as described in Section 4.I] Note that here we set the poison budget Ep. = 32/255. As shown in Figure g the poisons will not affect the semantics of images nor modify the true labels.
14

---

Under review as a conference paper at ICLR 2022
clean
2/255
4/255
8/255
16/255
24/255
32/255
64/255
Figure 7: The noise generated from PGD200 attack with a pre-trained robust model under different ép-ball.
attack with pre-trained robust model. Ten images are randomly selected from each of the class in CIFAR-I0. Each row shows ten pairs of clean and poisoned images. Here the poison budget Ep is set to 32/255.
Figure 8: The poisoned images generated from PGD 200
15


Reviews:
[{"review_id": "fYhIKd6bhzu", "reviewer": "Reviewer_H39o", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is well written. It presents an effective approach to performing such poisoning; however, the work does not sufficiently explore the reasons for the success of the technique (ADVIN) or compare it to appropriate baselines. The work could also improve clarity in some areas.\n\nFleshing out technique/baselines: ADVIN, the main presented technique of the work, is not explained or motivated sufficiently. It is difficult to make sense of why this technique works, or even why it should work to begin with---I would have thought that changing the model objective so much would make the network relatively useless when poisoning new adversarially trained models. It would be good to start out with simple extensions of the AT-Pretrained method first: what happens when you ensemble and do the AT-Pretrained method, or adversarially train for the standard (i.e. proper labels with their corresponding perturbed $x$ images) objective in the loop?\n\nClarity: \n- It would be good to expand on Huang et al 2021 around equation (3) for greater context on methods used in this area (especially considering that the techniques in the work heavily build upon Huang et al 2021).\n- Minor point: the title of the forum page for this paper does not match the title of the pdf of this paper.", "summary_of_the_paper": "This work tackles the problem of data poisoning adversarial training in the (image classification) setting where the adversary can add $\\ell_p$ constrained perturbations to each training example. Others have previously studied this problem in the standard ERM setting by adding $\\ell_p$ perturbations to examples that minimize loss on pretrained models: when the downstream learner tries to minimize risk on these perturbed examples, they can do well on the train set but not on a held-out set from the original distribution (this work is all performed in CIFAR). It turns out that adversarially trained models can largely circumvent this class of attacks, as they can modify the adversarial $\\ell_p$ perturbations in the adversarial training process to make learning require appropriately generalizable features. \n\nThe authors try a new attack that focusses on adding \"robust\" features to examples: in the most basic attack, for each (example, label) combination the authors perturb the example to maximize the probability of a different (label-consistent --- i.e. each image of a given label is perturbed towards the same class, one such mapping could be target_label = label + 1 % num_classes ) label. The authors specialize the attack further by performing an alternating optimization routine in which one repeatedly (starting from a pretrained adv-trained model $f$): (a) completes the perturbation above on the examples using $f$ and then (b) adversarially trains on $f$ using the new label scheme. Using this technique the authors can force the model to perform well on the train set, but not on a held-out set. Using only step (a) (i.e. no alternating optimization routine) the authors can fool the model in the same sense but to a lesser extent.", "summary_of_the_review": "The work presents a new technique that effectively poisons adversarially trained models at train time. However, the authors do not properly motivate the technique, the baselines are inadequate, and the paper can be sometimes unclear. For these reasons I give a score of weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XUgNcyL91r", "reviewer": "Reviewer_eute", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The paper studies the failure of the existing method, for instance, how the data poison attack proposed in Huang et al. (2021) fails in the adversarial setting. This motivates a stronger type of attack that can fool adversarially learned models.\n\n2. The paper is clearly written. The logic and content are well organized.\n\n3. The paper presents many experiments to show the effectiveness of fooling adversarial training.\n\n## Weaknesses\n1. The findings that existing data poison attacks do not work well in adversarial training are quite trivial since those attacks focus on the normal setting but might not consider the existence of adversarial training. For instance, the error-minimization-noise in Huang et al. (2021) generates noise based on normally trained models and therefore it is very natural that it doesn't fool adversarial training. Therefore, such findings are not surprising.\n\n2. The novelty of the proposed poison algorithm is weak. The major idea in this paper follows the error-minimization-noise in Huang et al. (2021), but it considers updating the model based on the adversarial loss defined on the adversarial samples with poison noise instead of on the clean data with noise. Although this paper proposes other techniques for improvements such as the consistent label bias, overall, the novelty is quite weak.\n\n3. The paper doesn't discuss whether existing defense methods against data poison attacks can defend the proposed attack. Although the poison data looks visually similar to the original data, it is still possible to detect or remove such attacks. More discussion on this will be helpful.\n\n4. The data poison effect is not that strong as shown in the main paper. If I understand correctly, in most of the experiments in the paper, the poison ratio is 100%. In practice, it is unlikely that we can control and poison the whole dataset. However, Table 8 shows that when the poison ratio is lower, the poison effect is much weaker. For instance, when 20% / 40% / 60% of the data samples are poisoned, the robust accuracy only decreases by 0% / 3% / 7%. Therefore, this suggests the proposed attack might not be very strong in practice. \n\n", "summary_of_the_paper": "The paper proposes a type of data poison attack called adversarially inducing noise (ADVIN) that degrades the performance of adversarial training. It shows the necessity of considering the adversarial learning in the poison process and the experiments show that ADVIN can successfully make adversarial training ineffective.\n\n", "summary_of_the_review": "The paper proposes a data poison attack to fool the adversarial training algorithms. The novelty of the proposed method is weak and the attack seems to be weak when the poison ratio is lower. I would like to recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ObPImVJ3nhf", "reviewer": "Reviewer_Hbey", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- This paper is the first to study the problem of poisoning the performance of adversarial training.\n- The proposed poisoning attack can successfully make adversarial training with small $\\epsilon$ ineffective.\n\nCons:\n\n- This paper lacks an adequate discussion/reference of a few closely related works, including [1], [2], [3], [4], and [5]. In short, they all studied the threat that aims to degrade test accuracy by perturbing training data.\n- The main drawback of this paper is that using such a large perturbation range $\\epsilon_p=32/255$ for generating poisons may be not so reasonable. Actually, The commonly used $\\ell_{\\infty}$ perturbation budget in the literature is 0.3 for MNIST and 8/255 for CIFAR-10, SVHN, and CIFAR-100. However, it has been shown in [6] that perturbations with a large budget for MNIST may modify an input’s true label. Though [6] only conduct experiments on MNIST, the same conclusion may be true for other commonly used datasets. The few examples in Figure 7 are not enough to support the claim \"it will not affect the semantics of the raw images\".\n- In section 4.1, could you tell me the budget of training perturbations for adversarial training?\n- It may not be suitable to claim the success of the poisons when the the budget of training perturbations $\\epsilon_t=8/255$ is much small than the poison budget $\\epsilon_p=32/255$, since adversarial training needs a large $\\epsilon$ to guarantee its performance (cf. Theorem 1 in [2]).\n- As a reminder, [6] was the first to point out that adversarial example noise can be used as poisons. Later, this idea was experimented in [2] and [7].\n- As a reminder, on Page 10, the reference to MART appears twice.\n\n[1] Feng et al., Learning to Confuse: Generating Training Time Adversarial Data with Auto-Encoder, NeurIPS 2019  \n[2] Tao et al., Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training, NeurIPS 2021  \n[3] Fowl et al., Preventing Unauthorized Use of Proprietary Data: Poisoning for Secure Dataset Release, arxiv preprint, 2021  \n[4] Yuan et al., Neural Tangent Generalization Attacks, ICML 2021  \n[5] Evtimov et al., Disrupting Model Training with Adversarial Shortcuts, ICML 2021 Workshop  \n[6] Nakkiran, Adversarial Examples are Just Bugs, Too, Distill, 2019. https://distill.pub/2019/advex-bugs-discussion/response-5/  \n[7] Fowl et al., Adversarial examples make strong poisons, NeurISP 2021  ", "summary_of_the_paper": "This paper proposes to reduce the adversarial robustness of adversarial training by perturbing the training data with large perturbation budget. Various strategies for assigning fooling labels are explored, which eventually leads to a strong poisoning attack named IAT. Extensive experiments show the effectiveness of the proposed poisoning algorithm.", "summary_of_the_review": "Overall, I thought the threat model under consideration is interesting and worth exploring, but the authors may need to explain more motivations for their choice of threat budget. I would increase the rating if an updated draft addresses the mentioned issues in the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "AkNSTUHmGFo", "reviewer": "Reviewer_r3Un", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My concern for this paper is along the following lines: The threat model is really not clear -- and to the degree of incomparable.\n\nFor example, when the paper first discusses data poisoning for standard training by Huang et al. (3), that threat model is about to say\nthat we do not modify the labels, but very small perturbations on the features **suffices** to fool the model. The fact that labels are not allowed to modify is the whole point, and the actual interesting part, of Huang's approach.\n\nBut this paper then silently switches to allow labels to be changed -- see (4). This is already a significant change of threat model, and if labels are allowed to change, why would it be surprising that AT can be fooled? In fact, if labels are allowed to change, then intuitively AT can be in an even worse situation than standard training: This is because that AT will insist to learn w.r.t. a wrong label, because we want \"robustness\". This seems to really diminish the \"interestingness\" of this paper.\n\nI understand that this paper investigated a somewhat deep issue of consistent label bias. This is interesting -- and as I said this paper has some interesting content. To this end, I feel this paper at least needs a significant revision to make clear how to compare with previous work, and why the studied threat model is interesting after all.", "summary_of_the_paper": "This paper studies a data poisoning method for adversarial training. I like this paper in general: The derivations are clear and empirical results are clearly presented. However, this paper may suffer from unclear formulations of threat model, as I will discuss below in more detail.", "summary_of_the_review": "As I mentioned in the detailed review, I think this paper silently changed the threat model and allow modification of labels -- this is a significant change, and actually puts AT to a worse situation compared to ST (at least there are intuitions that would support such an argument). More revisions are needed -- in fact I think a significant revision is needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that adversarial training can be fooled to have robust test accuracy < 1% with a new type of poisoning attack ADVIN on the CIFAR-10 dataset, even though the robust training accuracy > 90%. This requires 100% poisoning rate, though the claim is that the poisoned data is 'semantically similar' to the original data. This is an interesting research direction. Questions were raised about novelty as well as whether these poisoned data could be detected. During the rebuttal phase the authors provide some evidence that with adaptive attacks detection could be evaded (as expected). The authors are encouraged to take all comments into account and update the paper as indicated in the rebuttal for any future revision.
Decision: Reject