Adversarial Examples Make Strong Poisons
Authors: Liam H Fowl
Micah Goldblum
Ping-yeh Chiang
Jonas Geiping
Wojciech Czaja
Tom Goldstein
Venue: NeurIPS 2021 Poster (2021)
Keywords: Data Poisoning
Robustness
Security
Adversarial Examples
Data Release
Availability Attack

Abstract: The adversarial machine learning literature is largely partitioned into evasion attacks on testing data and poisoning attacks on training data.  In this work, we show that adversarial examples, originally intended for attacking pre-trained models, are even more effective for data poisoning than recent methods designed specifically for poisoning. In fact, adversarial examples with labels re-assigned by the crafting network remain effective for training, suggesting that adversarial examples contain useful semantic content, just with the "wrong" labels (according to a network, but not a human). Our method, adversarial poisoning, is substantially more effective than existing poisoning methods for secure dataset release, and we release a poisoned version of ImageNet, ImageNet-P, to encourage research into the strength of this form of data obfuscation.


Adversarial Examples Make Strong Poisons
Liam Fowl" Department Mathematics University of Maryland 1fow10umd. edu
Micah Goldblum'
Department of Computer Science University of Maryland
Ping-yeh Chiang
Jonas Geiping
Department of Computer Science University of Maryland
Department of Electrical Engineering University of Siegen
Wojtek Czaja Department Mathematics University of Maryland
Tom Goldstein
Department of Computer Science University of Maryland
Abstract
The adversarial machine learning literature is largely partitioned into evasion attacks on testing data and poisoning attacks on training data. In this work, we show that adversarial examples, originally intended for attacking pre-trained models, are even more effective for data poisoning than recent methods designed specifically for poisoning. Our findings indicate that adversarial examples, when assigned the original label of their natural base image, cannot be used to train a classifier for natural images. Furthermore, when adversarial examples adversarial class label, they are useful for training. This suggests that adversarial examples contain useful semantic content, just with the "wrong" labels (according to a network, but not human). Our method, adversarial poisoning, is substantially more effective than existing poisoning methods for secure dataset release, and we release poisoned version of ImageNet, ImageNet-P, to encourage research into the strength of this form of data obfuscation.
assigned their
Introduction
Automated dataset scraping has become necessary to satisfy the exploding demands of cutting-edge deep models [Bonawitz et al., 2019, Brown al., 2020], but the same automation that enables massive performance boosts exposes these modcls to security vulnerabilities [Bagdasaryan et al., 2020, Chen et al., 2020]. Data poisoning attacks manipulate training data in order to cause the resulting models to misclassify samples during inference [Koh and Liang, 2017], while backdoor attacks embed exploits which can be triggered by pre-specified input features [Chen et al., 2017]. this work, we focus flavor of data poisoning known availability attacks, which aim degrade overall testing performance [Barreno et al., 2010, Biggio et al., 2012).
Adversarial attacks, on the other hand, focus manipulating samples at test-time, rather than during training [Szegedy et al., 3. this work, we connect adversarial and poisoning attacks by showing that adversarial examples form stronger availability attacks than any existing poisoning method, even though the latter were designed specifically for manipulating training data while adversarial examples were not. We compare method, adversarial poisoning, to existing availability attacks for neural
Authors contributed equally.
35th Conference on Noural Information Processing Systems (NeurIPS 2021).

---

networks, and we exhibit consistent performance boosts (i.e. lower test accuracy). In fact, models trained on adversarial examples may exhibit test-time performance below that of random guessing.
Intuitively, adversarial examples look dramatically different from their natural base images in the eye of neural networks, despite the two looking similar humans. Models trained only on such perturbed examples are completely unprepared for inference on the clean data. In support of this intuition, we observe that models trained on adversarially perturbed training data often fail to correctly classify the original clean training samples.
But does this phenomenon occur simply because adversarial examples are off the "natural image manifold" or because they actually contain informative features from other classes? Popular belief assumess that adversarial examples live off the natural image manifold, causing a catastrophic mis- match when digested by models trained on only clean data [Khoury and Hadfield-Menell, 2018, Stutz al., 2019, Zhao et al., 2017]. However, models trained on data with random additive noise (rather than adversarial noise) perform well on noiseless data, suggesting that there may be more to the effects of adversarial examples than simply moving off the manifold (see Table 2). We instead find that since adversarial attacks inject features that a model associates with incorrect labels, training on these examples similar to training on mislabeled training data, After re-labeling adversarial examples with the "wrong" prediction of the network on which they were crafted, models trained on such label-flipped data perform substantially better than models trained on uncorrected adversarial examples and almost as well as models trained on clean images. While this label-correction is infeasible for a practitioner defending against adversarial poisoning, since it assumes possession of the crafting network which requires access to the clean dataset, this experiment strengthens the intuition that adversarial examples contain a strong training signal just from the "wrong" class.
2 Related Work
Data poisoning can generally be phrased bilevel optimization problem which minimizes loss with respect to parameters in the inner problem while maximizing some attack loss with respect to inputs in the outer problem (Biggio et al., 2012, Huang al., 2020a]. Poisoning comes in several flavors, including integrity attacks and availability attacks. The former aims to cause targeted misclassification on a small number of pre-selected datapoints, while the latter aims to degrade the overall performance (generalization ability) of victim network Barreno et al. [2010]. Classical approaches to poisoining attacks often focused on simple models, where the inner problem can sometimes be solved exactly [Biggio ct al., 2012, Mci and Zhu, 2015, Xiao et al., 2015, Chan-Hon-Tome 2019]. However, on neural networks, obtaining exact solutions is intractable. In this setting, Auñoz-González al. [2017] approximates solution to the inner problem using small number of descent steps, but the authors note that this method ineffective against deep neural networks.
Modern poisoning attacks adopt new methods and approximations, like gradient alignment [Geiping al., 2020, Souri al., 2021], computation graph unrolling (Huang al., 2020b], etc., but for integrity attacks on deep networks. On the availability attack side, still other heuristics have been adopted. For example, gradient alignment [Geiping et al., 2020] with modified indiscriminate objective was used in Fowl et al. [2021], while gradient cxplosion was suggested in Shen et al. [2019]. Other availability attacks have used auto-encoder generated perturbations [Feng et al., 2019a], as well as loss minimization objectives [Huang et al., 2021]. Notably, it was previously believed that adversarial (loss maximization) objectives were not suitable to availability attacks [Huang et al., 2021].
Still other related poisoning works hamness influence functions which estimate the impact of each training sample on a resulting model [Fang et al., 2020, Koh and Liang, 2017, Koh et al., 2018J. However, influence functions are brittle on deep networks whose loss surfaces are highly irregular [Basu et al., 2020]. A general overview of data poisoning methods can be found in Goldblum et al. [2020].
Adversarial examples. Adversarial attacks probe the blindspots of trained models where they catastrophically miselassify inputs that have undergone small perturbations [Szegedy et al., 2013]. Prototypical algorithms for adversarial attacks simply maximize loss with respect to the input while constraining perturbations. The resulting adversarial examples exploit the fact that as inputs are even slightly perturbed in just the right direction, their corresponding deep features and logits change dramatically, and gradient-based optimizers can efficiently find these directions. The literature
2

---

contains a wide array of proposed loss functions and optimizers for improving the effectiveness of attacks [Carlini and Wagner, 2017, Gowal al., 2019). number of works suggest that adversarial examples are off the image manifold, and others propose methods for producing on-manifold attacks [Khoury and Hadfield-Menell, 2018. Stutz et al., 2019, Zhao et al., 2017].
Adversarial training, The most popular method for producing neural networks that are robust to attacks involves crafting adversarial versions of each mini-batch and training on these versions [Madry et al., 2017]. On the surface, it might sound as if adversarial training is very similar to training on poisons crafted via adversarial attacks. After all, they both involve training on adversarial examples. However, adversarial training ensures that the robust model classifies inputs correctly within a ball surrounding each training sample. This is accomplished by updating perturbations to inputs throughout training. This process desensitizes the adversarially trained model to small perturbations to its inputs. contrast, a model trained on adversarially poisoned data is only encouraged to fit the exact, fixed perturbed data.
Adversarial Examples as Poisons
3.1 Threat Model and Motivation
We introduce two parties: the poisoner (sometimes called the attacker), and the victim. The poisoner has the ability to perturb the victim's 's training data but does not know the victim's 's model initialization, training routine, architecture. The victim then trains new model from scratch the poisoned data. The poisoner's 's success is determined by the accuracy of the victim model on clean data.
Early availability attacks that worked best in simple settings, like SVMs, often modified only small portion of the training data. However, recent availability attacks that work more complex settings have instead focused on applications such secure data release where the poisoner has access to, and modifies, all data used by the victim [Shen et al., 2019, Feng et al., 2019a, Huang et al., 2021, Fowl et al., 2021].
These methods manipulate the entire training set cause poor gencralization deep learning models trained on the poisoned data. This setting is relevant to practitioners such as social media companies who wish maintain the competitive advantage afforded to them by access to large amounts of user data, while also protecting user privacy by making scraped data useless for training models. Practically speaking, companies could employ methods this domain imperceptibly modify user data before dissemination through social media sites in order to degrade performance of any model which is trained on this disseminated data.
To compare our method to recent works, we focus our experiments in this setting where the poisoner can perturb the entire training set. However, we also poison lower proportions of the data in Tables 5, 4. We find that on both simple and complex datasets, our method produces poisons which are useless for training, and models trained on data including poisons would have performed just as well had they identified and thrown out the poisoned data altogether.
3.2 Problem Setup
Formally stated, availability poisoning attacks aim to solve the following bi-level objective in terms of perturbations ( to elements x, of a dataset
B(X.))~D C [(F(x;
max dES
s.t. ((6) E arg min
Σ
C(F(x, + + 6,; ().))
(2)
(A..U)ET
where S denotes the constraint set of the perturbations. As is common in both the adversarial attack and poisoning literature, we employ an f.o bound on each 6; Unless otherwise stated, attacks are bounded by fac-norm both adversarial and poisoning literature [Madry al., 2017, Geiping al., 2020]. Simply put, the attacker wishes to cause a network, F, trained on the poisons to generalize poorly to distribution D from which T was sampled.
8/255 as is standard practice on CIFAR-10, and ImageNet data in
3

---

Directly solving this optimization problem is intractible for neural networks as it requires unrolling the entire training procedure found the inner objective (Equation (2)) and backpropagating through it to perform a single step of gradient descent on the outer objective. Thus, the attacker must approximate the bilevel objective. Approximations to this objective often involve heuristics, as previously described. For example, TensorClog [Shen et al., 2019] aims to cause gradient vanishing in order to disrupt training, while more recent work aims to align poison gradients with an adversarial objective [Geiping ct al., 2020].
We opt for an entirely different strategy and instead replace the bi-level problem with two empirical loss maximization problems an approach that was believed to be suboptimal for availability poisoning (Huang et al., J. This turns the poison generation problem into an adversarial example problem. Specifically, we optimize the following untargeted (UT) objective:
+,; Σ ((F(x,+6;;)) +,;;0,,)) (E..U))T
max BES
(3)
where 8* denotes the parameters of model trained on clean data, which fixed during poison generation. We call this model the crafting model.
We also optimize an objective which defines a class targeted (CT) adversarial attack. This modified objective is defined by:
min SES
Σ C(F(2, C(F(a,+6,00), +6,00),
(4)
(BL.UET
where g is permutation (with no fixed points) the label space of S. Fittingly, call methods adversarial poisoning. Note that the class targeted objective was previously (independently) hypothesized to produce potent poisons in Nakkiran [2019], and also tested in work concurrent ours [Tao et al., 2021].
Projected Gradient Descent (PGD) has become the standard method for generating adversarial examples for deep networks [addy et al., 2017]. Accordingly, we craft our poisons with 250 steps of PGD on this loss-maximization objective. In addition to the adversarial attack introduced in Madry et al. [2017], we also experiment with other attacks such as FGSM [Goodfellow et al., 2014] and Carlini-Wagner [Carlini and Wagner, 2017] in Appendix Table 2. We find that while other adversaries do produce effective poisons, a PGD based attack is the most effective in generating poisons. Finally, borrowing from recent targeted data poisoning works, we also employ differentiable data augmentation in the crafting stage [Geiping et al., 2020] (see section 3.3).
An aspect of note for our method is the ease of crafting perturbations we use straightforward adversarial attack on a fixed pretrained network to generate the poisons. This in contrast to previous works which require pretraining an adversarial auto-encoder [Feng et al., 2019b] (5 7 GPU days for simple datasets), or require iteratively updating the model and perturbations [Huang et al., 2021], which requires access to the entire training set all at once assumption that does not hold for practitioners like social media companics who acquire data sequentially. In addition to the performance boosts our method offers, it is also the most flexible compared to each of the availability attacks with which we compare.
3.3 Technical Details for Successful Attacks
As we will see in the following sections, adversarial objectives can indeed produce powerful poisons. As previously stated, such loss maximization approaches to availability attacks were thought be suboptimal [Huang et al., 2021]. However, we find that diffcrentiable data augmentation during crafting, along with more PGD steps, greatly improves the potency of the generated poisons. In Appendix Fig. 6, we see that augmentation is very helpful for the untargeted objective at higher number of PGD steps, but is less important for the class-targeted objective.
Furthermore, we find that the variability of the untargeted attack is generally higher than the class- targeted attack. That is, the untargeted attack is more sensitive random initialization of the poisons. For purposes of comparison, we do modest hyperparameter searching to find a reasonable poisoned dataset on which to evaluate victim model. We discuss this further in Appendix Sec. A.9.
4

---

3.4 Baseline CIFAR-10 10 Results
We first experiment relatively simple setting, CIFAR-10 [Krizhevsky al., 2009], consisting of 50, 000 low-resolution images from 10 classes. All poisons in this setting are tested on variety of architectures setting adapted from commonly used repository 2. We find that our poisoning method reliably degrades the test accuracy of wide variety of popular models including VGG19 [Simonyan and Zisserman, 2014]. ResNet-18 [He et al., 2013], GoogLeNet [Szegedy et al., 2015], DenseNet-121 [Huang al., 2016], and MobileNetV2 [Sandler et al., 2018]. Even under very tight lo constraints, our poisons more than halve the test accuracy of these models. These results are presented in Table 1.
Table 1: Comparison of different e-bounds for our adversarial poisoning method. All poisons generated by ResNet-18 crafted with 250 steps of PGD, with differentiable data augmentation. CT denotes poisons crafted with the class targeted adversarial attack. Note that we would expect random guessing to achieve 10% validation accuracy.
BOUND VICTTM
VGG19
RESNET-
GOOGLENET
DENSENET-122 121
MOBILENETV2
CLEAN
0.16
95.33
95.38 = 0.11
95.51
92.42
E 4/255 (UT)
64.71 + 0.76
0.77
01990.46
47.72 0.23
E 8/255 (UT)
10.98 + 0.27
6.15
0.12
7.16
6.11 + 0.17
e 4/255 (CT)
30.26 1 0.55
26.49
29.53 0.51
26.63 1 0.59
21.64 + 0.54
8/255 (CT)
10.32 4 0.35
+.44
-.26
7.85 + 0.40
8.36 1 0.31
Additionally, we study the effect of the optimizer, data augmentation, number of steps, and crafting network on poison generation in Appendix Tables 10, 11, and 12. These tables demonstrate that a wide variety of adversarial attacks with various crafting networks and hyperparameters yield effective poisons. Also note that while the results we present here are for poisons generated with a ResNet18, we find that the poisons remain effective when generated from other network architectures (see Appendix Table 12).
In this popular setting, we can compare our method to existing availability attacks including Tensor- Clog [Shen al., 2019], Loss Minimization [Huang al., 2021], and a gradient alignment based method [Fowl et al., 2021]. Our method widely outperforms these previous methods. Compared with the previous best method (loss minimization), we degrade the validation accuracy of a victim network by a factor of more than three.
Table 2: Validation accuracies of models trained on data from different availability attacks. Tested on randomly initialized ResNet-18 models on CIFAR-10. All crafted with 8/255.
METHOD
VALIDATION ACCURACY (%, 4)
NONE (CLEAN)
94.56
RANDOM NOISE
90.52
TENSORCLOG [SHEN ET AL., 2019]
84.24
ALIGNMENT [FOWL AL., 2021]
53.67
UNLEARNABLE EXAMPLES [HUANG ET AL., 2021]
19.85
DEEPCONFUSE [FENG ET AL., 2019B]
31.10
ADVERSARIAL POISONING UNTARGETED (OURS)
11.94
ADVERSARIAL POISONING CLASS TARGETED (OURS)
8.69
Note that test method in a completely black-box setting wherein the poisoner has knowledge of the victim network's 's initialization, architecture, learning rate scheduler, optimizer, etc. We find our adversarial poisons transfer across these settings and reliably degrade the validation accuracy of all the models tested. See Appendix A.1 for more details about training procedures.
3.5 Large Scale Poisoning
In addition to validating our method on CIFAR-10, we also conduct experiments on ImageNet (ILSVRC2012), consisting of over million images coming from 1000 different classes [Russakovsky
htwptttww....mmemmmiiii///rrrrrr cifar.

---

et al., 2015]. This setting tests whether adversarial poisons can degrade accuracy on industrial- scale, high resolution datasets. We discover that while untargeted attacks successfully reduce the generalization, our class-targeted attack (CT) degrades clean validation accuracy significantly further (see Table 3). Furthermore, even at an almost imperceptible perturbation level 4/255, the class targeted poisons cripple the validation accuracy of the victim model to 3.57%. Visualizations of the poisons can be found in Figure
Table 3: Validation accuracies of models trained on poisoned ImageNet data. Tested on randomly initialized ResNet-1K 18 models.
METHOD
VALIDATION ACCURACY ())
NONE (CLEAN)
66.56
4/255 (UT)
45.07
8/255 (UT)
36.63
4/255 (CT)
3.57
8/255 (CT)
1.45
Figure 1: Randomly selected clean image (left), with perturbed counterparts level 4/255 (center), and = = 8/255 (right). The clean image is taken from class "hen" and poisons are generated via perturbations into class "ostrich".
We hypothesize that the notcable superiority of the class largeted attack in this setting arises from the larger label-space of ImageNet. Specifically, we find that untargeted adversarial attacks can result in concentrated attack matrix ie. attacks perturb data into relatively small number of classes. This could lead to the perturbations being less effective at degrading generalization because they are not discriminatively useful for network and are thus ignored. In contrast, class-targeted attack ensures that the adversarially perturbed features can be associated with an incorrect class label when learned by victim network.
3.6 Facial Recognition
In our third experimental setting, we test whether our poisons are effective against facial recognition models, along the lines of the motivations introduced Section 3.1, where companies like social media sites could employ recognition networks as demonstrated in [Cherepanova et al., 2021]. For the purposes of comparison, constrain attack to the same setting Huang al. [2021]. In this setting, only assume the ability to modify a subset of the face images, specifically, the face images of the users that want to protect. We test on the Webface dataset, and use pretrained Webface model from Yi et al. [2014] to generate the class targeted adversarial examples with & = 8/255. Our class targeted adversarial examples are exceedingly effective in this domain. Our method reduces the protected classes' accuracy down to 8%, half the classification success of the unlearnable examples method of Huang et al. [2021] (see Table 4). We visualize perturbations in Appendix Figure 5. Note that these images are produced using the same bound as unlearnable examples for comparison, and for more discrete perturbations, one can easily turn down the radius ensure more high fidelity user images.
method to protect user data from being scraped and used by facial

---

Table 4: Comparison of poisoning methods for facial recognition. All poisons generated with
8 = 8/ 255
POTSON METHOD
AVERAGE ACCURACY (%,
CLEAN ACCURACY
86,00
UNLEARNABLE EXAMPLES [HUANG ET AL., 2021]
16.00
OURS (CT)
8.00
3.7
Less Data
So far, our comparisons to previous methods have been in the setting of full dataset poisoning, as was tested in Shen et al. [2019]. Feng et al. [2019b], Huang et al. [2021], Fowl et al. [2021]. This setting relevant to practitioners like social media companies who wish to modify all user data to prevent others from using data scraped from their platforms.
However, even in this setting, it may be the case that an actor scraping data has access to an amount of unperturbed data either through previous scraping before poisoning method was employed, or data scraped from another source. A concern that follows is how this mixture affects model training, Thus, we test the effectiveness of our poisons when different proportions of clean and perturbed data are used to train the victim model. Poisons are then considered effective if their use does not significantly increase performance over training on the clean data alone. In Tables 5, 6 we find that, poisoned data does not significantly improve results over training on only the clean data and often degrades results below what one would achieve using only the clean data. This is in comparison to similar experiments in Huang et al. [2021] where poisoned data was observed to be slightly helpful when combined with clean data in training a model.
Table 5: Effects of adjusting the amount of clean data included in the poisoned CIFAR-10.
POISON CLEAA PROPORTION NONE (ONLY CLEAN DATA) E 8/255
0.1
0.2
0.5
0.8
82.30 E 0.08
87.90 + 0.04
92.48 + 0.07
93.90 F 0.06
85.34 + 0.09
88.23 + 0.09
92.20 = 0.08
93.65 + 0.07
Table 6: Effects of adjusting the amount of clean data included in the poisoned ImageNet.
POISON METHOD CLEAN PROPORTION
0.1
0.2
0.5
0.8
NONE (ONLY CLEAN DATA)
45.
53.48
62.79
64.65
65.65
C = 8/255
47.81
54.24
61.20
63.95
3.8 Defenses
We have demonstrated the effects of poisons in settings where the victim trains "normally". However, there have been several defenses proposed against poisoning attacks that a victim could potentially utilize. Thus, in this section, we test the effectiveness of several popular defenses against our method.
Adversarial training: Because poisons constrained in an foo ball around the clean inputs, and because we find that clean inputs are themselves adversarial examples for networks trained on poisoned data (more this in the next section), it is possible that adversarial training could "correct" the poisoned network's s behavior on the clean distribution. This is hypothesized Tao et al. [2021], where it is argued that adversarial training can be effective defense against delusive poisoning. We find in Table that this is indeed the case. In fact, it is known that adversarial training effectively mitigates the success of several previous availability attacks including Huang et al. [2021], Feng et al. [2019b], Fowl et al. [2021]. However, is worth noting that this might not be ideal solution for the victim as adversarial training expensive computationally, and it degrades natural accuracy to a level well below that of standard training. For example, on a large scale dataset like ImageNet, adversarial training can result in significant drop in validation

---

accuracy. With a ResNet-50, adversarial training results in validation accuracy dropping from 76.13% to 47.91% a drop that would be further exacerbated when adversarial training on poisoned data (cf. https://githen cow
Data Augmentation: Because our poisoning method relies upon slight perturbations to training data in order to degrade victim performance, it is conceivable that further modification of data during training could counteract the effects of the perturbations. This has recently been explored in Borgnia et al. [2021] and Geiping et al. [2021]. We test several data augmentations nor known to the poisoner during crafting, although adaptive attacks have been shown to be effective against some augmentations [Geiping et al., 2021]. Our augmentations range from straightforward random additive noise (of the same e-bound as poisons) to Gaussian smoothing. We also include popular training augmentations such as Mixup, which mixes inputs and input labels during training [Zhang et al., 2017], Cutmix [Yun et al., 2019], which mixes patches of one image with another, and Cutout [Cubuk et al., 2019] which excises certain parts of training images.
DPSGD: Differentially Private SGD (DPSGD) [Abadi al., 2016] was originally developed as a differential privacy tool for deep networks which aims to inure a dataset to small changes in training data, which could make it an effective antitode data poisoning. This defense adds noise to training gradients, and clips them. It was demonstrated to be successful in defending against targeted poisoning some settings (Hong et al., 2020, Geiping et al., 2020]. However, this defense often results in heavily degraded accuracy. We test this defense with a clipping parameter 1.0 and noise parameter 0.005.
Table 7: Effects of defenses against adversarial poisons. All results averaged over 5 runs of a ResNet-18 8 victim model trained on class targeted, & 8/255 poisons. Although adversarial training improves results, none of these effectively recover the accuracy of model trained on clean data.
DEFENSE
VALIDATION ACCURACY (%)
BASELINE (CLEAN)
94.56
ADV. TRAINING
83.01
GAUSSIAN SMOOTHING
11.94
RANDOM NOISE
6.35
MIXUP
15.86
CUTMIX
10.09
CUTOUT
DPSGD
24.61
Other than adversarial training, which has drawbacks discussed above, we find that previous proposed defenses are ineffective against our adversarial poisons, and can even degrade victim performance below the level of a model trained without any defense.
Analysis
Why do adversarial examples make such potent poisons? Section 1, we motivate the effectiveness of adversarial poisons with the explanation that the perturbed data contains semantically useful information, but for the wrong class. For instance, an adversarially perturbed "dog" might be labeled as "cat" by the crafting network because the perturbations contain discriminatory features useful for the "cat" class. In Ilyas et al. [2019], the authors discover that there exist image features which are both brittle under adversarial perturbations and useful for classification. llyas et al. [2019] explores these implications for creating robust models, as well as "mislabeled" datasets which produce surprisingly good natural accuracy. We investigate the implications that the existence of non-robust features have on data- data--oisoning attacks. We hypothesize that adversarial examples might poison networks so effectively because, for example, they teach the network to associate "cat" features found in perturbed data with the label "dog" of the original, unperturbed sample. Then, when the network tries to classify clean test-time data, it leverages the "mislabeled" features found in the perturbed data and displays low test-time accuracy. We confirm this behavior by evaluating how data is misclassified at test-time. We find that the distribution of predictions clean data closely mimics the distributions of labels assigned by the network used for crafting after adversarial attacks, even though these patterns are generated from different networks.

---

To tease apart these effects, we conduct several experiments. First, we verify that the victim network does indeed train i.e. reach a region of low loss on the adversarial examples. This is in contrast to the motivation of [Shen et al., 2019, Huang et al., 2021] which try prevent the network from training on poisoned data. We find that the victim network is able to almost perfectly fit the adversarial data. However, the accuracy of the victim network on the original, unperturbed training data is just as low as aceuracy on the clean validation data, revealing an interesting duality clean training data adversarial examples for networks trained on their perturbed counterparts (sec Table 8). But are adversarial examples simply so different from clean examples that learning on onc is useless for performing inference the other? Or do adversarial examples contain useful features but for the wrong class?
To investigate these hypotheses, we train models on the adversarial poisons crafted with the class targeted objective found in Equation 4 so that every image from given class is perturbed into the same target class. We then observe the classification patterns of the victim network and compare these to the attack patterns of the crafting network. We find that poisoned networks confuse exactly the same classes the crafting network. Specifically, network trained into the cat class will then misclassify clean cat images as dogs at test time as the network learns to associate "clean" cat features found in the perturbations to the training dog images with the label dog. This behavior is illustrated in Figure 2.
dog images perturbed
To further confirm our hypothesis, we employ the re-labeling trick introduced in Ilyas et al. [2019], and we train the victim network on the "incorrect" labels assigned to the poisons by the crafting network. For example, if dog image is perturbed into the "cat" class by adversarial attack crafting network, we train on the perturbed dog image but assign the "cat" label. We find that this simple re-labeling trick boosts validation accuracy significantly, for example from 6. 25% to 75.69% on a victim ResNet-18 model (see Appendix Table 14). This confirms the finding of Ilyas et al. [2019] concerning non-robust features. One might think that this observation is useful for defending against adversarial poisons. However, in order to perform this label correction, the victim must have access to the crafting model which requires the victim to already possess the original non-poisoned dataset.
the
plane
plano
077
06
tue lube
duer
0.6
Groporcoon
deer
05
04
frog
forse
horse
02
0.2
stp
shp
01
buck
plane
bird
cst
deer
dog to
horse
ship
truck
plane
bind
deer
dog
frog
home
ship truck
target
label
true label
(a) Adversarial attack predictions of network used to craft adversarial poisons.
(b) Test time predictions of network trained on adver sarial poisons.
Figure 2: Classification heatmaps. Left - heatmap of predictions after the adversarial attack on the network used for crafting. Right- heatmap of clean test predictions after training new network on adversarial poisons.
One further intricacy remains; even if adversarial examples contained useful features, they may still encode decision boundary information that accounts for the increased validation accuracy- behaviour that has previously been demonstrate with -utt---diitriuttion data in Nayak et al. [2019]. However, we find that training on data from a drastically different distribution (SVHN), labeled with the CIFAR- 10 crafting network's predictions, fails to achieve comparable CIFAR-10 validation accuracy. This confirms that the adversarial CIFAR-10 images contain useful features for the CIFAR-10 distribution but are simply mislabeled.

---

Table 8: Testing the victim clean adversarial training images. Poisons crafted CIFAR-10 10 trained ResNet-18 with 250 steps of PGD and differentiable data augmentation.
MEASUREMENT VICTIM
VGG19
RESNET-18
GoogLENET
DENSENET- 121
MOBILENETV2
TRAINING ACC. ON POISONS
99.95 = 0.00
99 0.00
99.99 0.00
99.99 0.00
99.94
ACC.
10.95 0.32
6.16 16
6.80 0.08
7:06 0.13
0.15
5 Limitations
While our adversarial poisoning method does outperform current art on availability poisoning attacks, there are limitations to method that warrant future investigation. We identify two primary areas of future research. First is the requirement
clean trained model to generate adversarial examples, iee. the 8 parameter vector which we craft poisons in Equation assumption for practitioners like social media companies, but it does make the attack less general purpose. However, this is also constraint with previous poisoning methods which require access all the data being perturbed [Huang et al., 2011] or clean data to train an autoencoder [Feng et al., 2019b], large amount of clean data to estimate an adversarial target gradient [Fowl al., 2021]. It has been shown that adversarial examples often transfer across initialization, or even dataset, so it inconceivable that this limitation of our method could be overcome. The second limitation the success of adversarial training as defense against delusive poisoning. This appears to be the trend across all availability attacks against deep networks date. This defense not optimal though as it leads to a severe drop in natural accuracy, and can be computationally expensive.
This may be reasonable
6 Conclusions
There is a rising interest availability attacks against deep networks due to their potential use in protecting publicly released user data. Several techniques have been introduced leveraging heuristics such as loss minimization, and auto-encoder based noise. However, we find that adversarial attacks against fixed network accuracy below random guess levels. We study the effects of these poisons in multiple settings and analyze why these perturbations make such effective poisons. Our observations confirm fundamental property of adversarial examples; they contain discriminatory features but simply for the wrong class. Our class-targeted atlack leverages this property effectively poison models on a variety of datasets.
potent availability poisons than existing methods, often degrading
7
Acknowledgements
This work was supported by DARPA GARD, the Office and Naval Research, and the National Science Foundation Division Mathematical Sciences. Addition support was provided by the Sloan Foundation, JP Morgan Chase, and Capital One Bank.
10

---

References
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS 16, pages 308-318, Vienna, Austria, October 2016. Association for Computing Machinery. ISBN 978- 978-1-4503-413999
doi: 10.1145/2976749
Bugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938 2948. PMLR, 2020.
Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. The security of machine learning. Mach. Learn., (1): 102)::211111 148, November 2010. ISSN 0885-6125, doi: 10. 1097009000900000 010-5188----
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning fragile. arXiv preprint 2006v 2020.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv:2206.6388 2012.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi. H Brendan McMahan, et al. Towards federated learning scale: System design. arXiv preprint arXiv: 1902. 01046, 2019.
Eitan Borgnia, Jonas Geiping, Valeriaa Cherepanova, Liam Fowl, Arjun Gupta, Amin Ghiasi, Furong Huang, Micah Goldblum, and Tom Goldstein. Dp-instahide: Provably defusing poisoning and backdoor attacks with differentially private data augmentations. arXiv preprint arXiv:2103. 02079, 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models few-shot learners. arXiv preprint ar 200: 14165, 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee
symposium on security and privacy (sp). pages 39 57. IEEE, 2017.
Adrien Chan-Hon Tong. An algorithm for generating invisible data poisoning using adversarial noise that breaks image classification deep learning. Machine Learning and Knowledge Extraction, 1(1):192-204, 2019.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. Badnl: Backdoor attacks against nlp models. arXiv preprint arXiv:2006. 2000022000000043,,, 2020.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 1712.0556,, 2017.
Valeriia Cherepanova, Micah Goldblum, Harrison Foley, Shiyuan Duan, John Dickerson, Gavin Taylor, and Tom Goldstein. Lowkey: leveraging adversarial attacks protect social media users from facial recognition. arXiv preprint arXiv:2101.0790 2021.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Lc. AutoAugment: Learning Augmentation Policies from Data. ArXiv1805095H Cs Stut, April 2019.
Minghong Fang, Neil Zhenqiang Gong, and Jia Liu. Influence function based data poisoning attacks top-n recommender systems. Proceedings of The Web Conference 2020, pages 3019-3025, 2020.
Feng, Qi-Zhi Cai, and Zhi-Hua Zhou. Learning to Confuse: Generating Training Time Adversarial Data with Auto-Encoder. ArXiv/90509027 Cs Stat, May 2019a
Ji Feng, Qi-Zhi Cai, and Zhi-Hua Zhou. Learning to Confuse: Generating Training Time Adversarial Data with Auto Encoder. In Advances Neural Information Processing Systems 32, pages 11994 12004. Curran Associates, Inc., 2019b.
Liam Fowl, Ping-ych Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use of proprictary data: Poisoning for secure dataset release. arXiv preprint 1rXiv:2103.02682 2021.
Jonas Geiping, Liam Fowl, W Ronny Huang. Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches brew: Industrial scale data poisoning via gradient matching. arXiv preprint arXiv:2009. 02276, 2020.
11

---

Jonas Geiping, Liam Fowl, Gowthami Somepalli, Micah Goldblum, Michael Moeller, and Tom Goldstein. What doesn't 't kill you makes you robust (er): Adversarial training against poisons and backdoors. arXiv preprint arXiv:2102. 13244, 2021.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song. Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning. backdoor attacks, and defenses. arXiv preprint 2xX.:20122.0544. 2020.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples. arXiv preprint arXiv: /412:::41166577 2014.
Sven Gowal, Jonathan Uesato, Chongli Qin, Po-Sen Huang, Timothy Mann, and Pushmeet Kohli. An alternative surrogate loss for pgd-based adversarial testing. arXiv preprint arXiv: 1910.09338, 2019.
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep Residual Learning for Image Recognition. ArXiv151203385 Cs, December 2015.
Sanghyun Hong, Varun Chandrasekaran, Yigitean Kaya, Tudor Dumitrag, and Nicolas Papernot. On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping. ArXiv200211497 Cs, February 2020.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely Connected Convolutional Networks. August 2016.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. arXiv preprint arXiv:2101 210.09980 2021.
Ling Huang, Anthony D. Joseph, Blaine Nelson, Benjamin L.P. Rubinstein, and J. D. Tygar. Adversarial Machine Learning. In Proceedings of 4th ACM Workshop on Security and Artificial Intelligence, AlSec '11, pages 43-58, New York, NY, USA, 2011. ACM. ISBN 978-1 177-1-44503-003--1 doi: 10.1 (445//006684220466699
W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general-purpose clean-label data poisoning, arXiv preprint arXiv:2004.000 00025, 2020a.
W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstcin. MctaPoison: Practical General-purpose Clean-label Data Poisoning. ArXiv200400225 Cs Star, April 2020b.
Andrew Tlyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they features. arXiv preprint arXtv:1905.02175, 2019.
Mare Khoury and Dylan Hadfield-Mened On the geometry of adversarial examples. arXiv:1811.00525, 2018
arXiv preprint
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. International Conference on Machine Learning, pages 1885-1894, PMLR, 2017.
Pang Wei Koh, Jacob Stcinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. arXiv preprint 1xXi::8811.0044/ 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 06083, 2017.
Shike Mei and Xiaojin Zhu. Using machine teaching to identify optimal training set attacks on machine learnrs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 29, 2015.
Luis Muñoz González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. Proceedings of the 10th ACM Workshop on Artificial Intelligncee and Security, pages 27-38, 2017.
Preetum Nakkiran. A discussion of "adversarial examples are not bugs, they are features': Adversarial exam- ples are just bugs, too. Distill, 2019. doi: 10.2391 S/distill.0001155. discussion/rsspose-...
http://ddiaiil.puu//00 wwtp:ppp//////x/bbbbbbaaa /auvex-
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj. Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In International Conference Machine Learning, pages 4743 4751. PMLR, 2019.
12

---

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Adilya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Comput Vis, 115(3):11-252, December 2015. ISSN 1573-140S, doi: 10.1007/sl 12-3-01--011.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. ArXiv/8010438/ Cs, January 2018.
Shen, Zhu, and Ma. TensorClog: An Imperceptible Poisoning Attack on Deep Neural Network
Applications. IEEE Access, 7:41498-41506, 2019. ISSN 2169-3536. doi: 10. 10.1109/ACCESS 2019.2905915.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. ArXiv14091556 Cs, September 2014.
Hossein Souri, Micah Goldblum, Liam Fowl, Rama Chellappa, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. arXiv preprint urXiv:2/06.0080 2021,
David Stutz, Matthias Hein, and Bert Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6976 6987, 2019.
C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Ra- binovich. Going deeper with convolutions. In 20/5 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, June 2015. doi: 10. (O0.///VR..01..7298555..
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:13/2.6199, 2013.
Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap human level performance face verification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 1701-1708, 2014. doi: 10. 10.1109/ 2014.220
Lue Tao, Lei Feng, Jinfeng Sueng-Jun Huang, and Songcan Chen. Provable defense against delusive poisoning. arXiv preprint (xX.x:2102.047/1 2021.
Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is feature selection secure against training data poisoning? In International Conference on Machine Learning, pages 1689-1698. PMLR, 2015.
Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv: 1411.4417222 1.223, 2014.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6023-6032, 2019.
Hongyi Zhang, Moustapha Cissc, Yann N. Dauphin, and David Lopez Paz. Mixup: Beyond Empirical Risk Minimization. ArXiv/71009412 Cs Stat, Ootober2017.
Zhengli Zhao, Dheeru Dua. and Sameer Singh. Generating natural adversarial examples. arXiv preprint arxiv:1710.I 2017.
13


Reviews:
[{"review_id": "vh5G9W-xp1s", "reviewer": "Reviewer_gJ9i", "summary": "This paper investigates adversarial examples as a data poisoning method and offers insights as to why they are effective. The idea of using adversarial examples for poisoning is motivated by the intractability of standard data poisoning (cast as a bi-level optimization problem) for neural networks. The paper focuses on _availability attacks_ where the goal of poisoning is to maximize the expected loss of the model. A suite of experiments demonstrates the effectiveness of adversarial examples for poisoning. Experiments include: comparisons with baselines on CIFAR-10 and a facial recognition application; testing attack transfer to different model architectures; testing effectiveness on ImageNet (as a large-scale setting); and testing effectiveness for different poisoning ratios. Seven defenses are also tested, and the only effective defense is found to be adversarial training. Experiments are also run to test the authors’ hypothesis that adversarial examples are effective because they contain discriminative features for the “wrong” class.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is based on a very simple idea: seeing whether a well-known test-time attack (adversarial examples) is effective as a training-time attack. There is essentially no work required to adapt the attack to a training-time setting, however the authors do propose a novel _class targeted_  variant for which the class labels are permuted. Despite the simplicity of the idea – I see it as an obvious attack to try – it does not appear to have been investigated in the literature. The only prior work I could find is the following (not cited):\n\n> CHAN-HON-TONG A. An Algorithm for Generating Invisible Data Poisoning Using Adversarial Noise That Breaks Image Classification Deep Learning. Machine Learning and Knowledge Extraction. 2019; 1(1):192-204. https://doi.org/10.3390/make1010011\n\nAlthough the paper lacks new technical ideas, I think the empirical study is well done and offers important insights that ought to be shared with the community. I was surprised by the effectiveness of the approach, especially given its relative simplicity compared to existing approaches for poisoning neural networks. There are very few stones left unturned in the experiments – the attack is tested on multiple datasets (including at larger scales), transferability is assessed for different network architectures, defenses are tested, and there are further experiments testing the effect of various parameters in the appendices. It would have been interesting to see how the compute time varies for each type of attack, however this would be difficult to measure fairly on heterogeneous hardware.\n\nFor an empirical paper such as this one, I think it’s important that clearly documented scripts and code be made available to support reproducibility. I have not been able to carefully examine the repository included in the supplementary material, however the README in the root directory seems light on detail. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, this is covered adequately in Sections 3.1 and 5 of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "oFRcpNkMYO", "reviewer": "Reviewer_7LBn", "summary": "In this paper, the authors show that test-time adversarial examples, which are originally crafted to fool a well-trained model at inference time, can be used to construct strong availability attacks at training time. The resulting attacks achieve state-of-the-art results on multiple datasets and can successfully transfer across many settings, such as model initialization, architecture, learning rate scheduler, optimizer, etc.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### **Originality**\n\nMy main concern about the paper is its originality. The idea, adversarial examples can make availability attacks, is not new to the community. It was first proposed by Nakkiran [1]. He simply predicted the success of the idea through permutation-symmetry of the labels, and showed its effectiveness with numerical results. For attribution in academic contexts, please cite his work [1] properly. \n\nFurthermore, the exact idea was also experimented in previous works, such as the error-maximizing noise attack in Huang et al. [2] and the P1 attack in Tao et al. [3], just with slightly different optimization hyperparameters. These previous efforts about the idea should not be ignored in the main text.\n\n#### **Significance**\n\nSince the idea is not new, the main insight I can tell from the manuscript is that adversarial examples can make \"strong\" availability attacks. Contradicting to the results reported by Huang et al. [2], where they concluded that the error-maximizing attack is inferior to their proposed error-minimizing attack, this work shows that the same error-maximizing noise attack can actually surpass the error-minimizing attack (6.25% vs 19.85% in Table 2).\n\nI note that one possible crux is the number of PGD steps used for crafting poisons: [2] adopted only 20 steps for the error-maximizing attack, while this work executes 250 steps. From Table 10 in Appendix, it turns out that the number of PGD steps is crucial for achieving high performance. Therefore, I suggest the authors to clarify this earlier in the main text. This is an important point that differentiates this work from previous works.\n\n#### **Quality**\n\nMain claims are experimentally supported. However, neither a new method nor theoretical analysis is proposed.\n\n#### **Clarity**\n\nThe writing of this paper is mostly clear and straightforward. A minor concern is that Section 3 is too long. It would be better to separate Section 3 into two sections, one describing the problems and methods, the other showing the experiments.\n\nReference:\n\n[1] Nakkiran, https://distill.pub/2019/advex-bugs-discussion/response-5/\n\n[2] Huang et al., https://arxiv.org/abs/2101.04898\n\n[3] Tao et al., https://arxiv.org/abs/2102.04716\n\n\n-----\nAfter rebuttal:\n\nI am glad to see that my suggestions helped to identify the contributing factors for attack effectiveness. It turns out that three strong factors emerge during the rebuttal:\n- the variance of the attack effectiveness itself\n- the use of differentiable data augmentation\n- the number of PGD steps\n\nI would like to suggest the authors to reflect these key factors (and other minor factors) more clearly in the main body. Surely, achieving SOTA performance itself is a good contribution, but for this venue, it alone may not be enough. In my opinion, another contribution of this work would be to sufficiently clarify why prior implementations of adversarial examples cannot achieve SOTA. Thus, when combining the two contributions, I would prefer this paper.\n\nOverall, I am leaning to accept this paper on the condition that the authors can further clarify the confusion about the variance, the use of differentiable data augmentation, and the motivation for the choice of the number of PGD steps.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although many limitations of the work are studied through experiments, I still have some suggestions, as listed below.\n\n- It would be better if the authors could plot the data in Table 10 using curves, to show the trend of the decreasing accuracy as the number of PGD steps increases, with more fine-grained hyperparameters (e.g., plotting every 20 steps, up to 1000 steps). I'm curious about the limits of this poison.\n- [2] and [3] showed that class-wise noises perform better than sample-wise noises, for both error-maximizing attacks and error-minimizing attacks. It would be better if this work can also consider the class-wise noises.\n- Section 4 of the manuscript (and Figure 2) only explains why the targeted adversarial examples (i.e., Equation 4) work for poisoning. However, [1] provided a more concise explanation through permutation-symmetry, which directly implies the use of targeted adversarial examples as poisoning attacks.\n- It remains unclear why the untargeted adversarial examples (i.e., Equation 3) are also effective (sometimes more effective than targeted adversarial examples as shown in Table 1). Since the originality of this work is limited, it would be helpful to provide a novel and insightful analysis like this.\n- I note that, in Table 2, the results of UNLEARNABLE EXAMPLES and DEEPCONFUSE are different from those reported in their original papers. Please clarify this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "j97t98JIYB", "reviewer": "Reviewer_RuBF", "summary": "This paper proposes a method for poisoning the train dataset of a victim classification model. Even if the poisoner does not know any information about the target victim model training procedure, it can still seriously degrade the classification accuracy of the victim model on clean data. This is done by replacing total or partial training data as corresponding adversarial examples (AEs) with ground truth labels. It is verified that AEs with ground truth labels indeed degrade the classification performance of models.    ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper contributes some new ideas.\nClarity: The organization of this paper needs to be improved. Some experimental details are presented in the technical part, making it difficult to capture the core idea of the technical contributions.\nSignificance: The problem formulation of this paper is almost the same as that of the work [1]. Although this paper utilizes a surrogate model to approximately solve the problem, the work [1] attacks DNNs by back-gradient without any surrogate model, which is more challenging. Therefore, the novelty or improvement of the proposed work is rather limited.\nWeaknesses and my questions about this paper:\n1.\tThe experiment of this paper is not sufficient. Firstly, there is no comparison with other data poison methods, especially with [1], which is very similar to the proposed one. \n2.\tThis work utilizes existing attack methods on a surrogate model. It is similar to use the transferability of adversarial examples directly. The author needs to further claim the novelty and contribution of the proposed method.\n3.\tThe proposed method might be invalid when adversarial detections are involved. More precisely, the defender can utilize existing detection methods, such as like LID[2], MD[3], and KB[4], to remove those poisoned examples. Thus, there should be some tests on evaluating the robustness of the proposed method against adversarial detections.\n4.\tThe author has pointed out that their method performs unsatisfactorily to the defense of adversarial training techniques. In fact, such an limitation is fatal as  the adversarial training is not so expensive as the authors claimed. Some adversarial training method like FastAdv [5] improves the training speed significantly. \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "endx0dqj4Yh", "reviewer": "Reviewer_WUSL", "summary": "This paper proposes a data poisoning approach based on the adversarial attack technique. Specifically, to generate the perturbed samples, the poisoning model aims at maximizing the traditional cross-entropy loss or minimizing the loss with the wrong labels. Adversarial example generation approach PGD is adopted to generate the poisons. Experiments on several datasets show that training on the perturbed samples would significantly reduce the performance of various deep learning models.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this work is to leverage the idea of adversarial example generation for data poisoning. The perturbation generation is directly based on the existing approaches. The authors also conduct substantial experiments to show the effectiveness of the proposed approach.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned in the paper, the proposed approach assumes the knowledge of a pre-trained model to generate the perturbation, which makes the approach not very practical in the real world. Many attack strategies, such as backdoor attacks, usually assume to inject limited instances to training samples to damage the model. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a technique	to poison datasets with	adversarial examples.
This an	interesting and	new direction, and the results are (often order of magnitude)
more effective than prior work that relied on much more	sophisticated techniques.

While the reviewers are	concerned about	the tone of the	paper over-claiming in
its originality (and I would encourage the authors to compare to the prior work
the reviewers have pointed out), the results are sufficiently strong to	merit
publication.

Decision: Accept (Poster)