Just How Toxic is Data Poisoning?  A Benchmark for Backdoor and Data Poisoning Attacks
Authors: Avi Schwarzschild
Micah Goldblum
Arjun Gupta
John P Dickerson
Tom Goldstein
Venue:  (2021)
Keywords: Poisoning
backdoor
attack
benchmark

Abstract: Data poisoning and backdoor attacks manipulate training data in order to cause models to fail during inference.  A recent survey of industry practitioners found that data poisoning is the number one concern among threats ranging from model stealing to adversarial attacks. However, we find that the impressive performance evaluations from data poisoning attacks are, in large part, artifacts of inconsistent experimental design. Moreover, we find that existing poisoning methods have been tested in contrived scenarios, and many fail in more realistic settings. In order to promote fair comparison in future work, we develop standardized benchmarks for data poisoning and backdoor attacks.

Under review as a conference paper at ICLR 2021
JUST How TOXIC IS DATA POISONING? A BENCH- MARK FOR BACKDOOR AND DATA POISONING AT- TACKS
Anonymous authors Paper under double-blind review
ABSTRACT
Data poisoning and backdoor attacks manipulate training data in order to cause models to fail during inference. A recent survey of industry practitioners found that data poisoning is the number one concern among threats ranging from model stealing to adversarial attacks. However, we find that the impressive performance evaluations from data poisoning attacks are, in large part, artifacts of inconsistent experimental design. Moreover, we find that existing poisoning methods have been tested in contrived scenarios, and many fail in more realistic settings. In order to promote fair comparison in future work, we develop standardized benchmarks for data poisoning and backdoor attacks.
INTRODUCTION
Data poisoning is security threat to machine learning systems in which attacker controls the behavior of a system by manipulating its training data. This class of threats is particularly germane deep learning systems because they require large amounts of data train and are therefore often trained (or pre-trained) on large datasets scraped from the web. For example, the Open Images and the Amazon Products datasets contain approximately 9 million and 233 million samples, respectively, that are scraped from wide range of potentially insecure, and in many cases unknown, sources (Kuznetsova et al., 2020; Ni. 2018). At this scale, often infeasible to properly content. Furthermore, many practitioners create datasets by harvesting system inputs (e.g., emails received, files uploaded) or scraping user-created content (e.g., profiles, text messages, advertisements) without any mechanisms to bar malicious actors from contributing data. The dependence of industrial AI systems on datasets that are not manually inspected has led to fear that corrupted training data could produce faulty models (Jiang al. 2017). In fact, recent survey of 28 industry organizations found that these companies are significantly more afraid of data poisoning than other threats from adversarial machine learning (Kumar al. 2020).
A spectrum of poisoning attacks exists in the literature. Backdoor data poisoning causes a model misclassify test-time samples that contain frigger visual feature in images
particular character sequence in the natural language setting (Chen et al. 2017 Dai et al. 2019: Saha et al. 2019 Turner 2018). For example, one might tamper with training images so that a vision system Tails to Identify any person wearing a shirt with the trigger symbol printed on it. In this threat model, the attacker modifies data both train time (by placing poisons) and at inference time (by inserting the trigger). Triggerless poisoning attacks, on the other hand, do not require modification at inference time (Biggio al., 20122 Huang et al. 2020 Muñozz González et al. 2017 Shafahi et al., 2018 Zhu et al. 2019: Aghakhani et al.) 2020b Geiping et al.) (220)). A variety of innovative backdoor and triggerless poisoning attacks and defenses have emerged in recent years, but inconsistent and perfunctory experimentation has rendered performance evaluations and comparisons misleading.
In this paper, we develop a framework for benchmarking and evaluating wide range of poison attacks on image classifiers. Specifically, we provide way to compare attack strategies and shed light on the differences between them.
Our goal is to address the following weaknesses in the current literature. First, observe that the reported success of poisoning attacks in the literature is often dependent on specific (and sometimes unrealistic) choices of network architecture and training protocol, making it difficult assess the

---

Under review as a conference paper at ICLR 2021
viability of attacks in real-world scenarios. Second, we find that the percentage of training data that an attacker can modify, the standard budget measure in the poisoning literature, is not a useful metric for comparisons. The flaw in this metric invalidates comparisons because even with fixed percentage of the dataset poisoned, the success rate of an attack can still be strongly dependent on the dataset size, which is not standardized across experiments date. Third, find that some attacks that claim to be "clean label," such that poisoned data still appears natural and properly labeled upon human inspection, are not.
Our proposed benchmarks measure the effectiveness of attacks in standardized scenarios using modern network architectures. We benchmark from- training scenarios and also white-box and black-box transfer learning settings. Also, we constrain poisoned images to be clean in the sense of small perturbations. Furthermore, our benchmarks are publicly available as proving ground for existing and future data poisoning attacks.
The data poisoning literature contains attacks in a variety of settings including image classification, facial recognition, and text classification (Shafahi al. [2018] Chen et al. 2017 Dai et al. 2019). Attacks on the fairness of models, on on speech recognition, and recommendation engines have also been developed (Solans et al. 2020 Aghakhani et al. 2020a; Li et al. 2016 Fang et al., 2018 Hu iet al., 2019 Fang et al.) (020). While we acknowledge the merits of studying poisoning in a range of modalities, our benchmark focuses on image classification since it is by far the most common setting in the existing literature.
A SYNOPSIS OF TRIGGERLESS AND BACKDOOR DATA POISONING
Early poisoning attacks targeted support vector machines and simple neural networks (Biggio et al., 2012; Koh& & Liang 2017). As poisoning gained popularity, various strategies for triggerless attacks on deep architectures emerged Munoz-GGonzález et al. 2017:; Shafahi al. 2018 Zhu al. 2019
Huang et al. 2020 Aghakhani et al.] 2020b] Geiping et al. (020). The early backdoor attacks contained triggers in the poisoned data and in some cases changed the label, thus were not clean-label
(Chen et al. 2017: Gu et al. 2017 Liu et al. 2017). However, methods that produce poison examples
which don't visibly contain a trigger also show positive results (Chen et al. 2017; Turner al.
2018; (ahaa et al. 2019). Poisoning attacks have also precipitated several defense strategies, but
sanitization- based defenses may be overwhelmed by some attacks (Koh et al. 2018 Liu et al.. 2018
Chacon et al. 2019 Peri et al. 2019).
We focus on attacks that achieve targeted misclassification. That is, under both the triggerless and backdoor threat models, the end goal of an attacker is to cause a target sample to be misclassified as another specified class. Other objectives, such as decreasing overall test accuracy, have been studied, but less work exists on this topic with respect to neural networks (Xiao et al. 2015 Liu et al. 2019). In both triggerless and backdoor data poisoning, the clean images, called base images, that are modified by an attacker come from single class, the base class. This class is often chosen to be precisely the same class into which the attacker wants the target image or class to be misclassified.
There are two major differences between triggerless and backdoor threat models in the literature. First and foremost, backdoor attacks alter their targets during inference by adding a trigger. In the works we consider, triggers take the form of small patches added an image (Turner et al., 2018 8 Saha et al. 2019). Second, these works on backdoor attacks cause a victim to misclassify any image containing the trigger rather than a particular sample. Triggerless attacks instead cause the victim to misclassify an individual image called the target image (Shafahi et al., 2018 Zhu et al. 2019 Aghakhani et al., 2020b] Geiping et al. 2020). This second distinction between the two threat models is not essential; for example, triggerless attacks could be designed to cause the victim to misclassify collection of images rather than a single target. To be consistent with the literature at large, we focus on triggerless attacks that target individual samples and backdoor attacks that target whole classes of images.
We focus on the clean-label backdoor attack and the hidden trigger backdoor attack, where poisons are crafted with optimization procedures and do not contain noticeable patches (Saha et al. 2019 Turner et al. 2018). For triggerless attacks, we focus the feature collision and convex polytope methods, the most highly cited attacks of the last two years that have appeared at prominent ML conferences (Shafahi et al. 2018 Zhu et al. 2019). We include the recent triggerless methods
2

---

Under review as a conference paper at ICLR 2021
Bullseye Polytope (BP) and Witches' Brew (WiB) in the section where we present metrics benchmark problems (Aghakhami et al. 2020b; Geiping et al. 2020). The following section details the attacks that serve the subjects of experiments.
Technical details Before formally describing various poisoning methods, we begin with notation. Let X be the set of all clean training data, and let Xp {-p)) )/=1 denote the set of J poison examples with corresponding clean base image [-/] )/= Let xy be the target image. Labels are denoted by y and Y for single image and a set of images, respectively, and are indexed to match the data. We use f to denote feature extractor network.
Feature Collision (FC) Poisons in this attack are crafted by adding small perturbations to base images so that their feature representations lie extremely close to that of the target (Shafahi et al. Formally, each poison is the solution to the following optimization problem.
2018
z argmin [lf( fxx))1 + B|| 1
When we enforce l -nom constraints, we drop the last term in Equation (T) and instead enforce || 1
I E, Vi by projecting onto the &o ball after each iteration.
Convex Polytope (CP) This attack crafts poisons such that the target's feature representation is convex combination of the poisons' feature representations by solving the following optimization problem (Zhu et al. 2019).
[1((m)) E/ E/ (,
X= argmin (15))
[5(x))
(2)
21=1 = and oj 2 O. j. and ||,(] - (1) 11:0 S S
subject to
Clean Label Backdoor (CLBD)
This backdoor attack begins by computing an adversarial pertur- bation each base image Turner et 2018). Formally,
#6 1bb
(1) argmax [(x))) (1) + 6, 6,y())))
1
where L denotes cross-entropy loss. Then, a patch added to each image in (a to generate the ( The patched image is subject an -nom constraint.
final poisons {xp²)
Hidden Trigger Backdoor (HTBD) A backdoor analogue the FC attack, where poisons are crafted to remain close the base images but collide in feature space with patched image from the target class (Saha al. 2019p. Let _x(5) denote a patched training image from the target class (this image is not clean), then we solve the following optimization problem to find poison images.
argmin (l](z) - (1 s.t. || 2 11) oo Se
x
WHY DO WE NEED BENCHMARKS?
Backdoor and triggerless attacks have been tested in wide range of disparate settings. From model architecture to target/base class pairs, the literature is inconsistent. Experiments are also lacking in the breadth of trials performed, sometimes using only one model initialization for all experiments, or testing against one single target image. We find that inconsistencies in experimental settings have a large impact on performance evaluations, and have resulted in comparisons that are difficult to interpret. For example, in CP the authors compare their consstraad attack to FC, which is crafted with an &e penalty. other words, these methods have never been compared on a level playing field.
To study these attacks thoroughly and rigorously, employ sampling techniques that allow to draw conclusions about the attacks taking into account variance across model initializations and class choice. For single trial, sample one of ten checkpoints of a given architecture, then randomly select the target image, base class, and base images. In Section all figures are averages from 100 trials with sampling techniques.
3

---

Under review as a conference paper at ICLR 2021
Disparate evaluation settings from the literature To understand how differences in evaluation settings impact results, we re-create the various original performance tests for each of the methods described above within our common evaluation framework. We try to be as faithful as possible to the original works, however we employ our own sampling techniques described above to increase statistical significance. Then, we tweak these experiments component a time revealing the fragility of each method to changes in experimental design.
Establishing baselines For the FC setting, following of the main setups in the original paper, we craft 50 poisons on an AlexNet variant (for details on the specific architecture, see (Krizhevsky) al. 2012; Shafahi et al. 2018)) pre-trained on CIFAR-I0 (Krizhevsky et al. 2009), and we use the 22-norm penalty version of the attack. We then evaluate poisons on the same AlexNet, using the same CIFAR-10 10 data to train for 20 more epochs to "fine tune" the model end to end. Note that this not really transfer learning in the usual sense, as the fine tuning utilizes the same dataset as pre-training, except with poisons inserted (Shafahi et al. (018)).
The CP setting involves crafting 5 poisons using ResNet-18 model pre-trained on CIFAR-10, 10, and then fine tuning the linear layer of the same R8sNet-18 model with subset of the CIFAR-TO training comprising 50 images per class (including the poisons) (He et al., 2016). This setup is also not representative of typical transfer learning. as the fine-tuning data sub-sampled from the pre-training dataset. In this baseline set = 25.5/255 matching the original work (Zhu al. 2019).
One of the original evaluation settings for CLBD uses 500 poisons. We craft these on an adversarially trained ResNet- 18 and modify them with 3 x 3 patch in the lower right-hands corner. The pertur- bations are bounded with & = 16/255. We then train a narrow ResNet model from scratch with the CIFAR-10 training set (including the poisons) (Turner et al. 2018).
For the HTBD setting, we generate 800 poisons with another modified AlexNet (for architectural details, see Appendix A.13) which is pre-trained on CIFAR-10 1- dataset. Then, an 8 8 trigger patch is added to the lower right corner of the target image, and the perturbations are bounded with = 16/256. 255. We the entire CIFAR-10 10 dataset (including the poisons) to fine tune the last fully connected layer of the same model used for crafting, Once again, the fine-tuning data in this setup is not disjoint from the pre-training data (Saha al. 2019). See the left-most bars of Figure for all bascline results.
Inconsistencies in previous work The baselines defined above do not serve fair comparison across methods, since the original works to which we try and stay faithful are inconsistent. Table []] summarizes experimental settings in the original works. If a particular component (column header) was considered anywhere in the original paper's s experiments, we mark a (V), leaving exes (x) when something was not present in any experiments. Table I shows the presence of data normalization and augmentation as well as optimizers (SGD or ADAM). also shows which learning setup the original works considered: frozen feature extractor (FFE), end-to-end fine tuning (E2E), or crom-scratch training (FST), as well as which threat levels were tested, white, grey or black box (WB, GB, BB). out of 255 and represent the smallest bound considered in the papers; note FC uses an f2 penalty so no bound is enforced despite the attack being called "clean-label" in the original work. We conclude from Table that experimental design in this field is extremely inconsistent.
We also consider whether not ensembled attack used. The values reported
Table I: Various experimental designs used in data poisoning research.
Data
Opt.
Transfer Learning
Threat Model
Attack
Norm.
Aug.
SGD
FFE
E2E
FST
WB
GB
BB
Ensembles
FC
X
x
X
X
CP
V
X
X
V
X
X
V
V
V
25.5
CLBD
X
V
V
X
X
V
X
X
V
X
8
HTBD
V
X
Y
V
X
X
X
X
8
4

---

Under review as a conference paper at ICLR 2021
4 JUST HOW TOXIC ARE POISONING METHODS REALLY?
In this section, we look weaknesses and inconsistencies in existing experimental setups, and how these lead to potentially misleading comparisons between methods. We use our testing framework to put triggerless and backdoor attacks to the test under variety of cireumstances, and get a tighter grip on just how reliable existing poisoning methods are.
Training without SGD or data augmentation Both FC and CP attacks have been tested with victim models pre-trained with the ADAM optimizer. However, SGD with momentum has become the dominant optimizer for training CNNs (Wilson et al. (017). Interestingly, we find that models trained with SGD are significantly harder poison, rendering these attacks ineffective in practical settings. Moreover, none of the baselines include simple data augmentation such as horizontal flips and random crops. We find that data augmentation, standard in the deep learning literature, also greatly reduces the effectiveness of all of the attacks. For example, FC and CP success rates plummet in this setting to 51.00% and 19.09%, respectively. Complete results including hyperparameters, success rates, and confidence intervals are reported in Appendix A.3] We conclude that these attacks may be significantly less effective against real practitioner than originally advertised.
Two attacks, FC and HTBD, originally tested AlexNet variants, and CLBD is tested with a narrow ResNet. These models are not widely used, and they are unlikely to be employed by a realistic victim. We observe that many attacks are significantly less effective against ResNet- 18 victims. See Figure 3 where for example, the success rate of HTBD on these victims is as low as 18%. See Appendix [A.4 for table of numerical results. These ablation studies are conducted in the baseline settings but with a 18 victim architecture. These ResNet experiments serve as an example of how performance can be highly dependent on the selection of architecture.
Victim architecture matters
"Clean" attacks are sometimes dirty Each of the original works consider purports to produce "clean-label" poison examples that look like natural images. However these methods often produce easily visible image artifacts and distortions due the large values of used. See Figure [ for examples generated by two of the methods, where FC perturbs a clean "cat" into an unrecognizable poison (left), and CP generates an extremely noisy poison from a base in the "airplane" class (right). These images are not surprising since the FC method is tested with an lg penalty in the original work, and CP is fo constrained with large radius of 25.5/233.
In many contexts, avoiding detection by automated sys- tems may be more important than maintaining perceptual similarity. In our work, we focus on perceptual similarity as defined by the &o constraint as this reflects the explicit goal of most of the attacks we examine, and it is, in gen- eral, much more common area of study. Adaptive attacks that avoid defense or detection is relatively unexplored and an interesting area for future research (Koh et al.) 2018).
Figure 1: Bases (top) and poisons (bot- tom).
Borrowing from common practice in the evasion attack and defense literature, we test each method with an Eoo constraint of radius 8/255 and find that the effectiveness of every attack is significantly diminished (Madry et al.)
2017 Dong et al. 2020). Thus, standardized constraint on poison examples is necessary for fair comparison of attacks, and these previous attacks are not nearly as threatening under constraints that enforce clean poisons. See Figure] 3, and see Appendix A.5 for a table of numerical results.
Proper transfer learning is less vulnerable Of the attacks we study here, FC, CP, and HTBD were originally proposed in settings referred to as "transfer learning." Each particular setup varies, but none are true transfer learning since the pre-training datasets and fine-tuning datasets overlap. For example, FC uses the entire CIFAR-10 training dataset for both pre-training and fine tuning. Thus, their threat model entails allowing an adversary to modify the training dataset but only for the last few epochs. Furthermore, these attacks use inconsistently sized fine-tuning datasets.
5

---

Under review as a conference paper at ICLR 2021
To simulate transfer learning, test each attack with ResNet-18 feature extractors pre- pre-trained on CIFAR- 100. We fine tune with CIFAR- 10 data in both cases, showing that these methods actually perform better in the setting with real transfer learning, i.e. where the pre-training data and fine-tuning data are not from the same datasets and do not contain the same classes. In Figure X every attack aside from CP shows worse performance when transfer learning is done data that is disjoint from the pre-training dataset. The attacks designed for transfer learning may not work as advertised in more realistic transfer learning settings, See Appendix A.6
Performance is not invariant to dataset size Existing work on data poisoning measures an at- tacker's 's budget in terms of what percentage of the training data they may modify. This begs the question whether percentage alone is enough to characterize the budget. Does the actual size of the training set matter? We find the number of images in the training set has a large impact on attack performance, and that performance curves for FC and CP intersect. When we hold the percentage poisoned constant at 1,, but we change the number of poisons and the size of the training set accordingly, we see no consistent trends in how the attacks are affected. Figure Sauure/]sooo shows the success of each attack as a function of dataset size (shaded region is one standard error). This observation suggests that one cannot compare attacks tested on different sized datasets by only fixing the percent of the dataset poisoned. See Appendix A.7
A.7
Black-box performance low Whether con- sidering transfer learning training from scratch, testing these methods against a black- box victim is surely one of the most realistic tests of the threat they pose. Since, FC, CP and HTBD do not consider the black-box scenario in the original works, we take the poisons crafted using baseline methods and evaluate them on models of different architectures than those used for crafting. The attacks show much lower per- formance in the black-box settings than in the baselines, particular FC, CP, and HTBD have success rates lower than 20%. See Figure 3 and see Appendix A.8 for more details.
Figure 2: Scaling the dataset size while fixing the poison budget.
100
Success Rate (%)
50
25
Small sample sizes and non-random targets On top of inconsistencies in experimental setuss, existing work on data poisoning often test only on specific target/base class pairs. For example, FC largely uses "frog" as the base class and "airplane" as the target class. CP, on the other hand, only uses "ship" and "frog" as the base and target classes, respectively. Neither work contains experiments where each trial consists of a randomly selected target/base class pair. We find that the success rates are highly class pair dependent and change dramatically under random class pair sampling. Thus, random sampling is critical for performance evaluation. See Appendix A.99 for comparison of the specific class pairs from these original works with randomly sampled class pairs.
Alldd
BC
C7
CL30
1730
10,000 20,000 30,000
50,000
Trainset Size (subsets of CIFAR-10)
In addition to inconsistent class pairs, data poisoning papers often evaluate performance with very few trials since the methods are computationally expensive. In their original works, FC and CP use 30 and 50 trials, respectively, for each experiment, and these experiments pre-trained models each time. And while HTBD does test randomized pairs, they only show results for ten trials on CIFAR-10. These small sample sizes yield wide error bars in performance evaluation. We choose to run 100 trials per experiment in our own work. While we acknowledge that larger number would be even more compelling, 100 a compromise between thorough experimentation and practicality since each trial requires re-training classifier.
performed on the same exact
Attacks are highly specific to the target image Triggerless attacks have been proposed as a threat against systems deployed in the physical world. For example, blue Toyota sedans may go undetected by a poisoned system so that an attacker may fly under the radar. However, triggerless attacks are generally crafted against a specific target image, while physical object may appear differently under

---

Under review as a conference paper at ICLR 2021
difference real-world cireumstances. We upper-bound the robustness of poison attacks by applying simple horizontal flips to the target images, and we find that poisoning methods are significantly less successful when the exact target image is unknown. For example, FC is only successful 7% of the time when simply flipping the target image. See Figure 3 and Appendix A0
Backdoor success depends on patch size Backdoor attacks add a patch to target images to trigger misclassification. In real-world world scenarios, small patch may be critical to avoid being caught. The original HTBD attack uses an x patch, while the CLBD attack originally uses a 3 patch (Saha et al.. 2019 Turner al. 2018). In order understand the impact attack performance, different patch sizes. We find a strong correlation between the patch size and attack performance, see Appendix A. We conclude that backdoor attacks must be compared using identical patch sizes.
test
We show the fragility of poisoning methods to experimental design. This figure depicts baselines along with the results ablation studies. Different methods respond differently these testing scenarios, supporting the need for consistent and thorough testing. Horizontal lines denote performance on baselines described Section 3 and bars represent the results of changing specific feature in an individual method's baseline. Tables of these results with confidence intervals can be found in the appendices.
Figure 3:
Collision
Conves Polytope
Label Backdoor
Hidden Trigger
100
Success Rate (%)
75
50
25
Baseline
Data Aug.
SGD
ResNet-18
Transfer
8/255
Black-box
Flip
UNIFIED BENCHMARKS FOR DATA POISONING ATTACKS
Our Benchmark We propose new benchmarks for measuring the efficacy of both backdoor and triggerless data poisoning attacks. We standardize the datasets and problem settings for our benchmarks below Target and base images are chosen from the testing and training sets, respectively, according to seeded/reproduuible random assignment. Poison examples crafted from the bases must remain within the fg -bal radius 5/255 centered at the corresponding base images. Seeding test against significant number of different random choices
the random assignment allows of base/target, while always using the same choices for each method, thus removing source of variation from the results. We consider two different training modes:
Transfer Learning: A feature extractor pre-trained clean data is frozen and used while training a linear classification head on a disjoint set of training data that contains poisons.
II Training From Scratch: A network trained from random initialization on data containing poison examples in the training set.
To further standardize these tests, we provide pre-trained architectures to test against. The parameters of one model are given to the attacker. then evaluate the strength of the attacks in white-box and black-box scenarios. For white-box tests in the transfer learning benchmarks, feature extractor that is given to the attacker for evaluation. While in the black-box setting, we craft poisons using the known model but test the models the attacker has not seen, averaging the results. When training from scratch, models are trained from a random initialization on the poisoned
the same frozen
Cooee available (suppressed for anonymity).

---

Under review as a conference paper at ICLR 2021
dataset. We report averages over 100 independent trials for each test. Backdoor attacks use any 5 x 5 patch. Note that the number of attacker-victin network pairs is kept small in our benchmark because each of the 100 trials requires re- -eatrinning (in some cases from scratch), and want to keep the benchmark within reach for researchers with modest computing resources.
CIFAR-10 benchmarks Models are pretrained on CIFAR-I00, and the fine-tuning data is a subset of CIFAR- 10. We choose this subset to be the first 250 images from each class, allowing for 25 poison examples. This amount of data motivates the use of transfer learning, since training from scratch on only 2,500 images yields poor generalization. See Appendix [A.13 examples. We allow 500 poisons when training from scratch, see Appendix AA.15 for a case-study in which we investigate how many poisons an attacker may be able to place in dataset compiled by querying the internet for images. We allow the attacker access to a ResNet-18, and we do black-box tests on a VGG (Simonyan & Zisserman 2014), and a MobileNetV2 (Sandler et al. 2018), and we use one of each model when training from scratch and report the average.
TinyImageNet benchmarks Additionally, we pre-train VGG16, ResNet-34, MobileNetV2 models on the first 100 classes of the TinylmageNer dataset (Le & Yang 2015). We fine tune these models on the second half of the dataset, allowing for 250 poison images. As above, the attacker has access the particular VGGI6 model, and black-box tests are done on the other two models. In the from-scratch setting, we train a VGG16 model on the entire Imayene dataset with 250 images poisoned
Benchmark hyperparameters We pre-train models on CIFAR-100 with SGD for 400 epochs starting with learning rate of 0.1, which decays by a factor of 10 after epochs 200, 300, and 350. Models pre-trained on the first half of TinyImageNe are trained with SGD for 200 epochs starting with a learning rate of O. I, which decays by a factor of 10 after epochs 100 and 150. In both cases, we apply per-channel data normalization, random crops, and horizontal flips, and use batches of 128 images (augmentation is also applied to the poisoned images). We then fine tune with poisoned data for 40 epochs with learning rate that starts 0.01 and drops to 0.001 after the 30th epoch (this applies to the transfer learning settings).
When training from scratch CIFAR-10, we include the 500 perturbed poisons in the standard training set. We use SGD and train for 200 epochs with batches of 128 images and an initial learning rate of 0.1 that decays by factor of 10 after epochs 100 and 150. Here too, use data normalization and augmentation as described above. When training from scratch on Tiny ImageN,,,,,, we allow for 250 poisoned images. All other hyperparameters are identical.
Our evaluations of six different attacks are shown in Table These attacks are not easily ranked, as the strongest attacks in some settings are not the strongest others. Witches' Brew (WiB) is not evaluated in the transfer learning settings, since it is not considered in the original work (Geiping) jet al., (020).) See Appendix 16 for tables with confidence intervals. We find that by using disjoint and standardized datasets for transfer learning, and common training practices like data normalization and scheduled learning rate decay, we overcome the deficits in previous work. Our benchmarks can provide useful evaluations of data poisoning methods and meaningful comparisons between them.
Table 2: Benchmark success rates (%) (best in each column is in bold).
CIFAR-10
TinyImage
Transfer WB
From Scratch
Transfer WB
From Scratch
Attack
BB
BB
FC
22.0
7.0
1.33
49.0
2.0
4.0
CP
33.0
7.0
0.67
14.0
1.0
0.0
BP
85.0
8.5
2.33
100.0
10.5
44.0
WiB
26.0
32.0
CLBD
5.0
6.5
1.00
3.0
1.0
0.0
HTBD
10.0
9.5
2.67
3.0
0.5
0.0
2The TinylmageNet from-scratch benchmark is done with 25 independent trials to keep this problem within reach for researchers with modest resources.
8

---

Under review as a conference paper at ICLR 2021
CONCLUSION
The threat of data poisoning is at the forefront of fears around emerging ML systems (Siva Kumar al., 2020). While many of the methods claiming do so do not pose a practical threat, some of the recent methods are cause for practitioner concern. With real threats emerging, there is need for fair comparison. The diversity of attacks, and in particular the difficulty ordering them by efficacy, calls for diverse set of benchmarks. With those we present here, practitioners and researchers can compare attacks on level playing field and gain an understanding of how existing methods match up with one another and where they might fail.
Since the future advancement of these methods is inevitable, our benchmarks will also serve the data poisoning community as a standardized test problem on which to evaluate and future attack methodologies. As even stronger attacks emerge, trepidation on the part practitioners will be matched by the potential harm of poisoning attacks. We are arming the community with the high quality metrics this evolving situation calls for.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hojjat Aghakhani, Thorsten Fisenhofer, Lea Schönherr, Dorothea Kolossa, Thorsten Holz, Christo- pher Kruegel, and Giovanni Vigna. Venomave: Clean-label poisoning against speech recognition, 2020a.
Hojjat Aghakhani, Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferability, 2020b.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp. 1467-1474, USA, 2012. Omnipress. ISBN 978-1 1503--11--1. URL
http: / /dl. acm. o ci eitatiiinn...... c wtt.::ccmmm2m1212111111 304766
H. Chacon, S. Silva, and P. Rad. Deep learning poison data attack detection. In 2019 JEEE 3 International Conference on Tools with Artificial Intelligence (ICTAI), pp. 971-978, 2019.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:/7/2.05520 2017.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against istm-based text classification systems. IEEE Access, 2-138872-138875 2019.
Yinpeng Dong, Qi-An Fu, Xiao Yang. Tianyu Pang. Hang Su, Zihao Xiao, and Jun Zhu. Benchmark- ing adversarial robustness on image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 321-331, 2020.
Minghong Fang, Guolei Yang, Neil Zhenqiang Gong, and Jia Liu. Poisoning attacks to graph- based recommender systems. In Proceedings of the 34th Annual Computer Security Applications Conference, pp. 381 381-392, 2018.
Minghong Fang, Neil Zhenqiang Gong, and Jia Liu. Influence function based data poisoning attacks to top-n recommender systems. In Proceedings of The Web Conference 2020, pp. 3019-3025, 2020.
Jonas Geiping, Liam Fowl, W. Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching, 2020.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint urXiv::708.0000 06733, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Rui Hu, Yuanxiong Guo, Miao Pan, and Yanmin Gong. Targeted poisoning attacks on social recommender systems. In 2019 IEEE Global Communications Conference (GLOBECOM), pp. 1-6. IEEE, 2019.
W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general-purposs clean-label data poisoning, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv:1712.05055. 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning-Voi Volume 70, pp. 1885-1894. JMLR. org, 2017.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses, 2018
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
10

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky, lly Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine learning-industry perspectives. arXiv preprint 0rXiv:2002.05646 2020.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, pp. 1-26, 2020.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7, 2015.
Bo Li, Yining Wang, Aarti Singh, and Yevgeniy Vorobeychik. Data poisoning attacks on factorization- based collaborative filtering, 2016.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposiun on Research in Attacks, Intrusions, and Defenses. pp. 273-294. Springer, 2018
Sijia Liu, Songtao Lu, Xiangyi Chen, Yao Feng, Kaidi Xu, Abdullah Al-Dujaili, Minyi Hong, and Una-May Obelilly. Min-max optimization without gradients: Convergence and applications to adversarial ml, 2019.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1706.060866 2017.
Luis GonizzGG,zález, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back- becdggra optimization. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 27-38. ACM, 2017.
Jianmo Ni. Amazon Review Data, 2018 https://ni https:/nijjanmo..ger wtw.:///....mmmooiiiieee.e. html
Neehar Peri, Neal Gupta, W. Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John Dickerson. Deep k-nn defense against clean-label data poisoning attacks, 2019.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. arXiv preprint arXiv: 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520 2018.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems, pp. 6103-6113, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image 2014.
recognition. arXiv preprint
Ram Shankar Siva Kumar, Magnus Nystrom, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine learning industry perspectives. SSRN Electronic Jourmal, 2020. ISSN 1556-5068. doi: 10.21399/sm..5324777 URL http: http://22...1..rrr/..0.333/////3
3532474
David Solans, Battista Biggio, and Carlos Castillo. Poisoning attacks on algorithmic faireess. arXiv preprint arXiv:2004.07401 2020.
11

---

Under review as a conference paper at ICLR 2021
Alexander Tumer, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. 2018.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in neural information processing systems, pp. 4148-4158, 2017.
Huang Xiao, Battista Biggio, Blaine Nelson, Han Xiao, Claudia Eckert, and Fabio Roli, Support vector machines under adversarial label contamination. Neurocomputing, 160:53-62 2015.
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer. and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets, In International Conference on Machine Learning, pp. 7614-7623, 2019.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1
TECHNICAL SETUP
We report confidence intervals radius one standard error, & vp p)/N, where ý is the observed probability of success, and N is the number of trials. If there are fewer than five observed successes or failures, we set p = 1/2 to upper-bound standard error.
Hyperparameters We one of seven sets of hyperparameters when training models. We refer to these by name throughout this appendix, and Table shows each setup. For all models trained with SGD, set the momentum coefficient 0.9. We always use batches of 128 images and weight decay with a coefficient of 2 x 10m4. The "Decay Schedule" column details the epochs after which the learning rate drops by the corresponding decay factor.
Table Hyperparameters.
Learning rate
Setup
Initial
Decay Factor
Decay Schedule
Epochs
Optimizer
A
0.001
0.5
32, 64, 96, 128, 160, 192
200
ADAM
B
0.010
0.1
100, 150
200
SGD
C
0.100
0.1
100, 150
200
SGD
DE
0.100
0.1
200, 300, 350
400
SGD
0.100
0.1
40, 60
100
SGD
FF
0.100
0.1
75, 90
100
SGD
0.010
0.1
30
40
SGD
A.2 BASELINES
Table shows the baseline performance of each attack. This table reports averages over 100 independent trials with confidence intervals of width one standard error. The experimental setups are summarized in Section 3 and we report additional details here. When we say that an experiment uses a particular architecture, we mean that each trial randomly selects one
pre-trained models of this type. The average performances for these sets of pre-traines models are reported in Table [6 below where the hyperparameters and training routines are detailed.
Feature Collision The FC baseline uses AlexNet variant without data normalization data augmentation. We use the unconstrained version of this attack with the l2 penalty in the optimization problem. The algorithm presented in the original work has hyperparameters which set as follows (Safahi et al. (2018). We add a watermark of the target image with 30% opacity to each base before the optimization and we use step size of 0.0001 with the maximum number of iterations to 1,200. When fine tuning on the poisoned data, we train for 20 epochs with ADAM and fixed learning rate of 0.001 0.5% 0.00015625, which is the smallest learning rate used when pre-training.
Convex Polytope The CP baseline uses ResNet-18 with data normalization (without data aug- mentation). In the poison crafting procedure, we use the ADAM optimizer with a learning rate of 0.04 for maximum of 4,000 iterations when the CP loss is less than equal to 1 10-6 We bound the perturbations with E = 25.5/255. Then, we fine tune the model with ADAM for 10 epochs on the poisoned data with a learning rate of 0.1.
Clean Label Backdoor The CLBD baseline is training from scratch scenario. The model used for crafting is an adversarially trained ResNet-18 18, and we use 20-step PGD with step size of 42ss and of 16/2s to compute the adversarial perturbations (Madry al. 2017). Then we train a narrow ResNet model (see (Turner et al. 2018) for architectural details) from scratch using hyperparameter set E as defined in Table 3
13

---

Under review as a conference paper at ICLR 2021
Hidden Trigger Backdoor The HTBD baseline uses modified AlexNet model with data normal- ization. Poisons are crafted using SGD for a maximum of 5,000 iterations with initial learning rate of 0.001, which decays patched with an x 8 patch in the bottom right corner. Then we fine-tune the last linear layer of the network using SGD for 20 epochs with initial learning rate of 0.5, which decays by factor of 0. after epochs 5, 10, and 13.
factor of 0.95 every 2,000 iterations with 16/255. The target image is
Table 4: Baseline performance.
Attack
Success Rate (%)
FC
92.00 2.71
CP
88.00 = 3.25
CLBD
86.00 3.47
HTBD
69.00 = 4.62
A.3 TRAINING WITHOUT SGD OR DATA AUGMENTATION
We add data normalization and augmentation the pre-training processes in each attack. For FC and CP, which were originally tested with ADAM, we show results from experiments where normalization and augmentation used with ADAM as well as when pre-train with SGD.
Table 5: Data normalization and augmentation + ADAM.
Attack Success Rate (%)
Diff. From Baseline (%)
FC
42.00 + 4.94
50.00
CP
37.00 4.83
51.00
Table 6: Data normalization and augmentation + SGD.
Attack
Success Rate (%)
Diff. From Baseline (%)
FC
51.00 5.00
-41.00
CP
12.00 + 3.25
76.00
CLBD
1.00 5.00
-85.00
HTBD
11.00 + 3.13
-58.00
A.4 VICTIM ARCHITECTURE MATTERS
We test each method on ResNet-18 victims. Note that CP shows no change from the baseline, as our baseline set-up for CP uses ResNet-18 victim model.
Table ResNer- 18 victims.
Attack
Success Rate (%)
Diff. From Baseline (%)
FC
4.00 = 5.00
-8.00
CP
88.00 + 3.25
0.00
CLBD
80.00 4.00
66.00
HTBD
18.00 3.84
-1.00
A.5 "CLEAN" ATTACKS ARE SOMETIMES DIRTY
We test each attack with lo -orm constrained perturbation with 8/255. Note that HTBD shows no change form the baseline since this was the e values used in our baseline for this attack. See Table
14

---

Under review as a conference paper at ICLR 2021
Table 8: Poisons crafted with &
8/255.
Attack
Success Rate (%)
Diff. From Baseline (%)
FC
7.00 2.55
85.00
CP
80.00 + 4.00
- 8.00
CLBD
10.00 3.00
76.00
HTBD
59.00 + 4.96
10.00
A.6 PROPERLY TRANSFER LEARNED MODELS ARE VULNERABLE
We use feature extractors pre-trained to classify CIFAR- 100 data to craft the poisons. Then, we use those same feature extractors in the fine-tuning stage when we train the models classify CIFAR-10 data. See Table
Table 9: Transfer learned victims.
Attack
Success Rate (%)
Diff. From Baseline (%)
FC
16.00 3.67
76.00
CP
100.00 + 5.00
+12.00
CLBD
2.00 5.00
-84.00
HTBD
25.00 + 4.33
444.00
A.7 PERFORMANCE IS NOT INVARIANT TO DATASET SIZE
We study the effect of scaling the dataset size while holding the percentage of data that is poisoned constant. We test each attack with 5 poisons and 500 training images and increment both the number of poison examples and the training set size until reach 500 poisons and 50,000 training images (the entire CIFAR-10 training set). For every training set size, we allow the attacker to perturb 1% of the data and we see that the strength of poisoning attacks does not scale with any generality in some cases we see success rates drop with increase in dataset size, and some attacks are more successful with more data. See Table 10 for complete numerical results with confidence intervals of width one standard error.
In addition, we investigate these dynamics with exactly the CIFAR-10 transfer learning benchmark set up. This way, we can evaluate each attack in exactly the same setting, as opposed to above, where we use respective baseline setups. Figure 4 shows that when tested in the exact same evaluation setting, these attacks scale differently with the size of the dataset. [I complete numerical results with confidence intervals width one standard error. This experiment, whose results are perhaps best presented in Figure 2 shows that discussing the poison budget only percentage of the data does not allow for fair comparison.
A.8 BLACK-BON PERFORMANCE IS LOW
When tested in the black-box setting, all methods except for CLBD show dramatically lower perfor- mance, CLBD is intended for use in the training from scratch case, and the particular architectures for crafting and testing are different. So for this experiment, For FC, CP, and HTBD we craft poisons on the architectures used in the baselines. The black-bo box victims for FC and HTBD are ResNet-18 models, whereas the CP baseline used a ResNet- 18 victim, so we use MobileNetV2 for the black-box victim. See Table 12
consider the CLBD baseline black-box.
A.9 SMALL SAMPLE SIZES AND NON-RANDOM TARGETS
We test FC and CP with the specific target/base class pairs studied in the original works. We find the performance of each attack measured only these classes differs from our baseline. See Table [3 This fact alone sufficient evidence that the comparisons done in the poison literature are lacking consistency, and that this field needs benchmark problem.
15

---

Under review as a conference paper at ICLR 2021
Table 10: Success rates (%) with varying dataset sizes and number of poisons.
Number of Poisons
Attack
5
10
25
50
100
FC
46.30 = 6.79
47.06 = 6.99
66.67 = 6.60
62.75 6.77
CP
100.00 + 7.07
100.00
7.07
98.00 + 7.07
88.00 + 4.60
82.00 5.43
CLBD
5.88 + 7.00
4.08
7.14
2.00 + 7.07
9.80 + 7.00
3.92 + 7.00
HTBD
26.00 + 4.39
35.00 = 4.77
43.00 + 4.95
50.00 = 5.00
55.00 + 4.97
Number of Poisons
Attack
200
300
400
500
FC
76.00 6.04
71.70 + 6.19
80.39 = 5.56
78.00
5.86
CP
76.00 + 6.04
55.10
+ 7.11
42.86 + 7.07
26.00 + 6.20
CLBD
4.00
7.07
30.00
+ 6.48
68.00 + 6.60
92.00 7.07
HTBD
56.00 = 4.96
59.00 + 4.92
62.00 + 4.85
53.00 = 4.99
Scaling 1% Poison Budget- Benchmark Setting
Attack
FC
err
Success Rate (%)
25
20,000 Trainset Size (subsets of CIFAR-10)
50,000
Figure 4: Scaling the dataset size in the benchmark setting.
A.10 ATTACKS ARE HIGHLY SPECIFIC TO THE TARGET IMAGE
We consider the where the target object is photographed in slightly different environment than in the particular image the attacker uses while crafting poisons. Perhaps, the attacker is trying to keep their own red from being classified different image than the specific photograph to which the attacker has access. We are unable to get choose to upper bound performance
In reality, the deployed model may see
new photographs of the exact objects in CIFAR- images, on highly modified images by simply flipping the target images horizontally during evaluation. In this setting, we observe that triggerless attacks severely impaired, supporting our conclusion that they pose less practical threat in physical settings than suggested in previous work. See Table [4]
16

---

Under review as a conference paper at ICLR 2021
Table 11: Success rates (%) with varying dataset size in the benchmark setting.
Number of Poisons
Attack
5
10
25
50
FC
2.30 5.00
14.0 + 3.47
22.0 - 4.14
10.0 + 3.00
CP
19.0
29.0 + 4.61
33.0
4.70
19.0 3.92
CLBD
1.00
2.0 + 5.00
5.0 5.00
2.0 + 5.00
HTBD
1.0
6.0 + 2.37
10.0 + 3.00
4.0 + 5.00
Number of Poisons
Attack
100
250
500
FC
23.0
4.21
14.0 3.47
= 3.84
CP
19.0 + 3.92
2.0 + 5.00
4.0+55000
CLBD
3.0 5.00
1.0 5.00
1.0 5.00
HTBD
2.0 : 5.00
3.0 5.00
Table 12: Black-box victim.
Attack
Success Rate (%)
Diff. From Baseline (%)
FC
4.00 + 5.00
-88.00
CP
16.00 3.67
72.00
CLBD
86.00 = 3.47
0.00
HTBD
2.00 5.00
-67.00
Table 13: Success with specific class pairs.
Attack
Target
Base
Success Rate (%)
Diff. From Baseline (%)
FC
plane
frog
80.00 4.00
12.00
CP
frog
ship
83.00 3.76
-5.00
Table 14: Success flipped targets.
Attack
Success Rate (%)
Diff. From Baseline (%)
FC
7.00 +: 2.55
-5.00
CP
40.00 4.90
48.00
A.11 ENSEMBLIZING BOOSTS THE ATTACKER
We study the impact of ensemblizing attacks, where the attacker several architectures while crafting poisons. This was suggested and tested with CP in the original work (Zhu et al.] 2019). In that study however, the comparison between CP and FC is incomplete. We show that ensemblizing helps both attacks and that FC outperforms CP in the white-box setting with enough poisons (both in the single model and the ensemblized attacks). See he rows in Table 22 corresponding to the ensembled FC attack (FC-Ens.) and the ensembled CP attack (CP-Ens.).
A.12 BACKDOOR SUCCESS DEPENDS ON PATCH SIZE
In order to determine the effect of the particular size of the patch used backdoor attacks, we test the backdoor methods with variety of patch sizes. We see strong correlation between patch size and success rate. See Table I5 Note that dashes correspond to an attack's 's baseline performance, see Table
17

---

Under review as a conference paper at ICLR 2021
Table 15: Success rates (%) of backdoor attacks with varying patch sizes.
Patch Size
Attack
3 x 3
5 x 5
8 x 8
HTBD CLBD
20.00 = 4.0
33.00 = 4.70
97.00 + 5.00
100 + 5.0
A.13 MODEL TRAINING AND PERFORMANCES
19 we show the training setups, including references to sets of hyperparameters outlined in Table and the training and testing accuracy of the models use in this study. Each row in these two tables shows averages of ten models trained from random intializations with identical training setups. Note that the models called "AlexNet" the variants introduced in the original FC paper, that work for details (Shafahi et al., (018). Models named "HTBD AlexNet" are the modified AlexNet architecture we used for the HTBD experiments and the details below.
Models trained for our experiments In Tables T6
Architectures we use Four of the five architectures use in this study are widely used and/or detailed other works. For modified AlexNet used in FC experiments, see (Shafahi et 2018). For ResNet-18 architecture details, see (He et al. 2016). For MobileNetV2, see (Sandler et al. 2018). For VGG11, see (Simonyan Zisserman 2014).
HTBD AlexNet The model used in the original HTBD work was simplified version of AlexNet. But for our baseline experiments we adapt the ImageNet AlexNet model to CIFAR-10 dataset. We modify the kernel size and stride in the first convolution layer to and 2, respectively, order to take 32 32 x 3 input images. For deeper layers we use stride of 1. The width of the network at then end the convolutional layers is 256.
Table 16: CIFAR- 10 models.
Model
Norm.
Aug.
Hyperparam.
Train Acc. (%)
Test Acc. (%)
AlexNet
99.99
73.96
AlexNet
99.99
74.45
AlexNet
A
90.35
82.36
AlexNet
B
98.77
85.91
BB
HTBD AlexNet
100.00
77.35
HTBD AlexNet
V
98.80
84.30
R8
X
C
100.00
87.05
R8sNet-18
X
100.00
87.10
ResNet-18
C
99.99
94.96
MobileNetV2
X
C
99.99
82.11
MobileNerV2
C
99.99
82.04
MobileNetV2
99.88
93.36
Table 17: CIFAR-100 models.
Model
Norm.
Aug.
Hyperparam.
Train Acc. (%)
Test Acc. (%)
R8
D
99.97
74.37
MobileNetV2
D
99.95
71.81
VGG11
/
D
99.97
67.87
Transfer learning We also train models of each architecture from scratch the first 250 images per class of CIFAR-10. By comparing these models to transfer learned models on the same data, we the benefit of transfer learning for this quantity of data. Each of Table 20 shows averages of
18

---

Under review as a conference paper at ICLR 2021
Table 18: TinylmageNet models (first 100 classes).
Model
Norm.
Aug.
Hyperparam.
Train Acc. (%)
Test Acc. (%)
VGG16
C
99.99
63.12
MobileNetV2
C
99.99
67.
ResNet-34
/
v
C
99.99
62.11
Table 19: TinylmageNet models (all classes).
Model
Norm.
Aug.
Hyperparam.
Train Acc. (%)
Test Acc. (%)
VGGI6
C
99.98
58.77 72
ResNet-34
99.98
58.08
10 models. For the transfer learned models we use exactly the feature extractors from the benchmark, and the pre-trained models' performances are in Table [7] clear from Table 20 that with so little data, training from scratch leads to less-than-optimal test accuracy. This motivates the transfer learning benchmark tests since transfer learning does improve performance in this setting.
Table 20: CIFAR-IO models with Trainset size 2500.
Model
Norm.
Aug.
Hyperparam.
Train Acc. (%)
Test Acc. (%
From Scratch
ResNet-18 MobileNetV2
F
99.12
59.71
/
F
98.99
68.55
VGG11
/
97.98
60.72
Transfer learned
ResNet-18
G
99.98
80.38
MobileNetV2
99.97
79.53
VGG11
G
99.93
74.95
A. ADDITIONAL EXPERIMENTS
Backdoor triggers Different backdoor attacks use different patch images as trigger. Does the performance of the attack depends the patch used? In order to study the impact, swap the patches of CLBD and HTBD. We resize the CLBD patch to 8 x 8 and the HTBD patch to 3 x which the baseline sizes, see Figure We observe that patch image does matter and can have significant effect on the performance of an attack. See Table 21
Figure 5: CLBD patch (left), and HTBD patch (right).
Table 21: Success rates (%) of backdoor attacks with swapped patch images.
Attack
Success Rate (%)
Diff. From Baseline (%)
HTBD w/ CLBD patch
51.00 + 4.99
-8.00
CLBD w/ HTBD patch
2.00 + 5.00
-84.0
Poison budget We conduct an additional experiment to assess the impact of the budget in our benchmark. We test each attack in the same setting at the benchmark, where do 100 trials each with 50, 100, and 250 poisons, holding the dataset size constant. See Table 22 We see the expected rise in success rate with increased budget, however note that these increases are almost always
19

---

Under review as a conference paper at ICLR 2021
small. We choose to use 25 poisons in the benchmark tests for the following two reasons. First, want the evaluation of an attack to be accessible to those with modest computing resources. Second, as discussed in Appendix A.13 we find 25 images out of 250 images per class to be large budget in realistic settings.
Table 22: CIFAR-10 Transfer learning benchmark tests with varying budget.
Success Rates (%)
Attack
Budget
White-box
Black-box
FC
23.0 = 4.21
7.0 + 1.80
100
59.0 = 4.92
5.0 + 1.56
250
93.0 + 2.55
17.0 + 3.76
FC-Ens.
50
17.0
19.5 2.80
100
40.0 4.90
3
250
79.0 4.07
69.0 + 4.62
CP
50
24.0 4.27
8.0
100
38.0 4.85
2.5
250
=.00
5.0 1.54
CP-Ens.
50
22.0
22.0 + 2.93
100
3.70
250
47.0
37.0 + 4.83
CLBD
50
5.00
3.0 1.20
100
2.0
2.0
250
5.00
HTBD
50
7.0
7
100
2.0 5.00
6.5 1.74
250
6.0 2.37
A. HOW MANY IMAGES
When scraping data from Google, the sources images are diverse. If each source responsible for very few images, then adversary may have difficult time poisoning significant amount of data scraped by their victim. In order to investigate the diversity of sources, we query Google Images for each CIFAR-10 class label and measure how many images come from each source. Table 23 shows how many images the first through fifth most represented source are each responsible for in the first 100 search results for each class. Entries represent the number of images from a particular source within the first 100 search results for that query. The first column represents the source responsible for the most images. The second column represents the source responsible for the second most images, etc. We find that sources generally are not highly dominant, and each source responsible for few images. Poisoning methods which only perturb data from the target class may only be able to poison very small percentage of the victim's 's total data, especially when the number of classes is high. For example, in a 1000-class problem like ImageNet, even if the attacker could poison 10% of the target class, this would only represent 0.01% of the total dataset. This percentage far smaller than the percentages studied the data poisoning literature.
particular class coming
20

---

Under review as a conference paper at ICLR 2021
Table 23: Google Images case study.
Number of Images
Search term
Source
Source 2
Source 3
Source 4
Source 5
airplane automobile
5
5
3
3
6
6
bird cat deer dog frog
5
4
4
4
8
5
4
4
6
5
4
14
9
6
4
3
4
4
3
horse
5
4
3
3
ship
5
5
truck
6
6
5
4
A.16 BENCHMARK RESULTS
We present complete benchmark results with confidence intervals in Tables here are success rates of attacks reported as percentages.
24
25
and
26
All figures
Table 24: Complete CIFAR-10 transfer learning benchmark results.
Transfer learning
Attack
WB
BB
FC
16.0
3.5 + 1.30
CP
4.27
4.5 1.47
BP
85.0
8.5 1.97
CLBD HTBD
3.5 5.00
2.00
4.0 1.39
Table 25: Complete CIFAR-10 from-scratch benchmark results.
Training from scratch
Attack
ResNet-18
MobileNetV2
VGG11
Average
FC
0.0 + 5.00
1.0 5.00
3.0
1.3 + 5.00
CP
0.0 5.00
5.00
5.00
0.7 0.7 5.00
BP
3.0 + 5.00
3.0
1.0
2.3 5,00
WiB
45.0 + 4.97
25.0
8.07
26.0 + 4.38
CLBD
0.0 5.00
1.0
2.0
1.0 + 5.00
HTBD
0.0 + 5.00
4.0 5.00
1.0
2.7 + 5.00
Table 26: Complete TinyImageNet benchmark results.
Transfer learning
From scratch
Attack
WB
BB
VGG16
FC
49.0 + 4.99
2.0 + 5.00
4.0 10.0
CP
14.0 3.47
1.0 5.00
0.0 10.0
BP
100.0 5.0
10.5 3.06
44.0
WiB
32.0
CLBD
3.0
1.0
0.0
HTBD
3.0
0.0
21


Reviews:
[{"review_id": "oApdA7_MN4x", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** This paper studies 4 different previously proposed (clean label and targeted) poisoning attacks and compares them in a systematic way. The authors argue that the experimental setup in these papers are significantly different and comparing them with each other is hard. This incentivizes them to use a benchmark to test all in a unified way. They show that these 4 attacks are not robust to changing parameters of learning algorithm or the experimental setting. To fix this issue, the paper suggest a benchmark for testing poisoning attacks. Their benchmark is only for \"clean label\" and \"targeted\" poisoning attacks. They identify several issues with the experimental settings of previous poisoning attacks and suggest some guidelines on  for evaluating poisoning attacks. \n\n**Evaluation:** I like the paper's analysis of different attacks and showing when they fail and more importantly, when they succeed. However, In my opinion, I don't find the findings of the paper in lack of robustness for the attacks surprising. These attacks are all proof of concepts that show a serious security concerns for real world applications. It is not surprising that none of them cannot succeed in all scenarios and in a robust manner. Attackers can always make tweaks to their algorithm for individual settings (adaptive attacks). \n\nIn general, I think designing adaptive benchmarks for attacks is not a very good idea. There might not exist a single attack that succeeds in all the scenarios in benchmark and this can give a false sense of security. Even if there is an attack that passes all the tests in the benchmark, one can try and find a scenario where the attack fails and add it to the benchmark. The authors might argue that the goal of the benchmark is not to show that the security threat is not real or that the attacks are weak, but to give a way to compare different attacks and provide insight for when attacks are (un)successful. I think this goal would be much more valuable as it poses a lot of open question for future research, but the current tone of the paper seems not to focus on that. \n\nComment to authors:\n-The paper starts great by talking about the relevance of poisoning attacks in real world applications, but then the benchmarks are only for image classification. I think the focus on image classification is disproportionate to real world applications of machine learning. \n- Any particular reason why you don't consider indiscriminative attacks?\n-In the description of attacks it is not specified how the labels of the poison points are selected.\n-In the blackbox setting, does the attacker know the clean training set?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OvP4cdmNanw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Reject\n\nThis paper consists of two main parts, essentially inspired from the argumentation of the Baconian method. In particular, the first part criticizes current work in the area of data poisoning attacks (pars destruens), while the second part tries to overcome the limits described in the first half of the paper (pars construens).\n\nIn their pars destruens, the authors heavily criticize the lack of standardized evaluations across different papers, claiming that: \"inconsistent and perfunctory experimentation has rendered performance evaluations and comparisons misleading\". The authors then continue by highlighting the inconsistencies observed across different papers and summarize them in Table 1.\n\nFirst, I think that this criticism is not fully justified. None of the considered papers had the goal of proposing a common benchmark evaluation. They were all proposing different backdoor attacks, under different scenarios. Hence, it is unfair to claim that their evaluations are inconsistent. Of course they are. Every paper considers a more or less different experimental setups according to their hypotheses, to validate or reject them. And this is true also for many other different research topics and areas. I believe that the authors should revise the presentation of their paper, acknowledging that a benchmark methodology is lacking and it is required, but without blaming the others because they did not develop it. Their goal was different.\n\nSecond, this work does not consider the whole family of poisoning attacks, but only \"targeted\" (or better, integrity) ones - this includes backdoor attacks or attacks aimed to misclassify only specific test samples, but not poisoning attacks that aim to indiscriminately increase the test error (i.e., availability / denial-of-service attacks). In addition, the whole paper only considers deep neural networks, and not other models. This should be thus clarified since the beginning in the paper, and better reflected also in the title. A clearer taxonomy of the whole family of data poisoning attacks should also be reported (e.g., in the form of a table to help understand the existing different types of threat in the area of data poisoning).\nI recommend the authors to refer to these papers that may help categorizing data poisoning attacks (and systematize nomenclature):\n- https://arxiv.org/abs/1910.03137\n- https://arxiv.org/abs/1712.03141\n- https://dl.acm.org/doi/10.1145/2046684.2046692\n\n\nThe main inconsistencies/issues identified by the authors in the evaluation of backdoor attacks are delineated in Sect. 4:\n1. Training without SGD or data augmentation;\n2. Victim architecture matters;\n3. \"Clean\" attacks are sometimes dirty;\n4. Proper transfer learning is less vulnerable;\n5. Performance is not invariant to dataset size;\n6. Black-box performance is low;\n7. Small sample sizes and non-random targets;\n8. Attacks are highly specific to the target image;\n9. Backdoor success depends on patch size.\n\nThese 9 causes, according to the authors of this work, hinder the impact of the 4 backdoor attacks (FC, CP, CLBD, HTBD) considered in this paper.\nI am quite convinced that in some specific cases, as the ones identified in Sect. 4, the attacks may fail, and I agree with the arguments posed by the authors in this section. \n\nI am only concerned by Issue no. 3 about the need of \"clean-label\" attacks in realistic settings. This is a common criticism/misconception also related to adversarial examples with imperceptible perturbations.\nWhy the perturbation should be required to be small? Are there practical scenarios where humans are going to observe the samples and be trained to detect that these samples are \"dangerous\"?\nAs the authors of this work seem to be quite concerned on the realism of these attacks, this point should be better discussed, as well as the need of considering the perturbation model to be l-inf with size 8/255 (or anyway fixed). In this respect, note that one more pertinent motivation for requiring small perturbations may be the detectability of the attack by automatic tools (rather than imperceptibility to the human eye); see, e.g., the discussion in https://arxiv.org/abs/1802.07295 and consider expanding the paper to discuss this issue. \n\nA final comment for the part destruens is that, eventually, it is not well systematized. Besides the 9 issues delineated above, a clear systematization/taxonomy of the potential issues is lacking. For example, issues can be related to the model (architecture), training algorithm (SGD/Adam, etc.), training hyperparameters, etc. Unfortunately, this step is lacking in the paper.\nAnd, as we will see, this impacts the development of a proper evaluation framework.\n\nAfter their part destruens, in Section 5, the authors move to the part construens of their argumentation, in which they propose a standardized benchmark for evaluation of clean-label and hidden trigger backdoor attacks. Again, I agree that providing a benchmark to assess poisoning attack effectiveness is a valuable contribution, and the authors have done a good job highlighting the factors which may impact the performance of these attacks.\n\nHowever, I am also concerned about the proposed benchmarking framework.\nIn particular, as the authors have shown, factors such as the type of the target model, the training dataset size, and the size of the perturbation that the attacker can inject into the poisoning samples substantially impact the attack effectiveness. However, in the proposed framework, those factors assume a single value that may unreasonably favor an approach rather than another. \nWhen a factor has a substantial impact on the results, it is recommended to analyze the performance when that factor assumes different values, as it is usually done for the size of the perturbation that an attacker can add to evasion samples (see, e.g., http://arxiv.org/abs/1902.06705). \nMore generally, a clear evaluation procedure or methodology is neither discussed nor provided, and this stems from the fact that a clear systematization of the causes of failure is lacking in the previous part of the paper.\nIf we identify, e.g., that the model architecture, the training algorithm and the perturbation size are all affecting the attack impact, a proper evaluation framework for attacks (and defenses) should then consider variants of all these factors, which means:\n- testing the attack/defense on different models;\n- (for each model) testing with different training algorithms;\n- (for each model, training algorithm) testing with different perturbation sizes.\nThis would indeed give a much more detailed understanding of how an attack/defense performs w.r.t. previously-proposed or existing ones.\n\nTo conclude, I mostly liked the idea presented in the paper, but a much higher level of systematization is required to propose a comprehensive framework for evaluation of poisoning attacks and defenses, as well as clarifying that the scope of the framework is also restricted to backdoor/integrity attacks on DNNs. I would anyway encourage the authors to continue working on this benchmark to make it more systematic, fairer and inclusive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9VhGi03_y6M", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses many issues in the data poisoning literature. They call into question the real world applicability of data poisoning attacks and discuss the short coming in the literature. They present several issues in the comparison of such attacks. Notably they show that many attributes of a data poisoning attack assumed to be irrelevant to the success of the attack are in fact statistically significant, such as a dataset size. Finally they present a new benchmark which attempts to standardize the comparison between data poisoning attacks.  \n\nI think this paper will benefit the data poisoning literature by illustrating its short comings and presenting a path forward. \n\n----------------------------\nSuggestions: \n\nin the '“Clean” attacks are sometimes dirty ' section the authors present two examples, but a human study could be set up in order to establish how many 'clean' attacks are actually identifiable. A small Mechanical turk study would greatly increase the strength of this argument and should not be financially prohibitive. \n\nThe section on 'Performance is not invariant to dataset size ' convinces the reader that this attribute thought to be irrelevant to the success of the attack is relevant, but proposes no mechanism by which the data set size could effect the attack. A discussion section attempting to provide an answer to this question would be interesting, but is perhaps out of scope.  \n\nThe section on data augmentation does not mention if the augmentations are also applied to the poisoned inputs. If the clean data is augmented but the poisoned data is not, it seems expected that the success of the attack would drop. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9DWRvIxVPQu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study a number of existing data poisoning attacks, ablating different design choices of these attacks and evaluating them on a common benchmark. They find that many of these attacks are quite brittle to changes in their original experimental evaluation and fail to generalize to more realistic/challenging scenarios. Then, they propose a unified benchmarks for evaluating these attacks.\n\nOverall, I found the paper quite interesting. As far as I can tell, the experimental methodology is solid and clearly demonstrates the brittleness of many of these attacks. Similarly, the benchmark proposed presents a reasonable and realistic setting to evaluate attacks on. I believe that such unified benchmarks are important for making progress in such an actively growing field. While early attack approaches serve as proof of concept for different vulnerabilities, understanding when these attacks fail and where should one focus to improve them is necessary to make progress.\n\nOther comments (not affecting score):\n- S2, 1st paragraph: Early backdoor attacks were not clean-label because they also explicitly flip the label of the poisoned examples.\n- S2, 3rd paragraph: Backdoor attacks cause the victim to misclassify everything _except_ the target class (adding the trigger causes the model to predict the target class).\n- The CLBD attack does examine a modified trigger that works in the presence of data augmentation.\n- Is there some intuition as to why the success rate of CP goes down when adding more training samples? If I understand correctly, the success of CLBD increases since the overall accuracy of the model improves with additional data (the model is more likely to rely on the trigger to perform well on the poisoned examples which, after all are inconsistent with the clean classifier).\n\n====== POST-RESPONSE UPDATE ======\nAfter reading the authors' responses and the rest of the reviews, I still stand by my original score.\n\nThat being said, I do agree with the other reviewers that the framing of certain discussion points could be improved to avoid misconceptions. Specifically:\n- There are places where the narrative feels accusatory towards earlier attacks (e.g., words like \"flaws\").\n- It would be good to explicitly state the scope of the paper relatively early (e.g., \"clean-label attacks on deep learning for images\") and explain this choice.\n- Clarify that this is not meant to be the ultimate poisoning benchmark but rather a first step that allows us to make progress in terms of attack development and evaluation.\n\nIt would be great if these could be incorporated in the next revision of the manuscript.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper generated significant discussion and division amongst the reviewers. On the positive side, some reviewers enjoyed both contributions, feeling the further empirical investigation of existing attacks to be interesting, and the creation of a benchmark to be very useful. On the negative side, no new positive results were proposed, criticism of previous attacks were considered to be unjust, the focus was somewhat narrow, and a benchmark could plausibly be misleading and detrimental.

Given the highly competitive nature of ICLR and the many other excellent submissions, the committee was unable to accept the paper at this time. Below are some suggestions for future submissions.

The content of the paper is generally fine, as long as the caveats and the "tone" are appropriate: we would hope to not mislead potential readers.
Here are some (strong) recommendations:
- Previous works were proof of concept attacks, and the authors should be careful to not frame them as being "broken" -- they perhaps were not meant to be robust to these modifications.
- The scope is somewhat narrow. There should be some explicitly statement and justification of the scope, and what in particular is *not* covered by the investigation.
- Importantly, a single benchmark can't be a unique gold standard, for many reasons discussed by reviewers. Please state these caveats clearly and prominently in the paper and/or code release, as otherwise the presence of a benchmark could do more harm than good. In particular, Reviewer 4 brought up the following philosophical concern with a benchmark, which I believe is quite reasonable, and I reproduce verbatim. The authors should try to address this in the next version: "This kind of benchmark can push the research to a wrong direction. In my view, the point of attacks are to create an alarm for using machine learning in critical applications. Developing these benchmarks would push the competition in the direction of making existing attacks "better" (whatever "better" means in the benchmark) instead of focusing on designing defense techniques or showing the severity of attacks in other situations. This benchmark could also have a bad effect on future attacks (attacks that want to show a new threat, not the one that try to improve the performance of clean label targeted poisoning attacks on deep neural nets) to gain attention from community as they probably will not pass all the criteria of this benchmark."

As another comment (I believe mentioned by other reviewers), it would be nice if all the terms and settings were defined clearly and precisely. For a benchmarking paper, it is important that the reader can clearly understand the threat model, and what does and does not count as a valid attack.

Finally, many of the reviewers gave detailed comments and concerns. The authors should please note and discuss these concerns in future versions (or at least in a supplement or an arXiv version).
Decision: Reject