Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching
Authors: Jonas Geiping
Liam H Fowl
W. Ronny Huang
Wojciech Czaja
Gavin Taylor
Michael Moeller
Tom Goldstein
Venue: ICLR 2021 Poster (2021)
Keywords: Data Poisoning
ImageNet
Large-scale
Gradient Alignment
Security
Backdoor Attacks
from-scratch
clean-label

Abstract: Data Poisoning attacks modify training data to maliciously control a model trained on such data.
In this work, we focus on targeted poisoning attacks which cause a reclassification of an unmodified test image and as such breach model integrity. We consider a
particularly malicious poisoning attack that is both ``from scratch" and ``clean label", meaning we analyze an attack that successfully works against new, randomly initialized models, and is nearly imperceptible to humans, all while perturbing only a small fraction of the training data. 
Previous poisoning attacks against deep neural networks in this setting have been limited in scope and success, working only in simplified settings or being prohibitively expensive for large datasets.
The central mechanism of the new attack is matching the gradient direction of malicious examples. We analyze why this works, supplement with practical considerations. and show its threat to real-world practitioners, finding that it is the first poisoning method to cause targeted misclassification in modern deep networks trained from scratch on a full-sized, poisoned ImageNet dataset.
Finally we demonstrate the limitations of existing defensive strategies against such an attack, concluding that data poisoning is a credible threat, even for large-scale deep learning systems.

Published as a conference paper at ICLR 2021
WITCHES' BREW: INDUSTRIAL SCALE DATA POISON- ING VIA GRADIENT MATCHING
Jonas Geiping*
Liam Fowl"
Dep. of Electr. Eng. and Computer Science
Department of Mathematics
University of Siegen
University of Maryland 1 1fowll @umddede
jonas. i pilipinngggn siegen. .e
W. Ronny Huang
Wojciech Czaja
Gavin Taylor
Department of Computer Science University of Maryland
Department of Mathematics University of Maryland wo wojtek @ma th. umd edu
Computer Science US Naval Academy tayl or uuana. edu
pronnyhuung@@mai com
Michael Moeller!
Tom Goldstein'
Dep. of Elect. Eng. and Computer Science
Department of Computer Science
University of Siegen
University of Maryland
michael.moellerreeeeee www.:moeeeeele..eee moleemmemeeeeeeee-eeeeeen de
tomg@urd und. edu
ABSTRACT
Data Poisoning attacks modify training data to maliciously control a model trained such data. In this work, we focus targeted poisoning attacks which cause a reclassification of an unmodified test image and as such breach model integrity. We consider particularly malicious poisoning attack that is both "from scratch" and "clean label", meaning we analyze an attack that successfully works against new, randomly initialized models, and is nearly imperceptible while perturbing only a small fraction of the training data. Previous poisoning attacks against deep neural networks in this setting have been limited in scope and success, working only in simplified settings or being prohibitively expensive for large datasets. The central mechanism of the new attack matching the gradient direction of malicious examples. We analyze why this works, supplement with practical considerations. and show threat real-world practitioners, finding that it is the first poisoning method to cause targeted misclassificatio in modern deep networks trained from scratch on a full-sized, poisoned ImageNet dataset. Finally demonstrate the limitations of existing defensive strategies against such an attack, concluding that data poisoning is a credible threat, even for large-scale deep learning systems.
humans, all
INTRODUCTION
Machine learning models have quickly become the backbone of many applications from photo processing mobile devices and ad placement security and surveillance (LeCun al.) 2015). These applications often rely on large training datasets that aggregate samples of unknown origins, and the security implications of this are not yet fully understood (Papernot) 2018). Data is often sourced in a way that lets malicious outsiders contribute to the dataset, such as scraping images from the web, farming data from website users, using large academic datasets scraped from social media (Taigman et al. 2014). Data Poisoning is security threat in which an attacker makes imperceptible changes data that then be disseminated through social media, or public datasets without being caught by human supervision. The goal of a poisoning attack is to modify the final model achieve malicious goal. In this work we focus on targeted attacks
devices,
Authors contributed equally.
Authors contributed equally.

---

Published as a conference paper at ICLR 2021
Step 1: Poison Dataset
Step 2: Train Victim Network from Scratch
Step 3: Victim is Fooled!
Good Dogs
0.1%
Clean Target Image
Small
IMAGENET
Deep Net
Prediction: Labrador Retriever X
Bad Dogs
The poisoning pipeline. Poisoned images (labrador retriever class) are inserted into dataset and cause a newly trained victim model to mis-classify a target (otter) image. We show successful poisons for threat model where 0.1% of training data is changed within an &oo bound of E == 8. Further visualizations of poisoned data can be found in the appendix.
Figure 1:
that achieve mis-classification of some predetermined target data as in Suciu et al. (2018) Shafahi al. (2018), effectively implementing a backdoor that is only triggered for a specific image. Yet. other potential goals of the attacker can include denial-of-service (Steinhardt et al. 2017 Shen et al. 2019). concealment of users (Shan et al. 2020), introduction of fingerprint information (Lukas
et al. 2020). These attacks are applied in scenarios such as social recommendation (Hu et al.
2019
content management (Li et al. 2016, Fang et al. 2018), algorithmic fairness (Solans et al. 2020)
and biometric recognition (Lovisotto et al. 2019 Accordingly, industry practitioners ranked data poisoning as the most serious attack on ML systems in a recent survey of corporations (Kumar et al. 2020).
We show that efficient poisoned data causing targeted misclassfication can be created even in the setting of deep neural networks trained large image classification tasks, such as ImageNet (Russakovsky et al., 2013). Previous work on targeted data poisoning has often focused on either linear classification tasks (Biggio et al. 2012 Xiao et al. 2015 Koh al. 2018) or poisoning of transfer learning and fine tuning (Shafahi et al. 2018 Koh & Liang 2017) rather than a full end-to-end training pipeline. Attacks deep neural networks (and especially on trained from scratch) have proven difficult in Muñoz Munoz-Gonzzáez González et al. (2017) and Shafahi et al. (2018). Only recently were targeted attacks against neural networks retrained from scratch shown to possible in Huang et al. (2020) for CIFAR-10 10 however with costs that render scaling to larger datasets, like the ImageNet dataset, prohibitively expensive.
We formulate targeted data poisoning as the problem of solving a gradient matching problem and analyze the resulting novel attack algorithm that scales to unprecedented dataset size and effectiveness. Crucially, the new poisoning objective is orders-of-magnitude more efficient than a previous formulation based on on meta-learning (Huang al. 2020) and succeeds more often. We conduct an experimental evaluation, showing that poisoned datasets created by this method robustly compromised and significantly outperform other attacks on CIFAR-10 on the benchmark of Schwarzschild et al. (2020). We then demonstrate reliably successful attacks common ImageNet models in realistic training scenarios. For example, the attack successfully compromises a ResNet-34 by manipulating only 0.1 0.1% of the data points with perturbations less than pixel values in formn We close by discussing previous defense strategies and how strong differential privacy (Abadi et al. 2016) is the only existing defense that partially mitigate the effects of the attack.
2 RELATED WORK
The task of data poisoning is closely related to the problem of adversarial attacks at test time, also referred to as evasion attacks (Szegedy et al. 2013 Madry et al., (017). where the attacker alters target test image to fool an already-trained model. This attack is applicable in scenarios where the attacker has control over the target image, but not over the training data. In this work we are specifically interested in targeted data poisoning attacks attacks which aim to specific target
2

---

Published as a conference paper at ICLR 2021
test image (or set of target test images) to be mis-classified. For example, an attack may cause certain target image of otter not part of the training set to be classified as a dog by victim models at test time. This attack is difficult to detect, because it does not noticeably degrade either training or validation accuracy (Shafahi et al., 2018 Huang et al. 2020) and is effectively invisible until is triggered. From security standpoint, these attacks break the integrity of machine learning model and are as such also called poison integrity attacks in Barreno et al. (2010) in contrast to poison availability attacks which reduce validation accuracy in general and are not a focus of this work.
In comparison to evasion attacks, targeted data poisoning attacks generally consider a setting where the attacker can modify training data within limits, but cannot modify test data and chooses specific target data a-priori. A related intermediary between data poisoning attacks we consider and evasion attacks are backdoor trigger attacks (Turner et al., 2018 Saha et al., 2019). These attacks involve inserting a trigger- often an image patch into training data, which is later activated by also applying the trigger to test images. Backdoor attacks require perturbations to both training and test-time data- the more permissive threat model is a trade-off that allows for unknown target images.
Two basic schemes for targeted poisoning are label flipping (Barreno et al.) 2010; Paudice et al. 2019), and watermarking (Suciu et al.) 2018; (Shafahi et al., 2018). In label flipping attacks, an attacker is allowed to change the label of examples, whereas in a tiatermarking attack, the attacker perturbs the training image, not label, by superimposing a target image onto training images. These attacks can be successful, yet they are easily detected by supervision such as Papernot & McDaniel (2018). This is in contrast to clean-label attacks which maintain the semantic labels of data.
Mathematically speaking, data poisoning is a bilevel optimization problem (Bard & Falk 1982 Biggio et al. 2012); the attacker optimizes image pixels enforce (malicious) criteria the resulting network parameters, which are themselves the solution to an "inner" optimization problem that minimizes the training objective. Direct solutions to the bilevel problem of data poisoning have been proposed where feasible, for example. SVMs in Biggio et al. (2012) or logistic regression in Demontis et al. (2019). However. direct optimization of the poisoning objective is intractable for deep neural networks because it requires backpropagating through the entire SGD training procedure, see Slunoz-Gonzálee et al. (2017). As such, the bilevel objective has to be approximated. Recently, MetaPoison (Huang et al. 2020) proposed to approximately solve the bi-level problem based on methods from the meta-learning community (Finn et al. 2017). The bilevel gradient is approximated by backpropagation through several unrolled gradient descent steps. This is the first attack to succeed against deep networks trained from scratch on CIFAR-10 10 as well as providing transferability to other models. Yet. Huaag et al. (2020) uses a complex loss function averaged over wide range of models trained to different epochs and a single unrolling step necessarily involves both clean and poisoned data, making it roughly costly one epoch of standard training. With an ensemble of 24 models, Huang et al. (2020) requires 3 (2 unrolling steps + clean update step) x 2 (backpropagation through unrolled steps) x 60 (first-order optimization steps) 24 (ensemble of models) equivalent epochs of normal training to attack, as well as EEin k = 253) epochs of pretraining. All in all, this equates to 8893 training epochs. While this can be mitigated by smart caching and parallelization strategies, unrolled ensembles remain costly.
In contrast to bilevel approaches stand heuristics for data poisoning of neural networks. The most prominent heuristic is feature collision, as in Poison Frogs (Shafahi et al. 2018), which seeks to cause target test image to be misclassified by perturbing training data to collide with the target image in feature space. Modifications surround the target image in feature space with a convex polytope (Zhu et al. 2019) collection of poisons (Aghakhani et al. 2020 and consider model ensembles (7hu et al. 2019). These methods are efficient, but designed to attack fine-tuning scenarios where the
feature extractor is nearly fixed and not influenced by poisoned data. When applied to deep networks
trained from scratch, their performance drops significantly.
3 EFFICIENT POISON BREWING
In this section, we will discuss intriguing weakness of neural network training based first-order optimization and derive an attack against it. This attack modifies training images that so they produce malicious gradient signal during training, even while appearing inconspicuous. This is done by matching the gradient of the target images within eod bounds. Because neural networks are trained by gradient descent, even minor modifications of the gradients can be incorporated into the final model.
3

---

Published as a conference paper at ICLR 2021
This attack compounds the strengths of previous schemes, allowing for data poisoning efficiently as in Poison Frogs (Shafahi et al. 2018). requiring only a single pretrained model and a time budget on the order of one epoch of training for optimization but still capable of poisoning the from-scratch setting considered in Huang et al. (020). This combination allow an attacker to "brew" poisons that successfully attack realistic models on ImageNet.
3.1 THREAT MODEL.
These discussed components of clean-label targeted data poisoning attack fit together into the following exemplary threat scenario: Assume a security system that classifies luggage images. An attacker wants this system to classify their particular piece of luggage, the target as safe, but can modify only small part of the training set. The attacker modifies this subset to be clean-label poisoned. Although the entire training set validated by human observer, the small subset of minorly modified images pass cursory inspection and receive their correct label. The security system trained on secretly compromised data, evaluated on validation data as normal and deployed. Until the target is evaluated and mis-classified as safe, the system appears to be working fine.
Formally, we define two parties, the attacker, which has limited control over the training data, and the victim, which trains model based on this data. We first consider gray-box setting, where the attacker has knowledge of the model architecture used by their victim. The attacker is permitted to poison fraction of the training dataset (usually less than 1%) by changing images within an so -ornm E-bound (e.g. with E < 16). This constraint enforces clean-label attacks, meaning that the semantic label of poisoned image is still unchanged. The attacker has no knowledge of the training procedure- neither about the initialization of the victim's 's model, nor about the (randomized) mini mini-batching and data augmentation that is standard in the training of deep learning models.
We formalize this threat model as bilevel problem for a machine learning model F(xx, 9) with inputs E Rn and parameters E IR, and loss function L. We denote the N training samples by (wi, y4 Bi) from which a subset of P samples are poisoned. For notation simplicity we assume the first P training images are poisoned by adding a perturbation A; to the hth training image. The perturbation is constrained to be smaller than in the &xo -orm. The task is to optimize A so that set of target samples (x yt)
T is reclassified with the new adversarial labels y!ah:
min Σ (F ((F(())(A))))))) ((A)), s.t. 0(A) E arg min AEC
1 N
L(F(x; (.,0).
0
We subsume the constraints in the set C ER main objective on the left the adversarial loss, and the objective that appears in the constraint on the right is the training loss. For the remainder, we consider a single target image (T = 1) as in Shafahi et al. 2018), but stress that this not a general limitation as shown in the appendix.
|||| 100 S = 0 Vi P}. We call the
3.2 MOTIVATION
What the optimal alteration of the training set that causes victim neural network F(z, 0) to mis-classify a specific target image s:²? We know that the expressivity of deep networks allows them to fit arbitrary training data (Zhang et al., 2016). Thus, if an attacker was unconstrained, a straightforward way to cause targeted missclassiiiiation of image is to insert the target image, with the incorrect label yu into the victim network's training set. Then, when the victim minimizes the training loss they simultaneously minimize the adversarial loss, based on the gradient information about the target image. In our threat model however, the attacker is not able to insert the mis-labeled target. They can, however, still mimic the gradient of the target by creating poisoned data whose training gradient correlates with the adversarial target gradient. If the attacker can enforce
1 E VoL(F(z. A., +A,0). P
VoL(F(E' 0), (0
to hold for any 9 encountered during training, then the victim's gradient steps that minimize the training loss on the poisoned data (right hand side) will also minimize the attackers adversarial loss on the targeted data (left side).
4

---

Published as a conference paper at ICLR 2021
3.3 THE CENTRAL MECHANISM: GRADIENT ALIGNMENT
Gradient magnitudes vary dramatically across different stages of training, and so finding poisoned images that satisfy eq. (2) for all 0 encountered during training is infeasible. Instead we align the target and poison gradients in the same direction, that is we minimize their negative cosine similarity, We do this by taking clean model F with parameters 6, keeping 9 fixed, and then optimizing
B 0) = -1 (Vo((F(2',)) y 3rdn), E VoL(F(X; + A).0).91)
(3)
(V yاdr EE VoL(F + A 9,,11
We optimize ((A) using signed Adam updates with decaying step size, projecting onto C after every step. This produces an alignment between the averaged poison gradients and the target gradient. In contrast to Poison Frogs, all layers of the network are included (via their parameters) in this objective, not just the last feature layer.
Each optimization step of this attack requires only a single differentiation of the parameter gradient w.r.t to its input compute the objective, instead of requiring the computation and evaluation of several unrolled steps as in MetaPoison to compute the objective. Furthermore, as in Poison Frogs we differentiate through a loss that only involves the (small) subset of poisoned data instead of involving the entire dataset, such that the attack is especially fast if the budget is small. As a side effect this also means that precise knowledge of the full training set is not required, only the poisoned subset and a trained parameter vector A. Finally. the method is able to create poisons using only a single parameter vector, 9 (like Poison Frogs in fine-tuning setting, but not the case for MetaPoison) and does not require updates of this parameter vector after each poison optimization step.
Remark. We find cosine similarity to be exceedingly effective for the classification models considered in this work. However, for small (and especially thin) models there exist regime which computing the matching term using squared Euclidean loss between gradients, directly optimizing eq. (2), is stronger attack. We conduct an ablation study in fig. 12 showing this for thin ResNets.
3.4 MAKING ATTACKS THAT TRANSFER AND SUCCEED "IN THE WILD"
practical and robust attack must be able poison different random initializations of network parameters and variety of architectures. To this end, we employ several techniques:
Differentiable Data Augmentation and Resampling: Data augmentation is standard tool in deep learning, and transferable image perturbations must survive this process. At each step minimizing eq. 3 we randomly draw a translation, crop, and possibly a horizontal flip for each poisoned image, then use bilinear interpolation to resample to the original resolution. When updating A, differentiate through this grid sampling operation as in Jaderberg et al. (2015) This creates an attack which is robust to data augmentation and leads to increased transferability.
Restarts: The efficiency we gained section 333 allows us to incorporate restarts, a common technique in the creation of evasion attacks (Qin al. 2019| Mosbach al.) (201)). We minimize eq. 33) several times from random starting perturbations, and select the set of poisons that give us the lowest alignment loss (()). This allows us to trade off reliability with computational effort.
Algorithm 1 Poison Brewing via the discussed approach.
1: Require Pretrained clean network {F(, 0)}. training set of images and labels (æ;, y1 3:) target (x", 1(dh),,,))))))))"))) < N poison budget, perturbation bound s, restarts R, optimization steps M
2: Begin
3: Select P training images with label ndv
4: For r 1,..., R restarts:
5: Randomly initialize perturbations A" C
6: For 1,..., M optimization steps:
7: Apply data augmentation to all poisoned samples (x; + AD
8: Compute the average costs, B(A", .0) as in eq. ( over all poisoned samples
9: Update A" with step of signed Adam and project onto ||''| I
Choose the optimal A as A with minimal value in B(
11: Return Poisoned dataset (z: + A W
5

---

Published as a conference paper at ICLR 2021
Model Ensembles: A known approach to improving transferability is to attack an ensemble of model instances trained from different initializations (Liu et (017] (Zuu et al. [019 Huang et al., (020).
However, ensembles highly expensive, increasing the pre- vre-training cost for only modest, but
stable, increase in performance.
We show the effects of these techniques via CIFAR-10 10 experiments (see table 1 and section 5.1) To keep the attack within practical reach, we do not consider ensembles for our experiments on ImageNet data, opting for the cheaper techniques of restarts and data augmentation. A summarizing description of the attack can be found in algorithm [ Lines 8 and 9 of algorithm | are done in a stochastic umini-batch) setting (which we omitted in algorithm for notation simplicity).
THEORETICAL ANALYSIS
Can gradient alignment cause network parameters to converge to a model with low adversarial loss? To simplify presentation, we denote the adversarial loss and normal training loss of eq. ① as (0) =: ((F((z), 0), (d and L()) n Σm= [(x;, y/ 0), respectively. Also, recall that 1-B( (A, 0h)) ) defined in eq. (3). measures the cosine similarity between the gradient of the adversarial
loss and the gradient normal training loss. We adapt a classical result of Zoutendijk (Nocedal &
Wright 2006 Thm. 3.2) to shed light on why data poisoning can work even though the victim only performs standard training on a poisoned dataset:
Proposition (Adversarial Descent). Let (udr(0) be bounded below and have a Lipschitz continuous gradient with constant L > 0 and assume that the victim model is trained by gradient descent with step sizes Og iee, 0 = 0² On VL(0)) If the gradient descent steps Og > 0 satisfy
(V((0 II
0(),0")) for some fixed 1, then Lado Lado(("). If in addition 3 > 0, Ko so that Vk 2
a&L < (1-1(1-A,))) <1
(4)
I Ladh (0*)
ku B(A, 8h) < 1 - E, then
lim 11 Laar - O. 10000
(5)
Proof. See supp. material.
Put simply, our poisoning method aligns the gradients of training loss and adversarial loss. This enforces that the gradient of the main objective is a descent direction for the adversarial objective, which, when combined with conditions on the step sizes, causes a victim to unwittingly converge to stationary point of the adversarial loss, i.e. optimize the original bilevel objective locally.
The strongest assumption in Proposition]] is that gradients are almost always aligned, B(A, 0%) < 1 e, k kip We directly maximize alignment during creation of the poisoned data, but only for selected 0", and not for all 0 encountered during gradient descent from any possible initialization. However, poison perturbations made from one parameter vector, 0, can transfer to other param- eter vectors encountered during training. For example, if one allows larger perturbations, and
Clean ResNet11
10-1
Poisoned ResNet18
10-2.
in the limiting case, unbounded perturbations, our objective is minimal if the poison data is identical to the target image, which aligns train- ing and adversarial gradients at every 9 encoun- tered. Empirically, we see that the proposed "poison brewing" attack does indeed increase gradient alignment. In fig. [ we see that in the first phase of training all alignments are positive, but only the poisoned model maintains pos- itive similarity for the adversarial target-label gradient throughout training. The clean model consistently shows that these angles are nega- tively aligned- i.e. normal training clean
10-3
0-
-10-5
-10-
-10-1
-100
5
10
15
20
25
30 35
40
Figure 2: Average batch cosine similarity, per epoch, between the adversarial gradient V V (0) and the gradient of each mini-batch V VC()) for poisoned and a clean ResNet- Crucially, the gradient alignment is strictly positive.

---

Published as a conference paper at ICLR 2021
Table 1: CIFAR-IO ablation. ε 16, budget is 1%. Differentiable data augmentation is able to replace large 8-model ensemble, without increasing computational effort.
Ensemble
Diff. Data Aug.
Victim does data aug.
Poison Accuracy (%(+SE))
100.00% (+0.00)
32.50% (#12.27)
78.75% (#11.77)
91 91.25% (+6.14)
dataset will increase adversarial loss. However, after the inclusion of poisoned data, the gradient alignment is modified enough to change the prediction for the target.
5 EXPERIMENTAL EVALUATION
We evaluate poisoning approaches in each experiment by sampling 10 random poison-target cases. We compute poisons for each and evaluate them on 8 newly initialized victim models (see supp. material Sec. A.1 for details of our methodology). We refer only the correct classification of each target as its adversarial class as a success and report as avg. poison success the average success rate over all 10 cases, each including 8 poisoned models. We apply algorithm [ with the following hyperparameters for all our experiments: T 0.1. R = 8, M = 250. We train victim models in realistic setting, considering data augmentation. SGD with momentum, weight decay and learning rate drops. Code for all experiments can be found at lilttps::.. //gi//giiit thub. com/ poonasGeiping boooiingggraiiett--nnnn
5.1 EVALUATIONS ON CIFAR-10
R=8,K=2
As a baseline on CIFAR-10, the inset figure (right) visualizes the number of restarts R and the number of ensembled models K, showing that the proposed method successful in cre- ating poisons even with just a single model (in- stead of an ensemble). The inset figure shows poison success versus time necessary to com- pute the poisoned dataset for a budget of 1%,
R=8, K=4
Avg. Poison Success
0.98
R=1,K=4
R=8/Kk
0.96
0.94
R=1
R-8 K-1
0.92
0.9
R=1 K-8
t=1 K=1
0.88
8 = 16 on CIFAR-10 for a ResNet-18, 18. We find that as the number of ensemble models, K, in- creases, it is beneficial increase the number of restarts as well, but increasing the number of restarts independently also improves perfor- mance. We validate the differentiable data augmentation discussed in section 3.4 table finding it crucial for scalable data poisoning, being efficient large model ensemble in facilitating robustness.
0.86
5
100
Time (Minutes)
Next, test different poisoning methods, fix our "brewing" framework efficient data poisoning. with only a single network and diff. data augmentation. We evaluate the discussed gradient matching cost function, replacing with either the feature-collision objective of Poison Frogs or the bullseye objective of Aghaahani et al. (202), thereby effectively replicating their methods, but in our context of from-scratch training.
The results of this comparison are collated in table 2 While Poison Frogs and Bullseye succeeded in finetuning settings, we find that their feature collision objectives are only successful in the shallower network in the from-scratch setting. Gradient matching further outperforms MetaPoison on CIFAR- 10, while faster (see appendix), in particular as K 24 for MetaPoison.
Benchmark results on 10: To evaluate our results against wider range of poison attacks,
we consider the recent benchmark proposed in Schwarzschild et al. (2020) in table S In the category "Training From Scratch". this benchmark evaluates poisoned CIFAR- 10 datasets with a budget of 1% and и against various model architectures, averaged over 100 fixed scenarios. We find that the discussed gradient matching attack, for K 1 is significantly more potent in the more difficult
7

---

Published as a conference paper at ICLR 2021
Table 2: CIFAR-I0 Comparison other poisoning objectives with a budget of 1% within our framework (columns to 3), for a 6-layer ConvNet and an 18-layer ResNet. MetaPoison" denotes the full framework of Huang et al. 2020). Each cell shows the avg. poison success and its standard error.
Proposed
Bullseye
Poison Frogs
MetaPoisson* 35.00% (=11.001) 42.50 (#8.33)
ConvNet (E = 32)
86.25% (#9.433
78.75% (+7.66)
52.50% (=1..8 1.25% (=1.19)
ResNet-18 16)
90.00% (#3.87)
3.75% (=3.5
Table 3: Results on the benchmark of Schwarzschild al. 2020 Avg. accuracy of poisoned CIFAR-
10 (budget 1%, & = 8) over 100 trials is shown. (*) denotes rows replicated from Schwarzschild et al. (2020). Poisons created with ResNet-18 except for the last row, where the ensemble consists of two models each architecture.
Attack
ResNet-18
MobileNet- V2
VGG11
Average
Poison Frogs* (Shafahi et al.
2018
0%
1%
3%
1.33%
Convex Polytopes (Zhuu et al.
2019
0%
1%
1%
0.67%
Clean-Label Backd. * (Trner et
2018
0%
1%
2%
1.00%
Hidden- Hidden-Trigger Backd. (Saha et al.
2019
0%
4%
1%
2.67%
Proposed Attack (K
45%
36%
8%
29.67%
Proposed Attack (K' = 4)
55%
37%
7%
33.00%
Proposed Attack (K 6, Heterogeneous)
49%
38%
35%
40.67%
benchmark setting. An additional feature of the benchmark is transferability. Poisons are created using ResNet-18 18 model, but evaluated also two other architectures. We find that the proposed attack transfers to the similar MobileNet-V2 architecture, but not as well to VGG11. However, we also show that this advantage can be easily circumvented by using an ensemble of different models as in Zhu et al. (2019). If we use an ensemble of K = 6, consisting of ResNet-18, 2 MobileNet-V and 2 VGGI I models (last row), then the same poisoned dataset can compromise all models and generalize across architectures.
5.2 POISONING IMAGENET MODELS
The ILSVRC2012 challenge, "ImageNet", consists of over million training examples, making it in- feasible for most actors to train large model ensembles or run extensive hyperparameter optimizations. However, the new gradient matching attack requires only single sample of pretrained parameters 0, and operates only on the poisoned subset, it can poison ImageNet images using publicly available pretrained models without ever training an ImageNet classifier. Poisoning ImageNet with previous methods would be infeasible. For example, following the calculations in section 2 it would take over 500 GPU days (relative to our hardware) to create poisoned ImageNet for a ResNet- 18 via MetaPoison. In contrast, the new attack can poison ImageNet in less than four GPU hours.
Figure shows that standard ImageNet models trained from scratch on poisoned dataset "brewed" with the discussed attack, are reliably compromised with examples of successful poisons shown (left). We first study the effect of varying poison budgets, and e-bounds (top right). Even a budget of 0.05% and e- b-bound of 8, the attack poisons randomly initialized ResNet- 18 80% of the time. These results extend to other popular models, such as MobileNet-v2 and ResNet50 (bottom right). Poisoning Cloud AutoML: To verify that the discussed attack can compromise models in practically relevant black-box setting, we test against Google's ' Cloud AutoMI. This a cloud framework that provides access black-box ML models based an uploaded dataset. In [Huang al.) (2020) Cloud AutoML was shown to be vulnerable for CIFAR-10. We upload a poisoned ImageNet dataset 32) for our first poison-target test case and upload the dataset.
(base: ResNet18. budget 0.1%, Even in this scenario, the attack is measurably effective, moving the adversarial label into the top-5 predictions of the model in of runs, and the top-1 prediction in 1 out of 5 runs.
5.3 DEFICIENCIES OF DEFENSE STRATEGIES
Previous defenses against data poisoning (Steinhardt et al., 2017 Paudice et al. 2018 Peri et al.) 2019) have relied mainly data sanitization, i.e. trying to find and remove poisons by outlier
8

---

Published as a conference paper at ICLR 2021
P:16
b-d
brn0ts
VOG1S
ReaNe50
Architectune
Figure 3: Poisoning ImageNet. Left: Clean images (above), with their poisoned counterparts (below) from successful poisoning of randomly initialized ResNet- 18 trained ImageNet for poison budget of 0.1% and an &o bound of 8. Right Top: ResNet- 18 results for different budgets and varying g-bounds. Right Bot.; More architectures (Simonyan & Zisserman 2014 He al. 2015
Sandler et al. 2018) with budget of 0.1% and = = 16.
T6tMl pRIDe CR5S
botal paison
0.9
polcree
Validation Accuraty
Posso Success
0.9
CouPT (00ormaaeee
0.6
0.85
0.2
0.75
Acturwy
Poson Sucoess (Defanse Knowledee)
10
100
0.001
0.01
0.6
00
0.5
21.2
25.5
27.0 27.5
Gradient Noise
7. /dSdaacce class.class cendriid
1 ddddacceet ree inovee feat.re
(a) Feature space distance to base class centroid, and target image feature, for victim model CIFAR-10. 4.0% budget, 16, showing sanitization defenses failing and no feature collision as in Poison Frogs.
(b) Defending through differential privacy. CIFAR-I1 10,
1% budget, 16, ResNet-18. Differential privacy is only able to limit the success poisoning via trade-off' with significant drops accuracy.
Figure 4: Defense strategies against poisoning.
detection (often in feature space). We demonstrate why sanitization methods fail in the face of the attack discussed in this work in fig. 4a Poisoned data points are distributed like clean data points, reducing filtering based methods almost-random guessing (see supp. material, table 6).
Differentially private training is a different defense. diminishes the impact of individual training samples, in turn making poisoned data less effective (Ma et al.) 2019 Hong et al. 2020). However, this come at a significant cost. Figure 4b shows that to push the Poison Success below 15%, one has sacrifice over 20% validation accuracy, model even more challenging. From this aspect, differentially private training can be compared adversarial training (Madry al. (017) against evasion attacks. Both methods can mitigate the effectiveness of an adversarial attack, but only by significantly impeding natural accuracy.
CIFAR- 10. Training diff. private ImageNet
CONCLUSION
We investigate targeted data poisoning via gradient matching and discover that this mechanism allows for data poisoning attacks against fully retrained models that unprecedented in scale and effectiveness. We motivate the attack theoretically and empirically, discuss additional mechanisms like differentiable data augmentation and experimentally investigate modern deep neural networks in realistic training scenarios, showing that gradient matching attacks compromise even models trained ImageNet. We close with discussing the limitations of current defense strategies.

---

Published as a conference paper at ICLR 2021
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS 16, pp. 308-318, Vienna, Austria, October 2016. Association for Computing Machinery. ISBN 978-1-4503-413944 doi: 00.144//9966499.9788318
Hojjat Aghakhani, Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye Polytope: A Scalable Clean-Label Poisoning Attack with Improved Transferability. arXiv:2005.0019/ [es, stat), April 2020.
Jonathan F. Bard and James E. Falk. An explicit solution to the multi-level programming problem. Computers & Operations Research, 9(1::77-100, January 1982. ISSN 0305-0548. doi: 10.1016/ 038550054862))9000777
Marco Barreno, Blaine Nelson, Anthony D. Joseph, and D. Tygar. The security of machine learning. Mach. Learn., 81(2):221-14, November 2010. ISSN 0885-6125. doi: 10.1007/
510994-0010-588-5.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Machines. ArXiv12066389 Cs Stat, June 2012.
Nicholas Carlini and David Wagner. Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec '17 pp. 3-14, Dallas, Texas, USA, November 2017. Association for Computing Machinery. ISBN 178-1-4503-5202-44 doi: 10.114553128572.114000
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input Similarity from the Neural Network Perspective. In Advances in Neural Information Processing Systems 32, pp-
5342-5351. Curran Associates, Inc., 2019.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotare and Fabio Roli. Why Do Adversarial Attacks Transfer? Explaining Trans- ferability of Evasion and Poisoning Attacks. In 28th (USENIX) Security Symposium (/USENIX) Security 19), pp. 321-338, 2019. ISBN 978- 1-939133-06-9.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Rotation and Translation Suffice: Fooling CNNs with Simple Transformations. ArXiv171202777 Cs Stat, December 2017.
Minghong Fang, Guolei Yang. Neil Zhenqiang Gong, and Jia Liu. Poisoning Attacks Graph-Basee Recommender Systems. In Proceedings of the 34th Anmual Computer Security Applications Conference, ACSAC '18, pp. 381-392, San Juan, PR, USA, December 2018. Association for Computing Machinery. ISBN 978-1 178-1-45033-569--7 doi: 10. 10.1145/32749943377766
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. ArXiv/70303400 Cs, March 2017.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting Gradients How easy is it to break privacy in federated learning? ArXiv2003 1405 Cs, March 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. ArX13151203385 Cs, December 2015.
Wen Heng, Shuchang Zhou, and Tingting Jiang. Harmonic Adversarial Attack Method. ArXiv180710590 Cs, July 2018.
Sanghyun Hong, Varun Chandrasekaran, Yigitcan Kaya, Tudor Dumitraş, and Nicolas Papernot. On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping. ArXiv200211497 Cs, February 2020.
Rui Hu, Yuanxiong Guo, Miao Pan, and Yanmin Gong. Targeted Poisoning Attacks Social Recommender Systems. In 2019 IEEE Global Communications Conference (GLOBECOM), pp. 1-6, December 2019. doi: 10. 00B.C/.0.BBCO0884470099900133350
10

---

Published as a conference paper at ICLR 2021
W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. MetaPoison: Clean-label Data Poisoning. ArXiv200400225 Cs Stat, April 2020.
Practical
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial Transformer Networks. In Advances in Neural Information Processing Systems 28, pp. 2017-2025. Curran Associates, Inc., 2015.
Danny Karmon, Daniel Zoran, and Yoav Goldberg. LaVAN: Localized and Visible Adversarial Noise. A812v180102608 Cs, January 2018.
Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. In International Conference Machine Learning, pp- 1885-1894, July 2017.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger Data Poisoning Attacks Break Data Sanitization Defenses. ArXiv181100741 Cs Stat, November 2018.
Alex Krizhevsky, lly Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial Machine Learning - Industry Perspectives. ArXiv200205646 Cs Stat, May 2020.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):43-4444 May 2015. ISSN 1476-4687. doi: 10.1038/nature145539
Bo Li, Yining Wang, Aarti Singh, and Yevgeniy Vorobeychik. Data Poisoning Attacks Factorization- Based Collaborative Filtering. In Advances in Neural Information Processing Systems 29, pp. 1885-1893. Curran Associates, Inc., 2016.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into Transferable Adversarial Examples and Black-box Attacks. ArXiv161 102770 Cs, February 2017.
Giulio Lovisotto, Simon Eberz, and Ivan Martinovic. Biometric Backdoors: A Poisoning Attack Against Unsupervised Template Updating. ArXiv190509162 Cs, May 2019.
Nils Lukas, Yuxuan Zhang, and Florian Kerschbaum. Deep Neural Network Fingerprinting by Conferrable Adversarial Examples. ArXiv Cs Stat, February 2020.
Yuzhe Ma, Xiaojin Zhu, and Justin Hsu. Data Poisoning against Differentially-Priin Privatt Learners: Attacks and Defenses. ArXiv190309860 Cs, July 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. ArXiv/7060608: Cs Stat, June 2017.
Marius Mosbach, Maksym Andriushchenko, Thomas Trost, Matthias Hein, and Dietrich Klakow, Logit Pairing Methods Can Fool Gradient-Baad -aased Attacks. ArXiv/810/2042 Cs Stat, March 2019.
Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, and Fabio Roli. Towards Poisoning of Deep Learning Algorithms with Back- gradient Optimization. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, AlSec '17, pp. 27-38, New York, NY, USA, 2017. ACM. ISBN 978- 100-----3--222-2-2 doi: 10.11455/123572.314445..
Luis Muñoz-González, Bjarne Plitzner, Matteo Russo, Javier Carnerero- and Emil C. Lupu. Poisoning Attacks with Generative Adversarial Nets. ArXiv190607773 Cs Stat, June 2019.
Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer Series in Operations Research. Springer, New York, 2nd edition, 2006. ISBN 978-0-387-30303-1
Nicolas Papemot. A Marauder's Map of Security and Privacy in Machine Learning. ArX1v181101134 Cs, November 2018
11

---

Published as a conference paper at ICLR 2021
Nicolas Papernot and Patrick McDaniel. Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. A Cs Stat, March 2018.
Andrea Paudice, Luis Muñoz-Gonzälez, Andras Gyorgy, and Emil C. Lupu. Detection of Adversarial Training Examples in Poisoning Attacks through Anomaly Detection. ArXiv/8020304] Cs Stat, February 2018.
Andrea Paudice, Luis Muñoz-GGnzález, and Emil C. Lupu. Label Sanitization Against Label Flipping Poisoning Attacks. In ECML PKDD 2018 Workshops, Lecture Notes Computer Science, pp. 5-15, Cham, 2019. Springer International Publishing. ISBN 978-3-033-13445-2. doi: 10.1007/978- 3-030- 13453-2_1.
Neehar Peri, Neal Gupta, W. Ronny Huang. Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John P. Dickerson. Deep k-nn defense against clean-label data poisoning attacks, 2019.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial Robustness through Local Lincarization. ArXiv190702610 Cs Stat, October 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Int Compur Vis, 115(3):211252. December 2015.
ISSN 1573-1405. doi: 10. 12300/510
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden Trigger Backdoor Attacks. ArXiv/91000033 Cs, December 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bileNetV2: Inverted Residuals and Linear Bottlenecks. ArXiv/8010438/ Cs, January 2018.
Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P. Dickerson, and Tom Goldstein. Just How Toxic is Data Poisoning? Unified Benchmark for Backdoor and Data Poisoning Attacks. ArXiv2006/2557 Cs Stat, June 2020.
Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. ArXiv180400792 Cs Stat, April 2018.
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Fawkes: Protecting Personal Privacy against Unauthorized Deep Learning Models. ArXhv200208327 Cs Stat, February 2020.
J. Shen, X. Zhu, and D. Ma. TensorClog: An Imperceptible Poisoning Attack on Deep Neural Network Applications. IEEE Access, 7:41498-41500 2019. ISSN 2169-3536. doi: 10.1 SCCESSS2019,290599)
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. ArXiv14091550 Cs, September 2014.
David Solans, Battista Biggio, and Carlos Castillo. Poisoning Attacks on Algorithmic Fairness. ArXiv200407400 CsLG, April 2020.
Jacob Steinhardt, Pang Wei W Koh, and Perey S Liang. Certified Defenses for Data Poisoning Attacks. In Advances in Neural Information Processing Systems 30, pp. 3517-3529, Curran Associates, Inc., 2017.
Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daume lii, and Tudor Dumitras. When Does Machine Learning [FAIL]}? Generalized Transferability for Evasion and Poisoning Attacks. In 27th (USENIX) Security Symposium (/USENIX) Security 18), pp. 1299-1316, 2018. ISBN 978-1-939133-2
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. In 1312::3312.6699 [Cs], December 2013.
12

---

Published as a conference paper at ICLR 2021
Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, and Lior Wolf. DeepFace: Closing the Gap to Human-Level Performance in Face Verification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701-1708, Columbus, OH, USA, June 2014. IEEE. ISBN 978-1-4799-51 18-5. doi: P0.100//C
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-Label Backdoor Attacks. openre- view, September 2018.
Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is Feature Selection Secure against Training Data Poisoning? In International Conference on Machine Learning, pp. (689-1698, June 2015.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 10X5v161103530 Cs, November 2016.
Hanwei Zhang, Yannis Avrithis, Teddy Furon, and Laurent Amsaleg. Smooth Adversarial Examples. ArXiv/90311862 Cs, March 2019.
Chen Zhu, W. Ronny Huang, Ali Shafahi, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable Clean-Label Poisoning Attacks on Deep Neural Nets. ArXiv190505897 Cs Stat, May 2019.
ACKNOWLEDGEMENTS
We thank the University of Maryland Institute for Advanced Computer Studies, specifically Matthew Baney and Liam Monahan, for their help with the Center for Machine Learning cluster.
This work supported by DARPA's GARD, QED4RMI,, and Young Faculty Award program. Additional support was provided by the National Science Foundation Directory of Mathematical Sciences. GT supported by ONR grant N0001420W tion Program. WC is supported by NSF DMS 1738003 and MURI from the Army Research Office Grant No. W91
well as the DoD HPC Moderniza-
A REMARKS
Remark (Validating the approach in special case). Inner-produce loss functions like eq. (3) work well in other contexts. In Geiping et al. (2020). cosine similarity between image gradients was minimized uncover training images used in federated learning. If we disable our constraints, setting e 255, and consider a single poison image and single target, then we minimize the problem of recovering image data from a normalized gradient special case. In Geiping et al.) (2020). it was shown that minimizing this problem can recover the target image. This means that we can indeed return to the motivating case in the unconstrained setting the optimal choice of poison data insertion of the target image in an unconstrained setting for one image.
Remark (Transfer of gradient alignment). An analysis of how gradient alignment often transfers between different parameters and even between architectures has been conducted, c.g. in Charpiat al. (2019); Koh & Liang (0117) and Demontis et al. (001)). It was shown in Demontis al. (2019) that the performance loss when transferring an evasion attack to another model is governed by the gradient alignment of both models. In the same vein, optimizing alignment appears to be useful metric in the case of data poisoning. Furthermore Hong et al.) (2020) note that previous poisoning algorithms might already cause gradient alignment side effect, even without explicitly optimizing for it.
Remark (Poisoning is Credible Threat to Deep Neural Networks). It is important to understand the security impacts of using unverified data sources for deep network training. Data poisoning attacks up to this point have been limited in scope. Such attacks focus on limited settings such as poisoning SVMs, attacking transfer learning models, or attacking toy architectures (Biggio et al. al. 2019 Shafahi et al.) 2018). We demonstrate that data poisoning poses a threat to large-scale systems as well. The approach discussed in this work pertains only to the classification scenario, as a guinea pig for data poisoning. but upplications to variety of scenarios of practical interest have been considered in the literature, for example spam detectors mis mis-classifying a spam email benign, or poisoning face unlock based mobile security systems.
2012; Muñoz-Gonzzine
13

---

Published as a conference paper at ICLR 2021
The central message of the data poisoning literature can be described as follows: From security perspective, the data that is used to train a machine learning model should be under the same scrutiny as the model itself. These models can only be secure if the entire data processing pipeline is secure. This issue further cannot easily be solved by human supervision (due to the existence of clean-label attacks) or outlier detection (see fig. [4a). Furthermore, targeted poisoning is difficult detect as validation accuracy is unaffected. As such, data poisoning is best mitigated by fully securing the data pipeline.
So far we have considered data poisoning from the industrial side. From the perspective of a user, or individual under surveillance, however, data poisoning can be means of securing personal data shared on the internet, making it unusable for automated ML systems. For this setting, we especially refer to an interesting application study in [Shan al. (2020) in the context of facial recognition.
B EXPERIMENTAL SETUP
This appendix section details our experimental setup for replication purposes, A central question in the context of evaluating data poisoning methods is how to judge and evaluate "average" performance. Poisoning is in general volatile with respect to poison-target class pair, and to the specific target example, with some combinations and target images being in general easier to poison than others. However, evaluating all possible combinations is infeasible for all but the simplest datasets, given that poisoned data has to created for each example and then neural network has be trained from scratch every time. Previous works (Shafahi et al., 2018 Zhu et al., 2019) have considered select target pairs, e.g. "birds-dogs" and "airplanes-fross but this runs the risk of mis-estimating the overall success rates. Another source of variability arises, especially in the from-scratch setting: Due to both the randomness of the initialization of the neural network, the randomness of the order in which images are drawn during mini-batch SGD, and the randomness of data augmentations, fixed poisoned dataset might only be effective some of the time, when evaluating it multiple times.
In light of this discussion, we adopt the following methodology: For every experiment randomly select n (usually 10 in our case) settings consisting of random target class, random poison class, random target and random images to be poisoned. For each of these experiments we create single poisoned dataset by the discussed or a comparing method within limits of the given threat model and then evaluate the poisoned datasets mn times (8 for CIFAR-10 10 and for ImageNet) on random re-initializations of the considered architecture. To reduce randomness for a fair comparison between different runs of this setup, fix the random seeds governing the experiment and rerun different threat models or methods with the same random seeds. We have used CIFAR- 10 with random seeds 1000000000-1111111 2000000000-2111 11111100011111111111111111111111 for CIFAR- 10 experiments and 10000000001111111111
hyperparameter tuning and now evaluate on random seeds for ImageNet, with class pairs and target image IDs for reproduction given in tables and 5 For CIFAR-10, 10, the target ID refers to the canonical order of all images in the dataset as downloaded from https 7/www. cs toronto...............oo www.orottouuuuoeeoooooooo edu/ ci far html) ;; for ImageNet, the ID refers to an order of ImageNet images imagge where the syn-sets ordered by their increasing numerical value (as is the default in torchvis sion). However for future research we encourage the sampling of new target-poison pairs to prevent overfitting, ideally even in larger numbers given enough compute power.
For every measurement of avg. poison success in the paper, we measure in the following way: After retraining the given deep neural network to completion, measure if the target image is successfully classified by the network as its adversarial class. We do not count mere misclassification of the original label (but note that this usually happens even before the target is incorrectly classified by the adversarial class). Over the m validation runs we repeat this measurement of target classification success and then compute the average success rate for a single example. We then aggregate this average over our 10 chosen random experiments and report the mean and standard error of these average success rates as avg. poison success. All error bars in the paper refer to standard error of these measurements.
B. HARDWARE
We use a heterogeneous mixture of hardware for our experiments. CIFAR-10, and a majority of the ImageNet experiments, were on NVIDIA GEFORCE RTX 2080 Ti gpus. CIFAR-10 experiments
14

---

Published as a conference paper at ICLR 2021
Table 4: Target/poison class pairs generated from the initial random seeds for ImageNet experiments.
Target ID relative to CIFAR-10 validation dataset.
Target Class
Poison Class
Target ID
Random Seed
dog
frog
8745
2000000000
frog
truck
1565
2100000000
frog
bird
2138
2110000000
airplane
dog
5036
2111000000
airplane
ship
1183
2111100000
cat airplane
7352
2111110000
automobile frog
3544
2111111000
truck cat
3676
2111111100
automobile
ship 9882
2111111110
automobile cat 3028
2111111111
Table 5: Target/poison class pairs generated from the initial random seeds for ImageNet experiments.
Target Id relative to ILSVRC2012 validation dataset Russakovsky et al. (0015)
Target Class
Poison Class
Target ID
Random Seed
otter
Labrador retriever
18047
1000000000
warthog
bib
17181
1100000000
orange
radiator
37530
1110000000
theater curtain
maillot
42720
1111000000
hartebeest
capuchin
17580
1111100000
burrito
plunger
48273
1111110000
jackfruit
spider web
47776
1111111000
king snake
hyena
2810
1111111100
flat-coated retriever
alp
10281
1111111110
window screen
hard disc
45236
1111111111
were run on gpu, while ImageNet experiments were run on gpus. We also use NVIDIA Tesla P100 gpus for some ImageNet experiments. All timed experiments using 2080 Ti gpus.
B.2 MODELS
For our experiments on CIFAR-10 in section 5 we consider two models. In table 2, the "6-layer ConvNet", in close association with similar models used in Finn et al.] (2017) Krizhevsky et al.) (2012), we consider architecture of 5 convolutional layers (with kernel size 3), followed by linear layer. All convolutional layers are followed by a ReLU activation. The last two convolutional layers followed by max pooling with size 3. The output widths of these layers are given by 64, 128, 126, 256, 2304. In tables 1, 2, in the inset figure and Fig. 4 we consider a ResNet- model. We make the customary changes the model architecture for CIFAR-10, replacing the stem of the original model (which requires ImageNet-sized images) by a convolutional layer of size following by batch normalization and a ReLU. This is effectively equal to upsampling the CIFAR-I0 images before feeding them into the model. For experiments on ImageNet, we consider ResNet-18, ResNet-34 (He et al. 2015), MobileNet-v2 (Sandler et al. 2018 and VGG-16 (Simonyan & Zisserman 2014) in standard configuration.
We train the ConvNet, MobileNet-v2 and VGG-16 with initial learning rate 0.01 and the residual architectures with initial learning rate 0.1. We train for 40 epochs, dropping the learning rate by factor of 10 at epochs 14, 24, 35. We train with stochastic mini-bateh gradient descent with Nesterov momentum, with batch size 128 and momentum 0.9. Note that the dataset is shuffled in each epoch, so that where poisoned images appear in mini-batches is random and not known to the attacker. We add weight decay with parameter 5 x 10-4 For CIFAR-10 horizontal flipping with probability 0.5 and random crops size 32 x 32 with zero- aao-padding of 4. For ImageNet resize all images to 256 256 and crop the central 224 x 224 pixels. We also consider horizontal flipping with probability 0.5, and data augmentation with random crops of size 224 x 224 with zero- prrddpddding of 28.
add data augmentations using
15

---

Published as a conference paper at ICLR 2021
When evaluating ImageNet poisoning from-scratch use the described procedure. To cre- ate our poisoned datasets as detailed in Alg. 1, we download the respective pretrained model from t torchvision, see http:/ttts//pttrreeeeeeeee http://.........oooo/////a////te
models..ee html
B.3 CLOUD AUTOML SETUP
For the experiment using Google's cloud autoML, we upload a poisoned ILSVRC2012 dataset into google storage, and then use pettps:/ccood..ooe google. com/ vision. http://siioo//ttooo.lo. om. to train classification model. Due to autoML limitations examples from each class (reaching a training set size slightly smaller than 950 000, which allows for an upload of the 50 000 validation images). We use ResNet- 18 model as surrogate for the black-box learning within autoML, pretrained autoML dataset and specify the vision model to be mobile le e high-accurace accuracy- which we train to 10 000 milli-node hours, five times. After training the model, evaluate its performance on the validation set and target image. The trained models all reach a 69% clean top- accuracy on the ILSVRC2012 validation set.
million images, we only upload up to 950
the full ILSVRC201 as before. We create MULTICLASS
C PROOF OF PROPOSITION 1
Proof of Prop. I. Consider the gradient descent update
gh+1 = l* ag VL((t)
Firstly, due to Lipschitz smoothness of the gradient of the adversarial loss Cadv we can estimate the value O++1 by the descent lemma
(0 S Leai (0t) (ara VLar VL((0)) a
If we further the cosine identity:
Lad (0*), VC(O")) = |VV(0> (0th( (oo0)/
denoting the angle between both vectors by 7² we find that
Ladv (Bb+1) < Lady (B+)
|| (05)) + +
(ax
L (0*)1
= Ladr (
|| cos(T*) az. |/
akk
As such, the adversarial loss decreases for nonzero step sizes if
cos(7*) > aq L
i.c.
| Ladv (0h)) cos(>^)
L3 <
|
for some xc. This follows from our assumption the parameter 33 in the statement of the proposition. Reinserting this estimate into the descent inequality reveals that
cos(>*))
Cutv (Bh+2) (8t)
|
C'L
for 1 2. Due monotonicity we may sum over all descent inequalities, yielding
Latr (8")
Latt (0*+1)=
1
Σ VV )|2 cos(7²)
CLL
5=0
As Lady is bounded below, we may consider the limit of k: to find
v Ladd (f0)))) oc.
16

---

Published as a conference paper at ICLR 2021
If for all, except finitely many iterates the angle between adversarial and training gradient is less than 90", i.. cos(?") is bounded below by some fixed 0, as assumed, then the convergence to stationary point follows:
lim Vadr )1 +
In fig. 5 we visualize measurements of the computed bound from an actual poisoned training. The classical gradient descent converges only ag < so we can find an upper bound this value by 1, even if the actual Lipschitz constant of the neural network training objective is not known to us.
Poisoned ResNet18
3.5
Clean ResNet18
Lower Bound
Bound value
2.5
0.5
00
10
30
Epochs
Figure 5: The bound considered in Prop. 1, evaluated during training of poisoned and clean model, using practical estimation of the lower bound via ar L 1. This an upper bound ox L as ag < T. is necessary for the convergence of (clean) gradient descent.
D POISONED DATASETS
We provide access to poisoned datasets as part of the supplementary material, allowing for replication of the attack. To save space however, we provide only the subset of poisoned images and not the full dataset. We hope that this separation also aids in the development of defensive strategies. To train model using these poisoned data points, you can use our code (using -save full) our your own to export either CIFAR- 10 or ImageNet into an image folder structure, where the clean images can then be replaced by poisoned images according to their ID. Note that the given IDs refer to the dataset ordering as discussed above.
E VISUALIZATIONS
We visualize poisoned sample from our ImageNet runs in figs. 6 and 7 label" effect. Poisoned data is only barely distinguishable from clean data, even in the given setting where the clean data shown to the observer. subset of poisoned images used to poison Cloud autoML with 32 can be found in fig.
noting especially the "clean
realistic setting, this is significantly harder. A
We concentrate only on small f perturbations to the training data as this is the most common setting for adversarial attacks. However, there exist other choices for attacks in practical settings. Previous works have already considered additional color transformations (Huang et al., 2020) or watermarks (Shatahi al. 2018). Most techniques that create adversarial attacks at test time within various
17

---

Published as a conference paper at ICLR 2021
Figure 6: Clean images (above), with their poisoned counterparts (below) from successful poisoning of ResNet-18 model trained on ImageNet. The poisoned images (taken from the Labrador Retriever class) successfully caused missclassification of target (otter) image under a threat model given by budget 0.1 % and foo bound of 8.
Figure Clean images (above), with their poisoned counterparts (below) from a successful poisoning of a randomly initialized ResNet- 18 trained on ImageNet. The poisoned images (taken from the Labrador Retriever class) successfully caused mis-classification of target (otter) image under threat model given by a budget of 0.1% and an foo bound of 16.
constraints (Engstrom et al. 2017 Zhang et al., 2019 Heng et al. 2018 Karmon et al.) 2018) are likely transfer into the data poisoning setting. Likewise, do not consider hiding poisoned images further by minimizing perceptual scores and relate to the large literature of adversarial attacks that evade detection (Carlini & Wagner 2017).
In fig. we visualize how the adversarial loss and accuracy behave during an exemplary training run, comparing the adversarial label with the original label of the target image.
F ADDITIONAL EXPERIMENTS
This section contains additional experiments.
FULL-SCALE METAPOISON COMPARISONS ON CIFAR-10
Removing all constraints for time and memory, visualize time/accuracy of approach against other poisoning approaches in fig. 10 Note that attacks, like MetaPoison, which succeed on CIFAR- 10 only after removing these constraints, cannot be used on ImageNet-sized datasets due to the
18

---

Published as a conference paper at ICLR 2021
Figure 8: Clean images (above), with their poisoned counterparts (below) from successful poisoning of a Google Cloud AutoML model trained on ImageNet. The poisoned images (taken from the I Labrador Retriever class) successfully caused mis-classification of a target (otter) image. This is accomplished with poison budget of 0. 0.1% and an foo bound of 32 the black-box attack against autoML requires an increased perturbation magnitude, in contrast to the other gray-box experiments this work.
Prisoned ResNiet Cean ResNec18
Prisored Reshett8
Adversarial Loss (Log.
Original Target Loss (Logg)
ResNett8
0.01
0.00
100g
Epochs
Epochs
Polsoned Reshiet!
Polsoned Reshiet1
Cean
Claan
Adversarial Sasssarlllccc Acc.
0.8
0.8
Target Acc.
0.5
0.6
0.4
0.4
0.2
0.2
Epochs
Epochs
Figure 9: Cross entropy loss (Top) and accuracy (Bottom) for a given target with its adversarial label (left), and with its original label (right) shown for poisoned and clean ResNet- 18. The clean model is used as victim for the poisoned model. The loss is averaged times for the poisoned model. Learning rate drops marked with gray horizontal bars.
significant computational effort required. For MetaPoison, we use the original implementation of Huang et al. 22020), but add our larger models. We find that with the larger architectures and different threat model (original MetaPoison considers color perturbation in addition the frD bound), our gradient matching technique still significantly outperforms MetaPoison. Note that for the ConvNet experiment MetaPoison in table 2 found that MetaPoison seems overfit with 32, and as
19

---

Published as a conference paper at ICLR 2021
Gradient Matching R-1
Gradient Matching, R Frogo
Aug. Poison Success
0.8
Water marks
0.6
MetaPaison
100
1000
Time (Minutes)
Figure 10: CIFAR-10 comparison without time and memory constraints for ResNet18 with realistic training. Budget 1%,- E = 16. Note that the x-axis is logarithmic.
such we show numbers running the MetaPoison code with & 16 in that column, which are about 8% better than = 16. This is possibly hyperparameter question for MetaPoison, which was optimized for E = and color perturbation.
F.2 DEFICIENCIES OF FILTERING DEFENSES
Defenses aim to sanitize training data of poisons by detecting outliers (often in feature space), and removing or relabeling these points (Steinhardt et al. 2017 Paudice et al. 2018 Peri et al. 2019). In some cases, these defenses in the setting of general performance degrading attacks, while others deal with targeted attacks. By in large, poison defenses up to this point are limited in scope. For example, many defenses that have been proposed are specific to simple models like linear classifiers and SVM, or the defenses are tailored to weaker attacks such as collision based attacks where feature space is well understood (Steinhardi al. 2017 Paudice al., 2018 Peri et al. 2019). However, data sanitization defenses break when faced with stronger attacks. Table 6 shows a defense by anomaly filtering. averaged over randomly seeded poisoning runs on CIFAR-IO (4% budget w/ E 16), we find that outlier detection is only marginally more successful than random guessing.
Table 6: Outlier detection close to random-guessin for poison detection on CIFAR-10.
10% filtering
20% filtering
Expected poisons removed (outlier method)
248
467
Expected clean removed (outlier method)
252
533
Expected poisons removed (random guessing)
200
400
Expected clean removed (random guessing)
300
600
F.3 DETAILS: DEFENSE BY DIFFERENTIAL PRIVACY
In fig. 4b we consider defense by differential privacy. According to Hong et al. (2020). gradient noise is the key factor that makes differentially private SGD (Abadi et al. 2016) useful as a defense. As such we keep the gradient clipping fixed to value of and only increase the gradient noise in fig. 4b) To scale differentially private SGD, we only consider this gradient clipping on the mini-batch level. not the example level. This is reflected the red, dashed line. A trivial counter-measure against this defense is shown the solid red line. If the level of gradient noise is known to the attacker, then the attacker can brew poisoned data by the approach shown algorithm I but also add gradient noise and gradient clipping to the poison gradient. We use naive strategy of redrawing the added noise every time the matching objective B(A, 0) is evaluated. turns out that this yields a good baseline counter-attack against the defense through differential privacy.
20

---

Published as a conference paper at ICLR 2021
Ste Sariet
Conesmiiy
Cornesintitt
Spothe
Sacche
Focms
(a) Alignment of (O) and (b) Zoom: Alignment of V Lut (c) Alignment (O,(0) (orig. (C(
and VZ( from epoch 14.
bel) and VZ(0)
Figure 11: Average batch cosine similarity, per epoch, between the adversarial gradient and the
gradient of each mini-batch (left), and with its clean counterpart (0) = VeL(#,, v²) (right) for poisoned and a clean ResNet-18. Each measurement averaged over an epoch. Learning rate drops are marked with gray vertical bars.
F.4 DETAILS: GRADIENT ALIGNMENT VISUALIZATION
Figure I1 visualizes additional details regarding fig. 2 Figure rap fig. 2 with linear scaling, whereas fig. [Tb]shows the behavior after epoch 14, which is the first learning rate drop. Note that in figures each measurement is averaged over an epoch and the learning rate drops are marked with gray vertical bars. Figure [Tc shows the opposite metric, that is the alignment of the original (non-adversarial) gradient. is important to note for these figures, that the positive alignment the crucial, whereas the magnitude of aligmment is not important. As this is the gradient averaged over the entire epoch, the contributions are from mini mini-batches can contain none or only a single poisoned example.
F.5 ABLATION STUDIES REDUCED BREWING/VICTHI TRAINING DATA
In order to further test the strength and possible limitations of the discussed poisoning method, we perform several ablation studies, where we reduce either the training set known the attacker or the set of poisons used by the victim, or both.
In many real world poisoning situations, it not reasonable assume that the victim will unwittingly add all poison examples to their training set, or that the attacker knows the full victim training set to begin with. For example, if the attacker puts 1000 poisoned images on social media, the victim might only scrape 300 of these. We test how dependent the method is on the victim training set by randomly removing a proportion of data (clean + poisoned) from the victim's 's training set. We then train the victim on the ablated poisoned dataset, and evaluate the target image to see if it is misclassified by the victim as the attacker's ' intended class. Then, we add another assumption the brewing network does not have access to all victim training data when creating the poisons (see tab We see that the attacker can still successfully poison the victim, even after a large portion of the victim's training data is removed, the attacker does not have access to the full victim training set.
Table 7: Average poisoning success under victim training data ablation. In the first regime, victim ablation, a proportion of the victim's training data (clean + poisoned) is selected randomly and then the victim trains this subset. In the second regime, pretrained + victim ablation, the pretrained network is trained on a randomly selected proportion of the data, and then the victim chose new random subset of clean poisoned data on which to train. All results averaged over runs on ImageNet.
70% data removed 50% data removed
victim ablation
60%
100%
pretrained + victim ablation
60%
80%
F.6 ABLATION STUDIES METHOD
Table shows different variations of the proposed method. While using the Carlini- -agger loss surrogate for cross entropy helped in Huang al. (2020), it does not help in setting. We
21

---

Published as a conference paper at ICLR 2021
Average Success, spa-16
Average Polson Sutces
Average Polcan Success
Aceraee Surres, spoet
0.6
0.2
deaan
ResNot Width
Epochs protrained
Figure 12: Ablation Studies. Left: avg, poison success for Euclidean Loss, cosine similarity and the Poison Frogs objective (Shafahi et al. 20188) for thin ResNet-18 variants. Right: Avg. poison success vs number of pretraining epochs.
Table CIFAR-10 ablation runs. e 16, budget is 1%. All values are computed for ResNet-18 models.
Setup
Avg. Poison Success %
Validation Acc.
Baseline (full data aug., R .M = 250
91.25% (#6..44 14)
92.20%
Carlini- Wagner loss instead of L
77.50 77.50% (#9.32)
92.08%
Fewer Opt. Steps (M = 50)
40.00% (=10.87)
92.05%
Euclidean Loss instead of cosine sim.
61.25% (+9.75)
92.09%
further find that running the proposed method for only 50 steps (instead of 250 as everywhere else in the paper) leads to significant loss in avg. poison success. Lastly investigate whether using euclidean loss instead cosine similarity would be beneficial. This would basically imply trying match eq. (2) directly. Euclidean loss amounts removing the invariance to gradient magnitude, in comparison to cosine similarity, which is invariant. We find that this is not beneficial in our experiments, and that the invariance with respect to gradient magnitude does allow for the construction of stronger poisoned datasets. Interestingly the discrepancy between both loss functions is related to the width of the network. In fig. [2 the left, we visualize avg. poison success for modified ResNet-18s. The usual base width of 64 is replaced by the width value shown on the x-axis. For widths smaller than 16, the Euclidean loss dominates, but its effectiveness does not increase with width. In contrast the cosine similarity is superior for larger widths and seems to be able to make use of the greater representative power the wider networks find vulnerabilities. fig. [12 on the right examines the impact of the pretrained model that is supplied to algorithm We compare avg. poison success against
number of pretraining epochs for a budget of 1%, first with e 16 and then with = 8. It turns out that for the easier threat model of & 8, even pretraining to only 20 epochs can be enough for the algorithm to work well, whereas in the more difficult scenario of e = 8, performance increases with pretraining effort.
F.7 TRANSFER EXPERIMENTS
In addition to the fully black-box pipeline of the AutoML experiments in appendix we test the transfcrability of our poisoning method against other commonly used architectures. Transfer results on CIFAR-I0 can be found in table 3 On Imagenet, we brew poisons with variety of networks, and test against other networks. We find that poisons crafted with one architecture can transfer and cause targeted mis-classification in other networks (see fig. 13).
F.8 MULTI-TARGET EXPERIMENTS
We also perform limited tests on poisoning multiple targets simultaneously. We find that while keeping the small poison budget of 1% fixed, are able to successfully poison more than one target while optimizing poisons simultancously. see table Effectively, however, every target image gradient has be matched with increasingly smaller budget. As the target images drawn
22

---

Published as a conference paper at ICLR 2021
Transfer results ImageNet
ResNet18
0.9
0.6
0.7
Avg. Poison Success
vicitm network
MobileNetv2
0.2
0.6
0.2
0.5
0.4
ResNet34
0.6
0.1
0.9
0.1
ResNet18
MobileNetv2
ResNet34
brewing network
Figure 13: Direct transfer results on common architectures. Averaged over 10 with budget of 0.1% and s-bound of 16. Note that for these transfer experiments, the model was only trained on the "brewing" network, without knowledge of the victim. This shows transferability unknown architectures.
Table 9: CIFAR-10 ablation runs. 16, budget is 1% with default parameters, modifying only the number targets and budget.
Targets
Budget
Avg. Poison Success
Budget / Target
Success
Target
1 %
90.00% (#44744)
1 1
90
I
4%
95.00% (#:4.77)
4%
95
I
6%
90.00% (1:6.71)
6%
90
2
%
65.00% (#:7.16)
0.5%
130
4
1%
36.25% (#:4500
0.25%
145
4
4%
65.00% (155112
1%
260
6%
46.25% (+-6.19)
1.5%
185
5
5%
44.00% (+5.40)
1%
220
6
6%
30.83% (#5.922
1%
185
8
4%
25.62% (1:3.59)
0.5%
205
16
4%
18.12% (+2.86)
0.25%
290
23

---

Published as a conference paper at ICLR 2021
Table 10: CIFAR-I0 baseline clean validation accuracy. : 16. budget 1%. All values are computed for ResNet- 18 models as in the baseline plot in section 5.1
Setting
Unpoisoned
Poisoned
K R
92.25%
(20.10)
92.12% (#0.05)
K = 2, R 92.16%
(#0.08)
92.06% (#0.04)
K 4,
92.18%
(=0.05)
92.08% (#0.04)
K = 8, R
92.16%
(+0.04)
92.20% (+0.03)
K 1,
92.22%
(+0.11)
92.08%
(+0.04)
K 2, R 8
92.27%
(#0.07)
92.03% (#0.05)
K 8, R 92.13% (+0.05)
92.04% (#0.03)
random and semantically similar (aside from their shared class), their synergy is limited. As such, we generally require larger budget for multiple targets. We show various combinations of number of targets and budget table While it is possible to reach gear-1009) avg. poison success for single target in the setting considered in this work, this value is not reached when optimizing for multiple targets, even when the budget is increased although the total number of erroneous classifications increases. We analyze this via the last column in table showing avg. poison success multiplied by number targets, where we find that multiple targets with increased budget can lead to more total mis-classifications, e.g. score of 290 for 16 targets and budget of 4%, yet the success for each individual target is only 18% on average.
No IMPACT ON VALIDATION ACCURACY F.9
The discussed attack does not significantly alter clean validation accuracy (i.e. validation accuracy on all validation images besides the targets), as the attack is specifically tailored to align only to specific target gradients, and as only small budget of images changed within æ bounds. We validate this by reporting the clean validation accuracy for the CIFAR- 10 baseline experiment in section 5.1 in table 10 finding that drop in validation accuracy is on the order of 0. 0.1%.
24


Reviews:
[{"review_id": "vhPesayC7X9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n- The paper proposes a novel data poisoning attack i.e., to perturb a small fraction of images in the victim's training dataset so as to cause targeted misclassification on certain examples at inference time.\n- The proposed approach works by perturbing the clean poison set to introduce a gradient direction which mimics the victim training their model on a targeted mislabeled set.\n- Experiments on CIFAR10 and ImageNet demonstrate that the model outperforms competing approaches.\n\n---\n\n## Strengths\n\n**1. Attack insight**\n- I appreciate the insight used to craft the perturbations for the poisoned instances. It seems reasonable to me to exploit the fact that the poisoned instances are used for training using a gradient-descent approach; using the gradient information as a yardstick to craft the perturbations is a nice insight that the paper leverages.\n\n**2. Thorough evaluation**\n- I am impressed by the thoroughness in the evaluation. The authors extensively evaluate numerous factors influencing the performances (e.g., size of ensembles, no. of restarts), compare with recent baselines, etc.\n- To add to it, the authors further evaluate on Imagenet and achieve strong results.\n\n**3. Writing**\n- I enjoyed the writing in the paper. I found the presentation clear and easy to follow.\n\n---\n\n## Concerns\n\n### Major Concerns\n\n**1. Attack assumes access to exact training data**\n- If I understand the approach correctly, it assumes access to the exact dataset used by the victim to train the model? Isn't this a really strong assumption?\n- Because if this is the case in the threat scenario, couldn't the attacker simply poison the entire dataset?\n- As a result, I wonder whether the attack also extends to the more interesting and practical case where the adversary has limited access to the victim's training set.\n\n**2. From scratch**\n- At many times in the paper, the authors remark that the attack works in spite of the targeted model being trained from scratch from an unknown initialization.\n- However, I would suspect that it is easier to tailor the poisoned instances with access to a strong gradient signal, such as early on during training. Are the authors aware whether the approach is robust to victim models that has been pretrained?\n\n### Minor Concerns\n\n**3. Test accuracies**\n- Could the authors comment on the difference in victim's test-set accuracy training with the clean and poisoned training set? I found this largely missing, since the focus primarily seems to be on the accuracy on the target set.\n- Because a minor concern I have is that the victim model might be overfitting to the poisoned instances by trading off test-set accuracy. It would be nice to know how severe this is.\n\n**4. \"single differentiation\"**\n- The authors claim that the attack requires only a \"single differentiation\". But doesn't the model have to be twice differentiable ($\\nabla_x \\nabla_\\theta \\mathcal{L}(\\cdot)$) to perform the updates?\n\n\n### Nitpicks\n\n**5. Strong focus on MetaPoison**\n- The paper makes many head-to-head comparisons with MetaPoison. I'm not sure why, since MetaPoison doesn't seem too closely-related. Especially in S5.2, it appears that it is singled out to demonstrate the computational overhead. This is understandable since it's relies on a meta-learning approach.\n\n**6. Writing**\n- Fig 4b is unreadable -- I recommend resizing the figure.\n- It seems surprising that the related work section claims that poisoning attacks, unlike backdoor attacks, do not require access to test data. As I'm aware, both require the same access to test-set -- specifically that a particular test instance is presented at inference time to cause misclassification. In fact I would think backdoors are more generalizable here since any test instance can be watermarked to introduce misclassification, unlike pre-specified instances in the case of poisoning.\n\n### Post-rebuttal update\nI thank the authors for their response -- this helps. Having read the other reviews, I am still leaning towards acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zy418x0vOS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a scalable data poisoning attack algorithm focusing on targeted attacks. The technique is based on gradient matching, where the intuition is to design the poisoning patterns such that their effect on the gradient of the training loss mimics the gradient as if the targeted test image is included in the training data.\n\nThe paper presents both theoretical intuitions behind the algorithm, as well as empirical reduction and simplification to make the algorithm scalable to ImageNet and applicable to even a black-box attack against the Google Cloud AutoML toolkit.\n\nThe algorithm proposed in this paper is practical and general, making it a realistic poisoning threat to modern deep learning systems. The presentation is clear and the theoretical justification is intuitive and easy to understand.\n\nOverall, I think this paper is a good contribution to the study of the large-scale poisoning attack.\n\nMinor typo:\n In proof of Prop 1, you need the angle between the two gradients to be almost always smaller than 90 degrees, not 180 degrees.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8S8UNSdyvc3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces a novel targeted clean-label poisoning attack, expected to be more efficient and scalable than current ones. The attack is formulated as a bilevel problem which is then solved with a (fast) heuristic approach based on aligning the gradients of the inner and outer objective functions. A theoretical analysis is also reported to show that this strategy consistently finds a descent direction for the outer objective, asymptotically converging to (a local) minimum. \n\nFirst of all, I like how the authors have derived their attack and its heuristic solution, and I'm wondering if this generalizes to other applications where bilevel problems are at the core, like meta learning or hyperparameter optimization. However, I have several concerns on the presentation and soundness of the reported results.\n\nFirst, I think that this paper makes confusion (at least in the reader's mind) when introducing the data poisoning problem. At the beginning, there is no clear distinction between the two main goals of data poisoning:\n(1) poisoning availability attacks, which aim to increase the test error causing a denial of service, and\n(2) poisoning integrity attacks (which are often referred to -in a misleading manner- as targeted attacks), which aim to allow specific intrusions/attacks at test time (backdoor attacks belong to this category). \nFor a clearer nomenclature/definition see: M. Barreno, B. Nelson, A. Joseph, and J. Tygar. The security of machine learning. Machine Learning, 81:121–148, 2010. \nThere, indeed, targeted/untargeted is referred to the victim user, not to the goal/security violation caused by the attack.\n\nWhile this work seems to claim that, in general, poisoning attacks are computationally demanding, a distinction should be made. While poisoning availability attacks are typically much more computationally demanding (they do require solving the bilevel optimization problem to work well) and this heavily hinders their scalability to large datasets, poisoning integrity attacks can be quite efficient (there's no need to solve a bilevel problem for them to work well, as in the case of most of the considered competing attacks in this paper, or anyway the approach can be simplified - see, e.g., Koh et al., ICML 2017, where the whole network was frozen and the bilevel problem was solved only by assuming that the parameters of the last layer were updated).\nI believe that this aspect should be clarified from the beginning. First, this paper focuses on *targeted* (or integrity) poisoning attacks, and this should also be clear from the title. Second, sentences related to the overwhelming complexity of targeted/integrity poisoning attacks should be revised (e.g., Koh et al., ICML 2017 also worked on the DogFish data which should be a subset of ImageNet, if I'm not mistaken). \n\n\nAnother important issue which I do not completely understand is what the authors mean with the word \"from scratch\", from the viewpoint of previous attacks. I agree with them that previous attacks designed to work on pre-trained models with fine tuning may not work against models which are trained from scratch, but what prevents the attacker to train a model from scratch on the clean data and then design their attack samples with fine tuning? The attack samples can then be added to the initial training set to see if, when learning again from scratch on the poisoned data, the attack remains effective or not. Is this the setup that the authors have considered in their paper for such attacks, or they run them against an \"untrained\" (or not fully trained) model? If we are in the second scenario, I don't think the analysis reported should be considered fair enough.\n\n\nThere are parts in the paper where it is claimed that 'clean-label' attacks are in some sense better than label flips or attacks that do not preserve image semantics. Why? Are we expecting human labelers to check the quality of our data?\nOr are we expecting that clean-label attacks are harder to spot?\nBoth questions are unaddressed in this paper.\nFirst, I don't think that in many realistic scenarios humans are expected to cleanup the whole training set, especially when it contains a lot of samples. Second, it's also true that the level of noise used in this paper is not so small. By zooming in Figs 6-8, the perturbation becomes quite visible even to the human eye.\nHence, we cannot only instruct humans to detect these patterns, but we can probably train detectors to do that automatically. Accordingly, I don't see in which practical, relevant application scenarios \"clean-label\" can be retained useful as a requirement.\nFinally, even though the authors have analyzed the robustness of their attack against some defense mechanisms, the defenses considered aim to detect mostly poisoning availability attacks and NOT backdoor attacks or targeted/integrity poisoning.\nI am even skeptical that such methods can detect label flips or even other current attacks. Have the authors tested such defenses against the competing approaches (poisoning frogs, convex polytopes, etc.)? Do these attacks work or not against them?\nHow do detection methods for backdoor attacks work against the proposed attack? For a list of such detection methods, see, e.g., Table 1 in https://arxiv.org/abs/1910.03137 (note that some detection methods should work against clean-label attacks too, there's no need to put a trigger on the image).\n\n\nFinally, the experimental section is missing key information for reproducing the experiments. The parameters \\tau, R and M are given a value but not a definition. The figure with the average accuracy vs. time is missing a caption and a figure label. It is extremely unclear what this figure is showing as 1) the parameters are missing descriptions 2) the metric used for evaluating the figure is described nowhere in the paper. This problem also extends to tables 1, 2 and 3: a clear definition of the \"evaluation metric\" should be given. Are we interested more in preserving accuracy or in the attack success? How is poisoning success defined? (this might be explained in the supplementary material, but it is important for understanding the whole results). Why not including a plot with poisoning success vs. accuracy of the model?\n\nTo summarize, the paper is promising, but important details and clarifications are still needed. The experimental section and the way results are presented needs major improvement, as it is hard to tell if the attack is working and how efficient and effective it is from the data presented in this paper. \n\n\n\n** Minor comments: ** \n\n* The Poison Frogs attack is described in Section 2, marking as a drawback the fact that it only works with fine-tuning. It is not clear however why this is a limitation, as one could train the model with normal training and add the poisoned data in the last epochs. \n\n* In Algorithm 1, step 9: what is the update being performed? It seems to me that the pseudo-code does not capture the entire processing steps, hence making the whole work hard to reproduce.\n\n* Figure 2 shows gradient alignment along epochs (please report the axis labels), however it does not seem \"flat\" in the end, it is slightly decreasing. What happens if we increase the number of epochs? Will the alignment disappear?\n\n* Sometimes the reader's expertise is taken for granted (e.g. define \"unrolled gradient\"). This might make it difficult for the paper to reach a broader audience.\n\n* Eq. 1 shows no constraints on the data points staying in the feature space after perturbations. Is it considered during the experiments?\n\n* It is observed that VGG11 on CIFAR10 is less transferable, but it would be interesting to read a possible explanation for this phenomenon.\n\n* Equations should distinguish vectors from scalars to improve readability.\n\n* Figure 4 is unreadable as the text in labels and legends is too small.\n\n\n** Comments after reading the authors' rebuttal **\n\nI would like to thank the authors for their clarifications. The threat model is now clearer to me - and I think it deserves clarifications in the paper as well.\n\nFirst of all, as far as I understand now, there's a net distinction between backdoors and clean-label attacks. Backdoor attacks assume that the attacker controls the design phase and the training process, and releases a backdoored model (which then someone else re-uses possibly with fine tuning). Hence, defenses against backdoors aim to detect whether models have been backdoored or not, and it is reasonable to expect that the defender doesn't know the training data as well as other design choices (as the attacker released the model). In this setting, clean-label attacks do not make sense (as the attacker controls the training labels too).\n\nClean-label attacks assume a different setup. Here the attacker only injects poisoning samples into the training set but does neither control the training process nor the training labels. Hence, clean-label attacks make sense in this setting. However, it also makes sense that the defender knows the training data (as the defender is the one that trains the algorithm, and the purpose is to either detect and remove the poisoning points or reduce their influence over training) - and hence I'm expecting the authors to do consider previous defenses that assume knowledge of the training set in their work.\n\nTo summarize, I think that:\n\n(1) the authors should clarify in the title that they restrict themselves to clean-label integrity/targeted poisoning attacks.\n\n(2) the authors should clarify the threat model, and clearly distinguish poisoning availability attacks (bilevel data poisoning) vs poisoning integrity attacks. Furthermore, in the poisoning integrity/targeted family, backdoor and clean-label attacks should be distinguished and the threat models clarified (in particular, w.r.t assumptions on what the attacker/defender know and have access to).\n\n(3) the authors should revise their sentences on the complexity of data poisoning (previous clean-label targeted attacks like poison frogs are not as complex as bilevel data poisoning attacks). A fairer comparison in terms of complexity should also be considered - how faster is this new attack w.r.t. poison frogs and the other clean-label targeted attacks? (poisoning availability should not be considered here as the goal is different in that case).\n\n(4) In general, there is need to disambiguate clean-label targeted poisoning attacks from the rest, and better position this work in context. Reading the paper in its current form, it seems that the authors are also able to improve scalability of poisoning availability attacks whereas this is not the goal of this work. \n\nI'm willing to revise my score if the authors agree on making these clarifications in the paper, better highlighting the net contributions of their work and the proper context of competing approaches (which do not include backdoors and poisoning availability attacks).\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "38h-AaEjjBd", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a simple yet effective approach for data poisoning attack targeting a few \"clean-label\" victim images, using the idea of gradient matching (cosine similarity maximization) between the gradients of adversarial and clean losses. Although the attack model still requires knowing the network architecture (gray-box setting), the resulting poisoned datasets are more effective against different initializations, and some techniques (e.g. model ensemble, multiple restarts) are proposed to further boost the attack performance. The attack results are significantly better than the compared poisoning attacks, and the authors show effective attacks on the ImageNet dataset as well as Google Cloud AutoML with the poisoned data. The authors also discussed the proposed attack on some defenses, showing that the poison has limited change to feature distribution, and differential privacy can mitigate the attack but at the cost of reduced utility (clean accuracy).\n\nOverall, this paper shows some new insights and sets new benchmarks for targeted data poisoning attacks, with practical threat assessment on ImageNet datasets and Google Cloud AutoML, which I deem as a significant contribution. The proposed gradient matching is simple, intuitive, yet very effective. One limitation from Appendix A.8 is that the proposal may not scale well to more than 1 target image, as indicated by the rapidly decreasing attack accuracy. It will be more meaningful to control the effective budget/target and check the resulting accuracy of different number of targets, in order to understand whether gradient matching is scalable to multiple-target setting.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a scalable data poisoning algorithm for targeted attacks, using the idea of designing poisoning patterns which "align" the gradients of the real objective and the adversarial objective. This intuition is supported by theoretical results, and the paper presents convincing experimental results about the effectiveness of the model.

The reviewers overall liked the paper. However, they requested a number of clarifications and some additional work, which should be incorporated in the final version (however, the authors are not required to use the wording as poison integrity/ poison availability). In particular, it would be great to see the experiment the authors suggested in their response to Reviewer 2 about the effectiveness of their method for multiple targets (this is important to better understand the limitations of the proposed approach).
Decision: Accept (Poster)